- Configuration changes
- Troubleshooting
Configure your GitLab Dedicated instance
The instructions on this page guide you through configuring your GitLab Dedicated instance, including enabling and updating the settings for available functionality.
Any functionality in the GitLab application that is not controlled by the SaaS environment can be configured by using the Admin area.
Examples of SaaS environment settings include gitlab.rb
configurations and access to shell, Rails console, and PostgreSQL console.
These environment settings cannot be changed by tenants.
GitLab Dedicated Engineers also don’t have direct access to tenant environments, except for break glass situations.
Configuration changes
Configure your instance using Switchboard
You can use Switchboard to make limited configuration changes to your GitLab Dedicated instance.
The following configuration settings are available in Switchboard:
Prerequisites:
- You must have the Admin role.
To make a configuration change:
- Sign in to Switchboard.
- At the top of the page, select Configuration.
- Follow the instructions in the relevant sections below.
For all other instance configurations, submit a support ticket according to the configuration change request policy.
Applying configuration changes in Switchboard
You can apply configuration changes made in Switchboard immediately or defer them until your next scheduled weekly maintenance window.
When you apply changes immediately:
- Deployment can take up to 90 minutes.
- Changes are applied in the order they’re saved.
- You can save multiple changes and apply them in one batch.
After deployment, you’ll receive an email notification. Check your spam folder if you don’t see it in your main inbox. All users with access to view or edit your tenant in Switchboard receive a notification for each change. For more information, see Manage Switchboard notification preferences.
View the configuration change log
You can use the configuration change log to track the changes made to your GitLab Dedicated instance, including:
- Configuration change: Name of the configuration setting that changed.
- User: Email address of the user that made the configuration change. For changes made by a GitLab Operator, this value will appear as
GitLab Operator
. - IP: IP address of the user that made the configuration change. For changes made by a GitLab Operator, this value will appear as
Unavailable
. - Status: Whether the configuration change is initiated, in progress, completed, or deferred.
- Start time: Start date and time when the configuration change is initiated, in UTC.
- End time: End date and time when the configuration change is deployed, in UTC.
Each configuration change has a status:
- Initiated: Configuration change is made in Switchboard, but not yet deployed to the instance.
- In progress: Configuration change is actively being deployed to the instance.
- Complete: Configuration change has been deployed to the instance.
- Delayed: Initial job to deploy a change has failed and the change has not yet been assigned to a new job.
To view the configuration change log:
- Sign in to Switchboard.
- Select your tenant.
- At the top of the page, select Configuration change log.
Configuration change request policy
This policy does not apply to configuration changes made by a GitLab Dedicated instance admin using Switchboard.
Configuration changes requested with a support ticket:
- Are applied during your environment’s weekly four-hour maintenance window.
- Can be requested for options specified during onboarding or for optional features listed on this page.
- May be postponed to the following week if GitLab needs to perform high-priority maintenance tasks.
- Can’t be applied outside the weekly maintenance window unless they qualify for emergency support.
Bring your own domain (BYOD)
You can add a custom hostname for your GitLab Dedicated instance. Optionally, you can also provide a custom hostname for the bundled container registry and KAS services.
Prerequisites:
- Access to your domain’s server control panel to set up DNS records.
Set up DNS records
Custom domains require a:
-
CNAME
record: Add aCNAME
record that points your custom hostname totenant_name.gitlab-dedicated.com
.gitlab.my-company.com. CNAME tenant_name.gitlab-dedicated.com
-
CAA
record: If your domain has an existingCAA
(Certification Authority Authorization) record, add aCAA
record for Let’s Encrypt. This allows Let’s Encrypt to also issue certificates for your domain.example.com. IN CAA 0 issue "pki.goog" example.com. IN CAA 0 issue "letsencrypt.org"
In this example, the
CAA
record defines Google Trust Services ("pki.goog"
) and Let’s Encrypt ("letsencrypt.org"
) as certificate authorities that are allowed to issue certificates for your domain.
Add a custom hostname
To add a custom hostname after your instance is created, submit a support ticket.
SMTP email service
You can configure an SMTP email service for your GitLab Dedicated instance.
To configure an SMTP email service, submit a support ticket with the credentials and settings for your SMTP server.
Inbound Private Link
AWS Private Link allows users and applications in your VPC on AWS to securely connect to the GitLab Dedicated endpoint without network traffic going over the public internet.
To enable the Inbound Private Link:
- Open a support ticket. In the body of your support ticket, include the IAM principals for the AWS users or roles in your AWS organization that are establishing the VPC endpoints in your AWS account. The IAM principals must be IAM role principals or IAM user principals. GitLab Dedicated uses these IAM Principals for access-control. These IAM principals are the only ones able to set up an endpoint to the service.
- After your IAM Principals have been allowlisted, GitLab creates the Endpoint Service and communicates the
Service Endpoint Name
on the support ticket. The service name is generated by AWS upon creation of the service endpoint.- GitLab handles the domain verification for the Private DNS name, so that DNS resolution of the tenant instance domain name in your VPC resolves to the PrivateLink endpoint.
- The endpoint service is available in two Availability Zones. These are either the zones you chose during onboarding or two randomly selected zones if you didn’t specify any.
- In your own AWS account, create an Endpoint Interface in your VPC, with the following settings:
- Service Endpoint Name: use the name provided by GitLab on the support ticket.
- Private DNS names enabled: yes.
- Subnets: choose all matching subnets.
- After you create the endpoint, use the instance URL provided to you during onboarding to securely connect to your GitLab Dedicated instance from your VPC, without the traffic going over the public internet.
Outbound Private Link
Outbound private links allow your GitLab Dedicated instance and the hosted runners for GitLab Dedicated to securely communicate with services running in your VPC on AWS without exposing any traffic to the public internet.
This type of connection allows GitLab functionality to access private services:
-
For the GitLab Dedicated instance:
- webhooks
- import or mirror projects and repositories
-
For hosted runners:
- custom secrets managers
- artifacts or job images stored in your infrastructure
- deployments into your infrastructure
Consider the following:
- You can only establish private links between VPCs in the same region. Therefore, you can only establish a connection in the regions specified for your Dedicated instance.
- The connection requires the Availability Zone IDs (AZ IDs) for the two Availability Zones (AZs) in the regions that you selected during onboarding.
- If you did not specify any AZs during onboarding to Dedicated, GitLab randomly selects both AZ IDs.
You can view the Reverse Private Link IAM Principal
attribute in the Tenant Details section of Switchboard.
To enable an Outbound Private Link:
-
Create the Endpoint service through which your internal service
will be available to GitLab Dedicated. Provide the associated
Service Endpoint Name
on a new support ticket. - Make sure you have configured a Network Load Balancer (NLB) for the endpoint service in the two AZs to which your Dedicated instance was deployed. If you did not specify these during onboarding to Dedicated, you must either:
- Submit a support ticket to request the AZ IDs required to enable the connection and ensure the NLB is enabled in those AZs.
- Ensure the NLB is enabled in every AZ in the region.
- In your support ticket, GitLab will provide you with the ARN of an
IAM role that will be initiating the connection to your endpoint service. You must ensure this ARN is included, or otherwise covered by other
entries, in the list of “Allowed Principals” on the Endpoint Service, as described by the AWS documentation.
Though it’s optional, you should you add it explicitly, allowing you to set
Acceptance required
to No so that Dedicated can connect in a single operation. If you leaveAcceptance required
as Yes, then you must manually accept the connection after Dedicated has initiated it. - To connect to services using the Endpoint, the Dedicated services require a DNS name. Private Link automatically creates an internal name, but
it is machine-generated and not generally directly useful. There are two options available:
- In your Endpoint Service, enable Private DNS name, perform the
required validation, and let GitLab know in the support ticket that you are using this option. If
Acceptance Required
is set to Yes on your Endpoint Service, also note this on the support ticket because Dedicated will need to initiate the connection without Private DNS, wait for you to confirm it has been accepted, and then update the connection to enable the use of Private DNS. - Dedicated can manage a private hosted zone (PHZ) within the Dedicated AWS account and alias DNS names to the endpoint, directing requests for those names to your endpoint service. These aliases are often referred to as PHZ entries. For more information, see Private hosted zones.
- In your Endpoint Service, enable Private DNS name, perform the
required validation, and let GitLab know in the support ticket that you are using this option. If
GitLab then configures the tenant instance to create the necessary Endpoint Interfaces based on the service names you provided. Any matching outbound connections made from the tenant instance are directed through the PrivateLink into your VPC.
Private hosted zones
You can use a private hosted zone (PHZ) if:
- You have multiple DNS names or aliases that will be accessed using a single endpoint. For example, if you are running a reverse proxy to connect to more than one service in your environment.
- The domain you want to use is not public and cannot be validated for use by private DNS.
To use private hosted zones, submit a support ticket. In the support ticket, provide a list of DNS names that should resolve to the endpoint service for the outbound private link. The list can be updated as needed.
When using your Dedicated instance’s domain as part of an alias, you must include two subdomains before the main domain. This is because:
- The first subdomain becomes the name of the PHZ.
- The second subdomain becomes the record entry for the alias.
For example:
- This is a valid PHZ entry:
subdomain2.subdomain1.<your-tenant-id>.gitlab-dedicated.com
. - This is an invalid PHZ entry:
subdomain1.<your-tenant-id>.gitlab-dedicated.com
.
If you don’t use the Dedicated instance domain, the PHZ name and a PHZ entry in the format phz-entry.phz-name.com
is still required.
Custom certificates
In some cases, the GitLab Dedicated instance can’t reach an internal service you own because it exposes a certificate that can’t be validated using a public Certification Authority (CA). In these cases, custom certificates are required.
Add a custom certificate with Switchboard
- Sign in to Switchboard.
- At the top of the page, select Configuration.
- Expand Custom Certificate Authorities.
- Select + Add Certificate.
- Paste the certificate into the text box.
- Select Save.
- Scroll up to the top of the page and select whether to apply the changes immediately or during the next maintenance window.
Add a custom certificate with a Support Request
If you are unable to use Switchboard to add a custom certificate, you can open a support ticket and attach your custom public certificate files to request this change..
Maximum number of reverse PrivateLink connections
GitLab Dedicated limits the number of reverse PrivateLink connections to 10.
IP allowlist
GitLab Dedicated allows you to control which IP addresses can access your instance through an IP allowlist. Once the IP allowlist has been enabled, when an IP not on the allowlist tries to access your instance an HTTP 403 Forbidden
response is returned.
IP addresses that have been added to your IP allowlist can be viewed on the Configuration page in Switchboard. You can add or remove IP addresses from your allowlist with Switchboard.
Add an IP to the allowlist with Switchboard
- Sign in to Switchboard.
- At the top of the page, select Configuration.
- Expand Allowed Source List Config / IP allowlist.
- Turn on the Enable toggle.
- Select Add Item.
- Enter the IP address and description. To add another IP address, repeat steps 5 and 6.
- Select Save.
- Scroll up to the top of the page and select whether to apply the changes immediately or during the next maintenance window. After the changes are applied, the IP addresses are added to the IP allowlist for your instance.
Add an IP to the allowlist with a Support Request
If you are unable to use Switchboard to update your IP allowlist, you can open a support ticket and specify a comma separated list of IP addresses that can access your GitLab Dedicated instance.
Enable OpenID Connect for your IP allowlist
Using GitLab as an OpenID Connect identity provider requires internet access to the OpenID Connect verification endpoint.
To enable access to the OpenID Connect endpoint while maintaining your IP allowlist:
- In a support ticket, request to allow access to the OpenID Connect endpoint.
The configuration is applied during the next maintenance window.
SAML
You can configure SAML single sign-on (SSO) for your GitLab Dedicated instance.
The following SAML SSO options are available:
Prerequisites:
- You must set up the identity provider (IdP) before you can configure SAML for GitLab Dedicated.
- To configure GitLab to sign SAML authentication requests, you must create a private key and public certificate pair for your GitLab Dedicated instance.
Activate SAML with Switchboard
To activate SAML for your GitLab Dedicated instance:
- Sign in to Switchboard.
- At the top of the page, select Configuration.
- Expand SAML Config.
- Turn on the Enable toggle.
- Complete the required fields:
- SAML label
- IdP cert fingerprint
- IdP SSO target URL
- Optional. To configure users based on SAML group membership or use group sync, complete the following fields:
- SAML group attribute
- Admin groups
- Auditor groups
- External groups
- Required groups
- Optional. To configure SAML request signing, complete the following fields:
- Name identifier format
- Issuer
- Attribute statements
- Security
- Select Save.
- Scroll up to the top of the page and select whether to apply the changes immediately or during the next maintenance window.
- Optional. To use group sync, configure the SAML group links.
- To verify the SAML configuration is successful:
- Check that the SSO button description is displayed on your instance’s sign-in page.
- Go to the metadata URL of your instance (
https://INSTANCE-URL/users/auth/saml/metadata
). This page can be used to simplify much of the configuration of the identity provider, and manually validate the settings.
Activate SAML with a Support Request
If you are unable to use Switchboard to activate or update SAML for your GitLab Dedicated instance, you can open a support ticket:
- To make the necessary changes, include the desired SAML configuration block for your GitLab application in your support ticket. At a minimum, GitLab needs the following information to enable SAML for your instance:
- IDP SSO Target URL
- Certificate fingerprint or certificate
- NameID format
- SSO login button description
"saml": { "attribute_statements": { //optional }, "enabled": true, "groups_attribute": "", "admin_groups": [ // optional ], "idp_cert_fingerprint": "43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8", "idp_sso_target_url": "https://login.example.com/idp", "label": "IDP Name", "name_identifier_format": "urn:oasis:names:tc:SAML:2.0:nameid-format:persistent", "security": { // optional }, "auditor_groups": [ // optional ], "external_groups": [ // optional ], "required_groups": [ // optional ], }
- After GitLab deploys the SAML configuration to your instance, you are notified on your support ticket.
- To verify the SAML configuration is successful:
- Check that the SSO login button description is displayed on your instance’s login page.
- Go to the metadata URL of your instance (
https://INSTANCE-URL/users/auth/saml/metadata
). This page can be used to simplify much of the configuration of the identity provider, as well as manually validate the settings.
Request signing
If SAML request signing is desired, a certificate must be obtained. This certificate can be self-signed which has the advantage of not having to prove ownership of an arbitrary Common Name (CN) to a public Certificate Authority (CA).
To enable SAML request signing:
- Open a support ticket and indicate that you want request signing enabled.
- GitLab will work with you on sending the Certificate Signing Request (CSR) for you to sign. Alternatively, the CSR can be signed with a public CA.
- After the certificate is signed, you can then use the certificate and its associated private key to complete the
security
section of the SAML configuration in Switchboard.
Authentication requests from GitLab to your identity provider can now be signed.
SAML groups
With SAML groups you can configure GitLab users based on SAML group membership.
To enable SAML groups, add the required elements to your SAML configuration in Switchboard or to the SAML block you provide in a support ticket.
Group sync
With group sync, you can sync users across identity provider groups to mapped groups in GitLab.
To enable group sync:
- Add the required elements to your SAML configuration in Switchboard or to the SAML configuration block you provide in a support ticket.
- Configure the Group Links.
Add users to an instance
Administrators can add Switchboard users to their GitLab Dedicated instance. There are two types of users:
- Read only: Users can only view instance data.
- Admin: Users can edit the instance configuration and manage users.
To add a new user to your GitLab Dedicated instance:
- From the Tenants page, select Manage next to the tenant instance.
- From the top of the page, select Users.
- Select New user.
- Enter the Email and select a Role for the user.
- Select Create.
An invitation to use Switchboard is sent to the user.
Manage notification preferences
You can specify whether you want to receive email notifications from Switchboard. You will only receive notifications after you:
- Receive an email invitation and first sign in to Switchboard.
- Set up a password and two-factor authentication (2FA) for your user account.
To manage your own email notification preferences:
- From any page, open the dropdown next to your user name.
- To stop receiving email notifications, select Toggle email notifications off.
- To resume receiving email notifications, select Toggle email notifications on.
You will see an alert confirming that your notification preferences have been updated.
Switchboard Tenant Admins can also manage email notifications for other users with access to their organization’s tenant:
- From the Users page, open the dropdown in the Email notifications column next to the user’s email.
- To turn off email notifications for that user, select No.
- To turn on email notifications for that user, select Yes.
You will see an alert confirming that your notification preferences have been updated.
Access to application logs
GitLab application logs are delivered to an S3 bucket in the GitLab tenant account, which can be shared with you. Logs stored in the S3 bucket are retained indefinitely, until the 1 year retention policy is fully enforced. GitLab team members can view more information in this confidential issue:
https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/issues/483
.
To gain read only access to this bucket:
- Open a support ticket with the title “Customer Log Access”. In the body of the ticket, include a list of IAM Principal ARNs (users or roles) that are fetching the logs from S3.
- GitLab then informs you of the name of the S3 bucket. Your nominated users/roles are then able to list and get all objects in the S3 bucket.
You can use the AWS CLI to verify that access to the S3 bucket works as expected.
Bucket contents and structure
The S3 bucket contains a combination of infrastructure logs and application logs from the GitLab log system. The logs in the bucket are encrypted using an AWS KMS key that is managed by GitLab. If you choose to enable BYOK, the application logs are not encrypted with the key you provide.
The logs in the S3 bucket are organized by date in YYYY/MM/DD/HH
format. For example, there would be a directory like 2023/10/12/13
. That directory would contain the logs from October 12, 2023 at 1300 UTC. The logs are streamed into the bucket with Amazon Kinesis Data Firehose.
Troubleshooting
Outbound Private Link
If you have trouble establishing a connection after the Outbound Private Link has been set up, there are a few things in your AWS infrastructure that could be the cause of the problem. The specific things to check will vary based on the unexpected behavior you’re seeking to fix. Things to check include:
- Ensure that cross-zone load balancing is turned on in your Network Load Balancer (NLB).
- Ensure that the Inbound Rules section of the appropriate Security Groups permits traffic from the correct IP ranges.
- Ensure that the inbound traffic is mapped to the correct port on the Endpoint Service.
- In Switchboard, expand Reverse Private Link Config and confirm that the details appear as you expect.
- Ensure that you have allowed requests to the local network from webhooks and integrations.