Troubleshooting GitLab tokens

When working with GitLab tokens, you might encounter the following issues.

Token appears active but requests fail

A token that is listed as active can still return 401 Unauthorized, 403 Forbidden, or 404 Not Found responses. The active status indicates only that the token exists and has not expired or been revoked. This status does not mean the token can make a given request.

A token’s permissions depend on its scopes and its role. A request can also fail for reasons outside the token: where the request comes from, the resource the request targets, and whether an administrator has turned off access tokens. None of these factors are apparent from the token itself. Personal, project, and group access tokens all use the same glpat- prefix. Two tokens that look identical can therefore behave differently.

An active token can fail for any of the following reasons:

To identify which cause applies, compare the details of the failing token with a token that works:

• Personal access tokens
• Group access tokens
• Project access tokens

The details include each token’s scopes, expiration date, and usage information. Group and project access tokens also show the assigned role.

If the token’s usage information does not update after you make a request, the request might not be reaching GitLab. GitLab updates usage times every 10 minutes and usage IP addresses every minute. If GitLab isn’t recording the usage after those intervals elapse, your request did not reach GitLab.

Expired access tokens

If an existing access token is in use and reaches the expires_at value, the token expires and:

• Can no longer be used for authentication.
• Is not visible in the UI.

Requests made using this token return a 401 Unauthorized response. Too many unauthorized requests in a short period of time from the same IP address result in 403 Forbidden responses from GitLab.com.

For more information on authentication request limits, see Git and container registry failed authentication ban.

Identify expired access tokens from logs

Prerequisites:

You must:

• Be an administrator.
• Have access to the api_json.log file.

To identify which 401 Unauthorized requests are failing due to expired access tokens, use the following fields in the api_json.log file:

When a user attempts to use an expired token, the meta.auth_fail_reason is token_expired. The following shows an excerpt from a log entry:

{
  "status": 401,
  "method": "GET",
  "path": "/api/v4/user",
  ...
  "meta.auth_fail_reason": "token_expired",
  "meta.auth_fail_token_id": "PersonalAccessToken/12",
}

meta.auth_fail_token_id indicates that an access token of ID 12 was used. From GitLab 18.9, meta.user will also be populated with any username associated with the token used for the failed request.

To find more information about this token, use the personal access token API. You can also use the API to rotate the token.

Replace expired access tokens

To replace the token:

1. Check where this token may have been used previously, and remove it from any automation that might still use the token.
   • For personal access tokens, use the API to list tokens that have expired recently. For example, go to https://gitlab.com/api/v4/personal_access_tokens, and locate tokens with a specific expires_at date.
   • For project access tokens, use the project access tokens API to list recently expired tokens.
   • For group access tokens, use the group access tokens API to list recently expired tokens.
2. Create a new access token:
   • For personal access tokens, use the UI or User tokens API.
   • For a project access token, use the UI or project access tokens API.
   • For a group access token, use the UI or group access tokens API.
3. Replace the old access token with the new access token. This process varies depending on how you use the token, for example if configured as a secret or embedded in an application. Requests made from this token should no longer return 401 responses.

Extend token lifetime

Delay the expiration of certain tokens with this script.

From GitLab 16.0, all access tokens have an expiration date. After you deploy at least GitLab 16.0, any non-expiring access tokens expire one year from the date of deployment.

If this date is approaching and there are tokens that have not yet been rotated, you can use this script to delay expiration and give users more time to rotate their tokens.

Extend lifetime for specific tokens

This script extends the lifetime of all tokens which expire on a specified date, including:

• Personal access tokens
• Group access tokens
• Project access tokens

For group and project access tokens, this script only extends the lifetime of these tokens if they were given an expiration date automatically when upgrading to GitLab 16.0 or later. If a group or project access token was generated with an expiration date, or was rotated, the validity of that token is dependent on a valid membership to a resource, and therefore the token lifetime cannot be extended using this script.

To use the script:

extend_expiring_tokens.rb

expiring_date = Date.new(2024, 5, 30)
new_expires_at = 6.months.from_now

total_updated = PersonalAccessToken
                  .not_revoked
                  .without_impersonation
                  .where(expires_at: expiring_date.to_date)
                  .update_all(expires_at: new_expires_at.to_date)

puts "Updated #{total_updated} tokens with new expiry date #{new_expires_at}"

Restore a personal access token

On GitLab Self-Managed or GitLab Dedicated instances, administrators can restore personal access tokens that were revoked accidentally. Restoration is not available on GitLab.com.

1. Open a Rails console.

2. Restore the token:

   token = PersonalAccessToken.find_by_token('<token_string>')
   token.update!(revoked:false)

   For example, to restore a token of token-string-here123:

   token = PersonalAccessToken.find_by_token('token-string-here123')
   token.update!(revoked:false)

Identify personal, project, and group access tokens expiring on a certain date

Access tokens that have no expiration date are valid indefinitely, which is a security risk if the access token is divulged.

To manage this risk, when you upgrade to GitLab 16.0 and later, any personal, project, or group access token that does not have an expiration date automatically has an expiration date set at one year from the date of upgrade.

In GitLab 17.3 and later, this automatic setting of expiry on existing tokens has been reverted, and you can disable expiration date enforcement for new access tokens.

If you are not aware of when your tokens expire because the dates have changed, you might have unexpected authentication failures when trying to sign into GitLab on that date.

To manage this issue, you should upgrade to GitLab 17.2 or later, because these versions contain a tool that assists with analyzing, extending, or remove token expiration dates.

If you cannot run the tool, you can also run scripts in GitLab Self-Managed instances to identify tokens that either:

• Expire on a specific date.
• Have no expiration date.

You run these scripts from your terminal window in either:

• A Rails console session.
• Using the Rails Runner.

The specific scripts you run differ depending on if you have upgraded to GitLab 16.0 and later, or not:

• If you have not yet upgraded to GitLab 16.0 or later, identify tokens that do not have an expiration date.
• If you have upgraded to GitLab 16.0 or later, use scripts to identify any of the following:
  • Tokens expiring on a specific date.
  • Tokens expiring in a specific month.
  • Dates when many tokens expire.

After you have identified tokens affected by this issue, you can run a final script to extend the lifetime of specific tokens if needed.

These scripts return results in the following format:

Expired group access token in Group ID 25, Token ID: 8, Name: Example Token, Scopes: ["read_api", "create_runner"], Last used:
Expired project access token in Project ID 2, Token ID: 9, Name: Test Token, Scopes: ["api", "read_registry", "write_registry"], Last used: 2022-02-11 13:22:14 UTC

For more information on this, see incident 18003.

Find all tokens expiring on a specific date

This script finds tokens that expire on a specific date.

Prerequisites:

• You must know the exact date your instance was upgraded to GitLab 16.0.

To use it:

expired_tokens.rb

This script requires you to know the exact date your GitLab instance was upgraded to GitLab 16.0.

# Change this value to the date one year after your GitLab instance was upgraded.

expires_at_date = "2024-05-22"

# Check for expiring personal access tokens
PersonalAccessToken.for_user_types(:human).where(expires_at: expires_at_date).find_each do |token|
  if token.user.blocked?
    next
    # Hide unusable, blocked PATs from output
  end

  puts "Expired personal access token ID: #{token.id}, User Email: #{token.user.email}, Name: #{token.name}, Scopes: #{token.scopes}, Last used: #{token.last_used_at}"
end

# Check for expiring project and group access tokens
PersonalAccessToken.project_access_token.where(expires_at: expires_at_date).find_each do |token|
  token.user.members.each do |member|
    type = member.is_a?(GroupMember) ? 'Group' : 'Project'

    puts "Expired #{type} access token in #{type} ID #{member.source_id}, Token ID: #{token.id}, Name: #{token.name}, Scopes: #{token.scopes}, Last used: #{token.last_used_at}"
  end
end

Find tokens expiring in a given month

This script finds tokens that expire in a particular month. You don’t need to know the exact date your instance was upgraded to GitLab 16.0. To use it:

expired_tokens_date_range.rb

# This script enables you to search for tokens that expire within a
# certain date range (like 1.month) from the current date. Use it if
# you're unsure when exactly your GitLab 16.0 upgrade completed.

date_range = 1.month

# Check for personal access tokens
PersonalAccessToken.for_user_types(:human).where(expires_at: Date.today .. Date.today + date_range).find_each do |token|
  puts "Expired personal access token ID: #{token.id}, User Email: #{token.user.email}, Name: #{token.name}, Scopes: #{token.scopes}, Last used: #{token.last_used_at}"
end

# Check for expiring project and group access tokens
PersonalAccessToken.project_access_token.where(expires_at: Date.today .. Date.today + date_range).find_each do |token|
  token.user.members.each do |member|
    type = member.is_a?(GroupMember) ? 'Group' : 'Project'

    puts "Expired #{type} access token in #{type} ID #{member.source_id}, Token ID: #{token.id}, Name: #{token.name}, Scopes: #{token.scopes}, Last used: #{token.last_used_at}"
  end
end

Identify dates when many tokens expire

This script identifies dates when most tokens expire. You can use it in combination with other scripts on this page to identify and extend large batches of tokens that may be approaching their expiration date, in case your team has not yet set up token rotation.

The script returns results in this format:

42 Personal access tokens will expire at 2024-06-27
17 Personal access tokens will expire at 2024-09-23
3 Personal access tokens will expire at 2024-08-13

To use it:

dates_when_most_of_tokens_expire.rb

PersonalAccessToken
  .select(:expires_at, Arel.sql('count(*)'))
  .where('expires_at >= NOW()')
  .group(:expires_at)
  .order(Arel.sql('count(*) DESC'))
  .limit(10)
  .each do |token|
    puts "#{token.count} Personal access tokens will expire at #{token.expires_at}"
  end

Find tokens with no expiration date

This script finds tokens that lack an expiration date: expires_at is NULL. For users who have not yet upgraded to GitLab version 16.0 or later, the token expires_at value is NULL, and can be used to identify tokens to add an expiration date to.

You can use this script in either the Rails console or the Rails Runner:

tokens_with_no_expiry.rb

This script finds tokens without a value set for expires_at.

# This script finds tokens which do not have an expires_at value set.

# Check for expiring personal access tokens
PersonalAccessToken.for_user_types(:human).where(expires_at: nil).find_each do |token|
  puts "Expires_at is nil for personal access token ID: #{token.id}, User Email: #{token.user.email}, Name: #{token.name}, Scopes: #{token.scopes}, Last used: #{token.last_used_at}"
end

# Check for expiring project and group access tokens
PersonalAccessToken.project_access_token.where(expires_at: nil).find_each do |token|
  token.user.members.each do |member|
    type = member.is_a?(GroupMember) ? 'Group' : 'Project'

    puts "Expires_at is nil for #{type} access token in #{type} ID #{member.source_id}, Token ID: #{token.id}, Name: #{token.name}, Scopes: #{token.scopes}, Last used: #{token.last_used_at}"
  end
end