Enable logging for self-hosted models

Tier: Ultimate with GitLab Duo Enterprise - Start a trial Offering: Self-managed Status: Beta
History
  • Introduced in GitLab 17.1 with a flag named ai_custom_model. Disabled by default.
  • Enabled on self-managed in GitLab 17.6.
  • Changed to require GitLab Duo add-on in GitLab 17.6 and later.
  • Feature flag ai_custom_model removed in GitLab 17.8

Prerequisites:

  • You must be an administrator for your self-managed instance.

To enable logging and access the logs, enable the feature flag: 

Feature.enable(:expanded_ai_logging)

Disabling the feature flag stops logs from being written.

Logs in your GitLab installation

The logging setup is designed to protect sensitive information while maintaining transparency about system operations, and is made up of the following components:

  • Logs that capture requests to the GitLab instance.
  • Feature flag and logging control.
  • The llm.log file.

Logs that capture requests to the GitLab instance

Logging in the application.json, production_json.log, and production.log files, among others, capture requests to the GitLab instance:

  • Filtered Requests: We log the requests in these files but ensure that sensitive data (such as input parameters) is filtered. This means that while the request metadata is captured (for example, the request type, endpoint, and response status), the actual input data (for example, the query parameters, variables, and content) is not logged to prevent the exposure of sensitive information.

  • Example 1: In the case of a code suggestions completion request, the logs capture the request details while filtering sensitive information: 

    {
  "method": "POST",
  "path": "/api/graphql",
  "controller": "GraphqlController",
  "action": "execute",
  "status": 500,
  "params": [
    {"key": "query", "value": "[FILTERED]"},
    {"key": "variables", "value": "[FILTERED]"},
    {"key": "operationName", "value": "chat"}
  ],
  "exception": {
    "class": "NoMethodError",
    "message": "undefined method `id` for {:skip=>true}:Hash"
  },
  "time": "2024-08-28T14:13:50.328Z"
}

    As shown, while the error information and general structure of the request are logged, the sensitive input parameters are marked as [FILTERED].

  • Example 2: In the case of a code suggestions completion request, the logs also capture the request details while filtering sensitive information: 

    {
  "method": "POST",
  "path": "/api/v4/code_suggestions/completions",
  "status": 200,
  "params": [
    {"key": "prompt_version", "value": 1},
    {"key": "current_file", "value": {"file_name": "/test.rb", "language_identifier": "ruby", "content_above_cursor": "[FILTERED]", "content_below_cursor": "[FILTERED]"}},
    {"key": "telemetry", "value": []}
  ],
  "time": "2024-10-15T06:51:09.004Z"
}

    As shown, while the general structure of the request is logged, the sensitive input parameters such as content_above_cursor and content_below_cursor are marked as [FILTERED].

Feature Flag and Logging Control

Feature Flag Dependency: You can control a subset of these logs by enabling or disabling the expanded_ai_logging feature flag. Disabling the feature flag disables logging for specific operations. For more information, see the Feature Flag section under Privacy Considerations.

The llm.log file

When the :expanded_ai_logging feature flag is enabled, the llm.log file in your GitLab instance captures code generation and Chat events that occur through your instance. The log file does not capture anything when the feature flag is not enabled. Code completion logs are captured directly in the AI gateway.

For more information on:

Logs in your AI gateway container

To specify the location of logs generated by AI gateway, run: 

docker run -e AIGW_GITLAB_URL=<your_gitlab_instance> \
 -e AIGW_GITLAB_API_URL=https://<your_gitlab_domain>/api/v4/ \
 -e AIGW_GITLAB_API_URL=https://<your_gitlab_domain>/api/v4/ \
 -e AIGW_LOGGING__TO_FILE="aigateway.log" \
 -v <your_file_path>:"aigateway.log"
 <image>

If you do not specify a filename, logs are streamed to the output and can also be managed using Docker logs. For more information, see the Docker Logs documentation.

Additionally, the outputs of the AI gateway execution can help with debugging issues. To access them:

  • When using Docker: 

    docker logs <container-id>

  • When using Kubernetes: 

    kubectl logs <container-name>

To ingest these logs into the logging solution, see your logging provider documentation.

Logs structure

When a POST request is made (for example, to the /chat/completions endpoint), the server logs the request:

  • Payload
  • Headers
  • Metadata

1. Request payload

The JSON payload typically includes the following fields:

  • messages: An array of message objects.
    • Each message object contains:
      • content: A string representing the user’s input or query.
      • role: Indicates the role of the message sender (for example, user).
  • model: A string specifying the model to be used (for example, mistral).
  • max_tokens: An integer specifying the maximum number of tokens to generate in the response.
  • n: An integer indicating the number of completions to generate.
  • stop: An array of strings denoting stop sequences for the generated text.
  • stream: A boolean indicating whether the response should be streamed.
  • temperature: A float controlling the randomness of the output.
Example request 
{
    "messages": [
        {
            "content": "<s>[SUFFIX]None[PREFIX]# # build a hello world ruby method\n def say_goodbye\n    puts \"Goodbye, World!\"\n  end\n\ndef main\n  say_hello\n  say_goodbye\nend\n\nmain",
            "role": "user"
        }
    ],
    "model": "mistral",
    "max_tokens": 128,
    "n": 1,
    "stop": ["[INST]", "[/INST]", "[PREFIX]", "[MIDDLE]", "[SUFFIX]"],
    "stream": false,
    "temperature": 0.0
}

2. Request headers

The request headers provide additional context about the client making the request. Key headers might include:

  • Authorization: Contains the Bearer token for API access.
  • Content-Type: Indicates the media type of the resource (for example, JSON).
  • User-Agent: Information about the client software making the request.
  • X-Stainless- headers: Various headers providing additional metadata about the client environment.
Example request headers 
{
    "host": "0.0.0.0:4000",
    "accept-encoding": "gzip, deflate",
    "connection": "keep-alive",
    "accept": "application/json",
    "content-type": "application/json",
    "user-agent": "AsyncOpenAI/Python 1.51.0",
    "authorization": "Bearer <TOKEN>",
    "content-length": "364"
}

3. Request metadata

The metadata includes various fields that describe the context of the request:

  • requester_metadata: Additional metadata about the requester.
  • user_api_key: The API key used for the request (anonymized).
  • api_version: The version of the API being used.
  • request_timeout: The timeout duration for the request.
  • call_id: A unique identifier for the call.
Example metadata 
{
    "user_api_key": "<ANONYMIZED_KEY>",
    "api_version": "1.48.18",
    "request_timeout": 600,
    "call_id": "e1aaa316-221c-498c-96ce-5bc1e7cb63af"
}

Example response

The server responds with a structured model response. For example: 

Response: ModelResponse(
    id='chatcmpl-5d16ad41-c130-4e33-a71e-1c392741bcb9',
    choices=[
        Choices(
            finish_reason='stop',
            index=0,
            message=Message(
                content=' Here is the corrected Ruby code for your function:\n\n```ruby\ndef say_hello\n  puts "Hello, World!"\nend\n\ndef say_goodbye\n    puts "Goodbye, World!"\nend\n\ndef main\n  say_hello\n  say_goodbye\nend\n\nmain\n```\n\nIn your original code, the method names were misspelled as `say_hell` and `say_gobdye`. I corrected them to `say_hello` and `say_goodbye`, respectively. Also, there was no need for the prefix',
                role='assistant',
                tool_calls=None,
                function_call=None
            )
        )
    ],
    created=1728983827,
    model='mistral',
    object='chat.completion',
    system_fingerprint=None,
    usage=Usage(
        completion_tokens=128,
        prompt_tokens=69,
        total_tokens=197,
        completion_tokens_details=None,
        prompt_tokens_details=None
    )
)

Logs in your inference service provider

GitLab does not manage logs generated by your inference service provider. See the documentation of your inference service provider on how to use their logs.

Logging behavior in GitLab and AI gateway environments

GitLab provides logging functionality for AI-related activities through the use of llm.log, which captures inputs, outputs, and other relevant information. However, the logging behavior differs depending on whether the GitLab instance and AI gateway are self-hosted or cloud-connected.

By default, the log does not contain LLM prompt input and response output to support data retention policies of AI feature data.

Logging Scenarios

GitLab self-managed and self-hosted AI gateway

In this configuration, both GitLab and the AI gateway are hosted by the customer.

  • Logging Behavior: Full logging is enabled, and all prompts, inputs, and outputs are logged to llm.log on the GitLab self-managed instance.
  • Expanded Logging: When the :expanded_ai_logging feature flag is activated, extra debugging information is logged, including:
    • Preprocessed prompts.
    • Final prompts.
    • Additional context.
  • Privacy: Because both GitLab and the AI gateway are self-hosted:
    • The customer has full control over data handling.
    • Logging of sensitive information can be enabled or disabled at the customer’s discretion.

GitLab self-managed and GitLab-managed AI gateway (cloud-connected)

In this scenario, the customer hosts GitLab but relies on the GitLab-managed AI gateway for AI processing.

  • Logging Behavior: Prompts and inputs sent to the AI gateway are not logged in the cloud-connected AI gateway to prevent exposure of sensitive information such as personally identifiable information (PII).
  • Expanded Logging: Even if the :expanded_ai_logging feature flag is enabled, no detailed logs are generated in the GitLab-managed AI gateway to avoid unintended leaks of sensitive information.
    • Logging remains minimal in this setup, and the expanded logging features are disabled by default.
  • Privacy: This configuration is designed to ensure that sensitive data is not logged in a cloud environment.

Feature Flag: :expanded_ai_logging

The :expanded_ai_logging feature flag controls whether additional debugging information, including prompts and inputs, is logged. This flag is essential for monitoring and debugging AI-related activities.

Behavior by Deployment Setup

  • GitLab self-managed and self-hosted AI gateway: The feature flag enables detailed logging to llm.log on the self-hosted instance, capturing inputs and outputs for AI models.
  • GitLab self-managed and GitLab-managed AI gateway: The feature flag enables logging on your self-managed instance. However, the flag does not activate expanded logging for the GitLab-managed AI gateway side. Logging remains disabled for the cloud-connected AI gateway to protect sensitive data. For more information, see the Feature Flag section under Privacy Considerations documentation.

Logging in cloud-connected AI gateways

To prevent potential data leakage of sensitive information, expanded logging (including prompts and inputs) is intentionally disabled when using a cloud-connected AI gateway. Preventing the exposure of PII is a priority.

Cross-referencing logs between the AI gateway and GitLab

The property correlation_id is assigned to every request and is carried across different components that respond to a request. For more information, see the documentation on finding logs with a correlation ID.

The Correlation ID can be found in your AI gateway and GitLab logs. However, it is not present in your model provider logs.