Setting up local development for Duo Workflow

Prerequisites

  • Vertex API access
    • You need access to the ai-enablement-dev-69497ba7 project in GCP. This should by available to all engineers at GitLab.
  • Docker
    • See which Docker tooling is approved for GitLab team members in the handbook.

Set up your local GitLab instance

  1. Configure the Duo Workflow Service URL in your local GitLab instance by updating the config/gitlab.yml file: 

    development:
  duo_workflow:
    service_url: 0.0.0.0:50052
    secure: false

  2. Restart the GitLab instance. 

    gdk restart rails

  3. In your local GitLab instance, enable the duo_workflow feature flag from the Rails console: 

    Feature.enable(:duo_workflow)
  4. Set up GitLab Runner with GDK so you can create CI jobs locally to test Workflow.
  5. Create a personal access token in your local GitLab instance with the api scope. Save this value and use it in the next step.
  6. Run GDK with an Ultimate license.

  7. Manually create a Workflow using the following curl request; the output will be a workflow ID that is referred to as $WORKFLOW_ID throughout the rest of these docs: 

    curl POST --verbose \
  --header "Authorization: Bearer $YOUR_GITLAB_PAT" \
  --header 'Content-Type: application/json' \
  --data '{
     "project_id": "$PROJECT_ID_FOR_RUNNING_WORKFLOW_AGAINST"
  }' \
  $YOUR_GDK_ROOT_URL/api/v4/ai/duo_workflows/workflows

Set up the Duo Workflow Service

+1. Clone the Duo Workflow Service repository. 

     git clone git@gitlab.com:gitlab-org/duo-workflow/duo-workflow-service.git

  1. Navigate to the Duo Workflow Service directory. 

    cd duo-workflow-service

  2. Install dependencies with poetry. 

    poetry install

  3. Copy the example env file in the Service repo. 

    cp .env.example .env

  4. Add your ANTHROPIC_API_KEY in the .env file.

  5. Setup gcloud on your system.

  6. Login using your GitLab Google account by running: 

    gcloud auth login

  7. Set the ai-enablement-dev-69497ba7 as active project by running: 

    gcloud config set project ai-enablement-dev-69497ba7

  8. Create the credentials for the application to use 

    gcloud auth application-default login --disable-quota-project

  9. Optional: You can disable auth for local development in the .env file. This disables authentication or the gRPC connection between the Duo Workflow Service and Duo Workflow Executor but a token will still be required for requests to your local GitLab instance. 

    DUO_WORKFLOW_AUTH__ENABLED=false

  10. Run the Duo Workflow Service server 

    poetry run python -m duo_workflow_service.server

  11. If you can correctly connect to Claude, you should see something like this in the output 

    2024-09-06 17:16:54 [info     ] Connected to model: claude-3-sonnet-20240229: You're talking to Claude, an AI assistant created by Anthropic.
2024-09-06 17:16:54 [info     ] Starting server on port 50052
2024-09-06 17:16:54 [info     ] Started server

Set up the Duo Workflow Executor

  1. Clone the Duo Workflow Executor repository 

      git clone git@gitlab.com:gitlab-org/duo-workflow/duo-workflow-executor.git

  2. Navigate to the Duo Workflow Executor directory 

    cd duo-workflow-executor

  3. Create a Dockerfile in the Duo Workflow Executor root directory with the following contents: 

    FROM alpine

RUN apk add go busybox-extras git bash

  4. Build a development image to use: 

    docker build -t alpine-dev-workflow .

  5. Run the executor with your GitLab token and workflow ID 

    make && \
./bin/duo-workflow-executor \
    --goal='Fix the pipeline for the Merge request 62 in the project 19.' \
    --insecure --debug \
    --workflow-id=$WORKFLOW_ID \
    --token=$YOUR_GITLAB_PAT \
    --base-url="$GDK_GITLAB_URL" \
    --user-id="1"

  6. Verify that the checkpoints for workflow have been created 

    curl --verbose \
  --header "Authorization: Bearer $YOUR_GITLAB_PAT" \
  $GDK_GITLAB_URL/api/v4/ai/duo_workflows/workflows/$WORKFLOW_ID/checkpoints

Configure the GitLab Duo Workflow extension for VS Code

The above steps show how to start a workflow directly from the Duo Workflow Executor.

If you would like to start Duo Workflow with the VS Code extension instead, follow these steps.

If you are debugging or making changes to the VSCode extension and need to run the extension in development mode, you can do that following these instructions.

Troubleshooting

Issues connecting to 50052 port

JAMF may be listening on the 50052 port which will conflict with Duo Workflow Service. 

$ sudo lsof -i -P | grep LISTEN | grep :50052
jamfRemot  <redacted>           root   11u  IPv4 <redacted>      0t0    TCP localhost:50052 (LISTEN)

To work around this,run the serveron 50053 with: 

PORT=50053 poetry run duo-workflow-service