Customizing analyzer settings

The API fuzzing behavior can be changed through CI/CD variables.

The API fuzzing configuration files must be in your repository’s .gitlab directory.

All customization of GitLab security scanning tools should be tested in a merge request before merging these changes to the default branch. Failure to do so can give unexpected results, including a large number of false positives.

Authentication

Authentication is handled by providing the authentication token as a header or cookie. You can provide a script that performs an authentication flow or calculates the token.

HTTP Basic Authentication

HTTP basic authentication is an authentication method built into the HTTP protocol and used in conjunction with transport layer security (TLS).

We recommended that you create a CI/CD variable for the password (for example, TEST_API_PASSWORD), and set it to be masked. You can create CI/CD variables from the GitLab project’s page at Settings > CI/CD, in the Variables section. Because of the limitations on masked variables, you should Base64-encode the password before adding it as a variable.

Finally, add two CI/CD variables to your .gitlab-ci.yml file:

FUZZAPI_HTTP_USERNAME: The username for authentication.
FUZZAPI_HTTP_PASSWORD_BASE64: The Base64-encoded password for authentication.

stages:
    - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick-10
  FUZZAPI_HAR: test-api-recording.har
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_HTTP_USERNAME: testuser
  FUZZAPI_HTTP_PASSWORD_BASE64: $TEST_API_PASSWORD

Raw password

If you do not want to Base64-encode the password (or if you are using GitLab 15.3 or earlier) you can provide the raw password FUZZAPI_HTTP_PASSWORD, instead of using FUZZAPI_HTTP_PASSWORD_BASE64.

Bearer Tokens

Bearer tokens are used by several different authentication mechanisms, including OAuth2 and JSON Web Tokens (JWT). Bearer tokens are transmitted using the Authorization HTTP header. To use bearer tokens with API fuzzing, you need one of the following:

A token that doesn’t expire
A way to generate a token that lasts the length of testing
A Python script that API fuzzing can call to generate the token

Token doesn’t expire

If the bearer token doesn’t expire, use the FUZZAPI_OVERRIDES_ENV variable to provide it. This variable’s content is a JSON snippet that provides headers and cookies to add to API fuzzing’s outgoing HTTP requests.

Follow these steps to provide the bearer token with FUZZAPI_OVERRIDES_ENV:

Create a CI/CD variable, for example TEST_API_BEARERAUTH, with the value {"headers":{"Authorization":"Bearer dXNlcm5hbWU6cGFzc3dvcmQ="}} (substitute your token). You can create CI/CD variables from the GitLab projects page at Settings > CI/CD, in the Variables section.

In your .gitlab-ci.yml file, set FUZZAPI_OVERRIDES_ENV to the variable you just created:

stages:
  - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick-10
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_OVERRIDES_ENV: $TEST_API_BEARERAUTH

To validate that authentication is working, run an API fuzzing test and review the fuzzing logs and the test API’s application logs. See the overrides section for more information about override commands.

Token generated at test runtime

If the bearer token must be generated and doesn’t expire during testing, you can provide to API fuzzing with a file containing the token. A prior stage and job, or part of the API fuzzing job, can generate this file.

API fuzzing expects to receive a JSON file with the following structure:

{
  "headers" : {
    "Authorization" : "Bearer dXNlcm5hbWU6cGFzc3dvcmQ="
  }
}

This file can be generated by a prior stage and provided to API fuzzing through the FUZZAPI_OVERRIDES_FILE CI/CD variable.

Set FUZZAPI_OVERRIDES_FILE in your .gitlab-ci.yml file:

stages:
     - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json

To validate that authentication is working, run an API fuzzing test and review the fuzzing logs and the test API’s application logs.

Token has short expiration

If the bearer token must be generated and expires prior to the scan’s completion, you can provide a program or script for the API fuzzer to execute on a provided interval. The provided script runs in an Alpine Linux container that has Python 3 and Bash installed. If the Python script requires additional packages, it must detect this and install the packages at runtime.

The script must create a JSON file containing the bearer token in a specific format:

{
  "headers" : {
    "Authorization" : "Bearer dXNlcm5hbWU6cGFzc3dvcmQ="
  }
}

You must provide three CI/CD variables, each set for correct operation:

FUZZAPI_OVERRIDES_FILE: JSON file the provided command generates.
FUZZAPI_OVERRIDES_CMD: Command that generates the JSON file.
FUZZAPI_OVERRIDES_INTERVAL: Interval (in seconds) to run command.

For example:

stages:
     - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick-10
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json
  FUZZAPI_OVERRIDES_CMD: renew_token.py
  FUZZAPI_OVERRIDES_INTERVAL: 300

To validate that authentication is working, run an API fuzzing test and review the fuzzing logs and the test API’s application logs.

API fuzzing profiles

GitLab provides the configuration file gitlab-api-fuzzing-config.yml. It contains several testing profiles that perform a specific numbers of tests. The runtime of each profile increases as the number of tests increases.

Profile	Fuzz Tests (per parameter)
Quick-10	10
Medium-20	20
Medium-50	50
Long-100	100

Overrides

API Fuzzing provides a method to add or override specific items in your request, for example:

Headers
Cookies
Query string
Form data
JSON nodes
XML nodes

You can use this to inject semantic version headers, authentication, and so on. The authentication section includes examples of using overrides for that purpose.

Overrides use a JSON document, where each type of override is represented by a JSON object:

{
  "headers": {
    "header1": "value",
    "header2": "value"
  },
  "cookies": {
    "cookie1": "value",
    "cookie2": "value"
  },
  "query":      {
    "query-string1": "value",
    "query-string2": "value"
  },
  "body-form":  {
    "form-param1": "value",
    "form-param2": "value"
  },
  "body-json":  {
    "json-path1": "value",
    "json-path2": "value"
  },
  "body-xml" :  {
    "xpath1":    "value",
    "xpath2":    "value"
  }
}

Example of setting a single header:

{
  "headers": {
    "Authorization": "Bearer dXNlcm5hbWU6cGFzc3dvcmQ="
  }
}

Example of setting both a header and cookie:

{
  "headers": {
    "Authorization": "Bearer dXNlcm5hbWU6cGFzc3dvcmQ="
  },
  "cookies": {
    "flags": "677"
  }
}

Example usage for setting a body-form override:

{
  "body-form":  {
    "username": "john.doe"
  }
}

The override engine uses body-form when the request body has only form-data content.

Example usage for setting a body-json override:

{
  "body-json":  {
    "$.credentials.access-token": "iddqd!42.$"
  }
}

Each JSON property name in the object body-json is set to a JSON Path expression. The JSON Path expression $.credentials.access-token identifies the node to be overridden with the value iddqd!42.$. The override engine uses body-json when the request body has only JSON content.

For example, if the body is set to the following JSON:

{
    "credentials" : {
        "username" :"john.doe",
        "access-token" : "non-valid-password"
    }
}

It is changed to:

{
    "credentials" : {
        "username" :"john.doe",
        "access-token" : "iddqd!42.$"
    }
}

Here’s an example for setting a body-xml override. The first entry overrides an XML attribute and the second entry overrides an XML element:

{
  "body-xml" :  {
    "/credentials/@isEnabled": "true",
    "/credentials/access-token/text()" : "iddqd!42.$"
  }
}

Each JSON property name in the object body-xml is set to an XPath v2 expression. The XPath expression /credentials/@isEnabled identifies the attribute node to override with the value true. The XPath expression /credentials/access-token/text() identifies the element node to override with the value iddqd!42.$. The override engine uses body-xml when the request body has only XML content.

For example, if the body is set to the following XML:

<credentials isEnabled="false">
  <username>john.doe</username>
  <access-token>non-valid-password</access-token>
</credentials>

It is changed to:

<credentials isEnabled="true">
  <username>john.doe</username>
  <access-token>iddqd!42.$</access-token>
</credentials>

You can provide this JSON document as a file or environment variable. You may also provide a command to generate the JSON document. The command can run at intervals to support values that expire.

Using a file

To provide the overrides JSON as a file, the FUZZAPI_OVERRIDES_FILE CI/CD variable is set. The path is relative to the job current working directory.

Here’s an example .gitlab-ci.yml:

stages:
     - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json

Using a CI/CD variable

To provide the overrides JSON as a CI/CD variable, use the FUZZAPI_OVERRIDES_ENV variable. This allows you to place the JSON as variables that can be masked and protected.

In this example .gitlab-ci.yml, the FUZZAPI_OVERRIDES_ENV variable is set directly to the JSON:

stages:
     - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_OVERRIDES_ENV: '{"headers":{"X-API-Version":"2"}}'

In this example .gitlab-ci.yml, the SECRET_OVERRIDES variable provides the JSON. This is a group or instance level CI/CD variable defined in the UI:

stages:
     - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_OVERRIDES_ENV: $SECRET_OVERRIDES

Using a command

If the value must be generated or regenerated on expiration, you can provide a program or script for the API fuzzer to execute on a specified interval. The provided script runs in an Alpine Linux container that has Python 3 and Bash installed.

You have to set the environment variable FUZZAPI_OVERRIDES_CMD to the program or script you would like to execute. The provided command creates the overrides JSON file as defined previously.

You might want to install other scripting runtimes like NodeJS or Ruby, or maybe you need to install a dependency for your overrides command. In this case, we recommend setting the FUZZAPI_PRE_SCRIPT to the file path of a script which provides those prerequisites. The script provided by FUZZAPI_PRE_SCRIPT is executed once, before the analyzer starts.

See the Alpine Linux package management page for information about installing Alpine Linux packages.

You must provide three CI/CD variables, each set for correct operation:

FUZZAPI_OVERRIDES_FILE: File generated by the provided command.
FUZZAPI_OVERRIDES_CMD: Overrides command in charge of generating the overrides JSON file periodically.
FUZZAPI_OVERRIDES_INTERVAL: Interval in seconds to run command.

Optionally:

FUZZAPI_PRE_SCRIPT: Script to install runtimes or dependencies before the analyzer starts.

To execute scripts in Alpine Linux you must first use the command chmod to set the execution permission. For example, to set the execution permission of script.py for everyone, use the command: chmod a+x script.py. If needed, you can version your script.py with the execution permission already set.

stages:
     - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json
  FUZZAPI_OVERRIDES_CMD: renew_token.py
  FUZZAPI_OVERRIDES_INTERVAL: 300

Debugging overrides

By default the output of the overrides command is hidden. If the overrides command returns a non zero exit code, the command is displayed as part of your job output. Optionally, you can set the variable FUZZAPI_OVERRIDES_CMD_VERBOSE to any value to display overrides command output as it is generated. This is useful when testing your overrides script, but should be disabled afterwards as it slows down testing.

It is also possible to write messages from your script to a log file that is collected when the job completes or fails. The log file must be created in a specific location and follow a naming convention.

Adding some basic logging to your overrides script is useful in case the script fails unexpectedly during typical running of the job. The log file is automatically included as an artifact of the job, allowing you to download it after the job has finished.

Following our example, we provided renew_token.py in the environmental variable FUZZAPI_OVERRIDES_CMD. Notice two things in the script:

Log file is saved in the location indicated by the environment variable CI_PROJECT_DIR.
Log filename should match gl-*.log.

#!/usr/bin/env python

# Example of an overrides command

# Override commands can update the overrides json file
# with new values to be used.  This is a great way to
# update an authentication token that will expire
# during testing.

import logging
import json
import os
import requests
import backoff

# [1] Store log file in directory indicated by env var CI_PROJECT_DIR
working_directory = os.environ.get( 'CI_PROJECT_DIR')
overrides_file_name = os.environ.get('FUZZAPI_OVERRIDES_FILE', 'api-fuzzing-overrides.json')
overrides_file_path = os.path.join(working_directory, overrides_file_name)

# [2] File name should match the pattern: gl-*.log
log_file_path = os.path.join(working_directory, 'gl-user-overrides.log')

# Set up logger
logging.basicConfig(filename=log_file_path, level=logging.DEBUG)

# Use `backoff` decorator to retry in case of transient errors.
@backoff.on_exception(backoff.expo,
                      (requests.exceptions.Timeout,
                       requests.exceptions.ConnectionError),
                       max_time=30)
def get_auth_response():
    authorization_url = 'https://authorization.service/api/get_api_token'
    return requests.get(
        f'{authorization_url}',
        auth=(os.environ.get('AUTH_USER'), os.environ.get('AUTH_PWD'))
    )

# In our example, access token is retrieved from a given endpoint
try:

    # Performs a http request, response sample:
    # { "Token" : "abcdefghijklmn" }
    response = get_auth_response()

    # Check that the request is successful. may raise `requests.exceptions.HTTPError`
    response.raise_for_status()

    # Gets JSON data
    response_body = response.json()

# If needed specific exceptions can be caught
# requests.ConnectionError                  : A network connection error problem occurred
# requests.HTTPError                        : HTTP request returned an unsuccessful status code. [Response.raise_for_status()]
# requests.ConnectTimeout                   : The request timed out while trying to connect to the remote server
# requests.ReadTimeout                      : The server did not send any data in the allotted amount of time.
# requests.TooManyRedirects                 : The request exceeds the configured number of maximum redirections
# requests.exceptions.RequestException      : All exceptions that related to Requests
except json.JSONDecodeError as json_decode_error:
    # logs errors related decoding JSON response
    logging.error(f'Error, failed while decoding JSON response. Error message: {json_decode_error}')
    raise
except requests.exceptions.RequestException as requests_error:
    # logs  exceptions  related to `Requests`
    logging.error(f'Error, failed while performing HTTP request. Error message: {requests_error}')
    raise
except Exception as e:
    # logs any other error
    logging.error(f'Error, unknown error while retrieving access token. Error message: {e}')
    raise

# computes object that holds overrides file content.
# It uses data fetched from request
overrides_data = {
    "headers": {
        "Authorization": f"Token {response_body['Token']}"
    }
}

# log entry informing about the file override computation
logging.info("Creating overrides file: %s" % overrides_file_path)

# attempts to overwrite the file
try:
    if os.path.exists(overrides_file_path):
        os.unlink(overrides_file_path)

    # overwrites the file with our updated dictionary
    with open(overrides_file_path, "wb+") as fd:
        fd.write(json.dumps(overrides_data).encode('utf-8'))
except Exception as e:
    # logs any other error
    logging.error(f'Error, unknown error when overwriting file {overrides_file_path}. Error message: {e}')
    raise

# logs informing override has finished successfully
logging.info("Override file has been updated")

# end

In the overrides command example, the Python script depends on the backoff library. To make sure the library is installed before executing the Python script, the FUZZAPI_PRE_SCRIPT is set to a script that installs the dependencies of your overrides command. As for example, the following script user-pre-scan-set-up.sh:

#!/bin/bash

# user-pre-scan-set-up.sh
# Ensures python dependencies are installed

echo "**** install python dependencies ****"

python3 -m ensurepip
pip3 install --no-cache --upgrade \
    pip \
    requests \
    backoff

echo "**** python dependencies installed ****"

# end

You have to update your configuration to set the FUZZAPI_PRE_SCRIPT to our new user-pre-scan-set-up.sh script. For example:

stages:
     - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_PRE_SCRIPT: user-pre-scan-set-up.sh
  FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json
  FUZZAPI_OVERRIDES_CMD: renew_token.py
  FUZZAPI_OVERRIDES_INTERVAL: 300

In the previous sample, you could use the script user-pre-scan-set-up.sh to also install new runtimes or applications that later on you could use in your overrides command.

Exclude Paths

When testing an API it can be useful to exclude certain paths. For example, you might exclude testing of an authentication service or an older version of the API. To exclude paths, use the FUZZAPI_EXCLUDE_PATHS CI/CD variable . This variable is specified in your .gitlab-ci.yml file. To exclude multiple paths, separate entries using the ; character. In the provided paths you can use a single character wildcard ? and * for a multiple character wildcard.

To verify the paths are excluded, review the Tested Operations and Excluded Operations portion of the job output. You should not see any excluded paths listed under Tested Operations.

2021-05-27 21:51:08 [INF] API Fuzzing: --[ Tested Operations ]-------------------------
2021-05-27 21:51:08 [INF] API Fuzzing: 201 POST http://target:7777/api/users CREATED
2021-05-27 21:51:08 [INF] API Fuzzing: ------------------------------------------------
2021-05-27 21:51:08 [INF] API Fuzzing: --[ Excluded Operations ]-----------------------
2021-05-27 21:51:08 [INF] API Fuzzing: GET http://target:7777/api/messages
2021-05-27 21:51:08 [INF] API Fuzzing: POST http://target:7777/api/messages
2021-05-27 21:51:08 [INF] API Fuzzing: ------------------------------------------------

Examples of excluding paths

This example excludes the /auth resource. This does not exclude child resources (/auth/child).

variables:
  FUZZAPI_EXCLUDE_PATHS: /auth

To exclude /auth, and child resources (/auth/child), we use a wildcard.

variables:
  FUZZAPI_EXCLUDE_PATHS: /auth*

To exclude multiple paths we can use the ; character. In this example we exclude /auth* and /v1/*.

variables:
  FUZZAPI_EXCLUDE_PATHS: /auth*;/v1/*

Exclude parameters

While testing an API you may might want to exclude a parameter (query string, header, or body element) from testing. This may be needed because a parameter always causes a failure, slows down testing, or for other reasons. To exclude parameters you can use one of the following variables: FUZZAPI_EXCLUDE_PARAMETER_ENV or FUZZAPI_EXCLUDE_PARAMETER_FILE.

The FUZZAPI_EXCLUDE_PARAMETER_ENV allows providing a JSON string containing excluded parameters. This is a good option if the JSON is short and can not often change. Another option is the variable FUZZAPI_EXCLUDE_PARAMETER_FILE. This variable is set to a file path that can be checked into the repository, created by another job as an artifact, or generated at runtime from a pre-script using FUZZAPI_PRE_SCRIPT.

Exclude parameters using a JSON document

The JSON document contains a JSON object which uses specific properties to identify which parameter should be excluded. You can provide the following properties to exclude specific parameters during the scanning process:

headers: Use this property to exclude specific headers. The property’s value is an array of header names to be excluded. Names are case-insensitive.
cookies: Use this property’s value to exclude specific cookies. The property’s value is an array of cookie names to be excluded. Names are case-sensitive.
query: Use this property to exclude specific fields from the query string. The property’s value is an array of field names from the query string to be excluded. Names are case-sensitive.
body-form: Use this property to exclude specific fields from a request that uses the media type application/x-www-form-urlencoded. The property’s value is an array of the field names from the body to be excluded. Names are case-sensitive.
body-json: Use this property to exclude specific JSON nodes from a request that uses the media type application/json. The property’s value is an array, each entry of the array is a JSON Path expression.
body-xml: Use this property to exclude specific XML nodes from a request that uses media type application/xml. The property’s value is an array, each entry of the array is a XPath v2 expression.

The following JSON document is an example of the expected structure to exclude parameters.

{
  "headers": [
    "header1",
    "header2"
  ],
  "cookies": [
    "cookie1",
    "cookie2"
  ],
  "query": [
    "query-string1",
    "query-string2"
  ],
  "body-form": [
    "form-param1",
    "form-param2"
  ],
  "body-json": [
    "json-path-expression-1",
    "json-path-expression-2"
  ],
  "body-xml" : [
    "xpath-expression-1",
    "xpath-expression-2"
  ]
}

Examples

Excluding a single header

To exclude the header Upgrade-Insecure-Requests, set the header property’s value to an array with the header name: [ "Upgrade-Insecure-Requests" ]. For instance, the JSON document looks like this:

{
  "headers": [ "Upgrade-Insecure-Requests" ]
}

Header names are case-insensitive, thus the header name UPGRADE-INSECURE-REQUESTS is equivalent to Upgrade-Insecure-Requests.

Excluding both a header and two cookies

To exclude the header Authorization and the cookies PHPSESSID and csrftoken, set the headers property’s value to an array with header name [ "Authorization" ] and the cookies property’s value to an array with the cookies’ names [ "PHPSESSID", "csrftoken" ]. For instance, the JSON document looks like this:

{
  "headers": [ "Authorization" ],
  "cookies": [ "PHPSESSID", "csrftoken" ]
}

Excluding a `body-form` parameter

To exclude the password field in a request that uses application/x-www-form-urlencoded, set the body-form property’s value to an array with the field name [ "password" ]. For instance, the JSON document looks like this:

{
  "body-form":  [ "password" ]
}

The exclude parameters uses body-form when the request uses a content type application/x-www-form-urlencoded.

Excluding a specific JSON nodes using JSON Path

To exclude the schema property in the root object, set the body-json property’s value to an array with the JSON Path expression [ "$.schema" ].

The JSON Path expression uses special syntax to identify JSON nodes: $ refers to the root of the JSON document, . refers to the current object (in our case the root object), and the text schema refers to a property name. Thus, the JSON path expression $.schema refers to a property schema in the root object. For instance, the JSON document looks like this:

{
  "body-json": [ "$.schema" ]
}

The exclude parameters uses body-json when the request uses a content type application/json. Each entry in body-json is expected to be a JSON Path expression. In JSON Path, characters like $, *, . among others have special meaning.

Excluding multiple JSON nodes using JSON Path

To exclude the property password on each entry of an array of users at the root level, set the body-json property’s value to an array with the JSON Path expression [ "$.users[*].paswword" ].

The JSON Path expression starts with $ to refer to the root node and uses . to refer to the current node. Then, it uses users to refer to a property and the characters [ and ] to enclose the index in the array you want to use, instead of providing a number as an index you use * to specify any index. After the index reference, we find . which now refers to any given selected index in the array, preceded by a property name password.

For instance, the JSON document looks like this:

{
  "body-json": [ "$.users[*].paswword" ]
}

The exclude parameters uses body-json when the request uses a content type application/json. Each entry in body-json is expected to be a JSON Path expression. In JSON Path characters like $, *, . among others have special meaning.

Excluding an XML attribute

To exclude an attribute named isEnabled located in the root element credentials, set the body-xml property’s value to an array with the XPath expression [ "/credentials/@isEnabled" ].

The XPath expression /credentials/@isEnabled, starts with / to indicate the root of the XML document, then it is followed by the word credentials which indicates the name of the element to match. It uses a / to refer to a node of the previous XML element, and the character @ to indicate that the name isEnable is an attribute.

For instance, the JSON document looks like this:

{
  "body-xml": [
    "/credentials/@isEnabled"
  ]
}

The exclude parameters uses body-xml when the request uses a content type application/xml. Each entry in body-xml is expected to be an XPath v2 expression. In XPath expressions, characters like @, /, :, [, ] among others have special meanings.

Excluding an XML element’s text

To exclude the text of the username element contained in root node credentials, set the body-xml property’s value to an array with the XPath expression [/credentials/username/text()" ].

In the XPath expression /credentials/username/text(), the first character / refers to the root XML node, and then after it indicates an XML element’s name credentials. Similarly, the character / refers to the current element, followed by a new XML element’s name username. Last part has a / that refers to the current element, and uses a XPath function called text() which identifies the text of the current element.

For instance, the JSON document looks like this:

{
  "body-xml": [
    "/credentials/username/text()"
  ]
}

The exclude parameters uses body-xml when the request uses a content type application/xml. Each entry in body-xml is expected to be a XPath v2 expression. In XPath expressions characters like @, /, :, [, ] among others have special meanings.

Excluding an XML element

To exclude the element username contained in root node credentials, set the body-xml property’s value to an array with the XPath expression [/credentials/username" ].

In the XPath expression /credentials/username, the first character / refers to the root XML node, and then after it indicates an XML element’s name credentials. Similarly, the character / refers to the current element, followed by a new XML element’s name username.

For instance, the JSON document looks like this:

{
  "body-xml": [
    "/credentials/username"
  ]
}

Excluding an XML node with namespaces

To exclude a XML element login which is defined in namespace s, and contained in credentials root node, set the body-xml property’s value to an array with the XPath expression [ "/credentials/s:login" ].

In the XPath expression /credentials/s:login, the first character / refers to the root XML node, and then after it indicates an XML element’s name credentials. Similarly, the character / refers to the current element, followed by a new XML element’s name s:login. Notice that name contains the character :, this character separates the namespace from the node name.

The namespace name should have been defined in the XML document which is part of the body request. You may check the namespace in the specification document HAR, OpenAPI, or Postman Collection file.

{
  "body-xml": [
    "/credentials/s:login"
  ]
}

Using a JSON string

To provide the exclusion JSON document set the variable FUZZAPI_EXCLUDE_PARAMETER_ENV with the JSON string. In the following example, the .gitlab-ci.yml, the FUZZAPI_EXCLUDE_PARAMETER_ENV variable is set to a JSON string:

stages:
     - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_EXCLUDE_PARAMETER_ENV: '{ "headers": [ "Upgrade-Insecure-Requests" ] }'

Using a file

To provide the exclusion JSON document, set the variable FUZZAPI_EXCLUDE_PARAMETER_FILE with the JSON file path. The file path is relative to the job current working directory. In the following example .gitlab-ci.yml file, the FUZZAPI_EXCLUDE_PARAMETER_FILE variable is set to a JSON file path:

stages:
     - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_EXCLUDE_PARAMETER_FILE: api-fuzzing-exclude-parameters.json

The api-fuzzing-exclude-parameters.json is a JSON document that follows the structure of exclude parameters document.

Exclude URLs

As an alternative to excluding by paths, you can filter by any other component in the URL by using the FUZZAPI_EXCLUDE_URLS CI/CD variable. This variable can be set in your .gitlab-ci.yml file. The variable can store multiple values, separated by commas (,). Each value is a regular expression. Because each entry is a regular expression, an entry such as .* excludes all URLs because it is a regular expression that matches everything.

In your job output you can check if any URLs matched any provided regular expression from FUZZAPI_EXCLUDE_URLS. Matching operations are listed in the Excluded Operations section. Operations listed in the Excluded Operations should not be listed in the Tested Operations section. For example the following portion of a job output:

2021-05-27 21:51:08 [INF] API Fuzzing: --[ Tested Operations ]-------------------------
2021-05-27 21:51:08 [INF] API Fuzzing: 201 POST http://target:7777/api/users CREATED
2021-05-27 21:51:08 [INF] API Fuzzing: ------------------------------------------------
2021-05-27 21:51:08 [INF] API Fuzzing: --[ Excluded Operations ]-----------------------
2021-05-27 21:51:08 [INF] API Fuzzing: GET http://target:7777/api/messages
2021-05-27 21:51:08 [INF] API Fuzzing: POST http://target:7777/api/messages
2021-05-27 21:51:08 [INF] API Fuzzing: ------------------------------------------------

Each value in FUZZAPI_EXCLUDE_URLS is a regular expression. Characters such as . , * and $ among many others have special meanings in regular expressions.

Examples

Excluding a URL and child resources

The following example excludes the URL http://target/api/auth and its child resources.

stages:
  - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_TARGET_URL: http://target/
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_EXCLUDE_URLS: http://target/api/auth

Excluding two URLs and allow their child resources

To exclude the URLs http://target/api/buy and http://target/api/sell but allowing to scan their child resources, for instance: http://target/api/buy/toy or http://target/api/sell/chair. You could use the value http://target/api/buy/$,http://target/api/sell/$. This value is using two regular expressions, each of them separated by a , character. Hence, it contains http://target/api/buy$ and http://target/api/sell$. In each regular expression, the trailing $ character points out where the matching URL should end.

stages:
  - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_TARGET_URL: http://target/
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_EXCLUDE_URLS: http://target/api/buy/$,http://target/api/sell/$

Excluding two URLs and their child resources

To exclude the URLs: http://target/api/buy and http://target/api/sell, and their child resources. To provide multiple URLs we use the , character as follows:

stages:
  - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_TARGET_URL: http://target/
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_EXCLUDE_URLS: http://target/api/buy,http://target/api/sell

Excluding URL using regular expressions

To exclude exactly https://target/api/v1/user/create and https://target/api/v2/user/create or any other version (v3,v4, and more), we could use https://target/api/v.*/user/create$. In the previous regular expression:

. indicates any character.
* indicates zero or more times.
$ indicates that the URL should end there.

stages:
  - fuzz

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_TARGET_URL: http://target/
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_EXCLUDE_URLS: https://target/api/v.*/user/create$

Header Fuzzing

Header fuzzing is disabled by default due to the high number of false positives that occur with many technology stacks. When header fuzzing is enabled, you must specify a list of headers to include in fuzzing.

Each profile in the default configuration file has an entry for GeneralFuzzingCheck. This check performs header fuzzing. Under the Configuration section, you must change the HeaderFuzzing and Headers settings to enable header fuzzing.

This snippet shows the Quick-10 profile’s default configuration with header fuzzing disabled:

- Name: Quick-10
  DefaultProfile: Empty
  Routes:
  - Route: *Route0
    Checks:
    - Name: FormBodyFuzzingCheck
      Configuration:
        FuzzingCount: 10
        UnicodeFuzzing: true
    - Name: GeneralFuzzingCheck
      Configuration:
        FuzzingCount: 10
        UnicodeFuzzing: true
        HeaderFuzzing: false
        Headers:
    - Name: JsonFuzzingCheck
      Configuration:
        FuzzingCount: 10
        UnicodeFuzzing: true
    - Name: XmlFuzzingCheck
      Configuration:
        FuzzingCount: 10
        UnicodeFuzzing: true

HeaderFuzzing is a boolean that turns header fuzzing on and off. The default setting is false for off. To turn header fuzzing on, change this setting to true:

    - Name: GeneralFuzzingCheck
      Configuration:
        FuzzingCount: 10
        UnicodeFuzzing: true
        HeaderFuzzing: true
        Headers:

Headers is a list of headers to fuzz. Only headers listed are fuzzed. To fuzz a header used by your APIs, add an entry for it using the syntax - Name: HeaderName. For example, to fuzz a custom header X-Custom, add - Name: X-Custom:

    - Name: GeneralFuzzingCheck
      Configuration:
        FuzzingCount: 10
        UnicodeFuzzing: true
        HeaderFuzzing: true
        Headers:
          - Name: X-Custom

You now have a configuration to fuzz the header X-Custom. Use the same notation to list additional headers:

    - Name: GeneralFuzzingCheck
      Configuration:
        FuzzingCount: 10
        UnicodeFuzzing: true
        HeaderFuzzing: true
        Headers:
          - Name: X-Custom
          - Name: X-AnotherHeader

Repeat this configuration for each profile as needed.

Customizing analyzer settings

Authentication

HTTP Basic Authentication

Raw password

Bearer Tokens

Token doesn’t expire

Token generated at test runtime

Token has short expiration

API fuzzing profiles

Overrides

Using a file

Using a CI/CD variable

Using a command

Debugging overrides

Exclude Paths

Examples of excluding paths

Exclude parameters

Exclude parameters using a JSON document

Examples

Excluding a single header

Excluding both a header and two cookies

Excluding a body-form parameter

Excluding a specific JSON nodes using JSON Path

Excluding multiple JSON nodes using JSON Path

Excluding an XML attribute

Excluding an XML element’s text

Excluding an XML element

Excluding an XML node with namespaces

Using a JSON string

Using a file

Exclude URLs

Examples

Excluding a URL and child resources

Excluding two URLs and allow their child resources

Excluding two URLs and their child resources

Excluding URL using regular expressions

Header Fuzzing

Help & feedback

Docs

Product

Feature availability and product trials

Get Help

Excluding a `body-form` parameter