Documentation testing

GitLab documentation is stored in projects with code and treated like code. Therefore, we use processes similar to those used for code to maintain standards and quality of documentation.

We have tests:

  • To lint the words and structure of the documentation.
  • To check the validity of internal links within the documentation suite.
  • To check the validity of links from UI elements, such as files in app/views files.

For the specifics of each test run in our CI/CD pipelines, see the configuration for those tests in the relevant projects:

We also run some documentation tests in the GitLab Development Kit project: https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/.gitlab/ci/test.gitlab-ci.yml.

Run tests locally

Similar to previewing your changes locally, you can also run these tests on your local computer. This has the advantage of:

  • Speeding up the feedback loop. You can know of any problems with the changes in your branch without waiting for a CI/CD pipeline to run.
  • Lowering costs. Running tests locally is cheaper than running tests on the cloud infrastructure GitLab uses.

To run tests locally, it’s important to:

Lint checks

Lint checks are performed by the lint-doc.sh script and can be executed as follows:

  1. Navigate to the gitlab directory.
  2. Run:

    MD_DOC_PATH=path/to/my_doc.md scripts/lint-doc.sh
    

Where MD_DOC_PATH points to the file or directory you would like to run lint checks for. If you omit it completely, it defaults to the doc/ directory. The output should be similar to:

=> Linting documents at path /path/to/gitlab as <user>...
=> Checking for cURL short options...
=> Checking for CHANGELOG.md duplicate entries...
=> Checking /path/to/gitlab/doc for executable permissions...
=> Checking for new README.md files...
=> Linting markdown style...
=> Linting prose...
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.
✔ Linting passed

This requires you to either:

  • Have the required lint tools installed on your computer.
  • A working Docker installation, in which case an image with these tools pre-installed is used.

To execute documentation link tests locally:

  1. Navigate to the gitlab-docs directory.
  2. Run the following commands:

    # Check for broken internal links
    bundle exec nanoc check internal_links
    
    # Check for broken external links (might take a lot of time to complete).
    # This test is set to be allowed to fail and is run only in the gitlab-docs project CI
    bundle exec nanoc check internal_anchors
    

The ui-docs-links lint job uses haml-lint to test that all documentation links from UI elements (app/views files, for example) are linking to valid pages and anchors.

To run the ui-docs-links test locally:

  1. Open the gitlab directory in a terminal window.
  2. Run:

    bundle exec haml-lint -i DocumentationLinks
    

If you receive an error the first time you run this test, run bundle install, which installs the dependencies for GitLab, and try again.

If you don’t want to install all of the dependencies to test the links, you can:

  1. Open the gitlab directory in a terminal window.
  2. Install haml-lint:

    gem install haml_lint
    
  3. Run:

    haml-lint -i DocumentationLinks
    

If you manually install haml-lint with this process, it does not update automatically and you should make sure your version matches the version used by GitLab.

Update linter configuration

Vale configuration and markdownlint configuration is under source control in each project, so updates must be committed to each project individually.

We consider the configuration in the gitlab project as the source of truth and that’s where all updates should first be made.

On a regular basis, the changes made in gitlab project to the Vale and markdownlint configuration should be synchronized to the other projects. In omnibus-gitlab, gitlab-runner, and charts/gitlab:

  1. Create a new branch.
  2. Copy the configuration files from the gitlab project into this branch, overwriting the project’s old configuration. Make sure no project-specific changes from the gitlab project are included. For example, RelativeLinks.yml is hard coded for specific projects.
  3. Create a merge request and submit it to a technical writer for review and merge.

Update linting images

Lint tests run in CI/CD pipelines using images from the gitlab-docs container registry.

If a new version of a dependency is released (like a new version of Ruby), we should update the images to use the newer version. Then, we can update the configuration files in each of our documentation projects to point to the new image.

To update the linting images:

  1. In gitlab-docs, open a merge request to update .gitlab-ci.yml to use the new tooling version. (Example MR)
  2. When merged, start a Build docs.gitlab.com every 4 hours scheduled pipeline.
  3. Go the pipeline you started, and manually run the relevant build-images job, for example, image:docs-lint-markdown.
  4. In the job output, get the name of the new image. (Example job output)
  5. Verify that the new image was added to the container registry.
  6. Open merge requests to update each of these configuration files to point to the new image. In each merge request, include a small doc update to trigger the job that uses the image.
  7. In each merge request, check the relevant job output to confirm the updated image was used for the test. (Example job output)
  8. Assign the merge requests to any technical writer to review and merge.

Local linters

To help adhere to the documentation style guidelines, and improve the content added to documentation, install documentation linters and integrate them with your code editor.

At GitLab, we mostly use:

markdownlint

markdownlint checks that Markdown syntax follows certain rules, and is used by the docs-lint test.

Our Documentation Style Guide and Markdown Guide elaborate on which choices must be made when selecting Markdown syntax for GitLab documentation. This tool helps catch deviations from those guidelines.

markdownlint configuration is found in the following projects:

This configuration is also used in build pipelines.

You can use markdownlint:

Vale

Vale is a grammar, style, and word usage linter for the English language. Vale’s configuration is stored in the .vale.ini file located in the root directory of projects.

Vale supports creating custom tests that extend any of several types of checks, which we store in the .linting/vale/styles/gitlab directory in the documentation directory of projects.

You can find Vale configuration in the following projects:

This configuration is also used in build pipelines, where error-level rules are enforced.

You can use Vale:

Vale result types

Vale returns three types of results:

  • Error - For branding and trademark issues, and words or phrases with ambiguous meanings.
  • Warning - For Technical Writing team style preferences.
  • Suggestion - For basic technical writing tenets and best practices.

The result types have these attributes:

Result type Displayed in CI/CD job output Causes CI/CD jobs to fail Vale rule link
error Yes Yes Error-level Vale rules
warning No No Warning-level Vale rules
suggestion No No Suggestion-level Vale rules

Vale spelling test

When Vale flags a valid word as a spelling mistake, you can fix it following these guidelines:

Flagged word Guideline
jargon Rewrite the sentence to avoid it.
correctly-capitalized name of a product or service Add the word to the vale spelling exceptions list.
name of a person Remove the name if it’s not needed, or add the vale exception code in-line.
a command, variable, code, or similar Put it in backticks or a code block. For example: The git clone command can be used with the CI_COMMIT_BRANCH variable. -> The `git clone` command can be used with the `CI_COMMIT_BRANCH` variable.
UI text from GitLab Verify it correctly matches the UI, then: If it does not match the UI, update it. If it matches the UI, but the UI seems incorrect, create an issue to see if the UI needs to be fixed. If it matches the UI and seems correct, add it to the vale spelling exceptions list.
UI text from a third-party product Rewrite the sentence to avoid it, or add the vale exception code in-line.

Vale uppercase (acronym) test

The Uppercase.yml test checks for incorrect usage of words in all capitals. For example, avoid usage like This is NOT important.

If the word must be in all capitals, follow these guidelines:

Flagged word Guideline
Acronym (likely known by the average visitor to that page) Add the acronym to the list of words and acronyms in Uppercase.yml.
Acronym (likely not known by the average visitor to that page) The first time the acronym is used, write it out fully followed by the acronym in parentheses. In later uses, use just the acronym by itself. For example: This feature uses the File Transfer Protocol (FTP). FTP is....
Correctly capitalized name of a product or service Add the name to the list of words and acronyms in Uppercase.yml.
Command, variable, code, or similar Put it in backticks or a code block. For example: Use `FALSE` as the variable value.
UI text from a third-party product Rewrite the sentence to avoid it, or add the vale exception code in-line.

Vale readability score

In ReadingLevel.yml, we have implemented the Flesch-Kincaid grade level test to determine the readability of our documentation.

As a general guideline, the lower the score, the more readable the documentation. For example, a page that s