API style guide

This style guide recommends best practices for API development.

GraphQL and REST APIs

We offer two types of API to our customers:

To reduce the technical burden of supporting two APIs in parallel, they should share implementations as much as possible. For example, they could share the same services.

Frontend

See the frontend guide on details on which API to use when developing in the frontend.

Instance variables

Don’t use instance variables, there is no need for them (we don’t need to access them as we do in Rails views), local variables are fine.

Entities

Always use an Entity to present the endpoint’s payload.

Documentation

Each new or updated API endpoint must come with documentation, unless it is internal or behind a feature flag. The docs should be in the same merge request, or, if strictly necessary, in a follow-up with the same milestone as the original merge request.

See the Documentation Style Guide RESTful API page for details on documenting API resources in Markdown as well as in OpenAPI definition files.

Methods and parameters description

Every method must be described using the Grape DSL (see environments.rb for a good example):

  • desc for the method summary. You should pass it a block for additional details such as:
    • The GitLab version when the endpoint was added. If it is behind a feature flag, mention that instead: This feature is gated by the :feature_flag_symbol feature flag.
    • If the endpoint is deprecated, and if so, its planned removal date
  • params for the method parameters. This acts as description, validation, and coercion of the parameters

A good example is as follows: 

desc 'Get all broadcast messages' do
  detail 'This feature was introduced in GitLab 8.12.'
  success Entities::System::BroadcastMessage
end
params do
  optional :page,     type: Integer, desc: 'Current page number'
  optional :per_page, type: Integer, desc: 'Number of messages per page'
end
get do
  messages = System::BroadcastMessage.all

  present paginate(messages), with: Entities::System::BroadcastMessage
end

Breaking changes

We must not make breaking changes to our REST API v4, even in major GitLab releases.

Our REST API maintains its own versioning independent of GitLab versioning. The current REST API version is 4. We commit to follow semantic versioning for our REST API, which means we cannot make breaking changes until a major version change (most likely, 5).

Because version 5 is not scheduled, we allow rare exceptions.

Accommodating backward compatibility instead of breaking changes

Backward compatibility can often be accommodated in the API by continuing to adapt a changed feature to the old API schema. For example, our REST API exposes both work_in_progress and draft fields.

Exceptions

The exception is only when:

This exception should be rare.

Even in this exception, rather than removing a field or argument, we must always do the following:

  • Return an empty response for a field (for example, "null" or []).
  • Turn an argument into a no-op.

What is a breaking change

Some examples of breaking changes are:

  • Removing or renaming fields, arguments, or enum values. In a JSON response, a field is any JSON key.
  • Removing endpoints.
  • Adding new redirects (not all clients follow redirects).
  • Changing the content type of any response.
  • Changing the type of fields in the response. In a JSON response, this would be a change of any Number, String, Boolean, Array, or Object type to another type.
  • Adding a new required argument.
  • Changing authentication, authorization, or other header requirements.
  • Changing any status code other than 500.

What is not a breaking change

Some examples of non-breaking changes:

  • Any additive change, such as adding endpoints, non-required arguments, fields, or enum values.
  • Changes to error messages.
  • Changes from a 500 status code to any supported status code (this is a bugfix).
  • Changes to the order of fields returned in a response.

Experimental, beta, and generally available features

You can add API elements as experimental and beta features. They must be additive changes, otherwise they are categorized as a breaking change.

API elements marked as experiment or beta are exempt from the ensuring backward compatibility policy, and can be changed or removed at any time without prior notice.

While in the experiment status:

While in the beta status:

When the feature becomes generally available:

Declared parameters

Grape allows you to access only the parameters that have been declared by your params block. It filters out the parameters that have been passed, but are not allowed.

– https://github.com/ruby-grape/grape#declared

Exclude parameters from parent namespaces

By default declared(params)includes parameters that were defined in all parent namespaces.

– https://github.com/ruby-grape/grape#include-parent-namespaces

In most cases you should exclude parameters from the parent namespaces: 

declared(params, include_parent_namespaces: false)

When to use declared(params)

You should always use declared(params) when you pass the parameters hash as arguments to a method call.

For instance: 

# bad
User.create(params) # imagine the user submitted `admin=1`... :)

# good
User.create(declared(params, include_parent_namespaces: false).to_h)
note
declared(params) return a Hashie::Mash object, on which you must call .to_h.

But we can use params[key] directly when we access single elements.

For instance: 

# good
Model.create(foo: params[:foo])

Array types

With Grape v1.3+, Array types must be defined with a coerce_with block, or parameters, fails to validate when passed a string from an API request. See the Grape upgrading documentation for more details.

Automatic coercion of nil inputs

Prior to Grape v1.3.3, Array parameters with nil values would automatically be coerced to an empty Array. However, due to this pull request in v1.3.3, this is no longer the case. For example, suppose you define a PUT /test request that has an optional parameter: 

optional :user_ids, type: Array[Integer], coerce_with: ::API::Validations::Types::CommaSeparatedToIntegerArray.coerce, desc: 'The user ids for this rule'

Usually, a request to PUT /test?user_ids would cause Grape to pass params of { user_ids: nil }.

This may introduce errors with endpoints that expect a blank array and do not handle nil inputs properly. To preserve the previous behavior, there is a helper method coerce_nil_params_to_array! that is used in the before block of all API calls: 

before do
  coerce_nil_params_to_array!
end

With this change, a request to PUT /test?user_ids causes Grape to pass params to be { user_ids: [] }.

There is an open issue in the Grape tracker to make this easier.

Using HTTP status helpers

For non-200 HTTP responses, use the provided helpers in lib/api/helpers.rb to ensure correct behavior (like not_found! or no_content!). These throw inside Grape and abort the execution of your endpoint.

For DELETE requests, you should also generally use the destroy_conditionally! helper which by default returns a 204 No Content response on success, or a 412 Precondition Failed response if the given If-Unmodified-Since header is out of range. This helper calls #destroy on the passed resource, but you can also implement a custom deletion method by passing a block.

Choosing HTTP verbs

When defining a new API route, use the correct HTTP request method.

Deciding between PATCH and PUT

In a Rails application, both the PATCH and PUT request methods are routed to the update method in controllers. With Grape, the framework we use to write the GitLab API, you must explicitly set the PATCH or PUT HTTP verb for an endpoint that does updates.

If the endpoint updates all attributes of a given resource, use the PUT request method. If the endpoint updates some attributes of a given resource, use the PATCH request method.

Here is a good example for PATCH: PATCH /projects/:id/protected_branches/:name Here is a good example for PUT: PUT /projects/:id/merge_requests/:merge_request_iid/approve

Often, a good PUT endpoint only has ids and a verb (in the example above, “approve”). Or, they only have a single value and represent a key/value pair.

The Rails blog has a detailed explanation of why PATCH is usually the most apt verb for web API endpoints that perform an update.

Using API path helpers in GitLab Rails codebase

Because we support installing GitLab under a relative URL, one must take this into account when using API path helpers generated by Grape. Any such API path helper usage must be in wrapped into the expose_path helper call.

For instance: 

- endpoint = expose_path(api_v4_projects_issues_related_merge_requests_path(id: @project.id, issue_iid: @issue.iid))

Custom Validators

In order to validate some parameters in the API request, we validate them before sending them further (say Gitaly). The following are the custom validators, which we have added so far and how to use them. We also wrote a guide on how you can add a new custom validator.

Using custom validators

  • FilePath:

    GitLab supports various functionalities where we need to traverse a file path. The FilePath validator validates the parameter value for different cases. Mainly, it checks whether a path is relative and does it contain ../../ relative traversal using File::Separator or not, and whether the path is absolute, for example /etc/passwd/. By default, absolute paths are not allowed. However, you can optionally pass in an allowlist for allowed absolute paths in the following way: requires :file_path, type: String, file_path: { allowlist: ['/foo/bar/', '/home/foo/', '/app/home'] }

  • Git SHA:

    The Git SHA validator checks whether the Git SHA parameter is a valid SHA. It checks by using the regex mentioned in commit.rb file.

  • Absence:

    The Absence validator checks whether a particular parameter is absent in a given parameters hash.

  • IntegerNoneAny:

    The IntegerNoneAny validator checks if the value of the given parameter is either an Integer, None, or Any. It allows only either of these mentioned values to move forward in the request.

  • ArrayNoneAny:

    The ArrayNoneAny validator checks if the value of the given parameter is either an Array, None, or Any. It allows only either of these mentioned values to move forward in the request.

  • EmailOrEmailList:

    The EmailOrEmailList validator checks if the value of a string or a list of strings contains only valid email addresses. It allows only lists with all valid email addresses to move forward in the request.

Adding a new custom validator

Custom validators are a great way to validate parameters before sending them to platform for further processing. It saves some back-and-forth from the server to the platform if we identify invalid parameters at the beginning.

If you need to add a custom validator, it would be added to it’s own file in the validators directory. Since we use Grape to add our API we inherit from the Grape::Validations::Validators::Base class in our validator class. Now, all you have to do is define the validate_param! method which takes in two parameters: the params hash and the param name to validate.

The body of the method does the hard work of validating the parameter value and returns appropriate error messages to the caller method.

Lastly, we register the validator using the line below: 

Grape::Validations.register_validator(<validator name as symbol>, ::API::Helpers::CustomValidators::<YourCustomValidatorClassName>)

Once you add the validator, make sure you add the rspecs for it into it’s own file in the validators directory.

Internal API

The internal API is documented for internal use. Keep it up to date so we know what endpoints different components are making use of.

Avoiding N+1 problems

In order to avoid N+1 problems that are common when returning collections of records in an API endpoint, we should use eager loading.

A standard way to do this within the API is for models to implement a scope called with_api_entity_associations that preloads the associations and data returned in the API. An example of this scope can be seen in the Issue model.

In situations where the same model has multiple entities in the API (for instance, UserBasic, User and UserPublic) you should use your discretion with applying this scope. It may be that you optimize for the most basic entity, with successive entities building upon that scope.

The with_api_entity_associations scope also automatically preloads data for Todo targets when returned in the to-dos API.

For more context and discussion about preloading see this merge request which introduced the scope.

Verifying with tests

When an API endpoint returns collections, always add a test to verify that the API endpoint does not have an N+1 problem, now and in the future. We can do this using ActiveRecord::QueryRecorder.

Example: 

def make_api_request
  get api('/foo', personal_access_token: pat)
end

it 'avoids N+1 queries', :request_store do
  # Firstly, record how many PostgreSQL queries the endpoint will make
  # when it returns a single record
  create_record

  control = ActiveRecord::QueryRecorder.new { make_api_request }

  # Now create a second record and ensure that the API does not execute
  # any more queries than before
  create_record

  expect { make_api_request }.not_to exceed_query_limit(control)
end

Testing

When writing tests for new API endpoints, consider using a schema fixture located in /spec/fixtures/api/schemas. You can expect a response to match a given schema: 

expect(response).to match_response_schema('merge_requests')

Also see verifying N+1 performance in tests.

Include a changelog entry

All client-facing changes must include a changelog entry. This does not include internal APIs.