• Installing applications
  • Helm
  • cert-manager
  • GitLab Runner
  • Ingress
    • Determining the external endpoint automatically
    • Determining the external endpoint manually
    • Using a static IP
    • Pointing your DNS at the external endpoint
    • Web Application Firewall (ModSecurity)
  • JupyterHub
    • Jupyter Git Integration
  • Knative
  • Prometheus
  • Crossplane
  • Elastic Stack
• Install using GitLab CI (alpha)
  • Usage
  • Install Ingress using GitLab CI
  • Install cert-manager using GitLab CI
  • Install Sentry using GitLab CI
  • Install GitLab Runner using GitLab CI
  • Install Cilium using GitLab CI
  • Install JupyterHub using GitLab CI
  • Install Elastic Stack using GitLab CI
• Upgrading applications
• Uninstalling applications
• Troubleshooting applications

GitLab Managed Apps

GitLab provides GitLab Managed Apps, a one-click install for various applications which can be added directly to your configured cluster.

These applications are needed for Review Apps and deployments when using Auto DevOps.

You can install them after you create a cluster.

Interested in contributing a new GitLab managed app? Visit the development guidelines page to get started.

Installing applications

Applications managed by GitLab will be installed onto the gitlab-managed-apps namespace.

This namespace:

• Is different from the namespace used for project deployments.
• Is created once.
• Has a non-configurable name.

To see a list of available applications to install. For a:

• Project-level cluster, navigate to your project’s Operations > Kubernetes.
• Group-level cluster, navigate to your group’s Kubernetes page.

Install Helm first as it’s used to install other applications.

The following applications can be installed:

• Helm
• Ingress
• cert-manager
• Prometheus
• GitLab Runner
• JupyterHub
• Knative
• Crossplane
• Elastic Stack

With the exception of Knative, the applications will be installed in a dedicated namespace called gitlab-managed-apps.

Helm

Helm is a package manager for Kubernetes and is required to install all the other applications. It is installed in its own pod inside the cluster which can run the helm CLI in a safe environment.

cert-manager

cert-manager is a native Kubernetes certificate management controller that helps with issuing certificates. Installing cert-manager on your cluster will issue a certificate by Let’s Encrypt and ensure that certificates are valid and up-to-date.

The chart used to install this application depends on the version of GitLab used. In:

• GitLab 12.3 and newer, the jetstack/cert-manager chart is used with a values.yaml file.
• GitLab 12.2 and older, the stable/cert-manager chart was used.

If you have installed cert-manager prior to GitLab 12.3, Let’s Encrypt will block requests from older versions of cert-manager.

To resolve this:

1. Uninstall cert-manager (consider backing up any additional configuration).
2. Install cert-manager again.

GitLab Runner

GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with GitLab CI/CD, the open-source continuous integration service included with GitLab that coordinates the jobs.

If the project is on GitLab.com, shared Runners are available (the first 2000 minutes are free, you can buy more later) and you do not have to deploy one if they are enough for your needs. If a project-specific Runner is desired, or there are no shared Runners, it is easy to deploy one.

Note that the deployed Runner will be set as privileged, which means it will essentially have root access to the underlying machine. This is required to build Docker images, so it is the default. Make sure you read the security implications before deploying one.

Ingress

Ingress provides load balancing, SSL termination, and name-based virtual hosting out of the box. It acts as a web proxy for your applications and is useful if you want to use Auto DevOps or deploy your own web apps.

The Ingress Controller installed is Ingress-NGINX, which is supported by the Kubernetes community.

In order to publish your web application, you first need to find the endpoint which will be either an IP address or a hostname associated with your load balancer.

To install it, click on the Install button for Ingress. GitLab will attempt to determine the external endpoint and it should be available within a few minutes.

Determining the external endpoint automatically

After you install Ingress, the external endpoint should be available within a few minutes.

If the endpoint doesn’t appear and your cluster runs on Google Kubernetes Engine:

1. Check your Kubernetes cluster on Google Kubernetes Engine to ensure there are no errors on its nodes.
2. Ensure you have enough Quotas on Google Kubernetes Engine. For more information, see Resource Quotas.
3. Check Google Cloud’s Status to ensure they are not having any disruptions.

Once installed, you may see a ? for “Ingress IP Address” depending on the cloud provider. For EKS specifically, this is because the ELB is created with a DNS name, not an IP address. If GitLab is still unable to determine the endpoint of your Ingress or Knative application, you can determine it manually.

Determining the external endpoint manually

If the cluster is on GKE, click the Google Kubernetes Engine link in the Advanced settings, or go directly to the Google Kubernetes Engine dashboard and select the proper project and cluster. Then click Connect and execute the gcloud command in a local terminal or using the Cloud Shell.

If the cluster is not on GKE, follow the specific instructions for your Kubernetes provider to configure kubectl with the right credentials. The output of the following examples will show the external endpoint of your cluster. This information can then be used to set up DNS entries and forwarding rules that allow external access to your deployed applications.

If you installed Ingress via the Applications, run the following command:

kubectl get service --namespace=gitlab-managed-apps ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Some Kubernetes clusters return a hostname instead, like Amazon EKS. For these platforms, run:

kubectl get service --namespace=gitlab-managed-apps ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

For Istio/Knative, the command will be different:

kubectl get svc --namespace=istio-system knative-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip} '

Otherwise, you can list the IP addresses of all load balancers:

kubectl get svc --all-namespaces -o jsonpath='{range.items[?(@.status.loadBalancer.ingress)]}{.status.loadBalancer.ingress[*].ip} '

The Ingress is now available at this address and will route incoming requests to the proper service based on the DNS name in the request. To support this, a wildcard DNS CNAME record should be created for the desired domain name. For example, *.myekscluster.com would point to the Ingress hostname obtained earlier.

Using a static IP

By default, an ephemeral external IP address is associated to the cluster’s load balancer. If you associate the ephemeral IP with your DNS and the IP changes, your apps will not be able to be reached, and you’d have to change the DNS record again. In order to avoid that, you should change it into a static reserved IP.

Read how to promote an ephemeral external IP address in GKE.

Pointing your DNS at the external endpoint

Once you’ve set up the external endpoint, you should associate it with a wildcard DNS record such as *.example.com. in order to be able to reach your apps. If your external endpoint is an IP address, use an A record. If your external endpoint is a hostname, use a CNAME record.

Web Application Firewall (ModSecurity)

A Web Application Firewall (WAF) is able to examine traffic being sent/received and can block malicious traffic before it reaches your application. The benefits of a WAF are:

• Real-time security monitoring for your application
• Logging of all your HTTP traffic to the application
• Access control for your application
• Highly configurable logging and blocking rules

Out of the box, GitLab provides you with a WAF known as ModSecurity

ModSecurity is a toolkit for real-time web application monitoring, logging, and access control. With GitLab’s offering, the OWASP’s Core Rule Set, which provides generic attack detection capabilities, is automatically applied.

This feature:

• Runs in “Detection-only mode” unless configured otherwise.

• Is viewable by checking your Ingress controller’s modsec log for rule violations. For example:

  kubectl logs -n gitlab-managed-apps $(kubectl get pod -n gitlab-managed-apps -l app=nginx-ingress,component=controller --no-headers=true -o custom-columns=:metadata.name) modsecurity-log -f
  

To enable ModSecurity, check the Enable Web Application Firewall checkbox when installing your Ingress application.

If this is your first time using GitLab’s WAF, we recommend you follow the quick start guide.

There is a small performance overhead by enabling ModSecurity. However, if this is considered significant for your application, you can disable it.

There is a small performance overhead by enabling ModSecurity. If this is considered significant for your application, you can disable ModSecurity’s rule engine for your deployed application by setting the deployment variable AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE to Off. This will prevent ModSecurity from processing any requests for the given application or environment.

To permanently disable it, you must uninstall and reinstall your Ingress application for the changes to take effect.

JupyterHub

JupyterHub is a multi-user service for managing notebooks across a team. Jupyter Notebooks provide a web-based interactive programming environment used for data analysis, visualization, and machine learning.

Authentication will be enabled only for project members for project-level clusters and group members for group-level clusters with Developer or higher access to the associated project or group.

We use a custom Jupyter image that installs additional useful packages on top of the base Jupyter. You will also see ready-to-use DevOps Runbooks built with Nurtch’s Rubix library.

More information on creating executable runbooks can be found in our Runbooks documentation. Note that Ingress must be installed and have an IP address assigned before JupyterHub can be installed.

Jupyter Git Integration

When installing JupyterHub onto your Kubernetes cluster, JupyterLab’s Git extension is automatically provisioned and configured using the authenticated user’s:

• Name.
• Email.
• Newly created access token.

JupyterLab’s Git extension enables full version control of your notebooks as well as issuance of Git commands within Jupyter. Git commands can be issued via the Git tab on the left panel or via Jupyter’s command line prompt.

You can clone repositories from the files tab in Jupyter:

Knative

Knative provides a platform to create, deploy, and manage serverless workloads from a Kubernetes cluster. It is used in conjunction with, and includes Istio to provide an external IP address for all programs hosted by Knative.

You will be prompted to enter a wildcard domain where your applications will be exposed. Configure your DNS server to use the external IP address for that domain. For any application created and installed, they will be accessible as <program_name>.<kubernetes_namespace>.<domain_name>. This will require your Kubernetes cluster to have RBAC enabled.

Prometheus

Prometheus is an open-source monitoring and alerting system useful to supervise your deployed applications.

GitLab is able to monitor applications automatically, using the Prometheus integration. Kubernetes container CPU and memory metrics are automatically collected, and response metrics are retrieved from NGINX Ingress as well.

To enable monitoring, simply install Prometheus into the cluster with the Install button.

Crossplane

Crossplane is a multi-cloud control plane useful for managing applications and infrastructure across multiple clouds. It extends the Kubernetes API using:

• Custom resources.
• Controllers that watch those custom resources.

Crossplane allows provisioning and lifecycle management of infrastructure components across cloud providers in a uniform manner by abstracting cloud provider-specific configurations.

The Crossplane GitLab-managed application:

• Installs Crossplane with a provider of choice on a Kubernetes cluster attached to the project repository.
• Can then be used to provision infrastructure or managed applications such as PostgreSQL (for example, CloudSQL from GCP or RDS from AWS) and other services required by the application via the Auto DevOps pipeline.

For information on configuring Crossplane installed on the cluster, see Crossplane configuration.

Elastic Stack

Elastic Stack is a complete end-to-end log analysis solution which helps in deep searching, analyzing and visualizing the logs generated from different machines.

GitLab is able to gather logs from pods in your cluster automatically. Filebeat will run as a DaemonSet on each node in your cluster, and it will ship container logs to Elasticsearch for querying. GitLab will then connect to Elasticsearch for logs instead of the Kubernetes API, and you will have access to more advanced querying capabilities.

Log data is automatically deleted after 30 days using Curator.

To enable log shipping, install Elastic Stack into the cluster with the Install button.

Install using GitLab CI (alpha)

This alternative method allows users to install GitLab-managed applications using GitLab CI. It also allows customization of the install using Helm values.yaml files.

Supported applications:

• Ingress
• cert-manager
• Sentry
• GitLab Runner
• Cilium
• JupyterHub
• Elastic Stack

Usage

You can find and import all the files referenced below in the example cluster applications project.

To install applications using GitLab CI:

1. Connect the cluster to a cluster management project.

2. In that project, add a .gitlab-ci.yml file with the following content:

    include:
      - template: Managed-Cluster-Applications.gitlab-ci.yml
   

3. Add a .gitlab/managed-apps/config.yaml file to define which applications you would like to install. Define the installed key as true to install the application and false to uninstall the application. For example, to install Ingress:

    ingress:
      installed: true
   

4. Optionally, define .gitlab/managed-apps/<application>/values.yaml file to customize values for the installed application.

A GitLab CI pipeline will then run on the master branch to install the applications you have configured. In case of pipeline failure, the output of the Helm Tiller binary will be saved as a CI job artifact.

Install Ingress using GitLab CI

To install Ingress, define the .gitlab/managed-apps/config.yaml file with:

ingress:
  installed: true

Ingress will then be installed into the gitlab-managed-apps namespace of your cluster.

You can customize the installation of Ingress by defining .gitlab/managed-apps/ingress/values.yaml file in your cluster management project. Refer to the chart for the available configuration options.

Install cert-manager using GitLab CI

cert-manager is installed using GitLab CI by defining configuration in .gitlab/managed-apps/config.yaml.

cert-manager:

• Is installed into the gitlab-managed-apps namespace of your cluster.
• Can be installed with or without a default Let’s Encrypt ClusterIssuer, which requires an email address to be specified. The email address is used by Let’s Encrypt to contact you about expiring certificates and issues related to your account.

The following configuration is required to install cert-manager using GitLab CI:

certManager:
  installed: true
  letsEncryptClusterIssuer:
    installed: true
    email: "user@example.com"

The following installs cert-manager using GitLab CI without the default ClusterIssuer:

certManager:
  installed: true
  letsEncryptClusterIssuer:
    installed: false

You can customize the installation of cert-manager by defining .gitlab/managed-apps/cert-manager/values.yaml file in your cluster management project. Refer to the chart for the available configuration options.

Install Sentry using GitLab CI

To install Sentry, define the .gitlab/managed-apps/config.yaml file with:

sentry:
  installed: true

Sentry will then be installed into the gitlab-managed-apps namespace of your cluster.

You can customize the installation of Sentry by defining .gitlab/managed-apps/sentry/values.yaml file in your cluster management project. Refer to the chart for the available configuration options.

We recommend you pay close attention to the following configuration options:

• email. Needed to invite users to your Sentry instance and to send error emails.
• user. Where you can set the login credentials for the default admin user.
• postgresql. For a PostgreSQL password that can be used when running future updates.

Here is an example configuration for Sentry:

# Admin user to create
user:
  # Indicated to create the admin user or not,
  # Default is true as the initial installation.
  create: true
  email: "<your email>"
  password: "<your password>"

email:
  from_address: "<your from email>"
  host: smtp
  port: 25
  use_tls: false
  user: "<your email username>"
  password: "<your email password>"
  enable_replies: false

ingress:
  enabled: true
  hostname: "<sentry.example.com>"

# Needs to be here between runs.
# See https://github.com/helm/charts/tree/master/stable/postgresql#upgrade for more info
postgresql:
  postgresqlPassword: example-postgresql-password

Install GitLab Runner using GitLab CI

GitLab Runner is installed using GitLab CI by defining configuration in .gitlab/managed-apps/config.yaml.

The following configuration is required to install GitLab Runner using GitLab CI:

gitlabRunner:
  installed: true

GitLab Runner is installed into the gitlab-managed-apps namespace of your cluster.

In order for GitLab Runner to function, you must specify the following:

• gitlabUrl - the GitLab server full URL (e.g., https://example.gitlab.com) to register the Runner against.
• runnerRegistrationToken - The registration token for adding new Runners to GitLab. This must be retrieved from your GitLab instance.

These values can be specifed using CI variables:

• GITLAB_RUNNER_GITLAB_URL will be used for gitlabUrl.
• GITLAB_RUNNER_REGISTRATION_TOKEN will be used for runnerRegistrationToken

You can customize the installation of GitLab Runner by defining .gitlab/managed-apps/gitlab-runner/values.yaml file in your cluster management project. Refer to the chart for the available configuration options.

Install Cilium using GitLab CI

Cilium is a networking plugin for Kubernetes that you can use to implement support for NetworkPolicy resources.

Enable Cilium in the .gitlab/managed-apps/config.yaml file to install it:

# possible values are gke, eks or you can leave it blank
clusterType: gke

cilium:
  installed: true

The clusterType variable enables the recommended Helm variables for a corresponding cluster type, the default value is blank. You can check the recommended variables for each cluster type in the official documentation:

• Google GKE
• AWS EKS

You can customize Cilium’s Helm variables by defining the .gitlab/managed-apps/cilium/values.yaml file in your cluster management project. Refer to the Cilium chart for the available configuration options.

By default, Cilium will drop all non-whitelisted packets upon policy deployment. The audit mode is scheduled for release in Cilium 1.8. In the audit mode non-whitelisted packets will not be dropped, instead audit notifications will be generated. GitLab provides alternative Docker images for Cilium with the audit patch included. You can switch to the custom build and enable the audit mode by adding the following to .gitlab/managed-apps/cilium/values.yaml:

global:
  registry: registry.gitlab.com/gitlab-org/defend/cilium
  policyAuditMode: true

agent:
  monitor:
    eventTypes: ["drop", "audit"]

The Cilium monitor log for traffic is logged out by the cilium-monitor sidecar container. You can check these logs via:

kubectl -n gitlab-managed-apps logs cilium-XXXX cilium-monitor

You can disable the monitor log via .gitlab/managed-apps/cilium/values.yaml:

agent:
  monitor:
    enabled: false

Install JupyterHub using GitLab CI

Enable JupyterHub in the .gitlab/managed-apps/config.yaml file to install it:

jupyterhub:
  installed: true
  gitlabProjectIdWhitelist: []
  gitlabGroupWhitelist: []

gitlabProjectIdWhitelist restricts GitLab authentication to only members of the specified projects. gitlabGroupWhitelist restricts GitLab authentication to only members of the specified groups. Specifying an empty array for both will allow any user on the GitLab instance to log in.

JupyterHub is installed into the gitlab-managed-apps namespace of your cluster.

In order for JupyterHub to function, you must setup an OAuth Application. Using the following values:

• “Redirect URI” to http://<JupyterHub Host>/hub/oauth_callback
• “Scope” to api read_repository write_repository

In addition the following variables must be specified using CI variables:

• JUPYTERHUB_PROXY_SECRET_TOKEN will set proxy.secretToken. Generate this using openssl rand -hex 32.
• JUPYTERHUB_COOKIE_SECRET will set hub.cookieSecret. Generate this using openssl rand -hex 32.
• JUPYTERHUB_HOST is the hostname used for the installation (e.g., jupyter.example.gitlab.com).
• JUPYTERHUB_GITLAB_HOST is the hostname of the GitLab instance used for authentication (e.g., example.gitlab.com).
• JUPYTERHUB_AUTH_CRYPTO_KEY will set auth.state.cryptoKey. Generate this using openssl rand -hex 32.
• JUPYTERHUB_AUTH_GITLAB_CLIENT_ID the “Application ID” for the OAuth Application.
• JUPYTERHUB_AUTH_GITLAB_CLIENT_SECRET the “Secret” for the OAuth Application.

By default JupyterHub will be installed using a default values file. You can customize the installation of JupyterHub by defining .gitlab/managed-apps/jupyterhub/values.yaml file in your cluster management project. Refer to the chart reference for the available configuration options.

Install Elastic Stack using GitLab CI

Elastic Stack is installed using GitLab CI by defining configuration in .gitlab/managed-apps/config.yaml.

The following configuration is required to install Elastic Stack using GitLab CI:

elasticStack:
  installed: true

Elastic Stack is installed into the gitlab-managed-apps namespace of your cluster.

You can check the default values.yaml we set for this chart.

You can customize the installation of Elastic Stack by defining .gitlab/managed-apps/elastic-stack/values.yaml file in your cluster management project. Refer to the chart for the available configuration options.

Upgrading applications

The applications below can be upgraded.

To upgrade an application:

1. For a:
   • Project-level cluster, navigate to your project’s Operations > Kubernetes.
   • Group-level cluster, navigate to your group’s Kubernetes page.
2. Select your cluster.
3. If an upgrade is available, the Upgrade button is displayed. Click the button to upgrade.

Uninstalling applications

The applications below can be uninstalled.

To uninstall an application:

1. For a:
   • Project-level cluster, navigate to your project’s Operations > Kubernetes.
   • Group-level cluster, navigate to your group’s Kubernetes page.
2. Select your cluster.
3. Click the Uninstall button for the application.

Support for uninstalling all applications is planned for progressive rollout. To follow progress, see the relevant epic.

Troubleshooting applications

Applications can fail with the following error:

Error: remote error: tls: bad certificate

To avoid installation errors:

• Before starting the installation of applications, make sure that time is synchronized between your GitLab server and your Kubernetes cluster.

• Ensure certificates are not out of sync. When installing applications, GitLab expects a new cluster with no previous installation of Helm.

  You can confirm that the certificates match via kubectl:

  kubectl get configmaps/values-content-configuration-ingress -n gitlab-managed-apps -o \
  "jsonpath={.data['cert\.pem']}" | base64 -d > a.pem
  kubectl get secrets/tiller-secret -n gitlab-managed-apps -o "jsonpath={.data['ca\.crt']}" | base64 -d > b.pem
  diff a.pem b.pem