Migrating from Helm v2 to Helm v3
Helm v2 was officially deprecated in November of 2020. Starting from GitLab Helm chart version 5.0 (GitLab App version 14.0), installation and upgrades using Helm v2.x are no longer supported. To get future GitLab updates, you will need to migrate to Helm v3.
Changes between Helm v2 and Helm v3
Helm v3 introduces a lot of changes that are not backwards compatible with Helm v2. Some of the major changes include the removal of Tiller requirements and how they store releases information on the cluster. Read more in the Helm v3 changes overview and the changes since Helm v2 FAQ.
The Helm chart you use to deploy the application might not be compatible with the newer / older versions of Helm. If you have multiple applications deployed and managed with Helm v2, you will need to find out if they are compatible with Helm v3 in case you want to convert them as well. GitLab Helm chart supports Helm v3.0.2 or higher starting with version v3.0.0 of GitLab Helm chart. Helm v2 is no longer supported.
From the standpoint of the application that is currently running, nothing is changed when you perform the migration from Helm v2 to v3. It’s generally pretty safe to perform the Helm v2 to v3 migration, however, be sure to take backups of Helm v2 as a precaution.
How to migrate from Helm v2 to Helm v3
You can use the Helm 2to3 plugin to migrate GitLab releases from Helm v2 to Helm v3. For a more detailed explanation with some examples about this migration plugin, refer to Helm blog post: How to migrate from Helm v2 to Helm v3.
If you have multiple people managing your GitLab Helm installation, you may need to do helm3 2to3 move config
on each local machine. You will only need to do helm3 2to3 convert
once.
Known Issues
“UPGRADE FAILED: cannot patch” error is shown after the migration
After migration the subsequent upgrades may fail with an error similar to the following:
Error: UPGRADE FAILED: cannot patch "..." with kind Deployment: Deployment.apps "..." is invalid: spec.selector:
Invalid value: v1.LabelSelector{...}: field is immutable
or
Error: UPGRADE FAILED: cannot patch "..." with kind StatefulSet: StatefulSet.apps "..." is invalid:
spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden
This is due to known issues with Helm 2 to 3 migration in Cert Manager
and Redis dependencies. In a nutshell, the heritage
label
on some Deployments and StatefulSets are immutable and can not be changed from Tiller
(set by Helm 2) to Helm
(set by Helm 3). So they must be replaced forcefully.
To work around this use the following instructions:
- Replace cert-manager Deployments (when enabled).
kubectl get deployments -l app=cert-manager -o yaml | sed "s/Tiller/Helm/g" | kubectl replace --force=true -f -
kubectl get deployments -l app=cainjector -o yaml | sed "s/Tiller/Helm/g" | kubectl replace --force=true -f -
- (Optional) Set
persistentVolumeReclaimPolicy
toRetain
on the PV that is claimed by Redis StatefulSet. This is to ensure that the PV won’t be deleted inadvertently.
kubectl patch pv <PV-NAME> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
- Set
heritage
label of the existing Redis PVC toHelm
.
kubectl label pvc -l app=redis --overwrite heritage=Helm
- Replace Redis StatefulSet without cascading.
kubectl get statefulsets.apps -l app=redis -o yaml | sed "s/Tiller/Helm/g" | kubectl replace --force=true --cascade=false -f -
RBAC issues after the migration when running Helm upgrade
You may face the following error when running Helm upgrade after the conversion has been completed:
Error: UPGRADE FAILED: pre-upgrade hooks failed: warning: Hook pre-upgrade gitlab/templates/shared-secrets/rbac-config.yaml failed: roles.rbac.authorization.k8s.io "gitlab-shared-secrets" is forbidden: user "your-user-name@domain.tld" (groups=["system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:[""], Resources:["secrets"], Verbs:["get" "list" "create" "patch"]}
Helm2 used the Tiller service account to perform such operations. Helm3 does not use Tiller anymore, and your user account should have proper RBAC permissions to run the command even if you are running helm upgrade
as a cluster admin. To grant full RBAC permissions to yourself, run:
kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=your-user-name@domain.tld
After that, helm upgrade
should work fine.