- Locate the GitLab Volumes
- Before making storage changes
- Making storage changes
- Make changes to the PersistentVolumeClaim
- Apply the changes to the GitLab chart
Managing Persistent Volumes
Some of the included services require persistent storage, configured through Persistent Volumes that specify which disks your cluster has access to. Documentation on the storage configuration necessary to install this chart can be found in our Storage Guide.
Storage changes after installation need to be manually handled by your cluster administrators. Automated management of these volumes after installation is not handled by the GitLab chart.
Examples of changes not automatically managed after initial installation include:
- Mounting different volumes to the Pods
- Changing the effective accessModes or Storage Class
- Expanding the storage size of your volume*1
1 In Kubernetes 1.11, expanding the storage size of your volume is supported
if you have allowVolumeExpansion
configured to true in your Storage Class.
Automating theses changes is complicated due to:
- Kubernetes does not allow changes to most fields in an existing PersistentVolumeClaim
- Unless manually configured, the PVC is the only reference to dynamically provisioned PersistentVolumes
-
Delete
is the default reclaimPolicy for dynamically provisioned PersistentVolumes
This means in order to make changes, we need to delete the PersistentVolumeClaim and create a new one with our changes. But due to the default reclaimPolicy, deleting the PersistentVolumeClaim may delete the PersistentVolumes and underlying disk. And unless configured with appropriate volumeNames and/or labelSelectors, the chart doesn’t know the volume to attach to.
We will continue to look into making this process easier, but for now a manual process needs to be followed to make changes to your storage.
Locate the GitLab Volumes
Find the volumes/claims that are being used:
kubectl --namespace <namespace> get PersistentVolumeClaims -l release=<chart release name> -ojsonpath='{range .items[*]}{.spec.volumeName}{"\t"}{.metadata.labels.app}{"\n"}{end}'
-
<namespace>
should be replaced with the namespace where you installed the GitLab chart. -
<chart release name>
should be replaced with the name you used to install the GitLab chart.
The command prints a list of the volume names, followed by the name of the service they are for.
For example:
$ kubectl --namespace helm-charts-win get PersistentVolumeClaims -l release=review-update-app-h8qogp -ojsonpath='{range .items[*]}{.spec.volumeName}{"\t"}{.metadata.labels.app}{"\n"}{end}'
pvc-6247502b-8c2d-11e8-8267-42010a9a0113 gitaly
pvc-61bbc05e-8c2d-11e8-8267-42010a9a0113 minio
pvc-61bc6069-8c2d-11e8-8267-42010a9a0113 postgresql
pvc-61bcd6d2-8c2d-11e8-8267-42010a9a0113 prometheus
pvc-61bdf136-8c2d-11e8-8267-42010a9a0113 redis
Before making storage changes
Note: The person making the changes needs to have administrator access to the cluster, and appropriate access to the storage solutions being used. Often the changes will first need to be applied in the storage solution, then the results need to be updated in Kubernetes.
Before making changes, you should ensure your PersistentVolumes are using
the Retain
reclaimPolicy so they don’t get removed while you are
making changes.
First, find the volumes/claims that are being used.
Next, edit each volume and change the value of persistentVolumeReclaimPolicy
under the spec
field, to be Retain
rather than Delete
For example:
kubectl --namespace helm-charts-win edit PersistentVolume pvc-6247502b-8c2d-11e8-8267-42010a9a0113
Editing Output:
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
kubernetes.io/createdby: gce-pd-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: "yes"
pv.kubernetes.io/provisioned-by: kubernetes.io/gce-pd
creationTimestamp: 2018-07-20T14:58:43Z
labels:
failure-domain.beta.kubernetes.io/region: europe-west2
failure-domain.beta.kubernetes.io/zone: europe-west2-b
name: pvc-6247502b-8c2d-11e8-8267-42010a9a0113
resourceVersion: "48362431"
selfLink: /api/v1/persistentvolumes/pvc-6247502b-8c2d-11e8-8267-42010a9a0113
uid: 650bd649-8c2d-11e8-8267-42010a9a0113
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 50Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: repo-data-review-update-app-h8qogp-gitaly-0
namespace: helm-charts-win
resourceVersion: "48362307"
uid: 6247502b-8c2d-11e8-8267-42010a9a0113
gcePersistentDisk:
fsType: ext4
pdName: gke-cloud-native-81a17-pvc-6247502b-8c2d-11e8-8267-42010a9a0113
# Changed the following line
persistentVolumeReclaimPolicy: Retain
storageClassName: standard
status:
phase: Bound
Making storage changes
First, make the desired changes to the disk outside of the cluster. (Resize the disk in GKE, or create a new disk from a snapshot or clone, etc).
How you do this, and whether or not it can be done live, without downtime, is dependent on the storage solutions you are using, and can’t be covered by this document.
Next, evaluate whether you need these changes to be reflected in the Kubernetes objects. For example: with expanding the disk storage size, the storage size settings in the PersistentVolumeClaim will only be used when a new volume resource is requested. So you would only need to increase the values in the PersistentVolumeClaim if you intend to scale up more disks (for use in additional Gitaly pods).
If you do need to have the changes reflected in Kubernetes, be sure that you’ve updated your reclaim policy on the volumes as described in the Before making storage changes section.
The paths we have documented for storage changes are:
Changes to an existing Volume
First locate the volume name you are changing.
Use kubectl edit
to make the desired configuration changes to the volume. (These changes
should only be updates to reflect the real state of the attached disk)
For example:
kubectl --namespace helm-charts-win edit PersistentVolume pvc-6247502b-8c2d-11e8-8267-42010a9a0113
Editing Output:
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
kubernetes.io/createdby: gce-pd-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: "yes"
pv.kubernetes.io/provisioned-by: kubernetes.io/gce-pd
creationTimestamp: 2018-07-20T14:58:43Z
labels:
failure-domain.beta.kubernetes.io/region: europe-west2
failure-domain.beta.kubernetes.io/zone: europe-west2-b
name: pvc-6247502b-8c2d-11e8-8267-42010a9a0113
resourceVersion: "48362431"
selfLink: /api/v1/persistentvolumes/pvc-6247502b-8c2d-11e8-8267-42010a9a0113
uid: 650bd649-8c2d-11e8-8267-42010a9a0113
spec:
accessModes:
- ReadWriteOnce
capacity:
# Updated the storage size
storage: 100Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: repo-data-review-update-app-h8qogp-gitaly-0
namespace: helm-charts-win
resourceVersion: "48362307"
uid: 6247502b-8c2d-11e8-8267-42010a9a0113
gcePersistentDisk:
fsType: ext4
pdName: gke-cloud-native-81a17-pvc-6247502b-8c2d-11e8-8267-42010a9a0113
persistentVolumeReclaimPolicy: Retain
storageClassName: standard
status:
phase: Bound
Now that the changes have been reflected in the volume, we need to update the claim.
Follow the instructions in the Make changes to the PersistentVolumeClaim section.
Update the volume to bind to the claim
In a separate terminal, start watching to see when the claim has its status change to bound, and then move onto the next step to make the volume available for use in the new claim.
kubectl --namespace <namespace> get --watch PersistentVolumeClaim <claim name>
Edit the volume to make it available to the new claim. Remove the .spec.claimRef
section.
kubectl --namespace <namespace> edit PersistentVolume <volume name>
Editing Output:
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
kubernetes.io/createdby: gce-pd-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: "yes"
pv.kubernetes.io/provisioned-by: kubernetes.io/gce-pd
creationTimestamp: 2018-07-20T14:58:43Z
labels:
failure-domain.beta.kubernetes.io/region: europe-west2
failure-domain.beta.kubernetes.io/zone: europe-west2-b
name: pvc-6247502b-8c2d-11e8-8267-42010a9a0113
resourceVersion: "48362431"
selfLink: /api/v1/persistentvolumes/pvc-6247502b-8c2d-11e8-8267-42010a9a0113
uid: 650bd649-8c2d-11e8-8267-42010a9a0113
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 100Gi
gcePersistentDisk:
fsType: ext4
pdName: gke-cloud-native-81a17-pvc-6247502b-8c2d-11e8-8267-42010a9a0113
persistentVolumeReclaimPolicy: Retain
storageClassName: standard
status:
phase: Released
Shortly after making the change to the Volume, the terminal watching the claim status should show Bound
.
Finally, apply the changes to the GitLab chart
Switching to a different Volume
If you want to switch to using a new volume, using a disk that has a copy of the appropriate data from the old volume, then first you need to create the new Persistent Volume in Kubernetes.
In order to create a Persistent Volume for your disk, you will need to locate the driver specific documentation for your storage type.
There are a couple of things to keep in mind when following the driver documentation:
- You need to use the driver to create a Persistent Volume, not a Pod object with a volume as shown in a lot of the documentation.
- You do not want to create a PersistentVolumeClaim for the volume, we will be editing the existing claim instead.
The driver documentation often includes examples for using the driver in a Pod, for example:
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
volumes:
- name: test-volume
# This GCE PD must already exist.
gcePersistentDisk:
pdName: my-data-disk
fsType: ext4
What you actually want, is to create a Persistent Volume, like so:
apiVersion: v1
kind: PersistentVolume
metadata:
name: test-volume
spec:
capacity:
storage: 400Gi
accessModes:
- ReadWriteOnce
gcePersist