Troubleshooting
All steps noted here are for DEVELOPMENT ENVIRONMENTS ONLY. Administrators may find the information insightful, but the outlined fixes are destructive and would have a major negative impact on production systems.
Passwords and secrets failing or unsynchronized
Developers commonly deploy, delete, and re-deploy a release into the same
cluster multiple times. Kubernetes secrets and persistent volume claims created by StatefulSets are
intentionally not removed by helm delete RELEASE_NAME
.
Removing only the Kubernetes secrets leads to interesting problems. For example, a new deployment’s migration pod will fail because GitLab Rails cannot connect to the database because it has the wrong password.
To completely wipe a release from a development environment including secrets, a developer must remove both the secrets and the persistent volume claims.
# DO NOT run these commands in a production environment. Disaster will strike.
kubectl delete secrets,pvc -lrelease=RELEASE_NAME
Database is broken and needs reset
The database environment can be reset in a development environment by:
- Delete the PostgreSQL StatefulSet
- Delete the PostgreSQL PersistentVolumeClaim
- Deploy GitLab again with
helm upgrade --install