Each experiment should have an experiment rollout issue to track the experiment from rollout through to cleanup and removal. The rollout issue is similar to a feature flag rollout issue, and is also used to track the status of an experiment.
When an experiment is deployed, the due date of the issue should be set (this depends on the experiment but can be up to a few weeks in the future). After the deadline, the issue needs to be resolved and either:
- It was successful and the experiment becomes the new default.
- It was not successful and all code related to the experiment is removed.
In either case, an outcome of the experiment should be posted to the issue with the reasoning for the decision.
When there is a case on GitLab.com (SaaS) that necessitates turning off all experiments, we have this control.
You can toggle experiments on SaaS on and off using the
gitlab_experiment feature flag.
This can be done via ChatOps:
/chatops run feature set gitlab_experiment false
/chatops run feature delete gitlab_experiment
- This allows the
default_enabledvalue of true in the yml to be honored.
You may already be familiar with the concept of feature flags in GitLab, but using
feature flags in experiments is a bit different. While in general terms, a feature flag
is viewed as being either
off, this isn’t accurate for experiments.
off means that when we ask if a feature flag is enabled, it will always
on means that it will always return
true. An interim state,
conditional, also exists. We take advantage of this trinary state of
feature flags. To understand this
conditional aspect: consider that either of these
settings puts a feature flag into this state:
- Setting a
percentage_of_actorsof any percent greater than 0%.
- Enabling it for a single user or group.
Conditional means that it returns
true in some situations, but not all situations.
When a feature flag is disabled (meaning the state is
off), the experiment is
considered inactive. You can visualize this in the decision tree diagram
as reaching the first
Running? node, and traversing the negative path.
When a feature flag is rolled out to a
percentage_of_actors or similar (meaning the
conditional) the experiment is considered to be running
where sometimes the control is assigned, and sometimes the candidate is assigned.
We don’t refer to this as being enabled, because that’s a confusing and overloaded
term here. In the experiment terms, our experiment is running, and the feature flag is
When a feature flag is enabled (meaning the state is
on), the candidate will always be
We should try to be consistent with our terms, and so for experiments, we have an
inactive experiment until we set the feature flag to
conditional. After which,
our experiment is then considered running. If you choose to “enable” your feature flag,
you should consider the experiment to be resolved, because everyone is assigned
the candidate unless they’ve opted out of experimentation.
As of GitLab 13.10, work is being done to improve this process and how we communicate about it.