Snowplow Guide

This guide provides an overview of how Snowplow works, and implementation details.

For more information about Product Intelligence, see:

More useful links:

What is Snowplow

Snowplow is an enterprise-grade marketing and Product Intelligence platform which helps track the way users engage with our website and application.

Snowplow consists of the following loosely-coupled sub-systems:

  • Trackers fire Snowplow events. Snowplow has 12 trackers, covering web, mobile, desktop, server, and IoT.
  • Collectors receive Snowplow events from trackers. We have three different event collectors, synchronizing events either to Amazon S3, Apache Kafka, or Amazon Kinesis.
  • Enrich cleans up the raw Snowplow events, enriches them and puts them into storage. We have an Hadoop-based enrichment process, and a Kinesis-based or Kafka-based process.
  • Storage is where the Snowplow events live. We store the Snowplow events in a flat file structure on S3, and in the Redshift and PostgreSQL databases.
  • Data modeling is where event-level data is joined with other data sets and aggregated into smaller data sets, and business logic is applied. This produces a clean set of tables which make it easier to perform analysis on the data. We have data models for Redshift and Looker.
  • Analytics are performed on the Snowplow events or on the aggregate tables.

snowplow_flow

Snowplow schema

We have many definitions of Snowplow’s schema. We have an active issue to standardize this schema including the following definitions:

Enabling Snowplow

Tracking can be enabled at:

  • The instance level, which enables tracking on both the frontend and backend layers.
  • User level, though user tracking can be disabled on a per-user basis. GitLab tracking respects the Do Not Track standard, so any user who has enabled the Do Not Track option in their browser is not tracked at a user level.

We use Snowplow for the majority of our tracking strategy and it is enabled on GitLab.com. On a self-managed instance, Snowplow can be enabled by navigating to:

  • Admin Area > Settings > General in the UI.
  • admin/application_settings/integrations in your browser.

Example configuration:

Name Value
Collector your-snowplow-collector.net
Site ID gitlab
Cookie domain .your-gitlab-instance.com

Snowplow request flow

The following example shows a basic request/response flow between the following components:

sequenceDiagram participant Snowplow JS (Frontend) participant Snowplow Ruby (Backend) participant GitLab.com Snowplow Collector participant S3 Bucket participant Snowflake DW participant Sisense Dashboards Snowplow JS (Frontend) ->> GitLab.com Snowplow Collector: FE Tracking event Snowplow Ruby (Backend) ->> GitLab.com Snowplow Collector: BE Tracking event loop Process using Kinesis Stream GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Log raw events GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Enrich events GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Write to disk end GitLab.com Snowplow Collector ->> S3 Bucket: Kinesis Firehose S3 Bucket->>Snowflake DW: Import data Snowflake DW->>Snowflake DW: Transform data using dbt Snowflake DW->>Sisense Dashboards: Data available for querying

Structured event taxonomy

When adding new click events, we should add them in a way that’s internally consistent. If we don’t, it is very painful to perform analysis across features since each feature captures events differently.

The current method provides several attributes that are sent on each click event. Please try to follow these guidelines when specifying events to capture:

attribute type required description
category text true The page or backend area of the application. Unless infeasible, please use the Rails page attribute by default in the frontend, and namespace + class name on the backend.
action text true The action the user is taking, or aspect that’s being instrumented. The first word should always describe the action or aspect: clicks should be click, activations should be activate, creations should be create, etc. Use underscores to describe what was acted on; for example, activating a form field would be activate_form_input. An interface action like clicking on a dropdown would be click_dropdown, while a behavior like creating a project record from the backend would be create_project
label text false The specific element, or object that’s being acted on. This is either the label of the element (e.g. a tab labeled ‘Create from template’ may be create_from_template) or a unique identifier if no text is available (e.g. closing the Groups dropdown in the top navigation bar might be groups_dropdown_close), or it could be the name or title attribute of a record being created.
property text false Any additional property of the element, or object being acted on.
value decimal false Describes a numeric value or something directly related to the event. This could be the value of an input (e.g. 10 when clicking internal visibility).

Examples

category* label action property** value
[root:index] main_navigation click_navigation_link [link_label] -
[groups:boards:show] toggle_swimlanes click_toggle_button - [is_active]
[projects:registry:index] registry_delete click_button - -
[projects:registry:index] registry_delete confirm_deletion - -
[projects:blob:show] congratulate_first_pipeline click_button [human_access] -
[projects:clusters:new] chart_options generate_link [chart_link] -
[projects:clusters:new] chart_options click_add_label_button [label_id] -

* It’s ok to omit the category, and use the default.
** Property is usually the best place for variable strings.

Reference SQL

Last 20 reply_comment_button events

SELECT
  event_id,
  v_tracker,
  event_label,
  event_action,
  event_property,
  event_value,
  event_category,
  contexts
FROM legacy.snowplow_structured_events_all
WHERE
  event_label = 'reply_comment_button'
  AND event_action = 'click_button'
  -- AND event_category = 'projects:issues:show'
  -- AND event_value = 1
ORDER BY collector_tstamp DESC
LIMIT 20

Web-specific parameters

Snowplow JS adds many web-specific parameters to all web events by default.

Implementing Snowplow JS (Frontend) tracking

GitLab provides Tracking, an interface that wraps the Snowplow JavaScript Tracker for tracking custom events. The simplest way to use it is to add data- attributes to clickable elements and dropdowns. There is also a Vue mixin (exposing a track method), and the static method Tracking.event. Each of these requires at minimum a category and an action. Additional data can be provided that adheres to our Structured event taxonomy.

field type default value description
category string document.body.dataset.page Page or subsection of a page that events are being captured within.
action string generic Action the user is taking. Clicks should be click and activations should be activate, so for example, focusing a form field would be activate_form_input, and clicking a button would be click_button.
data object {} Additional data such as label, property, value, and context as described in our Structured event taxonomy.

Usage recommendations

  • Use data attributes on HTML elements that emits either the click, show.bs.dropdown, or hide.bs.dropdown events.
  • Use the Vue mixin when tracking custom events, or if the supported events for data attributes are not propagating.
  • Use the Tracking class directly when tracking on raw JS files.

Tracking with data attributes

When working within HAML (or Vue templates) we can add data-track-* attributes to elements of interest. All elements that have a data-track-action attribute automatically have event tracking bound on clicks.

Below is an example of data-track-* attributes assigned to a button:

%button.btn{ data: { track: { action: "click_button", label: "template_preview", property: "my-template" } } }
<button class="btn"
  data-track-action="click_button"
  data-track-label="template_preview"
  data-track-property="my-template"
/>

Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows them to be properly handled on re-rendering and changes to the DOM. Note that because of the way these events are bound, click events should not be stopped from propagating up the DOM tree. If for any reason click events are being stopped from propagating, you need to implement your own listeners and follow the instructions in Tracking within Vue components or Tracking in raw JavaScript.

Below is a list of supported data-track-* attributes:

attribute required description
data-track-action true Action the user is taking. Clicks must be prepended with click and activations must be prepended with activate. For example, focusing a form field would be activate_form_input and clicking a button would be click_button. Replaces data-track-event, which was deprecated in GitLab 13.11.
data-track-label false The label as described in our Structured event taxonomy.
data-track-property false The property as described in our Structured event taxonomy.
data-track-value false The value as described in our Structured event taxonomy. If omitted, this is the element’s value property or an empty string. For checkboxes, the default value is the element’s checked attribute or false when unchecked.
data-track-context false The context as described in our Structured event taxonomy.

Available helpers

tracking_attrs(label, action, property) # { data: { track_label... } }

%button{ **tracking_attrs('main_navigation', 'click_button', 'navigation') }

Caveats

When using the GitLab helper method nav_link be sure to wrap html_options under the html_options keyword argument. Be careful, as this behavior can be confused with the ActionView helper method link_to that does not require additional wrapping of html_options

# Bad
= nav_link(controller: ['dashboard/groups', 'explore/groups'], data: { track_label: "explore_groups", track_action: "click_button" })

# Good
= nav_link(controller: ['dashboard/groups', 'explore/groups'], html_options: { data: { track_label: "explore_groups", track_action: "click_button" } })

# Good (other helpers)
= link_to explore_groups_path, title: _("Explore"), data: { track_label: "explore_groups", track_action: "click_button" }

Tracking within Vue components

There’s a tracking Vue mixin that can be used in components if more complex tracking is required. To use it, first import the Tracking library and request a mixin.

import Tracking from '~/tracking';
const trackingMixin = Tracking.mixin({ label: 'right_sidebar' });

You can provide default options that are passed along whenever an event is tracked from within your component. For instance, if all events within a component should be tracked with a given label, you can provide one at this time. Available defaults are category, label, property, and value. If no category is specified, document.body.dataset.page is used as the default.

You can then use the mixin normally in your component with the mixin Vue declaration. The mixin also provides the ability to specify tracking options in data or computed. These override any defaults and allow the values to be dynamic from props, or based on state.

export default {
  mixins: [trackingMixin],
  // ...[component implementation]...
  data() {
    return {
      expanded: false,
      tracking: {
        label: 'left_sidebar',
      },
    };
  },
};

The mixin provides a track method that can be called within the template, or from component methods. An example of the whole implementation might look like this:

export default {
  name: 'RightSidebar',
  mixins: [Tracking.mixin({ label: 'right_sidebar' })],
  data() {
    return {
      expanded: false,
    };
  },
  methods: {
    toggle() {
      this.expanded = !this.expanded;
      // Additional data will be merged, like `value` below
      this.track('click_toggle', { value: Number(this.expanded) });
    }
  }
};

The event data can be provided with a tracking object, declared in the data function, or as a computed property.

export default {
  name: 'RightSidebar',
  mixins: [Tracking.mixin()],
  data() {
    return {
      tracking: {
        label: 'right_sidebar',
        // category: '',
        // property: '',
        // value: '',
      },
    };
  },
};

The event data can be provided directly in the track function as well. This object will merge with any previously provided options.

this.track('click_button', {
  label: 'right_sidebar',
});

Lastly, if needed within the template, you can use the track method directly as well.

<template>
  <div>
    <button data-testid="toggle" @click="toggle">Toggle</button>

    <div v-if="expanded">
      <p>Hello world!</p>
      <button @click="track('click_action')">Track another event</button>
    </div>
  </div>
</template>

Testing example

import { mockTracking } from 'helpers/tracking_helper';
// mockTracking(category, documentOverride, spyMethod)

describe('RightSidebar.vue', () => {
  let trackingSpy;
  let wrapper;

  beforeEach(() => {
    trackingSpy = mockTracking(undefined, wrapper.element, jest.spyOn);
  });

  const findToggle = () => wrapper.find('[data-testid="toggle"]');

  it('tracks turning off toggle', () => {
    findToggle().trigger('click');

    expect(trackingSpy).toHaveBeenCalledWith(undefined, 'click_toggle', {
      label: 'right_sidebar',
      value: 0,
    });
  });
});

Tracking in raw JavaScript

Custom event tracking and instrumentation can be added by directly calling the Tracking.event static function. The following example demonstrates tracking a click on a button by calling Tracking.event manually.

import Tracking from '~/tracking';

const button = document.getElementById('create_from_template_button');

button.addEventListener('click', () => {
  Tracking.event('dashboard:projects:index', 'click_button', {
    label: 'create_from_template',
    property: 'template_preview',
  });
});

Testing example

import Tracking from '~/tracking';

describe('MyTracking', () => {
  let wrapper;

  beforeEach(() => {
    jest.spyOn(Tracking, 'event');
  });

  const findButton = () => wrapper.find('[data-testid="create_from_template"]');

  it('tracks event', () => {
    findButton().trigger('click');

    expect(Tracking.event).toHaveBeenCalledWith(undefined, 'click_button', {
      label: 'create_from_template',
      property: 'template_preview',
    });
  });
});

Implementing Snowplow Ruby (Backend) tracking

GitLab provides Gitlab::Tracking, an interface that wraps the Snowplow Ruby Tracker for tracking custom events.

Custom event tracking and instrumentation can be added by directly calling the GitLab::Tracking.event class method, which accepts the following arguments:

argument type default value description
category String   Area or aspect of the application. This could be HealthCheckController or Lfs::FileTransformer for instance.
action String   The action being taken, which can be anything from a controller action like create to something like an Active Record callback.
label String nil As described in Structured event taxonomy.
property String nil As described in Structured event taxonomy.
value Numeric nil As described in Structured event taxonomy.
context Array[SelfDescribingJSON] nil An array of custom contexts to send with this event. Most events should not have any custom contexts.
project Project nil The project associated with the event.
user User nil The user associated with the event.
namespace Namespace nil The namespace associated with the event.
extra Hash {} Additional keyword arguments are collected into a hash and sent with the event.

Tracking can be viewed as either tracking user behavior, or can be used for instrumentation to monitor and visualize performance over time in an area or aspect of code.

For example:

class Projects::CreateService < BaseService
  def execute
    project = Project.create(params)

    Gitlab::Tracking.event('Projects::CreateService', 'create_project', label: project.errors.full_messages.to_sentence,
                           property: project.valid?.to_s, project: project, user: current_user, namespace: namespace)
  end
end

Unit testing

Use the expect_snowplow_event helper when testing backend Snowplow events. See testing best practices for details.

Performance

We use the AsyncEmitter when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development.

Developing and testing Snowplow

There are several tools for developing and testing Snowplow Event

Testing Tool Frontend Tracking Backend Tracking Local Development Environment Production Environment Production Environment
Snowplow Analytics Debugger Chrome Extension
Snowplow Inspector Chrome Extension
Snowplow Micro
Snowplow Mini

Legend

Available, In progress, Not Planned

Snowplow Analytics Debugger Chrome Extension

Snowplow Analytics Debugger is a browser extension for testing frontend events. This works on production, staging and local development environments.

  1. Install the Snowplow Analytics Debugger Chrome browser extension.
  2. Open Chrome DevTools to the Snowplow Analytics Debugger tab.
  3. Learn more at Igloo Analytics.

Snowplow Inspector Chrome Extension

Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works on production, staging and local development environments.

  1. Install Snowplow Inspector.
  2. Open the Chrome extension by pressing the Snowplow Inspector icon beside the address bar.
  3. Click around on a webpage with Snowplow and you should see JavaScript events firing in the inspector window.

Snowplow Micro

Snowplow Micro is a very small version of a full Snowplow data collection pipeline: small enough that it can be launched by a test suite. Events can be recorded into Snowplow Micro just as they can a full Snowplow pipeline. Micro then exposes an API that can be queried.

Snowplow Micro is a Docker-based solution for testing frontend and backend events in a local development environment. You need to modify GDK using the instructions below to set this up.

  1. Ensure Docker is installed and running.

  2. Install Snowplow Micro by cloning the settings in this project:
  3. Navigate to the directory with the cloned project, and start the appropriate Docker container with the following script:

    ./snowplow-micro.sh
    
  4. Update your instance’s settings to enable Snowplow events and point to the Snowplow Micro collector:

    gdk psql -d gitlabhq_development
    update application_settings set snowplow_collector_hostname='localhost:9090', snowplow_enabled=true, snowplow_cookie_domain='.gitlab.com';
    
  5. Update DEFAULT_SNOWPLOW_OPTIONS in app/assets/javascripts/tracking.js to remove forceSecureTracker: true:

    diff --git a/app/assets/javascripts/tracking.js b/app/assets/javascripts/tracking.js
    index 0a1211d0a76..3b98c8f28f2 100644
    --- a/app/assets/javascripts/tracking.js
    +++ b/app/assets/javascripts/tracking.js
    @@ -7,7 +7,6 @@ const DEFAULT_SNOWPLOW_OPTIONS = {
       appId: '',
       userFingerprint: false,
       respectDoNotTrack: true,
    -  forceSecureTracker: true,
       eventMethod: 'post',
       contexts: { webPage: true, performanceTiming: true },
       formTracking: false,
    
    
  6. Update snowplow_options in lib/gitlab/tracking.rb to add protocol and port:

    diff --git a/lib/gitlab/tracking.rb b/lib/gitlab/tracking.rb
    index 618e359211b..e9084623c43 100644
    --- a/lib/gitlab/tracking.rb
    +++ b/lib/gitlab/tracking.rb
    @@ -41,7 +41,9 @@ def snowplow_options(group)
               cookie_domain: Gitlab::CurrentSettings.snowplow_cookie_domain,
               app_id: Gitlab::CurrentSettings.snowplow_app_id,
               form_tracking: additional_features,
    -          link_click_tracking: additional_features
    +          link_click_tracking: additional_features,
    +          protocol: 'http',
    +          port: 9090
             }.transform_keys! { |key| key.to_s.camelize(:lower).to_sym }
           end
    
  7. Update emitter in lib/gitlab/tracking/destinations/snowplow.rb to change protocol:

    diff --git a/lib/gitlab/tracking/destinations/snowplow.rb b/lib/gitlab/tracking/destinations/snowplow.rb
    index 4fa844de325..5dd9d0eacfb 100644
    --- a/lib/gitlab/tracking/destinations/snowplow.rb
    +++ b/lib/gitlab/tracking/destinations/snowplow.rb
    @@ -40,7 +40,7 @@ def tracker
             def emitter
               SnowplowTracker::AsyncEmitter.new(
                 Gitlab::CurrentSettings.snowplow_collector_hostname,
    -            protocol: 'https'
    +            protocol: 'http'
               )
             end
           end
    
    
  8. Restart GDK:

    `gdk restart`
    
  9. Send a test Snowplow event from the Rails console:

    Gitlab::Tracking.event('category', 'action')
    
  10. Navigate to localhost:9090/micro/good to see the event.

Snowplow Mini

Snowplow Mini is an easily-deployable, single-instance version of Snowplow.

Snowplow Mini can be used for testing frontend and backend events on a production, staging and local development environment.

For GitLab.com, we’re setting up a QA and Testing environment using Snowplow Mini.

Snowplow Schemas

gitlab_standard

We are including the gitlab_standard schema with every event. See Standardize Snowplow Schema for details.

The StandardContext class represents this schema in the application.

Field Name Required Type Description
project_id integer  
namespace_id integer  
environment string (max 32 chars) Name of the source environment, such as production or staging
source string (max 32 chars) Name of the source application, such as gitlab-rails or gitlab-javascript
extra JSON Any additional data associated with the event, in the form of key-value pairs

Default Schema

Field Name Required Type Description
app_id string Unique identifier for website / application
base_currency string Reporting currency
br_colordepth integer Browser color depth
br_cookies boolean Does the browser permit cookies?
br_family string Browser family
br_features_director boolean Director plugin installed?
br_features_flash boolean Flash plugin installed?
br_features_gears boolean Google gears installed?
br_features_java boolean Java plugin installed?
br_features_pdf boolean Adobe PDF plugin installed?
br_features_quicktime boolean Quicktime plugin installed?
br_features_realplayer boolean RealPlayer plugin installed?
br_features_silverlight boolean Silverlight plugin installed?
br_features_windowsmedia boolean Windows media plugin installed?
br_lang string Language the browser is set to
br_name string Browser name
br_renderengine string Browser rendering engine
br_type string Browser type
br_version string Browser version
br_viewheight string Browser viewport height
br_viewwidth string Browser viewport width
collector_tstamp timestamp Time stamp for the event recorded by the collector
contexts    
derived_contexts   Contexts derived in the Enrich process
derived_tstamp timestamp Timestamp making allowance for inaccurate device clock
doc_charset string Web page’s character encoding
doc_height string Web page height
doc_width string Web page width
domain_sessionid string Unique identifier (UUID) for this visit of this user_id to this domain
domain_sessionidx integer Index of number of visits that this user_id has made to this domain (The first visit is 1)
domain_userid string Unique identifier for a user, based on a first party cookie (so domain specific)
dvce_created_tstamp timestamp Timestamp when event occurred, as recorded by client device
dvce_ismobile boolean Indicates whether device is mobile
dvce_screenheight string Screen / monitor resolution
dvce_screenwidth string Screen / monitor resolution
dvce_sent_tstamp timestamp Timestamp when event was sent by client device to collector
dvce_type string Type of device
etl_tags string JSON of tags for this ETL run
etl_tstamp timestamp Timestamp event began ETL
event string Event type
event_fingerprint string Hash client-set event fields
event_format string Format for event
event_id string Event UUID
event_name string Event name
event_vendor string The company who developed the event model
event_version string Version of event schema
geo_city string City of IP origin
geo_country string Country of IP origin
geo_latitude string An approximate latitude
geo_longitude string An approximate longitude
geo_region string Region of IP origin
geo_region_name string Region of IP origin
geo_timezone string Timezone of IP origin
geo_zipcode string Zip (postal) code of IP origin
ip_domain string Second level domain name associated with the visitor’s IP address
ip_isp string Visitor’s ISP
ip_netspeed string Visitor’s connection type
ip_organization string Organization associated with the visitor’s IP address – defaults to ISP name if none is found
mkt_campaign string The campaign ID
mkt_clickid string The click ID
mkt_content string The content or ID of the ad.
mkt_medium string Type of traffic source
mkt_network string The ad network to which the click ID belongs
mkt_source string The company / website where the traffic came from
mkt_term string Keywords associated with the referrer
name_tracker string The tracker namespace
network_userid string Unique identifier for a user, based on a cookie from the collector (so set at a network level and shouldn’t be set by a tracker)
os_family string Operating system family
os_manufacturer string Manufacturers of operating system
os_name string Name of operating system
os_timezone string Client operating system timezone
page_referrer string Referrer URL
page_title string Page title
page_url string Page URL
page_urlfragment string Fragment aka anchor
page_urlhost string Host aka domain
page_urlpath string Path to page
page_urlport integer Port if specified, 80 if not
page_urlquery string Query string
page_urlscheme string Scheme (protocol name)
platform string The platform the app runs on
pp_xoffset_max integer Maximum page x offset seen in the last ping period
pp_xoffset_min integer Minimum page x offset seen in the last ping period
pp_yoffset_max integer Maximum page y offset seen in the last ping period
pp_yoffset_min integer Minimum page y offset seen in the last ping period
refr_domain_userid string The Snowplow domain_userid of the referring website
refr_dvce_tstamp timestamp The time of attaching the domain_userid to the inbound link
refr_medium string Type of referer
refr_source string Name of referer if recognised
refr_term string Keywords if source is a search engine
refr_urlfragment string Referer URL fragment
refr_urlhost string Referer host
refr_urlpath string Referer page path
refr_urlport integer Referer port
refr_urlquery string Referer URL query string
refr_urlscheme string Referer scheme
se_action string The action / event itself
se_category string The category of event
se_label string A label often used to refer to the ‘object’ the action is performed on
se_property string A property associated with either the action or the object
se_value decimal A value associated with the user action
ti_category string Item category
ti_currency string Currency
ti_name string Item name
ti_orderid string Order ID
ti_price decimal Item price
ti_price_base decimal Item price in base currency
ti_quantity integer Item quantity
ti_sku string Item SKU
tr_affiliation string Transaction affiliation (such as channel)
tr_city string Delivery address: city
tr_country string Delivery address: country
tr_currency string Transaction Currency
tr_orderid string Order ID
tr_shipping decimal Delivery cost charged
tr_shipping_base decimal Shipping cost in base currency
tr_state string Delivery address: state
tr_tax decimal Transaction tax value (such as amount of VAT included)
tr_tax_base decimal Tax applied in base currency
tr_total decimal Transaction total value
tr_total_base decimal Total amount of transaction in base currency
true_tstamp timestamp User-set exact timestamp
txn_id string Transaction ID
unstruct_event JSON The properties of the event
uploaded_at    
user_fingerprint integer User identifier based on (hopefully unique) browser features
user_id string Unique identifier for user, set by the business using setUserId
user_ipaddress string IP address
useragent string User agent (expressed as a browser string)
v_collector string Collector version
v_etl string ETL version
v_tracker string Identifier for Snowplow tracker