Experiment Query

An experiment query is a way of generating an A/B testing experiment. It’s compiled at runtime into a groupBy query, but has a vastly simpler syntax.

Example

This example compares Payscreen A and Payscreen B and will return metrics to judge which is more successful.

{
  "dataSource": "telemetry-signals",
  "granularity": "all",
  "queryType": "experiment",
  "sample1": {
    "filter": {
      "dimension": "type",
      "type": "selector",
      "value": "payScreenALaunched"
    },
    "name": "Payscreen A"
  },
  "sample2": {
    "filter": {
      "dimension": "type",
      "type": "selector",
      "value": "payScreenBLaunched"
    },
    "name": "Payscreen B"
  },
  "successCriterion": {
    "filter": {
      "dimension": "type",
      "type": "selector",
      "value": "paymentSucceeded"
    },
    "name": "Payment Succeeded"
  }
}

Properties

All default properties and the following properties are supported:

sample1 (required)

The first sample to compare, i.e. the control group in the experiment. It must have a name and a filter property. The filter property is used to select signals of all users that are part of the sample.

sample2 (required)

The second sample to compare, i.e. the test group in the experiment. It must have a name and a filter property. The filter property is used to select signals of all users that are part of the sample.

successCriterion (required)

The success criterion to use for the experiment. It must have a name and a filter property. The filter property is used to select signals of all users that are part of the sample.

filter (optional)

A Filter to apply to the data to be queried. The default is no filter apart from the base filters.

granularity (optional)

The granularity of the results. The default is all.

Output

{
  "timestamp": "2023-01-01T00:00:00.000Z",
  "cohort_1": 181,
  "cohort_2": 616,
  "success": 53,
  "cohort_1_success": 20,
  "cohort_2_success": 53,
  "zscore": 0.9444253447373949,
  "pvalue": 0.34495233502890077
}

The output of an experiment query is a list of objects with the following properties:

timestamp

The timestamp of the interval (or none if granularity all is used).

cohort_1

The number of users in the first sample.

cohort_2

The number of users in the second sample.

success

The number of users in any sample that reached the success criterion.

cohort_1_success

The number of users in the first sample that reached the success criterion.

cohort_2_success

The number of users in the second sample that reached the success criterion.

zscore

The z-score of the experiment.

pvalue

The p-value of the experiment.