TelemetryDeck v4.0 brings A/B Testing Experiments

Every single time someone asked me if TelemetryDeck supported A/B Testing and I had to decline, it hurt a little. We want people to use the best, more privacy-focused analytics and usage data service, but in return we have to deliver on features. Luckily, we're here to deliver!

Version 4.0 of TelemetryDeck ships the much-anticipated A/B Testing Feature. Our customers can now run unlimited experiments using our Dashboard UI and make their apps even better.

A/B Testing Experiments

We need to continuously improve our app’s user interface and user experience. Why?

Because that leads to users flowing better towards in app purchases and improves conversion rates, so it increases revenue. If users are less confused about our apps, that also means less support load! And good apps just make people happy!

Happy customers use ours apps more, give better App Store ratings, recommend us to their friends, and give us a warm glowy feeling in our bellies. Well, they do for me.

With TelemetryDeck's new A/B Testing feature, you can easily compare features, designs, behaviours and so on, and choose the one that helps you reach your goals faster.

Use your own cohort code

In this first version, we're not doing any cohort management for you. You'll have to decide, in your app's code, which cohort a user belongs to, and store that fact. Then, let us know by sending a user's cohort in relevant signal payloads.

Filters define cohorts on the server

Using our awesome filter system, define groups of users:

• Users who are in Cohort A (or the control group)
• Users who are in Cohort B (or the experimental group)
• Users who have succeeded in the experiment (we use this as succcess criteria for both cohorts)

We interpret the results for you

It pains me to admit, but most people are not experts in statistics. Which is why we'll not only calculate all relevant statistical metrics for you, but also give you a human-readable sentence that tells you at a glance what's going on. It includes a confidence rating and how different the two groups are.

A/B Testing Experiments are a feature of TQL

We're really proud of the UI, but we want to make sure that all this is usable programmatically as well. This is why experiments, just like funnels before, are fully fledged query types in TQL. This way, you can have even more power and dive even deeper into your data with custom granularities and hard-coded intervals.

Let’s go!

For a lot of our awesome customers, A/B tests are an important feature that they were eagerly waiting for, and it feels amazing to be finally able to deliver them.

A lot of preparation work went into this feature: from, from the transformation to the TelemetryDeck Query Language, over our sneaky addition of the Intersection Post-Aggregator, to the Filter Editor and query pre-compilation feature that the experiment queries are built on. And all of these base layers give us even more opportunities to build on them!

We can’t wait to see what you’ll build with A/B Tests! Please share your screenshots and stories, and let us know what you think.

Thanks for being awesome!
– Daniel