v2023-2 is all about Filters!

The latest version of TelemetryDeck – v2023-2 – is now available, and it comes with a bunch of exciting new features and updates! In this blog post, we'll take a look at the new filter editor, the new insight editor, changes to how queries are calculated, and the slow but inevitable transformation of the insight editor.

The Filter Editor

This is the most visible change of TelemetryDeck 2023-2: the filter editor. Previously, filtering data for insights was a pretty manual affair because you had to enter them by hand as JSON code, and you couldn’t really combine filters a lot in different ways: all filters had to apply, so you couldn’t for example say “give me one of these 5 signal types”.

The new filter editor makes it easier to apply advanced filters to query parameters, allowing you to refine query results with various tools.

You can nest filter conditions as deep as you want, and you can combine them using boolean “and”, “or” and “not” operations.

New Filter Condition Types

In addition to combining filter conditions, we’re also adding new condition types:

• Signal Type, which previously was its own control in the Insight Editor, is now just a filter condition, similar to any other condition. This makes it easier to combine multiple signal types using the boolean operations. Also, the signal type was always part of the filter system anyway, so it makes sense to put it there, instead of separately.
• Selector, a filter that allows you to compare any payload key (also known as a dimension) to a value. For example, if you send a shouldUseHealthKit parameter as part of your payload in signals, you can filter for shouldUseHealthKit = “true”. This used to be the only available way to filter.
• Regular Expressions, a programmers best friend! Sometimes you can’t match a value exactly. You might want to catch all iPhone models for example, and there are just too many to list individually. For that, you can now write regular expressions as filter conditions, so you can just filter against iPhone(.*) to get rid of these pesky iPads.

The Insight Editor’s new Layout

We’ve switched the insight editor to a new compact layout with three columns.

This fits nicely with the new filter editor, and it gives us way more room for your actual content, without spending as much space on descriptions and instructions. You’ll still be able to find helpful hints by hovering over the question mark icons next to each segment.

It’ll also give us some space for future cool features 😏

Changes to Query Construction

We’re completely overhauling the way queries to the TelemetryDeck database are constructed and executed.Previously, queries were just low-level objects that were generated just before they were executed, and then immediately discarded.

This has been changing for a while, ever since we allowed you to also start writing queries in the TelemetryDeck Query language for your insights: we’ll integrate TQL queries more into our workflow, and in the future, our visual user interfaces will generate the exact same queries that you’d write by hand, allowing both to be stronger together.

The Playground and TQL docs

Since TQL is becoming more and more important, we thought it’d be neat if you had an area where you could play around with it without always having to create an insight.

You can convert SQL into TQL (thanks to Ed Wellbrook for that suggestion!), and you can directly run any TQL query against your dataset. This is a wonderful quick way of trying out an idea you have, playing around with a complicated query, or just exploring and deep-diving into your dataset.

We also wrote a huge amount of documentation for the TelemetryDeck Query Language! This should help you get started with advanced concepts and help you get the most out of TelemetryDeck.

Updates to TQL

To support our various new developments, we’re adding support for additional features to the TelemetryDeck Query Language. We use these changes to declutter the filters property and make it more easy to work with.

These changes are just in the TQL language themselves for now, but we’re looking forward to incorporating them more into our visual editors as well.

• We’re extracting common filters into a new property, baseFilters. The baseFilters property allows you to switch between “thisApp”, “thisOrganization” and “exampleData”, so you can either query one app, all apps in your organization, or an example project if you don’t have your own data yet.
• You don’t have to set an appID filter any more: just use the appID property in combination with the thisAppbaseFilter.
• You don’t have to define a test mode filter any more: instead, set the testMode property to true or false.
• If you use any UI to enter TQL, such as the playground or an insight, you don’t need to set testMode or relativeIntervals any more. Your query will automatically use the settings of the Dashboard UI.
• If you’re running TQL from an insight, you don’t need to set the appID property or the thisApp baseFilter. The default is set to thisApp and the app ID is automatically injected.

Filters now belong directly to the TQL query

The new version of TelemetryDeck also includes changes to the way queries are generated from Insights.

The way insights work in TelemetryDeck is this:

1. You set up an insight with all the properties you’d like to see, such as insight type, title, chart type, unique-users, etc. and it’s stored in our metadata database.
2. The server generates a TQL query from that insight.
3. That query is then handed to the query calculation infrastructure which handles rights management, calculation, caching, precalculation, etc. and then hands the result back to the UI to display.

Step 2 is where we’ve changed things a lot. Previously, an insight would have all its settings stored in separate database fields, except for one special case: if an insight contained a hand-written TQL query, all database fields would get ignored and the query generation functions would just return that hand-written query.

Well, as of now we’re always storing the filters in the field for the TQL query, which doesn’t have to be hand-written any more. The filters are stored as TQL filters anyway, and the query generation functions will take into account filters from those fields and combine them with the rest of the query generated from the existing fields.

We’re deprecating the fields we’ve used for generating filters previously, and as soon as you open an insight using the new UI, we’re converting the old filter style into the new one and store it in the tql_query property where it belongs.

Why are we doing this? Because it allows us to better push for new features and we can update our visual editors to use the same language that the custom written TQL queries use. This way, any new development is available to both, and you can use them interchangeably!

Query pre-compilation and compilation

To support TQL’s new features, we’ve split up calculation of query results into multiple steps:

1. A valid TQL query is any query that adheres to the TQL specs, allowing you to write high-level queries that use features like base filters and funnel type queries and omit a lot of previously necessary boilerplate.
2. The query constructor then pre-compiles this into a more low-level Druid Query. TQL is a superset of the Druid Native Query Language, so in this step we’re replacing all the custom additions we’ve made with regular Druid elements which are a lot more verbose.
3. We can then run this pre-compiled query using our query executor, which takes care of queueing*, caching, and access control. When we’re passing on pre-compiled queries to be executed, we’re also injecting things like relative time intervals and test mode flags from your UI, so the queries reflect those global settings

This multi step process allows us to offer a very condensed and terse language and still be compatible with our backend time series databases, as well as allowing you to run all kinds of queries directly and still have excellent queueing*, caching, and make sure everyone is only accessing their own data.

*what a fun word to write

Slow Transformation of the Insight Editor

Finally, TelemetryDeck has undergone a slow transformation of the insight editor, making it easier to create and manage custom insights.

As I’ve said above, we’ve already streamlined the process by moving various keys and functionalities directly into the TQL query, and we’ll continue that process in the future. Next up is the breakdown_key and group_by properties, which are also duplicated on both sides right now.

With that process underway, we can also start working on an editor that will work the same, regardless of wether you’re writing a custom query or just assembling a quick insight. You’ll be able to use a graphical UI or a JSON text field to edit your TQL queries (at this point, we’ll probably have to rename the “Custom Query” mode to something like “advanced” mode).

The future

With the release of TelemetryDeck Version 2023-2, users can now take advantage of a range of new features and updates to make the most of their data. Whether it's using the new filter editor, taking advantage of custom query extensions, or quickly gaining insights using the improved insight editor, there's something for everyone!

I’m super happy to be able to finally release this into the world. Next up we’re full steam ahead of advanced features that build on these foundations, such as funnels, A/B testing features, or more.