App Tracing is a powerful feature which correlates information about applications (dashboards, orchestration tools) with cluster performance data.
App Tracing can answer questions like:
- which dashboard and user is contributing to this spike in queries?
- what is the average latency of a dashboard? Of all dashboards executed by a particular user?
- my Airflow or Pinball tasks experienced a spike in latency - what caused it? Which query slowed down and why?
- my Airflow or Pinball jobs are failing, how can I quickly find those queries in Intermix?
How it Works
Queries are tagged with SQL comments. The comment encodes metadata about the application which submitted this query. Intermix handles tags from popular dashboard tools, and has also published our own format.
The intermix.io platform automatically detects tagged queries. When we see a tagged query, any of the properties of that application are then accessible as first-class properties in our system. For example they can be used to build filters, aggregations, and are visible in in Throughput and Memory Analysis pages as additional workload attributes.
Airflow is supported via a custom plugin. See this article for documentation on getting started.
Looker support requires enabling a feature called "Context Comments for SQL Queries". See the following article for instructions:
Mode queries will be automatically detected without any required action on your part.
Report, run, and query ID properties will also be parsed from the URL.
Periscope queries will be automatically detected without any required action on your part.
Chartio queries will be automatically detected without any required action on your part.
Etleap queries will be automatically detected without any required action on your part.
Seeing data for Pinball DAGs and Jobs requires importing the Intermix Python Package and adding a line of code inside each of your Pinball tasks. See the article intermix.io - Python Plugin for documentation on doing that.