Introduction
App Tracing is a powerful feature which correlates information about applications (dashboards, orchestration tools) with cluster performance data.
App Tracing can answer questions like:
- which dashboard and user is contributing to this spike in queries?
- what is the average latency of a dashboard? Of all dashboards executed by a particular user?
- my Airflow or Pinball tasks experienced a spike in latency - what caused it? Which query slowed down and why?
- my Airflow or Pinball jobs are failing, how can I quickly find those queries in Intermix?
Related Articles
intermix.io - App Tracing Guide
intermix.io - Tagging Specification
intermix.io - Apache Airflow Plugin
How it Works
Queries are tagged with SQL comments. The comment encodes metadata about the application which submitted this query. Intermix handles tags from popular dashboard tools, and has also published our own format.
The intermix.io platform automatically detects tagged queries. When we see a tagged query, any of the properties of that application are then accessible as first-class properties in our system. For example they can be used to build filters, aggregations, and are visible in in Throughput and Memory Analysis pages as additional workload attributes.
Supported Apps
Apache Airflow
Airflow is supported via a custom plugin. See this article for documentation on getting started.
Supported properties:
| dag |
| task |
| user |
| classname |
| file |
| function |
| linenumber |
Looker
Looker support requires enabling a feature called "Context Comments for SQL Queries". See the following article for instructions:
https://docs.looker.com/admin-options/server/usage#sql_comments
Supported properties:
| user_id |
| history_id |
Mode Analytics
Mode queries will be automatically detected without any required action on your part.
Supported properties:
| user |
| url |
Periscope Data
Periscope queries will be automatically detected without any required action on your part.
Supported properties:
| user_email |
| chart_name |
| dashboard_id |
| query_source |
| user_cat_refresh |
| aggregation |
| priority |
| filter_values |
Chartio
Periscope queries will be automatically detected without any required action on your part.
Supported properties:
| user_email |
| chart_id |
| dashboard_id |
| dashboard_slug |
| organization_id |
| datasource_id |
| is_manual |
Pinterest Pinball
Seeing data for Pinball DAGs and Jobs requires importing the Intermix Python Package and adding a line of code inside each of your Pinball tasks. See the article intermix.io - Python Plugin for documentation on doing that.
| dag |
| task |
| user |
| classname |
| file |
| function |
| linenumber |
Talend ETL
Seeing data for Talend ETL jobs requires using the Tag Generator to add an Intermix tag explicitly to every Talend job. See the article intermix.io - Tag Generator for instructions on generating Intermix tags.
| dag |
| task |
| user |
| classname |
| file |
| function |
| linenumber |
Comments
0 comments
Please sign in to leave a comment.