Articles in this section

intermix.io - App Tracing Guide

Paul Lappas
  • Updated
Follow

Introduction

App Tracing is a powerful feature which correlates information about applications (dashboards, orchestration tools) with cluster performance data.

App Tracing can answer questions like:

  • which dashboard and user is contributing to this spike in queries?
  • what is the average latency of a dashboard? Of all dashboards executed by a particular user?
  • my Airflow or Pinball tasks experienced a spike in latency - what caused it? Which query slowed down and why?
  • my Airflow or Pinball jobs are failing, how can I quickly find those queries in Intermix?

 

Related Articles

intermix.io - App Tracing Guide

intermix.io - Tagging Specification

intermix.io - Python Plugin

intermix.io - Tag Generator

 

How it Works

Queries are tagged with SQL comments. The comment encodes metadata about the application which submitted this query. Intermix handles tags from popular dashboard tools, and has also published our own format.

The intermix.io platform automatically detects tagged queries. When we see a tagged query, any of the properties of that application are then accessible as first-class properties in our system. For example they can be used to build filters, aggregations, and are visible in in Throughput and Memory Analysis pages as additional workload attributes.

 

Supported Apps

Apache Airflow

Airflow is supported via a custom plugin. See this article for documentation on getting started.

Supported properties:

dag
task
user
classname
file
function
linenumber

 

Looker

Looker support requires enabling a feature called "Context Comments for SQL Queries". See the following article for instructions:

https://docs.looker.com/admin-options/server/usage#sql_comments

Supported properties:

user_id
history_id

 

Mode Analytics

Mode queries will be automatically detected without any required action on your part.

Supported properties:

user
url

 

Periscope Data

Periscope queries will be automatically detected without any required action on your part.

Supported properties:

user_email
chart_name
dashboard_id
query_source
user_cat_refresh
aggregation
priority
filter_values

 

Chartio

Chartio support requires using the Tag Generator to add an Intermix tag explicitly to every Chartio dashboard. See the article intermix.io - Tag Generator for instructions on generating Intermix tags.

Supported properties:

user
chart_name

 

Pinterest Pinball

Seeing data for Pinball DAGs and Jobs requires importing the Intermix Python Package and adding a line of code inside each of your Pinball tasks. See the article intermix.io - Python Plugin  for documentation on doing that.

dag
task
user
classname
file
function
linenumber

 

Talend ETL

Seeing data for Talend ETL jobs requires using the Tag Generator to add an Intermix tag explicitly to every Talend job. See the article intermix.io - Tag Generator for instructions on generating Intermix tags.

dag
task
user
classname
file
function
linenumber



 

 

Related articles

Comments

0 comments

Please sign in to leave a comment.