Tagging SQL queries with annotations allows you to leverage the intermix.io App Tracing suite of tools. Annotations tagged with the intermix.io - Tagging Specification will automatically be detected by intermix.io and visible in the dashboard.
App Tracing is a powerful feature which correlates information about applications (dashboards, orchestration tools) with cluster performance data. It can answer questions like:
- which dashboard and user is contributing to this spike in queries?
- what is the average latency of a dashboard? Of all dashboards executed by a particular user?
- my Airflow or Pinball tasks experienced a spike in latency - what caused it? Which query slowed down and why?
- my Airflow or Pinball jobs are failing, how can I quickly find those queries in Intermix?
When to use this
You can use this process to manually add annotations to any query. For example within BI tools or to update SQL strings manually via your ETL workflow system.
Three Ways to Generate Query Annotations
- Web-based tool
- Python SDK
Generating Annotions via "api_annotate" API endpoint
The following API responds with a tag.
The endpoint accepts up to 5 parameters. Here is a usage example:
curl -S -XPOST "api_annotate?app=app1&app_ver=1.1&dag=dag1&task=task1&user=user1"
"annotation": "/* INTERMIX_ID: eyJkYWciOiAiZGFnMSIsICJ2ZXJzaW9uIjogIjEiLCAiYXBwX3ZlciI6ICIxLjEiLCAidXNlciI6ICJ1c2VyMSIsICJmb3JtYXQiOiAiaW50ZXJtaXgiLCAiY2hhcnRfbmFtZSI6ICIiLCAiYXBwIjogImFwcDEiLCAibWV0YSI6IHt9LCAiYXQiOiAiIiwgInRhc2siOiAidGFzazEifQ== */",
Generating Annotations via Web Tool
- Log into intermix.io and go to 'Organization Settings' -> 'App Tracing'
- Enter the values that identify your SQL query and hit 'Create'.
- Paste the annotation string to the top of your SQL query.
Generating Annotations with Python SDK
See this article for instruction on using the Python SDK.