Streamlit-style learning doc
Air Quality Pulse Visual Guide
A browser-friendly project guide for the current data pipeline: flow, data, tools, engineering practices, learning outcomes, and source references in one clean page.
Flow
5 stages
source to dashboard
Data
Raw + curated
JSON and SQLite
Tools
Python stack
pandas, Altair, Streamlit
Outcome
DE basics
ingest, validate, serve
Pipeline Flow
1. Source
OpenAQ or sample data
Public air-quality locations and latest pollutant readings.
2. Extract
Python runner
Loads one or many locations through API calls or sample payloads.
3. Raw
JSON files
Preserves original source responses for replay and debugging.
4. Transform
Clean and validate
Adds typed fields, metadata, quality flags, and safety labels.
5. Store
SQLite
Creates analytics-ready measurement and pipeline run tables.
6. Visualize
Streamlit
Shows filters, separate pollutant charts, map points, and guides.
Raw data and curated data are intentionally separate. That is the core habit: keep the source response, then build a cleaner analytics table from it.
Data Shape
Main curated table
The dashboard reads from air_quality_measurements. Each row is one pollutant reading at one monitoring location.
| Field group | Examples |
|---|---|
| Location | location_id, location_name, country_code, coordinates |
| Pollutant | parameter, unit, value |
| Time | measured_at_utc, measured_at_local |
| Quality | quality_issues |
| Lineage | source_url, ingested_at_utc |
Run log table
pipeline_runs records each load so you can tell when it ran, which source URL was used, and how many rows were loaded.
| Tracking field | Purpose |
|---|---|
started_at_utc | When the run began. |
row_count | How many rows were loaded. |
invalid_row_count | How many rows need review. |
status | Whether the run succeeded. |
Layered Data Flow
Input
OpenAQ locations
OpenAQ latest measurements
sample India city payloads
Raw zone
data/raw/openaq_location_*.json
data/raw/openaq_latest_*.json
Curated zone
air_quality_measurements
pipeline_runs
Serving
separate pollutant charts
city comparison
risk-colored map
pollutant guide
Tools And Responsibilities
| Layer | Tool | Responsibility |
|---|---|---|
| Source | OpenAQ API | Public location metadata and air-quality measurements. |
| Extract | Python | Calls API endpoints or loads sample payloads. |
| Raw storage | JSON files | Preserves original responses before transformation. |
| Transform | Python modules | Enriches readings with sensor metadata, validates values, adds safety context. |
| Store | SQLite | Local analytics database for measurements and run logs. |
| Visualize | Streamlit, pandas, Altair | Interactive filters, separate pollutant plots, maps, guides, and reference tables. |
| Verify | unittest | Protects transformation, threshold, pollutant, and multi-location behavior. |
Best Practices
Keep raw responsesDebug and replay without calling the API again.
Create curated tablesMake analytics easier than querying nested API JSON.
Use idempotent keysPrevent duplicate rows when the same reading is loaded again.
Log each runTrack row counts, source URLs, and load status.
Validate rowsCatch missing timestamps, invalid coordinates, and non-numeric readings.
Separate pollutant scalesPM2.5 and ozone use different units, so each needs its own plot.
Safety colors
Safe
Caution
Unsafe
High Risk
Learning Outcomes
By the end of this project, you can explain not just the chart, but the engineering system behind it: where the data came from, how it was shaped, how it was checked, and why the visualization choices are trustworthy.
Outcome
API ingestion
Pull public data through repeatable code.
Outcome
Raw-to-curated
Separate source storage from analytics tables.
Outcome
Quality checks
Flag missing or invalid fields before analysis.
Outcome
Local serving
Serve dashboard data from SQLite.
Outcome
Unit-aware charts
Avoid misleading comparisons across pollutant units.
Outcome
Storytelling
Explain flow, meaning, caveats, and references.
Project File Map
| File or folder | Purpose |
|---|---|
scripts/run_air_quality_pipeline.py | Runs sample or live ingestion for one or many locations. |
scripts/discover_india_locations.py | Finds nearby live OpenAQ monitors for top Indian cities. |
src/air_quality_pulse/openaq_client.py | Handles OpenAQ API requests. |
src/air_quality_pulse/pipeline.py | Coordinates extraction, raw writes, transformation, and database load. |
src/air_quality_pulse/transform.py | Builds curated measurement rows and quality issues. |
src/air_quality_pulse/storage.py | Creates and writes SQLite tables. |
src/air_quality_pulse/thresholds.py | Adds nominal EPA/AirNow safety labels. |
src/air_quality_pulse/pollutants.py | Adds short pollutant effect descriptions. |
app/air_quality_dashboard.py | Displays readings, comparisons, map, legend, and guides. |
tests/ | Verifies transformation, threshold, pollutant, and pipeline behavior. |
References
| Reference | What it supports |
|---|---|
| OpenAQ API documentation | Public API behavior and endpoints. |
| OpenAQ API overview | Platform purpose and data context. |
| OpenAQ quick start | API-key setup and first requests. |
| OpenAQ locations endpoint | Finding monitoring locations. |
| AirNow AQI basics | Safety categories and AQI interpretation. |
| AirNow AQI colors | Green-to-red color scheme. |
| EPA particulate matter health effects | PM2.5 and PM10 effect summaries. |
| EPA ozone health effects | Ozone effect summaries. |
| EPA nitrogen dioxide basics | NO2 effect summaries. |