Streamlit-style learning doc

Air Quality Pulse Visual Guide

A browser-friendly project guide for the current data pipeline: flow, data, tools, engineering practices, learning outcomes, and source references in one clean page.

Flow
5 stages
source to dashboard
Data
Raw + curated
JSON and SQLite
Tools
Python stack
pandas, Altair, Streamlit
Outcome
DE basics
ingest, validate, serve

Pipeline Flow

1. Source
OpenAQ or sample data
Public air-quality locations and latest pollutant readings.
2. Extract
Python runner
Loads one or many locations through API calls or sample payloads.
3. Raw
JSON files
Preserves original source responses for replay and debugging.
4. Transform
Clean and validate
Adds typed fields, metadata, quality flags, and safety labels.
5. Store
SQLite
Creates analytics-ready measurement and pipeline run tables.
6. Visualize
Streamlit
Shows filters, separate pollutant charts, map points, and guides.
Raw data and curated data are intentionally separate. That is the core habit: keep the source response, then build a cleaner analytics table from it.

Data Shape

Main curated table

The dashboard reads from air_quality_measurements. Each row is one pollutant reading at one monitoring location.

Field groupExamples
Locationlocation_id, location_name, country_code, coordinates
Pollutantparameter, unit, value
Timemeasured_at_utc, measured_at_local
Qualityquality_issues
Lineagesource_url, ingested_at_utc

Run log table

pipeline_runs records each load so you can tell when it ran, which source URL was used, and how many rows were loaded.

Tracking fieldPurpose
started_at_utcWhen the run began.
row_countHow many rows were loaded.
invalid_row_countHow many rows need review.
statusWhether the run succeeded.

Layered Data Flow

Input
OpenAQ locations OpenAQ latest measurements sample India city payloads
Raw zone
data/raw/openaq_location_*.json data/raw/openaq_latest_*.json
Curated zone
air_quality_measurements pipeline_runs
Serving
separate pollutant charts city comparison risk-colored map pollutant guide

Tools And Responsibilities

LayerToolResponsibility
SourceOpenAQ APIPublic location metadata and air-quality measurements.
ExtractPythonCalls API endpoints or loads sample payloads.
Raw storageJSON filesPreserves original responses before transformation.
TransformPython modulesEnriches readings with sensor metadata, validates values, adds safety context.
StoreSQLiteLocal analytics database for measurements and run logs.
VisualizeStreamlit, pandas, AltairInteractive filters, separate pollutant plots, maps, guides, and reference tables.
VerifyunittestProtects transformation, threshold, pollutant, and multi-location behavior.

Best Practices

Keep raw responsesDebug and replay without calling the API again.
Create curated tablesMake analytics easier than querying nested API JSON.
Use idempotent keysPrevent duplicate rows when the same reading is loaded again.
Log each runTrack row counts, source URLs, and load status.
Validate rowsCatch missing timestamps, invalid coordinates, and non-numeric readings.
Separate pollutant scalesPM2.5 and ozone use different units, so each needs its own plot.

Safety colors

Safe
Caution
Unsafe
High Risk

Learning Outcomes

By the end of this project, you can explain not just the chart, but the engineering system behind it: where the data came from, how it was shaped, how it was checked, and why the visualization choices are trustworthy.
Outcome
API ingestion
Pull public data through repeatable code.
Outcome
Raw-to-curated
Separate source storage from analytics tables.
Outcome
Quality checks
Flag missing or invalid fields before analysis.
Outcome
Local serving
Serve dashboard data from SQLite.
Outcome
Unit-aware charts
Avoid misleading comparisons across pollutant units.
Outcome
Storytelling
Explain flow, meaning, caveats, and references.

Project File Map

File or folderPurpose
scripts/run_air_quality_pipeline.pyRuns sample or live ingestion for one or many locations.
scripts/discover_india_locations.pyFinds nearby live OpenAQ monitors for top Indian cities.
src/air_quality_pulse/openaq_client.pyHandles OpenAQ API requests.
src/air_quality_pulse/pipeline.pyCoordinates extraction, raw writes, transformation, and database load.
src/air_quality_pulse/transform.pyBuilds curated measurement rows and quality issues.
src/air_quality_pulse/storage.pyCreates and writes SQLite tables.
src/air_quality_pulse/thresholds.pyAdds nominal EPA/AirNow safety labels.
src/air_quality_pulse/pollutants.pyAdds short pollutant effect descriptions.
app/air_quality_dashboard.pyDisplays readings, comparisons, map, legend, and guides.
tests/Verifies transformation, threshold, pollutant, and pipeline behavior.

References

ReferenceWhat it supports
OpenAQ API documentationPublic API behavior and endpoints.
OpenAQ API overviewPlatform purpose and data context.
OpenAQ quick startAPI-key setup and first requests.
OpenAQ locations endpointFinding monitoring locations.
AirNow AQI basicsSafety categories and AQI interpretation.
AirNow AQI colorsGreen-to-red color scheme.
EPA particulate matter health effectsPM2.5 and PM10 effect summaries.
EPA ozone health effectsOzone effect summaries.
EPA nitrogen dioxide basicsNO2 effect summaries.