Akiba So
fc468464b2
Context: Build a spatial risk assessment system correlating air quality data with children's respiratory disease incidence across Wuhan. Approach: FastAPI backend serving PostGIS spatial queries, React frontend with Deck.gl maps, and a PyTorch SpatialTemporalGCN pipeline for multi-day (1d/3d/7d) risk prediction. Changes: - backend/ — FastAPI API with auth (JWT), alerts, risk analysis, geocoded case data, grid statistics, and report endpoints - frontend/ — React dashboard with interactive risk maps, alert monitoring, district comparison charts, and timeline player - models/ — SpatialTemporalGCN model with trained weights and ONNX export for inference - scripts/ — ETL pipeline for weather + medical data, grid generation, feature engineering, training, and daily inference - deploy/ — Docker Compose configs for backend, frontend, and MLflow - docs/ — API docs, deployment guide, user guide, and code review Impact: Enables spatial risk visualization, alert monitoring, and ML-driven health risk forecasting for environmental health teams.
1.7 KiB
1.7 KiB
Scripts — ML Pipeline & ETL
Purpose
All data processing, feature engineering, model training, and inference scripts.
Stack
- pandas, numpy, scipy (data processing)
- torch, torch_geometric (GCN model)
- MLflow (experiment tracking)
- geopandas, rasterio (spatial data)
Key Scripts
| Script | Purpose |
|---|---|
etl_weather.py |
Weather data ETL (wide→long, interpolation) |
etl_medical.py |
Medical case ETL (address standardization, geocoding) |
generate_grid.py |
100m grid generation |
generate_grid_features.py |
Grid-level feature engineering |
resample_spatial_features.py |
DEM/raster resampling to grid |
aggregate_cases_to_grid.py |
Aggregate cases to grid cells |
train_model.py |
Full training pipeline (PyTorch + MLflow) |
inference_grid.py |
Batch grid-level inference |
inference_daily.py |
Daily inference runner |
alert_engine.py |
Risk alert generation |
evaluate.py |
Model evaluation & metrics |
deploy_schema.sql |
PostGIS database schema |
Patterns
- Scripts are standalone:
if __name__ == '__main__': main() - Paths use
Path('processed/...')relative to project root - Run from project root:
python scripts/train_model.py - MLflow tracks experiments in
mlruns/andmlflow.db
Data Flow
Datas/ → etl_* → processed/ → train_model.py → models/
↘ inference_*.py → PostGIS → API
Anti-Patterns
- Don't hardcode absolute paths — use
Pathrelative to project root - Don't skip MLflow logging for new experiments
- Don't modify
processed/files manually — re-run ETL scripts - Don't import from
backend/— scripts are independent