Akiba So
fc468464b2
Context: Build a spatial risk assessment system correlating air quality data with children's respiratory disease incidence across Wuhan. Approach: FastAPI backend serving PostGIS spatial queries, React frontend with Deck.gl maps, and a PyTorch SpatialTemporalGCN pipeline for multi-day (1d/3d/7d) risk prediction. Changes: - backend/ — FastAPI API with auth (JWT), alerts, risk analysis, geocoded case data, grid statistics, and report endpoints - frontend/ — React dashboard with interactive risk maps, alert monitoring, district comparison charts, and timeline player - models/ — SpatialTemporalGCN model with trained weights and ONNX export for inference - scripts/ — ETL pipeline for weather + medical data, grid generation, feature engineering, training, and daily inference - deploy/ — Docker Compose configs for backend, frontend, and MLflow - docs/ — API docs, deployment guide, user guide, and code review Impact: Enables spatial risk visualization, alert monitoring, and ML-driven health risk forecasting for environmental health teams.
51 lines
1.7 KiB
Markdown
51 lines
1.7 KiB
Markdown
# Scripts — ML Pipeline & ETL
|
|
|
|
## Purpose
|
|
|
|
All data processing, feature engineering, model training, and inference scripts.
|
|
|
|
## Stack
|
|
|
|
- pandas, numpy, scipy (data processing)
|
|
- torch, torch_geometric (GCN model)
|
|
- MLflow (experiment tracking)
|
|
- geopandas, rasterio (spatial data)
|
|
|
|
## Key Scripts
|
|
|
|
| Script | Purpose |
|
|
|--------|---------|
|
|
| `etl_weather.py` | Weather data ETL (wide→long, interpolation) |
|
|
| `etl_medical.py` | Medical case ETL (address standardization, geocoding) |
|
|
| `generate_grid.py` | 100m grid generation |
|
|
| `generate_grid_features.py` | Grid-level feature engineering |
|
|
| `resample_spatial_features.py` | DEM/raster resampling to grid |
|
|
| `aggregate_cases_to_grid.py` | Aggregate cases to grid cells |
|
|
| `train_model.py` | Full training pipeline (PyTorch + MLflow) |
|
|
| `inference_grid.py` | Batch grid-level inference |
|
|
| `inference_daily.py` | Daily inference runner |
|
|
| `alert_engine.py` | Risk alert generation |
|
|
| `evaluate.py` | Model evaluation & metrics |
|
|
| `deploy_schema.sql` | PostGIS database schema |
|
|
|
|
## Patterns
|
|
|
|
- Scripts are standalone: `if __name__ == '__main__': main()`
|
|
- Paths use `Path('processed/...')` relative to project root
|
|
- Run from project root: `python scripts/train_model.py`
|
|
- MLflow tracks experiments in `mlruns/` and `mlflow.db`
|
|
|
|
## Data Flow
|
|
|
|
```
|
|
Datas/ → etl_* → processed/ → train_model.py → models/
|
|
↘ inference_*.py → PostGIS → API
|
|
```
|
|
|
|
## Anti-Patterns
|
|
|
|
- Don't hardcode absolute paths — use `Path` relative to project root
|
|
- Don't skip MLflow logging for new experiments
|
|
- Don't modify `processed/` files manually — re-run ETL scripts
|
|
- Don't import from `backend/` — scripts are independent
|