51 lines
1.7 KiB
Markdown
51 lines
1.7 KiB
Markdown
|
# Scripts — ML Pipeline & ETL
|
||
|
|
|
||
|
|
## Purpose
|
||
|
|
|
||
|
|
All data processing, feature engineering, model training, and inference scripts.
|
||
|
|
|
||
|
|
## Stack
|
||
|
|
|
||
|
|
- pandas, numpy, scipy (data processing)
|
||
|
|
- torch, torch_geometric (GCN model)
|
||
|
|
- MLflow (experiment tracking)
|
||
|
|
- geopandas, rasterio (spatial data)
|
||
|
|
|
||
|
|
## Key Scripts
|
||
|
|
|
||
|
|
| Script | Purpose |
|
||
|
|
|--------|---------|
|
||
|
|
| `etl_weather.py` | Weather data ETL (wide→long, interpolation) |
|
||
|
|
| `etl_medical.py` | Medical case ETL (address standardization, geocoding) |
|
||
|
|
| `generate_grid.py` | 100m grid generation |
|
||
|
|
| `generate_grid_features.py` | Grid-level feature engineering |
|
||
|
|
| `resample_spatial_features.py` | DEM/raster resampling to grid |
|
||
|
|
| `aggregate_cases_to_grid.py` | Aggregate cases to grid cells |
|
||
|
|
| `train_model.py` | Full training pipeline (PyTorch + MLflow) |
|
||
|
|
| `inference_grid.py` | Batch grid-level inference |
|
||
|
|
| `inference_daily.py` | Daily inference runner |
|
||
|
|
| `alert_engine.py` | Risk alert generation |
|
||
|
|
| `evaluate.py` | Model evaluation & metrics |
|
||
|
|
| `deploy_schema.sql` | PostGIS database schema |
|
||
|
|
|
||
|
|
## Patterns
|
||
|
|
|
||
|
|
- Scripts are standalone: `if __name__ == '__main__': main()`
|
||
|
|
- Paths use `Path('processed/...')` relative to project root
|
||
|
|
- Run from project root: `python scripts/train_model.py`
|
||
|
|
- MLflow tracks experiments in `mlruns/` and `mlflow.db`
|
||
|
|
|
||
|
|
## Data Flow
|
||
|
|
|
||
|
|
```
|
||
|
|
Datas/ → etl_* → processed/ → train_model.py → models/
|
||
|
|
↘ inference_*.py → PostGIS → API
|
||
|
|
```
|
||
|
|
|
||
|
|
## Anti-Patterns
|
||
|
|
|
||
|
|
- Don't hardcode absolute paths — use `Path` relative to project root
|
||
|
|
- Don't skip MLflow logging for new experiments
|
||
|
|
- Don't modify `processed/` files manually — re-run ETL scripts
|
||
|
|
- Don't import from `backend/` — scripts are independent
|