106 lines
3.0 KiB
Markdown
106 lines
3.0 KiB
Markdown
|
# Phase 3: Model Training Pipeline - Completion Report
|
||
|
|
|
||
|
|
**Date**: 2026-04-25
|
||
|
|
**Status**: Phase 3 infrastructure COMPLETE, training pending
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Deliverables Status
|
||
|
|
|
||
|
|
### 3.1 PyTorch Geometric Spatiotemporal Model ✓
|
||
|
|
- **File**: `models/spatiotemporal_gcn/model.py`
|
||
|
|
- **Architecture**:
|
||
|
|
- Transformer encoder: 3 layers, 4 heads, dim=48, ff_dim=192, dropout=0.2
|
||
|
|
- GCN: GCNConv(48, 128) → ReLU → Dropout → GCNConv(128, 64)
|
||
|
|
- Output: [N, 3] for 1-day, 3-day, 7-day risk
|
||
|
|
- **ONNX Export**: `models/spatiotemporal_gcn/model_1_3_7.onnx`
|
||
|
|
- **Verified**: Forward pass works on GPU
|
||
|
|
|
||
|
|
### 3.2 GraphSAINT Sampler ✓
|
||
|
|
- **File**: `models/spatiotemporal_gcn/sampler.py`
|
||
|
|
- **Config**: Layer depths [256, 128, 64], batch_size=256
|
||
|
|
- **Compatibility**: Works with base PyG (no torch-sparse required)
|
||
|
|
- **Verified**: Sampler produces valid mini-batches
|
||
|
|
|
||
|
|
### 3.3 MLflow Tracking Server ✓
|
||
|
|
- **File**: `deploy/docker-compose.mlflow.yml`
|
||
|
|
- **Services**: MLflow server + PostgreSQL with PostGIS
|
||
|
|
- **Endpoint**: http://localhost:5000
|
||
|
|
- **Status**: Docker compose file created
|
||
|
|
|
||
|
|
### 3.4 Baseline MAE Computation ✓
|
||
|
|
- **File**: `scripts/compute_baseline_mae.py`
|
||
|
|
- **Results** (validation set: 2023-07-01 to 2024-12-30):
|
||
|
|
|
||
|
|
| Horizon | Baseline MAE | Target (<0.9x) |
|
||
|
|
|---------|--------------|-----------------|
|
||
|
|
| 1-day | 0.2314 | < 0.2083 |
|
||
|
|
| 3-day | 0.5424 | < 0.4882 |
|
||
|
|
| 7-day | 0.6391 | < 0.5752 |
|
||
|
|
|
||
|
|
- **Report**: `reports/baseline_mae.md`
|
||
|
|
|
||
|
|
### 3.5 Training Run ✓
|
||
|
|
- **File**: `scripts/train_model.py`
|
||
|
|
- **Verified**: Data loading works (140k nodes, 23 stations, 9k medical records)
|
||
|
|
- **Configuration**:
|
||
|
|
- Learning rate: 1e-4
|
||
|
|
- Weight decay: 0.01
|
||
|
|
- Patience: 15
|
||
|
|
- Max epochs: 200
|
||
|
|
- Batch size: 1024
|
||
|
|
- **Status**: Ready to run training
|
||
|
|
|
||
|
|
### 3.6 Lambda Smooth Tuning ⏸️
|
||
|
|
- **Status**: Not yet implemented
|
||
|
|
- **Plan**: Search over [0.01, 0.05, 0.1, 0.2, 0.5]
|
||
|
|
|
||
|
|
### 3.7 ONNX Export ✓
|
||
|
|
- **Status**: Already included in model.py
|
||
|
|
- **Exported**: `models/spatiotemporal_gcn/model_1_3_7.onnx`
|
||
|
|
|
||
|
|
### 3.8 Evaluation on Test Set ⏸️
|
||
|
|
- **Status**: Pending - requires training to complete first
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Environment Verification
|
||
|
|
|
||
|
|
| Component | Status | Notes |
|
||
|
|
|-----------|--------|-------|
|
||
|
|
| PyTorch | ✓ | 2.10.0+cu128 |
|
||
|
|
| CUDA | ✓ | 12.8, RTX 3050 4GB |
|
||
|
|
| PyG | ✓ | 2.7.0 |
|
||
|
|
| Model | ✓ | Forward pass OK |
|
||
|
|
| Sampler | ✓ | Mini-batch OK |
|
||
|
|
| MLflow | ✓ | 3.11.1 installed |
|
||
|
|
| ONNX | ✓ | 1.21.0, Runtime 1.25.0 |
|
||
|
|
|
||
|
|
**GPU Memory**: 4GB VRAM (RTX 3050) - sufficient with GraphSAINT sampling
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## To Start Training
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Start MLflow (if not running)
|
||
|
|
docker-compose -f deploy/docker-compose.mlflow.yml up -d
|
||
|
|
|
||
|
|
# Run training
|
||
|
|
python scripts/train_model.py
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Next Steps
|
||
|
|
|
||
|
|
1. **Run training**: `python scripts/train_model.py`
|
||
|
|
- Expected time: Several hours on 4GB GPU
|
||
|
|
- Monitor via MLflow UI at http://localhost:5000
|
||
|
|
|
||
|
|
2. **After training completes**:
|
||
|
|
- Implement Phase 3.6 (Lambda smooth tuning)
|
||
|
|
- Run Phase 3.8 (evaluation on test set)
|
||
|
|
|
||
|
|
3. **Proceed to Phase 4** (Inference Pipeline)
|