feat: complete weather feature engineering with simplified approach (375 features)

Final weather feature set after two rounds of user feedback refinement.

Weather Data Collection:
- Collected 24-month weather data (51 points × 7 vars, 9.1 MB)
- 2,703 API requests to OpenMeteo Historical API (14 min runtime)
- 100% data completeness (894,744 records)
- Date range: Oct 2023 - Sep 2025

Weather Feature Engineering (375 features):
- 357 grid-level features (51 points × 7 weather variables)
- 12 temporal lags (temp/wind/solar × 1h/6h/12h/24h)
- 6 derived features (rate-of-change + stability)

Simplification 1 - Physics → Rate-of-Change (user feedback):
Removed overly complex physics-based features requiring calibration data:
× wind_power_potential (wind^3) - requires turbine power curves
× temp_deviation - arbitrary 15C reference
× solar_efficiency - requires solar panel specs

Replaced with simple rate-of-change features (hour-over-hour deltas):
✓ wind_rate_change - captures wind spikes/drops
✓ solar_rate_change - captures solar ramps (cloud cover)
✓ temp_rate_change - captures temperature swings

Kept stability features (detect volatility):
✓ wind_stability_6h, solar_stability_6h, temp_stability_6h

Simplification 2 - Removed Zone Aggregates (user feedback):
Removed zone-level aggregates (36 features) requiring capacity weighting:
× zone_temp_*, zone_wind_*, zone_solar_* for 12 zones
× Fatal flaw: Equal weighting without knowing generation capacity
× Example: Hamburg offshore (5 GW) ≠ Munich (0.1 GW)

Rationale:
- Model learns from granular grid-level data (357 features)
- Rate-of-change captures timing of weather events → grid adjustments
- No calibration data needed (turbine curves, asset locations, capacities)
- Simpler = more interpretable = easier to debug
- Zero-shot MVP: maximize raw signal, minimize engineered assumptions

Bug Fixes:
- Unicode emoji crash (Windows cp1252 compatibility)
- Polars completeness calculation (scalar extraction)
- Polars join deprecation (outer → left with coalesce)

Files:
- scripts/collect_openmeteo_24month.py (new)
- src/data_collection/collect_openmeteo.py (bug fixes)
- src/feature_engineering/engineer_weather_features.py (new, refined twice)
- data/raw/weather_24month.parquet (9.1 MB, not in git)
- data/processed/features_weather_24month.parquet (10.19 MB, not in git)

Feature Engineering COMPLETE:
- JAO: 1,698 features
- ENTSO-E: 296 features
- Weather: 375 features
- Total: 2,369 features ready for unification

Next: Feature unification → Zero-shot inference

doc/activity.md

@@ -2605,3 +2605,497 @@ Cleanup logic added at line 821-880:
 
 **Status**: ✅ ENTSO-E Features Clean & Ready - Moving to Weather Collection
 
+---
+
+## 2025-11-10 (Part 2) - Weather Data Collection Infrastructure Ready
+
+### Summary
+Prepared weather data collection infrastructure and fixed critical bugs. Ready for full 24-month collection (deferred to next session due to time constraints).
+
+### Weather Collection Scope
+**Target**: 52 strategic grid points × 7 weather variables × 24 months
+
+**Grid Coverage**:
+- Germany: 6 points (North Sea, Hamburg, Berlin, Frankfurt, Munich, Baltic)
+- France: 5 points (Dunkirk, Paris, Lyon, Marseille, Strasbourg)
+- Netherlands: 4 points (Offshore, Amsterdam, Rotterdam, Groningen)
+- Austria: 3 points (Kaprun, St. Peter, Vienna)
+- Belgium: 3 points (Offshore, Doel, Avelgem)
+- Czech Republic: 3 points (Hradec, Bohemia, Temelin)
+- Poland: 4 points (Baltic, SHVDC, Belchatow, Mikulowa)
+- Hungary: 3 points (Paks, Bekescsaba, Gyor)
+- Romania: 3 points (Fantanele, Iron Gates, Cernavoda)
+- Slovakia: 3 points (Bohunice, Gabcikovo, Rimavska)
+- Slovenia: 2 points (Krsko, Divaca)
+- Croatia: 2 points (Ernestinovo, Zagreb)
+- Luxembourg: 2 points (Trier, Bauler)
+- External: 8 points (CH, UK, ES, IT, NO, SE, DK×2)
+
+**Weather Variables**:
+- `temperature_2m`: Air temperature (C)
+- `windspeed_10m`: Wind at 10m (m/s)
+- `windspeed_100m`: Wind at 100m for generation (m/s)
+- `winddirection_100m`: Wind direction (degrees)
+- `shortwave_radiation`: Solar radiation (W/m2)
+- `cloudcover`: Cloud cover (%)
+- `surface_pressure`: Pressure (hPa)
+
+**Collection Strategy**:
+- OpenMeteo Historical API (free tier)
+- 2-week chunks (1.0 API call each)
+- 270 requests/minute (45% of 600/min limit)
+- Total: 2,703 HTTP requests
+- Estimated runtime: 10 minutes
+- Expected output: ~50-80 MB parquet file
+
+### Bugs Discovered and Fixed
+
+#### Bug 1: Unicode Emoji in Windows Console
+**Problem**:
+- Windows cmd.exe uses cp1252 encoding (not UTF-8)
+- Emojis (✓, ✗, ✅) in progress messages caused `UnicodeEncodeError`
+- Collection crashed at 15% after successfully fetching data
+
+**Root Cause**:
+```python
+# Line 281, 347, 372 in collect_openmeteo.py
+print(f"✅ {location_id}: {location_df.shape[0]} hours")  # BROKEN
+print(f"❌ Failed {location_id}")  # BROKEN
+```
+
+**Fix Applied**:
+```python
+print(f"[OK] {location_id}: {location_df.shape[0]} hours")
+print(f"[ERROR] Failed {location_id}")
+```
+
+**Files Modified**: `src/data_collection/collect_openmeteo.py:281,347,372`
+
+#### Bug 2: Polars Completeness Calculation
+**Problem**:
+- Line 366: `combined_df.null_count().sum()` returns DataFrame (not scalar)
+- Type error: `unsupported operand type(s) for -: 'int' and 'DataFrame'`
+- Collection completed 100% but failed at final save step
+- All 894,744 records collected but lost (not written to disk)
+
+**Root Cause**:
+```python
+# BROKEN - Polars returns DataFrame
+completeness = (1 - combined_df.null_count().sum() / (rows * cols)) * 100
+```
+
+**Fix Applied**:
+```python
+# Extract scalar from Polars
+null_count_total = combined_df.null_count().sum_horizontal()[0]
+completeness = (1 - null_count_total / (rows * cols)) * 100
+```
+
+**Files Modified**: `src/data_collection/collect_openmeteo.py:366-370`
+
+### Test Results
+
+**Test Scope**: 1 week × 51 grid points (minimal test)
+```bash
+Date range: 2025-09-23 to 2025-09-30
+Grid points: 51
+Total records: 9,792 (192 hours each)
+Test duration: ~20 seconds
+```
+
+**Test Output**:
+```
+Total HTTP requests: 51
+Total API calls consumed: 51.0
+Total records: 9,792
+Date range: 2025-09-23 00:00:00 to 2025-09-30 23:00:00
+Grid points: 51
+Completeness: 100.00%  ✅
+Output: test_weather.parquet
+File size: 0.1 MB
+```
+
+**Validation**:
+- ✅ All 51 grid points collected successfully
+- ✅ 100% data completeness (no missing values)
+- ✅ File saved and loaded correctly
+- ✅ No errors or crashes
+- ✅ Test file cleaned up
+
+### Files Modified
+
+**Scripts Created**:
+- `scripts/collect_openmeteo_24month.py` - 24-month collection script
+  - Uses existing `OpenMeteoCollector` class
+  - 2-week chunking
+  - Progress tracking with tqdm
+  - Output: `data/raw/weather_24month.parquet`
+
+**Bug Fixes**:
+- `src/data_collection/collect_openmeteo.py:281,347,372` - Removed Unicode emojis
+- `src/data_collection/collect_openmeteo.py:366-370` - Fixed Polars completeness calculation
+
+### Current Status
+
+**Weather Infrastructure**: ✅ Complete and Tested
+- Collection script ready
+- All bugs fixed
+- Tested successfully with 1-week sample
+- Ready for full 24-month collection
+
+**Data Collected**:
+- JAO: ✅ 1,698 features (24 months)
+- ENTSO-E: ✅ 296 features (24 months)
+- Weather: ⏳ Pending (infrastructure ready, ~10 min runtime)
+
+**Why Deferred**:
+User had time constraints - weather collection requires ~10 minutes uninterrupted runtime.
+
+### Next Session Workflow
+
+**IMMEDIATE ACTION** (when you return):
+```bash
+# Run 24-month weather collection (~10 minutes)
+.venv/Scripts/python.exe scripts/collect_openmeteo_24month.py
+```
+
+**Expected Output**:
+- File: `data/raw/weather_24month.parquet`
+- Size: 50-80 MB
+- Records: ~894,744 (51 points × 17,544 hours)
+- Features (raw): 12 columns (timestamp, grid_point, location_name, lat, lon, + 7 weather vars)
+
+**After Weather Collection**:
+1. **Feature Engineering** - Weather features (~364 features)
+   - Grid-level: `temp_{grid}`, `wind_{grid}`, `solar_{grid}` (51 × 7 = 357)
+   - Zone-level aggregation: `temp_avg_{zone}`, `wind_avg_{zone}` (optional)
+   - Lags: Previous 1h, 6h, 12h, 24h (key variables only)
+
+2. **Feature Unification** - Merge all sources
+   - JAO: 1,698 features
+   - ENTSO-E: 296 features
+   - Weather: ~364 features
+   - **Total: ~2,358 unified features**
+
+3. **Day 3: Zero-Shot Inference**
+   - Load Chronos 2 Large (710M params)
+   - Run inference on unified feature set
+   - Evaluate D+1 MAE (target: <150 MW)
+
+### Lessons Learned
+
+1. **Windows Console Limitations**: Never use Unicode characters in backend scripts on Windows
+   - Use ASCII alternatives: `[OK]`, `[ERROR]`, `[SUCCESS]`
+   - Emojis OK in: Marimo notebooks (browser-rendered), documentation
+
+2. **Polars API Differences**: Always extract scalars explicitly
+   - `.sum()` returns DataFrame in Polars
+   - Use `.sum_horizontal()[0]` to get scalar value
+
+3. **Test Before Full Collection**: Quick tests save hours
+   - 20-second test caught a bug that would have lost 10 minutes of collection
+   - Always test with minimal data (1 week vs 24 months)
+
+### Git Status
+
+**Committed**: ENTSO-E quality fixes (previous session)
+**Uncommitted**: Weather collection bug fixes (ready to commit)
+
+**Next Commit** (after weather collection completes):
+```
+feat: complete weather data collection with bug fixes
+
+- Fixed Unicode emoji crash (Windows cp1252 compatibility)
+- Fixed Polars completeness calculation
+- Collected 24-month weather data (51 points × 7 vars)
+- Created scripts/collect_openmeteo_24month.py
+- Output: data/raw/weather_24month.parquet (~50-80 MB)
+
+Next: Weather feature engineering (~364 features)
+```
+
+### Summary Statistics
+
+**Project Progress**:
+- Day 0: ✅ Setup complete
+- Day 1: ✅ Data collection (JAO, ENTSO-E complete; Weather ready)
+- Day 2: 🔄 Feature engineering (JAO ✅, ENTSO-E ✅, Weather ⏳)
+- Day 3: ⏳ Zero-shot inference (pending)
+- Day 4: ⏳ Evaluation (pending)
+- Day 5: ⏳ Documentation (pending)
+
+**Feature Count Tracking**:
+- JAO: 1,698 ✅
+- ENTSO-E: 296 ✅ (cleaned from 464)
+- Weather: 364 ⏳ (infrastructure ready)
+- **Projected Total: ~2,358 features**
+
+**Data Quality**:
+- JAO: 100% complete
+- ENTSO-E: 99.76% complete
+- Weather: TBD (expect >99% based on test)
+
+---
+
+## 2025-11-10 (Part 3) - Weather Feature Engineering Complete
+
+### Summary
+Completed weather data collection and feature engineering. All three feature sets (JAO, ENTSO-E, Weather) are now ready for unification.
+
+### Weather Data Collection
+**Execution**:
+- Ran `scripts/collect_openmeteo_24month.py`
+- Collection time: 14 minutes (2,703 API requests)
+- 51 grid points × 53 two-week chunks × 7 variables
+
+**Results**:
+- ✅ 894,744 records collected (51 points × 17,544 hours)
+- ✅ 100% data completeness
+- ✅ File: `data/raw/weather_24month.parquet` (9.1 MB)
+- ✅ Date range: Oct 2023 - Sep 2025 (24 months)
+
+**Bug Fixed** (post-collection):
+- Line 85-86 in script still had completeness calculation bug
+- Fixed `.sum()` to `.sum_horizontal()[0]` for scalar extraction
+- Data was saved successfully despite error
+
+### Weather Feature Engineering
+**Created**: `src/feature_engineering/engineer_weather_features.py`
+
+**Features Engineered** (411 total):
+1. **Grid-level features** (357): 51 grid points × 7 weather variables
+   - temp_<grid_point>, wind10m_<grid_point>, wind100m_<grid_point>
+   - winddir_<grid_point>, solar_<grid_point>, cloud_<grid_point>, pressure_<grid_point>
+
+2. **Zone-level aggregates** (36): 12 Core FBMC zones × 3 key variables
+   - zone_temp_<zone>, zone_wind_<zone>, zone_solar_<zone>
+
+3. **Temporal lags** (12): 3 variables × 4 time periods
+   - temp_avg_lag1h/6h/12h/24h
+   - wind_avg_lag1h/6h/12h/24h
+   - solar_avg_lag1h/6h/12h/24h
+
+4. **Derived features** (6):
+   - wind_power_potential (wind^3, proportional to turbine output)
+   - temp_deviation (deviation from 15C reference)
+   - solar_efficiency (solar output adjusted for temperature)
+   - wind_stability_6h, solar_stability_6h, temp_stability_6h (rolling std)
+
+**Output**:
+- File: `data/processed/features_weather_24month.parquet`
+- Size: 11.48 MB
+- Shape: 17,544 rows × 412 columns (411 features + timestamp)
+- Completeness: 100%
+
+**Bugs Fixed During Development**:
+1. **Polars join deprecation**: Changed `how='outer'` to `how='left'` with `coalesce=True`
+2. **Duplicate timestamp columns**: Used coalesce to prevent `timestamp_right` duplicates
+
+### Files Created
+- `scripts/collect_openmeteo_24month.py` (fixed bugs)
+- `src/feature_engineering/engineer_weather_features.py` (new)
+- `data/raw/weather_24month.parquet` (9.1 MB)
+- `data/processed/features_weather_24month.parquet` (11.48 MB)
+
+### Feature Count Update
+**Final Feature Inventory**:
+- JAO: 1,698 ✅ Complete
+- ENTSO-E: 296 ✅ Complete
+- Weather: 411 ✅ Complete
+- **Total: 2,405 features** (vs target ~1,735 = +39%)
+
+### Key Lessons
+1. **Polars API Evolution**: Deprecation warnings for join methods
+   - `how='outer'` → `how='left'` with `coalesce=True`
+   - Prevents duplicate columns in sequential joins
+
+2. **Feature Engineering Approach**:
+   - Grid-level: Maximum spatial resolution (51 points)
+   - Zone-level: Aggregated for regional patterns
+   - Temporal lags: Capture weather persistence
+   - Derived: Physical relationships (wind^3 for power, temp effects on solar)
+
+3. **Data Completeness**: 100% across all three feature sets
+   - No missing values to impute
+   - Ready for direct model input
+
+### Git Status
+**Ready to commit**:
+- Weather collection script (bug fixes)
+- Weather feature engineering module
+- Two new parquet files (raw + processed)
+
+**Next Commit**:
+```bash
+feat: complete weather feature engineering (411 features)
+
+- Collected 24-month weather data (51 points × 7 vars, 9.1 MB)
+- Engineered 411 weather features (100% complete)
+  * 357 grid-level features
+  * 36 zone-level aggregates
+  * 12 temporal lags (1h/6h/12h/24h)
+  * 6 derived features (wind power, solar efficiency, stability)
+- Created src/feature_engineering/engineer_weather_features.py
+- Output: data/processed/features_weather_24month.parquet (11.48 MB)
+
+Feature engineering COMPLETE:
+- JAO: 1,698 features
+- ENTSO-E: 296 features
+- Weather: 411 features
+- Total: 2,405 features ready for unification
+
+Next: Feature unification → Zero-shot inference
+```
+
+### Summary Statistics
+**Project Progress**:
+- Day 0: ✅ Setup complete
+- Day 1: ✅ Data collection complete (JAO, ENTSO-E, Weather)
+- Day 2: ✅ Feature engineering complete (JAO, ENTSO-E, Weather)
+- Day 3: ⏳ Feature unification → Zero-shot inference
+- Day 4: ⏳ Evaluation
+- Day 5: ⏳ Documentation + handover
+
+**Feature Count (Final)**:
+- JAO: 1,698 ✅
+- ENTSO-E: 296 ✅
+- Weather: 411 ✅
+- **Total: 2,405 features** (39% above target)
+
+**Data Quality**:
+- JAO: 100% complete
+- ENTSO-E: 99.76% complete
+- Weather: 100% complete
+
+---
+
+## 2025-11-10 (Part 4) - Simplified Weather Features (Physics → Rate-of-Change)
+
+### Summary
+Replaced overly complex physics-based features with simple rate-of-change features based on user feedback.
+
+### Problem Identified
+**User feedback**: Original derived features were too complex without calibration data:
+- `wind_power_potential` (wind^3) - requires turbine power curves
+- `temp_deviation` (from 15C) - arbitrary reference point
+- `solar_efficiency` (temp-adjusted) - requires solar panel specifications
+
+These require geographic knowledge, power curves, and equipment specs we don't have.
+
+### Solution Applied
+**Replaced 3 complex features with 3 simple rate-of-change features:**
+
+**Removed:**
+1. `wind_power_potential` (wind^3 transformation)
+2. `temp_deviation` (arbitrary 15C reference)
+3. `solar_efficiency` (requires solar panel specs)
+
+**Added (hour-over-hour deltas):**
+1. `wind_rate_change` - captures wind spikes/drops
+2. `solar_rate_change` - captures solar ramps (cloud cover)
+3. `temp_rate_change` - captures temperature swings
+
+**Kept (stability metrics - useful for volatility):**
+1. `wind_stability_6h` (rolling std)
+2. `solar_stability_6h` (rolling std)
+3. `temp_stability_6h` (rolling std)
+
+### Rationale
+**Rate-of-change features capture what matters:**
+- Sudden wind spikes → wind generation ramping → redispatch
+- Solar drops (clouds) → solar generation drops → grid adjustments
+- Temperature swings → demand shifts → flow changes
+
+**No calibration data needed:**
+- Model learns physics from raw grid-level data (357 features)
+- Rate-of-change provides timing signals for correlation
+- Simpler features = more interpretable = easier to debug
+
+### Results
+**Re-ran feature engineering:**
+- Total features: 411 (unchanged)
+- Derived features: 6 (3 rate-of-change + 3 stability)
+- File size: 11.41 MB (0.07 MB smaller)
+- Completeness: 100%
+
+### Key Lesson
+**Simplicity over complexity in zero-shot MVP:**
+- Don't attempt to encode domain physics without calibration data
+- Let the model learn complex relationships from raw signals
+- Use simple derived features (deltas, rolling stats) for timing/volatility
+- Save physics-based features for Phase 2 when we have equipment data
+
+---
+
+## 2025-11-10 (Part 5) - Removed Zone Aggregates (Final: 375 Weather Features)
+
+### Summary
+Removed zone-level aggregate features (36 features) due to lack of capacity weighting data.
+
+### Problem Identified
+**User feedback**: Zone aggregates assume equal weighting without capacity data:
+- Averaging wind speed across DE_LU grid points (6 locations)
+- No knowledge of actual generation capacity at each location
+- Hamburg offshore: 5 GW vs Munich: 0.1 GW → equal averaging = meaningless
+
+**Fatal flaw**: Without knowing WHERE wind farms/solar parks are located and their CAPACITY, zone averages add noise instead of signal.
+
+### Solution Applied
+**Removed zone aggregation entirely:**
+- Deleted `engineer_zone_aggregates()` function
+- Removed 36 features (12 zones × 3 variables)
+- Deleted GRID_POINT_TO_ZONE mapping (unused)
+
+**Final Feature Set (375 features):**
+1. **Grid-level**: 357 features (51 points × 7 variables)
+   - Model learns which specific locations correlate with flows
+2. **Temporal lags**: 12 features (3 variables × 4 time periods)
+   - Captures weather persistence
+3. **Derived**: 6 features (rate-of-change + stability)
+   - Simple signals without requiring calibration data
+
+### Rationale
+**Let the model find the important locations:**
+- 51 grid-level features give model full spatial resolution
+- Model can learn which points have generation assets
+- No false precision from unweighted aggregation
+- Cleaner signal for zero-shot learning
+
+### Results
+**Re-ran feature engineering:**
+- Total features: 375 (down from 411, -36)
+- File size: 10.19 MB (down from 11.41 MB, -1.22 MB)
+- Completeness: 100%
+
+### Key Lesson
+**Avoid aggregation without domain knowledge:**
+- Equal weighting ≠ capacity-weighted average
+- Geographic averages require knowing asset locations and capacities
+- When in doubt, keep granular data and let the model learn patterns
+- Zero-shot MVP: maximize raw signal, minimize engineered assumptions
+
+### Final Weather Features Breakdown
+1. **Grid-level (357)**:
+   - temp_*, wind10m_*, wind100m_*, winddir_*
+   - solar_*, cloud_*, pressure_* for each of 51 grid points
+
+2. **Temporal lags (12)**:
+   - temp_avg_lag1h/6h/12h/24h
+   - wind_avg_lag1h/6h/12h/24h
+   - solar_avg_lag1h/6h/12h/24h
+
+3. **Derived (6)**:
+   - wind_rate_change, solar_rate_change, temp_rate_change (hour-over-hour)
+   - wind_stability_6h, solar_stability_6h, temp_stability_6h (rolling std)
+
+---
+
+**NEXT SESSION BOOKMARK**: Feature unification (merge 2,369 features on timestamp), then zero-shot inference
+
+**Status**: ✅ All Feature Engineering Complete - Ready for Unification
+
+**Final Feature Count**:
+- JAO: 1,698
+- ENTSO-E: 296
+- Weather: 375
+- **Total: 2,369 features** (down from 2,405)

scripts/collect_openmeteo_24month.py

@@ -0,0 +1,159 @@
+"""
+Collect 24-Month Weather Data from OpenMeteo
+=============================================
+
+Collects hourly weather data from OpenMeteo Historical API for the full
+24-month period (Oct 2023 - Sept 2025) across 52 strategic grid points.
+
+7 Weather Variables:
+- temperature_2m: Air temperature at 2m (C)
+- windspeed_10m: Wind speed at 10m (m/s)
+- windspeed_100m: Wind speed at 100m (m/s) - for wind generation
+- winddirection_100m: Wind direction at 100m (degrees)
+- shortwave_radiation: Solar radiation (W/m2) - for solar generation
+- cloudcover: Cloud cover percentage
+- surface_pressure: Surface air pressure (hPa)
+
+Collection Strategy:
+- 52 grid points (covering all FBMC zones + neighbors)
+- 2-week chunks (1.0 API call each)
+- 270 requests/minute (45% of 600 limit)
+- Estimated runtime: ~5 minutes
+
+Output: data/raw/weather_24month.parquet
+Size: ~50-80 MB (52 points × 7 vars × 17,520 hours)
+Features: 364 (52 × 7) when engineered
+"""
+
+import sys
+from pathlib import Path
+
+# Add src to path
+sys.path.append(str(Path(__file__).parent.parent))
+
+from src.data_collection.collect_openmeteo import OpenMeteoCollector
+
+# Date range: Oct 2023 - Sept 2025 (24 months)
+START_DATE = '2023-10-01'
+END_DATE = '2025-09-30'
+
+# Output file
+OUTPUT_DIR = Path(__file__).parent.parent / 'data' / 'raw'
+OUTPUT_FILE = OUTPUT_DIR / 'weather_24month.parquet'
+
+print("="*80)
+print("24-MONTH WEATHER DATA COLLECTION")
+print("="*80)
+print()
+print("Period: October 2023 - September 2025 (24 months)")
+print("Grid points: 52 strategic locations across FBMC")
+print("Variables: 7 weather parameters")
+print("Estimated runtime: ~5 minutes")
+print()
+
+# Initialize collector with safe rate limiting
+print("Initializing OpenMeteo collector...")
+collector = OpenMeteoCollector(
+    requests_per_minute=270,  # 45% of 600 limit
+    chunk_days=14  # 1.0 API call per request
+)
+print("[OK] Collector initialized")
+print()
+
+# Run collection
+try:
+    df = collector.collect_all(
+        start_date=START_DATE,
+        end_date=END_DATE,
+        output_path=OUTPUT_FILE
+    )
+
+    if not df.is_empty():
+        print()
+        print("="*80)
+        print("COLLECTION SUCCESS")
+        print("="*80)
+        print()
+        print(f"Output: {OUTPUT_FILE}")
+        print(f"Shape: {df.shape[0]:,} rows x {df.shape[1]} columns")
+        print(f"Date range: {df['timestamp'].min()} to {df['timestamp'].max()}")
+        print(f"Grid points: {df['grid_point'].n_unique()}")
+        print(f"Weather variables: {len([c for c in df.columns if c not in ['timestamp', 'grid_point', 'location_name', 'latitude', 'longitude']])}")
+        print()
+
+        # Data quality summary
+        null_count_total = df.null_count().sum_horizontal()[0]
+        null_pct = (null_count_total / (df.shape[0] * df.shape[1])) * 100
+        print(f"Data completeness: {100 - null_pct:.2f}%")
+
+        if null_pct > 0:
+            print()
+            print("Missing data by column:")
+            for col in df.columns:
+                null_count = df[col].null_count()
+                if null_count > 0:
+                    pct = (null_count / len(df)) * 100
+                    print(f"  - {col}: {null_count:,} ({pct:.2f}%)")
+
+        print()
+        print("="*80)
+        print("NEXT STEPS")
+        print("="*80)
+        print()
+        print("1. Implement weather feature engineering:")
+        print("   - Create src/feature_engineering/engineer_weather_features.py")
+        print("   - Engineer ~364 features (52 grid points x 7 variables)")
+        print("   - Add spatial aggregation (zone-level averages)")
+        print()
+        print("2. Expected features:")
+        print("   - Grid-level: temp_{grid_point}, wind_{grid_point}, solar_{grid_point}, etc.")
+        print("   - Zone-level: temp_avg_{zone}, wind_avg_{zone}, solar_avg_{zone}, etc.")
+        print("   - Lags: Previous 1h, 6h, 12h, 24h for key variables")
+        print()
+        print("3. Final unified features:")
+        print("   - JAO: 1,698")
+        print("   - ENTSO-E: 296")
+        print("   - Weather: 364")
+        print("   - Total: ~2,358 features")
+        print()
+        print("[OK] Weather data collection COMPLETE!")
+    else:
+        print()
+        print("[ERROR] No weather data collected")
+        print()
+        print("Possible causes:")
+        print("  - OpenMeteo API access issues")
+        print("  - Rate limit exceeded")
+        print("  - Network connectivity problems")
+        print()
+        sys.exit(1)
+
+except KeyboardInterrupt:
+    print()
+    print()
+    print("="*80)
+    print("COLLECTION INTERRUPTED")
+    print("="*80)
+    print()
+    print("Collection was stopped by user.")
+    print()
+    print("NOTE: OpenMeteo collection does NOT have checkpoint/resume capability")
+    print("      (collection completes in ~5 minutes, so not needed)")
+    print()
+    print("To restart: Run this script again")
+    print()
+    sys.exit(130)
+
+except Exception as e:
+    print()
+    print()
+    print("="*80)
+    print("COLLECTION FAILED")
+    print("="*80)
+    print()
+    print(f"Error: {e}")
+    print()
+    import traceback
+    traceback.print_exc()
+    print()
+    sys.exit(1)

src/data_collection/collect_openmeteo.py

@@ -278,7 +278,7 @@ class OpenMeteoCollector:
             return df
 
         except requests.exceptions.RequestException as e:
-            print(f"❌ Failed {location_id} ({start_date} to {end_date}): {e}")
+            print(f"[ERROR] Failed {location_id} ({start_date} to {end_date}): {e}")
             return pl.DataFrame()
 
     def collect_all(
@@ -344,7 +344,7 @@ class OpenMeteoCollector:
                 if location_chunks:
                     location_df = pl.concat(location_chunks)
                     all_data.append(location_df)
-                    print(f"✅ {location_id}: {location_df.shape[0]} hours")
+                    print(f"[OK] {location_id}: {location_df.shape[0]} hours")
 
         # Combine all dataframes
         if all_data:
@@ -363,13 +363,18 @@ class OpenMeteoCollector:
             print(f"Total records: {combined_df.shape[0]:,}")
             print(f"Date range: {combined_df['timestamp'].min()} to {combined_df['timestamp'].max()}")
             print(f"Grid points: {combined_df['grid_point'].n_unique()}")
-            print(f"Completeness: {(1 - combined_df.null_count().sum() / (combined_df.shape[0] * combined_df.shape[1])) * 100:.2f}%")
+
+            # Calculate completeness (fix: extract scalar from Polars)
+            null_count_total = combined_df.null_count().sum_horizontal()[0]
+            completeness = (1 - null_count_total / (combined_df.shape[0] * combined_df.shape[1])) * 100
+            print(f"Completeness: {completeness:.2f}%")
+
             print(f"Output: {output_path}")
             print(f"File size: {output_path.stat().st_size / (1024**2):.1f} MB")
 
             return combined_df
         else:
-            print("❌ No data collected")
+            print("[ERROR] No data collected")
             return pl.DataFrame()
 
 

src/feature_engineering/engineer_weather_features.py

@@ -0,0 +1,263 @@
+"""Engineer 375 Weather features for FBMC forecasting.
+
+Transforms OpenMeteo weather data into model-ready features:
+1. Grid-level features (51 points × 7 vars = 357 features)
+2. Temporal lags (3 vars × 4 time periods = 12 features)
+3. Derived features (rate-of-change + stability = 6 features)
+
+Total: 375 weather features
+
+Weather Variables (7):
+- temperature_2m (C)
+- windspeed_10m (m/s)
+- windspeed_100m (m/s) - for wind generation
+- winddirection_100m (degrees)
+- shortwave_radiation (W/m2) - for solar generation
+- cloudcover (%)
+- surface_pressure (hPa)
+
+Author: Claude
+Date: 2025-11-10
+"""
+from pathlib import Path
+import polars as pl
+
+
+def engineer_grid_level_features(weather_df: pl.DataFrame) -> pl.DataFrame:
+    """Engineer grid-level weather features (51 points × 7 vars = 357 features).
+
+    For each grid point, pivot all 7 weather variables to wide format:
+    - temp_<grid_point>
+    - wind10m_<grid_point>
+    - wind100m_<grid_point>
+    - winddir_<grid_point>
+    - solar_<grid_point>
+    - cloud_<grid_point>
+    - pressure_<grid_point>
+    """
+    print("\n[1/5] Engineering grid-level features (51 points × 7 vars)...")
+
+    # Pivot each weather variable separately
+    features = None
+
+    weather_vars = [
+        ('temperature_2m', 'temp'),
+        ('windspeed_10m', 'wind10m'),
+        ('windspeed_100m', 'wind100m'),
+        ('winddirection_100m', 'winddir'),
+        ('shortwave_radiation', 'solar'),
+        ('cloudcover', 'cloud'),
+        ('surface_pressure', 'pressure')
+    ]
+
+    for orig_col, short_name in weather_vars:
+        print(f"  Pivoting {orig_col}...")
+
+        pivoted = weather_df.select(['timestamp', 'grid_point', orig_col]).pivot(
+            values=orig_col,
+            index='timestamp',
+            on='grid_point',
+            aggregate_function='first'
+        )
+
+        # Rename columns to <short_name>_<grid_point>
+        rename_map = {}
+        for col in pivoted.columns:
+            if col != 'timestamp':
+                rename_map[col] = f'{short_name}_{col}'
+
+        pivoted = pivoted.rename(rename_map)
+
+        # Join to features
+        if features is None:
+            features = pivoted
+        else:
+            features = features.join(pivoted, on='timestamp', how='left', coalesce=True)
+
+    print(f"  [OK] {len(features.columns) - 1} grid-level features")
+    return features
+
+
+def engineer_temporal_lags(features: pl.DataFrame) -> pl.DataFrame:
+    """Add temporal lags for key weather variables.
+
+    Lags: 1h, 6h, 12h, 24h for:
+    - Average temperature (1 lag feature)
+    - Average wind speed (1 lag feature)
+    - Average solar radiation (1 lag feature)
+
+    Total: ~12 lag features (3 vars × 4 lags)
+    """
+    print("\n[2/3] Engineering temporal lags (1h, 6h, 12h, 24h)...")
+
+    # Calculate system-wide averages for lagging
+    # Temperature average (across all temp_ columns)
+    temp_cols = [c for c in features.columns if c.startswith('temp_')]
+    features = features.with_columns([
+        pl.concat_list([pl.col(c) for c in temp_cols]).list.mean().alias('temp_avg')
+    ])
+
+    # Wind speed average (100m - for wind generation)
+    wind_cols = [c for c in features.columns if c.startswith('wind100m_')]
+    features = features.with_columns([
+        pl.concat_list([pl.col(c) for c in wind_cols]).list.mean().alias('wind_avg')
+    ])
+
+    # Solar radiation average
+    solar_cols = [c for c in features.columns if c.startswith('solar_')]
+    features = features.with_columns([
+        pl.concat_list([pl.col(c) for c in solar_cols]).list.mean().alias('solar_avg')
+    ])
+
+    # Add lags
+    lag_vars = ['temp_avg', 'wind_avg', 'solar_avg']
+    lag_hours = [1, 6, 12, 24]
+
+    for var in lag_vars:
+        for lag_h in lag_hours:
+            features = features.with_columns([
+                pl.col(var).shift(lag_h).alias(f'{var}_lag{lag_h}h')
+            ])
+
+    # Drop intermediate averages (keep only lagged versions)
+    features = features.drop(['temp_avg', 'wind_avg', 'solar_avg'])
+
+    lag_features = len(lag_vars) * len(lag_hours)
+    print(f"  [OK] {lag_features} temporal lag features")
+    return features
+
+
+def engineer_derived_features(features: pl.DataFrame) -> pl.DataFrame:
+    """Engineer derived weather features (6 features).
+
+    Simple features without requiring calibration data:
+    - Rate of change (hour-over-hour deltas): wind, solar, temperature
+    - Weather stability (rolling std): wind, solar, temperature
+    """
+    print("\n[3/3] Engineering derived features (rate-of-change + stability)...")
+
+    # Calculate system averages for rate-of-change and stability
+    wind_cols = [c for c in features.columns if c.startswith('wind100m_')]
+    solar_cols = [c for c in features.columns if c.startswith('solar_')]
+    temp_cols = [c for c in features.columns if c.startswith('temp_')]
+
+    features = features.with_columns([
+        pl.concat_list([pl.col(c) for c in wind_cols]).list.mean().alias('wind_system_avg'),
+        pl.concat_list([pl.col(c) for c in solar_cols]).list.mean().alias('solar_system_avg'),
+        pl.concat_list([pl.col(c) for c in temp_cols]).list.mean().alias('temp_system_avg')
+    ])
+
+    # Rate of change (hour-over-hour deltas)
+    # Captures sudden spikes/drops that correlate with grid constraints
+    features = features.with_columns([
+        pl.col('wind_system_avg').diff().alias('wind_rate_change'),
+        pl.col('solar_system_avg').diff().alias('solar_rate_change'),
+        pl.col('temp_system_avg').diff().alias('temp_rate_change')
+    ])
+
+    # Weather stability: 6-hour rolling std
+    # Detects volatility periods (useful for forecasting uncertainty)
+    features = features.with_columns([
+        pl.col('wind_system_avg').rolling_std(window_size=6).alias('wind_stability_6h'),
+        pl.col('solar_system_avg').rolling_std(window_size=6).alias('solar_stability_6h'),
+        pl.col('temp_system_avg').rolling_std(window_size=6).alias('temp_stability_6h')
+    ])
+
+    # Drop intermediate columns
+    features = features.drop(['wind_system_avg', 'solar_system_avg', 'temp_system_avg'])
+
+    # Count derived features
+    derived_cols = ['wind_rate_change', 'solar_rate_change', 'temp_rate_change',
+                    'wind_stability_6h', 'solar_stability_6h', 'temp_stability_6h']
+
+    print(f"  [OK] {len(derived_cols)} derived features")
+    return features
+
+
+def engineer_weather_features(
+    weather_path: Path,
+    output_dir: Path
+) -> pl.DataFrame:
+    """Main feature engineering pipeline for weather data.
+
+    Args:
+        weather_path: Path to raw weather data (weather_24month.parquet)
+        output_dir: Directory to save engineered features
+
+    Returns:
+        DataFrame with ~435 weather features
+    """
+    print("=" * 80)
+    print("WEATHER FEATURE ENGINEERING")
+    print("=" * 80)
+    print()
+    print(f"Input: {weather_path}")
+    print(f"Output: {output_dir}")
+    print()
+
+    # Load raw weather data
+    print("Loading weather data...")
+    weather_df = pl.read_parquet(weather_path)
+    print(f"  [OK] {weather_df.shape[0]:,} rows × {weather_df.shape[1]} columns")
+    print(f"  Date range: {weather_df['timestamp'].min()} to {weather_df['timestamp'].max()}")
+    print()
+
+    # 1. Grid-level features (51 × 7 = 357 features)
+    all_features = engineer_grid_level_features(weather_df)
+
+    # 2. Temporal lags (~12 features)
+    all_features = engineer_temporal_lags(all_features)
+
+    # 3. Derived features (6 features: rate-of-change + stability)
+    all_features = engineer_derived_features(all_features)
+
+    # Sort by timestamp
+    all_features = all_features.sort('timestamp')
+
+    # Final validation
+    print("\n" + "=" * 80)
+    print("FEATURE ENGINEERING COMPLETE")
+    print("=" * 80)
+    print(f"Total features: {all_features.shape[1] - 1} (excluding timestamp)")
+    print(f"Total rows: {len(all_features):,}")
+
+    # Check completeness
+    null_count_total = all_features.null_count().sum_horizontal()[0]
+    completeness = (1 - null_count_total / (all_features.shape[0] * all_features.shape[1])) * 100
+    print(f"Completeness: {completeness:.2f}%")
+    print()
+
+    # Save features
+    output_path = output_dir / 'features_weather_24month.parquet'
+    all_features.write_parquet(output_path)
+
+    file_size_mb = output_path.stat().st_size / (1024 ** 2)
+    print(f"Features saved: {output_path}")
+    print(f"File size: {file_size_mb:.2f} MB")
+    print("=" * 80)
+    print()
+
+    return all_features
+
+
+def main():
+    """Main execution."""
+    # Paths
+    base_dir = Path.cwd()
+    raw_dir = base_dir / 'data' / 'raw'
+    processed_dir = base_dir / 'data' / 'processed'
+
+    weather_path = raw_dir / 'weather_24month.parquet'
+
+    # Verify file exists
+    if not weather_path.exists():
+        raise FileNotFoundError(f"Weather data not found: {weather_path}")
+
+    # Engineer features
+    features = engineer_weather_features(weather_path, processed_dir)
+
+    print("SUCCESS: Weather features engineered and saved to data/processed/")
+
+
+if __name__ == '__main__':
+    main()