# GeoData Pipeline Architecture

Visual documentation showing how the GeoData pipeline transforms raw geospatial data into Unity-ready assets.

---

## Complete Pipeline Architecture

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                              INPUT LAYER                                     │
├──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┤
│  DGM1    │  DOM1    │  DOP20   │ CityGML  │   LPG    │   LPO    │ BDOM20   │
│  TIFF    │  TIFF    │   JP2    │   GML    │   XYZ    │   XYZ    │   LAZ    │
│ terrain  │ surface  │  ortho   │ buildings│  ground  │  object  │  RGB     │
│  10m     │  10m     │  20cm    │  LoD2    │  points  │  points  │  points  │
├──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┤
│                         raw/ directory (inputs)                             │
└────────────────────────────────────────┬────────────────────────────────────┘
                                         │
                                         ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           PROCESSING LAYER                                   │
├───────────────────┬────────────────────┬────────────────────────────────────┤
│                   │                    │                                    │
│   HEIGHTMAPS      │    BUILDINGS       │      VEGETATION & OBJECTS          │
│                   │                    │                                    │
│  ┌─────────────┐  │  ┌──────────────┐  │  ┌──────────────┐ ┌─────────────┐  │
│  │ DGM1 → VRT  │  │  │GML→CityJSON  │  │  │ DOM1-DGM1    │ │nDSM detect  │  │
│  │   ↓        │  │  │    ↓         │  │  │  = CHM       │ │   ↓        │  │
│  │ Warp 1025² │  │  │ Triangulate  │  │  │    ↓         │ │ Classify   │  │
│  │   ↓        │  │  │    ↓         │  │  │Local maxima  │ │(lamp/bench │  │
│  │ Scale 16bit│  │  │ Decimate     │  │  │    ↓         │ │ /sign)     │  │
│  │   ↓        │  │  │    ↓         │  │  │ Tree peaks   │ │            │  │
│  │ [LPG valid]│  │  │ Ground-snap  │  │  │ [LPO refine] │ │ [LPO ref]  │  │
│  └─────────────┘  │  │ [RANSAC roof]│  │  │ [DOP20 RGB]  │ └─────────────┘  │
│                   │  └──────────────┘  │  └──────────────┘                  │
├───────────────────┴────────────────────┴────────────────────────────────────┤
│                         work/ directory (temp)                              │
└────────────────────────────────────────┬────────────────────────────────────┘
                                         │
                                         ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                            OUTPUT LAYER                                      │
├─────────────────┬─────────────────┬─────────────────┬───────────────────────┤
│  height_png16/  │  ortho_jpg/     │ buildings_tiles/│  trees/ + furniture/  │
│  *.png (16-bit) │  *.jpg (2048²)  │     *.glb      │     *.csv             │
│  *.pgw          │  *.jgw          │                 │  trees_tiles/*.glb    │
├─────────────────┴─────────────────┴─────────────────┴───────────────────────┤
│  tile_index.csv (bounds + global/tile min/max + tile_key)                    │
├─────────────────────────────────────────────────────────────────────────────┤
│                      export_unity/ directory                                │
└────────────────────────────────────────┬────────────────────────────────────┘
                                         │
                                         ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           UNITY IMPORT                                       │
│  GeoTileImporter.cs reads tile_index.csv, creates:                          │
│  • Terrain tiles (heightmaps + ortho textures)                              │
│  • Building GLB instances                                                   │
│  • Tree proxies with canopy colors                                          │
│  • Street furniture placeholders                                            │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## Pipeline Dependency Graph

```
                    ┌─────────────────┐
                    │   --setup       │
                    │ materialize     │
                    │  archives       │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │   HEIGHTMAP     │◄──── MUST RUN FIRST
                    │  export_height  │      Creates tile_index.csv (+tile_key)
                    │     maps()      │      Creates work/dgm.vrt
                    └────────┬────────┘
                             │
              ┌──────────────┼──────────────┐
              │              │              │
              ▼              ▼              ▼
     ┌────────────┐  ┌────────────┐  ┌────────────┐
     │  TEXTURES  │  │ BUILDINGS  │  │   TREES    │
     │  ortho     │  │  citygml   │  │  dom1-dgm1 │
     │  photos    │  │   → glb    │  │  → csv+glb │
     └────────────┘  └────────────┘  └─────┬──────┘
                                           │
                                           ▼
                               ┌───────────────────┐
                               │ STREET FURNITURE  │
                               │  ndsm detection   │
                               │     → csv         │
                               └─────────┬─────────┘
                                         │
              ┌──────────────────────────┼──────────────┐
              │                          │              │
              ▼                          ▼              ▼
     ┌────────────────┐      ┌────────────────┐  ┌─────────────────┐
     │ HEIGHTMAP-     │      │ BUILDINGS-     │  │ TREES-          │
     │ ENHANCED       │      │ ENHANCED       │  │ ENHANCED        │
     │ +LPG valid     │      │ +RANSAC roofs  │  │ +LPO heights    │
     │ +quality map   │      │ +LPO heights   │  │ +DOP20 colors   │
     └────────────────┘      └────────────────┘  │ -furniture mask │
                                                 └─────────────────┘
```

---

## Data Flow Per Export Type

### 1. Heightmap (Standard)
```
raw/dgm1/*.tif ──► work/dgm.vrt ──► Per-tile warp (1025×1025)
                                          │
                                          ▼
                                    Scale to UInt16 (per-tile min/max by default)
                                          │
                                          ▼
                        export_unity/height_png16/{tile}.png
                        export_unity/tile_index.csv (manifest)
```

### 2. Orthophoto (Textures)
```
raw/dop20/jp2/*.jp2 ──► work/dop.vrt ──► Per-tile extract
         │                                     │
         │         tile_index.csv ────────────►│ (read bounds)
         │                                     │
         ▼                                     ▼
                               export_unity/ortho_jpg/{tile}.jpg (2048×2048)
```

### 3. Buildings (Standard)
```
raw/citygml/lod2/*.gml ──► citygml-tools ──► work/cityjson/*.json
                                                   │
                                                   ▼
                                             cjio triangulate
                                                   │
                                                   ▼
                                   work/cityjson_local/*.tri.json
                                                   │
         work/dgm.vrt ──────────────────────────►│ (ground-snap)
                                                   │
                                                   ▼
                           export_unity/buildings_tiles/{tile}.glb
```

### 4. Trees (Standard)
```
raw/dom1/*.tif ─────────► CHM = DOM1 - DGM1
raw/dgm1/*.tif ─────────►        │
                                 ▼
                         Local maxima detection
                                 │
raw/citygml/lod2/*.gml ─────────►│ (building mask)
                                 │
                                 ▼
                    export_unity/trees/{tile}.csv
                    export_unity/trees_tiles/{tile}_{chunk}.glb
```

### 5. Street Furniture
```
raw/dom1/*.tif ─────────► nDSM = DOM1 - DGM1
raw/dgm1/*.tif ─────────►        │
                                 ▼
                       Blob detection (0.3-8m)
                                 │
raw/lpo/*.xyz ──────────────────►│ (optional LPO refinement)
                                 │
                                 ▼
                      Classify: lamp/bench/sign/bollard
                                 │
                                 ▼
                 export_unity/street_furniture/{tile}.csv
```

### 6. Enhanced Pipelines
```
HEIGHTMAP-ENHANCED:
  heightmap output + raw/lpg/*.xyz ──► LPG validation ──► quality map

BUILDINGS-ENHANCED:
  buildings output + raw/bdom20rgbi/*.laz ──► RANSAC roof extraction
                   + raw/lpo/*.xyz ──────────► height refinement

TREES-ENHANCED:
  trees output + raw/lpo/*.xyz ──────► height refinement
              + raw/dop20/*.jp2 ──────► canopy RGB sampling
              + street_furniture/*.csv ► exclusion mask
```

---

### 7. River Erosion (Optional)
```
export_unity/height_png16/*.png ──► depth raster from HydroRIVERS (buffered by order)
export_unity/tile_index.csv ───────► bounds + scaling
                                  │
                                  ▼
Depth burn = MAX depth (emulated by sorting burns ascending + REPLACE)
                                  │
                                  ▼
export_unity/height_png16_river/{tile}.png
export_unity/tile_index_river_{vr|server}.csv
```
Notes:
- Uses `raw/hydrorivers/.../HydroRIVERS_v10_eu.shp` and runs after heightmap export.
- Rasterize merge supports only REPLACE/ADD; max-depth is achieved by ordered burns.

---

## File Format Reference

| Output | Format | Resolution | Purpose |
|--------|--------|------------|---------|
| `height_png16/*.png` | 16-bit PNG | 1025×1025 | Unity terrain heightmap |
| `ortho_jpg/*.jpg` | JPEG Q90 | 2048×2048 | Terrain texture |
| `buildings_tiles/*.glb` | glTF Binary | 200-350k tris | 3D building models |
| `trees/*.csv` | CSV | - | Tree positions (x,y,z,h,r) |
| `trees_tiles/*.glb` | glTF Binary | - | Proxy geometry chunks |
| `street_furniture/*.csv` | CSV | - | Furniture positions |
| `tile_index.csv` | CSV | - | Manifest (bounds, global/tile min/max, tile_key) |

---

## Tile Key Formula

`tile_key` is appended to `tile_index.csv` during heightmap export:

```
tile_key = f"{floor((xmin + overlap_x) / tile_size_x)}_{floor((ymin + overlap_y) / tile_size_y)}"
```

Defaults (in `[tile_key]`): `tile_size_x=1000.0`, `tile_size_y=1000.0`, `overlap_x=0.5`, `overlap_y=0.5`, `enabled=true`.

---

## CLI Commands

```bash
# Setup (extract archives, create directories)
uv run python geodata_to_unity.py --setup --build-from-archive

# Standard exports
uv run python geodata_to_unity.py --export heightmap    # First!
uv run python geodata_to_unity.py --export textures
uv run python geodata_to_unity.py --export buildings
uv run python geodata_to_unity.py --export trees
uv run python geodata_to_unity.py --export all          # All standard

# Enhanced exports (requires point cloud data)
uv run python geodata_to_unity.py --export heightmap-enhanced
uv run python geodata_to_unity.py --export buildings-enhanced
uv run python geodata_to_unity.py --export trees-enhanced
uv run python geodata_to_unity.py --export street-furniture
uv run python geodata_to_unity.py --export all-enhanced  # All enhanced
```

---

## Config Classes

```
Config
├── raw: RawConfig           # Input directories
├── archives: ArchiveConfig  # Archive staging
├── work: WorkConfig         # Temp/intermediate
├── export: ExportConfig     # Output directories
├── heightmap: HeightmapConfig       # out_res=1025, tile_size_m=1000, use_tile_minmax=true
├── ortho: OrthoConfig               # out_res=2048, quality=90
├── tile_key: TileKeyConfig          # tile_size_x/y=1000, overlap_x/y=0.5
├── buildings: BuildingConfig        # triangle budget 200-350k
├── trees: TreeConfig                # max_trees=5000, chunk_grid=4x4
├── pointcloud: PointCloudConfig     # LPG/LPO/BDOM directories
├── heightmap_enhanced: EnhancedHeightmapConfig
├── buildings_enhanced: EnhancedBuildingConfig
├── trees_enhanced: EnhancedTreeConfig
└── street_furniture: StreetFurnitureConfig
```

---

## Key Files

| Module | Purpose |
|--------|---------|
| `geodata_to_unity.py` | CLI entry point |
| `geodata_pipeline/config.py` | Configuration dataclasses |
| `geodata_pipeline/heightmaps.py` | DGM1 → PNG16 |
| `geodata_pipeline/orthophotos.py` | DOP20 → JPEG |
| `geodata_pipeline/buildings.py` | CityGML → GLB |
| `geodata_pipeline/trees.py` | CHM → CSV + GLB |
| `geodata_pipeline/street_furniture.py` | nDSM → CSV |
| `geodata_pipeline/pointcloud.py` | XYZ/LAZ utilities |
| `geodata_pipeline/*_enhanced.py` | Enhanced pipelines |
| `GeoTileImporter.cs` | Unity importer |
