273 lines
16 KiB
Markdown
273 lines
16 KiB
Markdown
# GeoData Pipeline Architecture
|
||
|
||
Visual documentation showing how the GeoData pipeline transforms raw geospatial data into Unity-ready assets.
|
||
|
||
---
|
||
|
||
## Complete Pipeline Architecture
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||
│ INPUT LAYER │
|
||
├──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┤
|
||
│ DGM1 │ DOM1 │ DOP20 │ CityGML │ LPG │ LPO │ BDOM20 │
|
||
│ TIFF │ TIFF │ JP2 │ GML │ XYZ │ XYZ │ LAZ │
|
||
│ terrain │ surface │ ortho │ buildings│ ground │ object │ RGB │
|
||
│ 10m │ 10m │ 20cm │ LoD2 │ points │ points │ points │
|
||
├──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┤
|
||
│ raw/ directory (inputs) │
|
||
└────────────────────────────────────────┬────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||
│ PROCESSING LAYER │
|
||
├───────────────────┬────────────────────┬────────────────────────────────────┤
|
||
│ │ │ │
|
||
│ HEIGHTMAPS │ BUILDINGS │ VEGETATION & OBJECTS │
|
||
│ │ │ │
|
||
│ ┌─────────────┐ │ ┌──────────────┐ │ ┌──────────────┐ ┌─────────────┐ │
|
||
│ │ DGM1 → VRT │ │ │GML→CityJSON │ │ │ DOM1-DGM1 │ │nDSM detect │ │
|
||
│ │ ↓ │ │ │ ↓ │ │ │ = CHM │ │ ↓ │ │
|
||
│ │ Warp 1025² │ │ │ Triangulate │ │ │ ↓ │ │ Classify │ │
|
||
│ │ ↓ │ │ │ ↓ │ │ │Local maxima │ │(lamp/bench │ │
|
||
│ │ Scale 16bit│ │ │ Decimate │ │ │ ↓ │ │ /sign) │ │
|
||
│ │ ↓ │ │ │ ↓ │ │ │ Tree peaks │ │ │ │
|
||
│ │ [LPG valid]│ │ │ Ground-snap │ │ │ [LPO refine] │ │ [LPO ref] │ │
|
||
│ └─────────────┘ │ │ [RANSAC roof]│ │ │ [DOP20 RGB] │ └─────────────┘ │
|
||
│ │ └──────────────┘ │ └──────────────┘ │
|
||
├───────────────────┴────────────────────┴────────────────────────────────────┤
|
||
│ work/ directory (temp) │
|
||
└────────────────────────────────────────┬────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||
│ OUTPUT LAYER │
|
||
├─────────────────┬─────────────────┬─────────────────┬───────────────────────┤
|
||
│ height_png16/ │ ortho_jpg/ │ buildings_tiles/│ trees/ + furniture/ │
|
||
│ *.png (16-bit) │ *.jpg (2048²) │ *.glb │ *.csv │
|
||
│ *.pgw │ *.jgw │ │ trees_tiles/*.glb │
|
||
├─────────────────┴─────────────────┴─────────────────┴───────────────────────┤
|
||
│ tile_index.csv (manifest with bounds + elevation range) │
|
||
├─────────────────────────────────────────────────────────────────────────────┤
|
||
│ export_unity/ directory │
|
||
└────────────────────────────────────────┬────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||
│ UNITY IMPORT │
|
||
│ GeoTileImporter.cs reads tile_index.csv, creates: │
|
||
│ • Terrain tiles (heightmaps + ortho textures) │
|
||
│ • Building GLB instances │
|
||
│ • Tree proxies with canopy colors │
|
||
│ • Street furniture placeholders │
|
||
└─────────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Pipeline Dependency Graph
|
||
|
||
```
|
||
┌─────────────────┐
|
||
│ --setup │
|
||
│ materialize │
|
||
│ archives │
|
||
└────────┬────────┘
|
||
│
|
||
▼
|
||
┌─────────────────┐
|
||
│ HEIGHTMAP │◄──── MUST RUN FIRST
|
||
│ export_height │ Creates tile_index.csv
|
||
│ maps() │ Creates work/dgm.vrt
|
||
└────────┬────────┘
|
||
│
|
||
┌──────────────┼──────────────┐
|
||
│ │ │
|
||
▼ ▼ ▼
|
||
┌────────────┐ ┌────────────┐ ┌────────────┐
|
||
│ TEXTURES │ │ BUILDINGS │ │ TREES │
|
||
│ ortho │ │ citygml │ │ dom1-dgm1 │
|
||
│ photos │ │ → glb │ │ → csv+glb │
|
||
└────────────┘ └────────────┘ └─────┬──────┘
|
||
│
|
||
▼
|
||
┌───────────────────┐
|
||
│ STREET FURNITURE │
|
||
│ ndsm detection │
|
||
│ → csv │
|
||
└─────────┬─────────┘
|
||
│
|
||
┌──────────────────────────┼──────────────┐
|
||
│ │ │
|
||
▼ ▼ ▼
|
||
┌────────────────┐ ┌────────────────┐ ┌─────────────────┐
|
||
│ HEIGHTMAP- │ │ BUILDINGS- │ │ TREES- │
|
||
│ ENHANCED │ │ ENHANCED │ │ ENHANCED │
|
||
│ +LPG valid │ │ +RANSAC roofs │ │ +LPO heights │
|
||
│ +quality map │ │ +LPO heights │ │ +DOP20 colors │
|
||
└────────────────┘ └────────────────┘ │ -furniture mask │
|
||
└─────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Data Flow Per Export Type
|
||
|
||
### 1. Heightmap (Standard)
|
||
```
|
||
raw/dgm1/*.tif ──► work/dgm.vrt ──► Per-tile warp (1025×1025)
|
||
│
|
||
▼
|
||
Scale to UInt16
|
||
│
|
||
▼
|
||
export_unity/height_png16/{tile}.png
|
||
export_unity/tile_index.csv (manifest)
|
||
```
|
||
|
||
### 2. Orthophoto (Textures)
|
||
```
|
||
raw/dop20/jp2/*.jp2 ──► work/dop.vrt ──► Per-tile extract
|
||
│ │
|
||
│ tile_index.csv ────────────►│ (read bounds)
|
||
│ │
|
||
▼ ▼
|
||
export_unity/ortho_jpg/{tile}.jpg (2048×2048)
|
||
```
|
||
|
||
### 3. Buildings (Standard)
|
||
```
|
||
raw/citygml/lod2/*.gml ──► citygml-tools ──► work/cityjson/*.json
|
||
│
|
||
▼
|
||
cjio triangulate
|
||
│
|
||
▼
|
||
work/cityjson_local/*.tri.json
|
||
│
|
||
work/dgm.vrt ──────────────────────────►│ (ground-snap)
|
||
│
|
||
▼
|
||
export_unity/buildings_tiles/{tile}.glb
|
||
```
|
||
|
||
### 4. Trees (Standard)
|
||
```
|
||
raw/dom1/*.tif ─────────► CHM = DOM1 - DGM1
|
||
raw/dgm1/*.tif ─────────► │
|
||
▼
|
||
Local maxima detection
|
||
│
|
||
raw/citygml/lod2/*.gml ─────────►│ (building mask)
|
||
│
|
||
▼
|
||
export_unity/trees/{tile}.csv
|
||
export_unity/trees_tiles/{tile}_{chunk}.glb
|
||
```
|
||
|
||
### 5. Street Furniture
|
||
```
|
||
raw/dom1/*.tif ─────────► nDSM = DOM1 - DGM1
|
||
raw/dgm1/*.tif ─────────► │
|
||
▼
|
||
Blob detection (0.3-8m)
|
||
│
|
||
raw/lpo/*.xyz ──────────────────►│ (optional LPO refinement)
|
||
│
|
||
▼
|
||
Classify: lamp/bench/sign/bollard
|
||
│
|
||
▼
|
||
export_unity/street_furniture/{tile}.csv
|
||
```
|
||
|
||
### 6. Enhanced Pipelines
|
||
```
|
||
HEIGHTMAP-ENHANCED:
|
||
heightmap output + raw/lpg/*.xyz ──► LPG validation ──► quality map
|
||
|
||
BUILDINGS-ENHANCED:
|
||
buildings output + raw/bdom20rgbi/*.laz ──► RANSAC roof extraction
|
||
+ raw/lpo/*.xyz ──────────► height refinement
|
||
|
||
TREES-ENHANCED:
|
||
trees output + raw/lpo/*.xyz ──────► height refinement
|
||
+ raw/dop20/*.jp2 ──────► canopy RGB sampling
|
||
+ street_furniture/*.csv ► exclusion mask
|
||
```
|
||
|
||
---
|
||
|
||
## File Format Reference
|
||
|
||
| Output | Format | Resolution | Purpose |
|
||
|--------|--------|------------|---------|
|
||
| `height_png16/*.png` | 16-bit PNG | 1025×1025 | Unity terrain heightmap |
|
||
| `ortho_jpg/*.jpg` | JPEG Q90 | 2048×2048 | Terrain texture |
|
||
| `buildings_tiles/*.glb` | glTF Binary | 200-350k tris | 3D building models |
|
||
| `trees/*.csv` | CSV | - | Tree positions (x,y,z,h,r) |
|
||
| `trees_tiles/*.glb` | glTF Binary | - | Proxy geometry chunks |
|
||
| `street_furniture/*.csv` | CSV | - | Furniture positions |
|
||
| `tile_index.csv` | CSV | - | Manifest (bounds, min/max) |
|
||
|
||
---
|
||
|
||
## CLI Commands
|
||
|
||
```bash
|
||
# Setup (extract archives, create directories)
|
||
uv run python geodata_to_unity.py --setup --build-from-archive
|
||
|
||
# Standard exports
|
||
uv run python geodata_to_unity.py --export heightmap # First!
|
||
uv run python geodata_to_unity.py --export textures
|
||
uv run python geodata_to_unity.py --export buildings
|
||
uv run python geodata_to_unity.py --export trees
|
||
uv run python geodata_to_unity.py --export all # All standard
|
||
|
||
# Enhanced exports (requires point cloud data)
|
||
uv run python geodata_to_unity.py --export heightmap-enhanced
|
||
uv run python geodata_to_unity.py --export buildings-enhanced
|
||
uv run python geodata_to_unity.py --export trees-enhanced
|
||
uv run python geodata_to_unity.py --export street-furniture
|
||
uv run python geodata_to_unity.py --export all-enhanced # All enhanced
|
||
```
|
||
|
||
---
|
||
|
||
## Config Classes
|
||
|
||
```
|
||
Config
|
||
├── raw: RawConfig # Input directories
|
||
├── archives: ArchiveConfig # Archive staging
|
||
├── work: WorkConfig # Temp/intermediate
|
||
├── export: ExportConfig # Output directories
|
||
├── heightmap: HeightmapConfig # out_res=1025, tile_size_m=1000
|
||
├── ortho: OrthoConfig # out_res=2048, quality=90
|
||
├── buildings: BuildingConfig # triangle budget 200-350k
|
||
├── trees: TreeConfig # max_trees=5000, chunk_grid=4x4
|
||
├── pointcloud: PointCloudConfig # LPG/LPO/BDOM directories
|
||
├── heightmap_enhanced: EnhancedHeightmapConfig
|
||
├── buildings_enhanced: EnhancedBuildingConfig
|
||
├── trees_enhanced: EnhancedTreeConfig
|
||
└── street_furniture: StreetFurnitureConfig
|
||
```
|
||
|
||
---
|
||
|
||
## Key Files
|
||
|
||
| Module | Purpose |
|
||
|--------|---------|
|
||
| `geodata_to_unity.py` | CLI entry point |
|
||
| `geodata_pipeline/config.py` | Configuration dataclasses |
|
||
| `geodata_pipeline/heightmaps.py` | DGM1 → PNG16 |
|
||
| `geodata_pipeline/orthophotos.py` | DOP20 → JPEG |
|
||
| `geodata_pipeline/buildings.py` | CityGML → GLB |
|
||
| `geodata_pipeline/trees.py` | CHM → CSV + GLB |
|
||
| `geodata_pipeline/street_furniture.py` | nDSM → CSV |
|
||
| `geodata_pipeline/pointcloud.py` | XYZ/LAZ utilities |
|
||
| `geodata_pipeline/*_enhanced.py` | Enhanced pipelines |
|
||
| `GeoTileImporter.cs` | Unity importer |
|