# GeoData Pipeline Architecture Visual documentation showing how the GeoData pipeline transforms raw geospatial data into Unity-ready assets. --- ## Complete Pipeline Architecture ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ INPUT LAYER │ ├──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┤ │ DGM1 │ DOM1 │ DOP20 │ CityGML │ LPG │ LPO │ BDOM20 │ │ TIFF │ TIFF │ JP2 │ GML │ XYZ │ XYZ │ LAZ │ │ terrain │ surface │ ortho │ buildings│ ground │ object │ RGB │ │ 10m │ 10m │ 20cm │ LoD2 │ points │ points │ points │ ├──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┤ │ raw/ directory (inputs) │ └────────────────────────────────────────┬────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ PROCESSING LAYER │ ├───────────────────┬────────────────────┬────────────────────────────────────┤ │ │ │ │ │ HEIGHTMAPS │ BUILDINGS │ VEGETATION & OBJECTS │ │ │ │ │ │ ┌─────────────┐ │ ┌──────────────┐ │ ┌──────────────┐ ┌─────────────┐ │ │ │ DGM1 → VRT │ │ │GML→CityJSON │ │ │ DOM1-DGM1 │ │nDSM detect │ │ │ │ ↓ │ │ │ ↓ │ │ │ = CHM │ │ ↓ │ │ │ │ Warp 1025² │ │ │ Triangulate │ │ │ ↓ │ │ Classify │ │ │ │ ↓ │ │ │ ↓ │ │ │Local maxima │ │(lamp/bench │ │ │ │ Scale 16bit│ │ │ Decimate │ │ │ ↓ │ │ /sign) │ │ │ │ ↓ │ │ │ ↓ │ │ │ Tree peaks │ │ │ │ │ │ [LPG valid]│ │ │ Ground-snap │ │ │ [LPO refine] │ │ [LPO ref] │ │ │ └─────────────┘ │ │ [RANSAC roof]│ │ │ [DOP20 RGB] │ └─────────────┘ │ │ │ └──────────────┘ │ └──────────────┘ │ ├───────────────────┴────────────────────┴────────────────────────────────────┤ │ work/ directory (temp) │ └────────────────────────────────────────┬────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ OUTPUT LAYER │ ├─────────────────┬─────────────────┬─────────────────┬───────────────────────┤ │ height_png16/ │ ortho_jpg/ │ buildings_tiles/│ trees/ + furniture/ │ │ *.png (16-bit) │ *.jpg (2048²) │ *.glb │ *.csv │ │ *.pgw │ *.jgw │ │ trees_tiles/*.glb │ ├─────────────────┴─────────────────┴─────────────────┴───────────────────────┤ │ tile_index.csv (manifest with bounds + elevation range) │ ├─────────────────────────────────────────────────────────────────────────────┤ │ export_unity/ directory │ └────────────────────────────────────────┬────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ UNITY IMPORT │ │ GeoTileImporter.cs reads tile_index.csv, creates: │ │ • Terrain tiles (heightmaps + ortho textures) │ │ • Building GLB instances │ │ • Tree proxies with canopy colors │ │ • Street furniture placeholders │ └─────────────────────────────────────────────────────────────────────────────┘ ``` --- ## Pipeline Dependency Graph ``` ┌─────────────────┐ │ --setup │ │ materialize │ │ archives │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ HEIGHTMAP │◄──── MUST RUN FIRST │ export_height │ Creates tile_index.csv │ maps() │ Creates work/dgm.vrt └────────┬────────┘ │ ┌──────────────┼──────────────┐ │ │ │ ▼ ▼ ▼ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ TEXTURES │ │ BUILDINGS │ │ TREES │ │ ortho │ │ citygml │ │ dom1-dgm1 │ │ photos │ │ → glb │ │ → csv+glb │ └────────────┘ └────────────┘ └─────┬──────┘ │ ▼ ┌───────────────────┐ │ STREET FURNITURE │ │ ndsm detection │ │ → csv │ └─────────┬─────────┘ │ ┌──────────────────────────┼──────────────┐ │ │ │ ▼ ▼ ▼ ┌────────────────┐ ┌────────────────┐ ┌─────────────────┐ │ HEIGHTMAP- │ │ BUILDINGS- │ │ TREES- │ │ ENHANCED │ │ ENHANCED │ │ ENHANCED │ │ +LPG valid │ │ +RANSAC roofs │ │ +LPO heights │ │ +quality map │ │ +LPO heights │ │ +DOP20 colors │ └────────────────┘ └────────────────┘ │ -furniture mask │ └─────────────────┘ ``` --- ## Data Flow Per Export Type ### 1. Heightmap (Standard) ``` raw/dgm1/*.tif ──► work/dgm.vrt ──► Per-tile warp (1025×1025) │ ▼ Scale to UInt16 │ ▼ export_unity/height_png16/{tile}.png export_unity/tile_index.csv (manifest) ``` ### 2. Orthophoto (Textures) ``` raw/dop20/jp2/*.jp2 ──► work/dop.vrt ──► Per-tile extract │ │ │ tile_index.csv ────────────►│ (read bounds) │ │ ▼ ▼ export_unity/ortho_jpg/{tile}.jpg (2048×2048) ``` ### 3. Buildings (Standard) ``` raw/citygml/lod2/*.gml ──► citygml-tools ──► work/cityjson/*.json │ ▼ cjio triangulate │ ▼ work/cityjson_local/*.tri.json │ work/dgm.vrt ──────────────────────────►│ (ground-snap) │ ▼ export_unity/buildings_tiles/{tile}.glb ``` ### 4. Trees (Standard) ``` raw/dom1/*.tif ─────────► CHM = DOM1 - DGM1 raw/dgm1/*.tif ─────────► │ ▼ Local maxima detection │ raw/citygml/lod2/*.gml ─────────►│ (building mask) │ ▼ export_unity/trees/{tile}.csv export_unity/trees_tiles/{tile}_{chunk}.glb ``` ### 5. Street Furniture ``` raw/dom1/*.tif ─────────► nDSM = DOM1 - DGM1 raw/dgm1/*.tif ─────────► │ ▼ Blob detection (0.3-8m) │ raw/lpo/*.xyz ──────────────────►│ (optional LPO refinement) │ ▼ Classify: lamp/bench/sign/bollard │ ▼ export_unity/street_furniture/{tile}.csv ``` ### 6. Enhanced Pipelines ``` HEIGHTMAP-ENHANCED: heightmap output + raw/lpg/*.xyz ──► LPG validation ──► quality map BUILDINGS-ENHANCED: buildings output + raw/bdom20rgbi/*.laz ──► RANSAC roof extraction + raw/lpo/*.xyz ──────────► height refinement TREES-ENHANCED: trees output + raw/lpo/*.xyz ──────► height refinement + raw/dop20/*.jp2 ──────► canopy RGB sampling + street_furniture/*.csv ► exclusion mask ``` --- ## File Format Reference | Output | Format | Resolution | Purpose | |--------|--------|------------|---------| | `height_png16/*.png` | 16-bit PNG | 1025×1025 | Unity terrain heightmap | | `ortho_jpg/*.jpg` | JPEG Q90 | 2048×2048 | Terrain texture | | `buildings_tiles/*.glb` | glTF Binary | 200-350k tris | 3D building models | | `trees/*.csv` | CSV | - | Tree positions (x,y,z,h,r) | | `trees_tiles/*.glb` | glTF Binary | - | Proxy geometry chunks | | `street_furniture/*.csv` | CSV | - | Furniture positions | | `tile_index.csv` | CSV | - | Manifest (bounds, min/max) | --- ## CLI Commands ```bash # Setup (extract archives, create directories) uv run python geodata_to_unity.py --setup --build-from-archive # Standard exports uv run python geodata_to_unity.py --export heightmap # First! uv run python geodata_to_unity.py --export textures uv run python geodata_to_unity.py --export buildings uv run python geodata_to_unity.py --export trees uv run python geodata_to_unity.py --export all # All standard # Enhanced exports (requires point cloud data) uv run python geodata_to_unity.py --export heightmap-enhanced uv run python geodata_to_unity.py --export buildings-enhanced uv run python geodata_to_unity.py --export trees-enhanced uv run python geodata_to_unity.py --export street-furniture uv run python geodata_to_unity.py --export all-enhanced # All enhanced ``` --- ## Config Classes ``` Config ├── raw: RawConfig # Input directories ├── archives: ArchiveConfig # Archive staging ├── work: WorkConfig # Temp/intermediate ├── export: ExportConfig # Output directories ├── heightmap: HeightmapConfig # out_res=1025, tile_size_m=1000 ├── ortho: OrthoConfig # out_res=2048, quality=90 ├── buildings: BuildingConfig # triangle budget 200-350k ├── trees: TreeConfig # max_trees=5000, chunk_grid=4x4 ├── pointcloud: PointCloudConfig # LPG/LPO/BDOM directories ├── heightmap_enhanced: EnhancedHeightmapConfig ├── buildings_enhanced: EnhancedBuildingConfig ├── trees_enhanced: EnhancedTreeConfig └── street_furniture: StreetFurnitureConfig ``` --- ## Key Files | Module | Purpose | |--------|---------| | `geodata_to_unity.py` | CLI entry point | | `geodata_pipeline/config.py` | Configuration dataclasses | | `geodata_pipeline/heightmaps.py` | DGM1 → PNG16 | | `geodata_pipeline/orthophotos.py` | DOP20 → JPEG | | `geodata_pipeline/buildings.py` | CityGML → GLB | | `geodata_pipeline/trees.py` | CHM → CSV + GLB | | `geodata_pipeline/street_furniture.py` | nDSM → CSV | | `geodata_pipeline/pointcloud.py` | XYZ/LAZ utilities | | `geodata_pipeline/*_enhanced.py` | Enhanced pipelines | | `GeoTileImporter.cs` | Unity importer |