GeoData/README.md

GeoData Toolkit

This repository converts DGM1 elevation tiles into Unity-ready 16-bit PNG heightmaps and a placement manifest. It relies on GDAL for mosaicking, resampling, and scaling to UInt16 ranges Unity expects.

Prerequisites

• uv installed. Use Python 3.10–3.12 for now (triangle2 from cjio[export] has no CPython 3.13 wheels).
• Optional: Java 17+ if you want to experiment with the bundled citygml-tools utilities (not needed for heightmaps/orthophotos).

Environment setup (uv)

• Install deps (creates .venv if missing): uv sync. You can skip manual activation by prefixing commands with uv run ...; if you prefer activation, run uv venv && source .venv/bin/activate.
• uv run <cmd> executes with the project environment (e.g., uv run python geodata_to_unity.py --setup). Use --directory to target another path if needed; --offline disables network fetches.
• If wheels fail to resolve, ensure system GDAL is present (e.g., brew install gdal or apt-get install gdal-bin libgdal-dev), then rerun uv sync.
• Create the default directory tree and config: uv run python geodata_to_unity.py --setup (or bash scripts/setup_dirs.sh for directories only).

Repository Layout

• raw/ — working inputs (not versioned): raw/dgm1/, raw/dop20/jp2/, raw/citygml/lod1/, raw/citygml/lod2/.
• archive/ — offline storage for untouched downloads (e.g., zipped DOP/CityGML tiles, dop20 filelist).
• work/ — intermediates such as dgm.vrt and _tmp.tif files; safe to delete/regenerate.
• export_unity/height_png16/ — final 16-bit PNG heightmaps for Unity import.
• export_unity/tile_index.csv — manifest mapping tile IDs to world bounds and global min/max used for scaling; built during heightmap export and required by orthophotos.
• export_unity/ortho_jpg/ — cropped orthophoto tiles aligned to the terrain grid (JPEG + worldfiles).
• geodata_to_unity.py — main CLI (uses geodata_pipeline/ library modules).
• scripts/ — helpers to create the directory tree and fetch DOP20 inputs.
• geodata_config.json — generated config (see geodata_config.example.json for defaults).
• AGENTS.md — contributor guide.

Quick Start

1. Activate the uv venv (source .venv/bin/activate) or prefix commands with uv run.
2. Initialize config + directories: uv run python geodata_to_unity.py --setup.
3. Export assets (builds VRTs automatically if missing):

   uv run python geodata_to_unity.py --export all
   # heightmaps only: uv run python geodata_to_unity.py --export heightmap
   # textures only:   uv run python geodata_to_unity.py --export textures
   

4. Import the PNGs into Unity Terrains using tile_index.csv for placement and consistent height scaling (0–65535).

How the export works

• Heightmaps: the pipeline builds work/dgm.vrt from all raw/dgm1/*.tif, computes a global min/max once, and warps each tile footprint to heightmap.out_res with srcNodata=-9999 and dstNodata set to the global min. PNGs are scaled to [0, 65535] with worldfiles and listed in export_unity/tile_index.csv.
• Orthophotos: work/dop.vrt is built from raw/dop20/jp2/*.jp2; the manifest drives the cropping bounds. JPEG tiles are written to export_unity/ortho_jpg/ with matching .jgw worldfiles. If the manifest is missing, the orthophoto export aborts—run the heightmap export first or use --export all.
• Archives: --build-from-archive expands every *.zip under archive/* into the matching raw/* directories and copies archive/dop20/filelist.txt next to raw/dop20/ for the downloader.
• Cleanup: temporary _tmp.tif and GDAL aux XML files under work/ and raw/dgm1/ are removed at the end of the heightmap export; avoid storing non-GDAL metadata in those folders.

Key Commands

• Refresh VRT: gdalbuildvrt work/dgm.vrt raw/dgm1/*.tif
• Run export pipeline: uv run python geodata_to_unity.py --export all
• Inspect an output tile: gdalinfo export_unity/height_png16/<tile>.png | head
• Override config paths: use --config <path>, --raw-dgm1-path <dir>, --raw-dop20-path <dir>.
• Build raws from archives: uv run python geodata_to_unity.py --build-from-archive --export all (unzips archive/*; dop20 filelist stays in archive/dop20/ for the downloader).
• Rebuild VRTs after moving data: add --force-vrt.

Workflow Notes

• The pipeline computes a global min/max from the VRT to scale all tiles consistently; adjust heightmap.out_res or heightmap.resample in geodata_config.json if your AOI or target resolution changes.
• _tmp.tif files in work/ are transient; you can delete work/ to force a clean rebuild.
• Keep file names stable to avoid churn in Unity scenes; re-exports overwrite in place.
• Large raw datasets are intentionally excluded from version control—document download sources or scripts instead of committing data.
• Additional inputs: download helper lives in scripts/dlscript_dop20.sh and pulls JP2/J2W/XML orthophotos listed in archive/dop20/filelist.txt (one URL per line); archive/ can hold zipped 3D building tiles for future use.
• Handoff to Unity: copy/sync export_unity/height_png16/ and export_unity/tile_index.csv into DTrierFlood/Assets/GeoData/ before running the Unity-side importer. Keep heightmap.out_res aligned with the importer’s expected resolution (currently 1025).

Orthophotos (textures)

1. Ensure DOP assets are present in raw/dop20/jp2/, raw/dop20/j2w/, and raw/dop20/meta/; use scripts/dlscript_dop20.sh to fetch JP2/J2W/XML entries listed in archive/dop20/filelist.txt (one URL per line).
2. From GeoData/, run:

   uv run python geodata_to_unity.py --export textures
   

   This builds work/dop.vrt if missing and writes export_unity/ortho_jpg/<tile>.jpg + .jgw aligned to tile_index.csv.
   • If you see Computed -srcwin ... falls partially outside source raster extent warnings, the DOP coverage is slightly smaller than the tile footprint; edge pixels will be filled with NoData/zeros. Add adjacent JP2s or shrink the requested window if you need to avoid the warning.
   • The download script relies on a Linux/OpenSSL toolchain with system CA bundle at /etc/ssl/certs/ca-certificates.crt; it builds a trust chain by fetching the geobasis intermediate. macOS/Windows users should either provide a combined CA via CURL_CA_BUNDLE or download with a browser/wget and place files manually.
   • Place companion .j2w and .xml files under raw/dop20/j2w/ and raw/dop20/meta/ if available; they are not required for the VRT but help provenance.

Buildings

LoD2 CityGML tiles can be converted to GLB per tile while preserving roof/wall semantics:

1. CityGML → CityJSON:

   for f in raw/citygml/lod2/*.gml; do
     base="$(basename "$f" .gml)"
     tools/citygml-tools-2.4.0/citygml-tools to-cityjson "$f" \
       -o "work/cityjson/${base}.city.json"
   done
   

   If citygml-tools outputs directories named <tile>.city.json/, they contain <tile>.json inside. If cjio later reports out-of-range vertex indices during triangulation, clean first:

   mkdir -p work/cityjson_clean
   uv run python scripts/clean_cityjson_vertices.py --input-dir work/cityjson --output-dir work/cityjson_clean
   

2. Clean + triangulate:

   mkdir -p work/cityjson_tri
   for f in work/cityjson_clean/*.city.json; do
     base="$(basename "$f" .city.json)"
     input_json="$f/$base.json"  # citygml-tools writes a .json inside the .city.json folder
     uv run cjio --ignore_duplicate_keys "$input_json" upgrade triangulate vertices_clean \
       save "work/cityjson_tri/${base}.tri.city.json"
   done
   

   If cjio still complains about duplicate object IDs, inspect the source GML; --ignore_duplicate_keys tells cjio to keep the last occurrence.
3. Split roof/wall semantics without reindexing vertices: uv run python scripts/split_semantics.py
4. Export GLBs (tile-local coords; place with export_unity/tile_index.csv):

   for f in work/cityjson_tri/*.tri.city.json; do
     base="$(basename "$f" .tri.city.json)"
     uv run cjio "$f" export --format glb "export_unity/buildings_glb/${base}.glb"
   done
   for f in work/cityjson_split/*.roof.city.json; do
     base="$(basename "$f" .roof.city.json)"
     uv run cjio "$f" export --format glb "export_unity/buildings_glb_split/${base}_roof.glb"
   done
   for f in work/cityjson_split/*.wall.city.json; do
     base="$(basename "$f" .wall.city.json)"
     uv run cjio "$f" export --format glb "export_unity/buildings_glb_split/${base}_wall.glb"
   done
   

5. Confirm intermediates/GLBs exist: uv run python scripts/verify_pipeline.py --mode both

Troubleshooting

• Empty raw directories cause VRT creation to fail fast (No sources available to build VRT); populate inputs or adjust --raw-* overrides.
• If you moved raw data or deleted work/, add --force-vrt to rebuild VRTs before exporting.
• Orthophoto export warnings like Computed -srcwin ... falls partially outside source raster extent indicate coverage gaps; add neighboring JP2s or accept the NoData edge fill.
• If GDAL Python bindings are missing, install system GDAL first and re-run uv sync so osgeo imports succeed.