GeoData Toolkit

This repository converts DGM1 elevation tiles into Unity-ready 16-bit PNG heightmaps and a placement manifest. It relies on GDAL for mosaicking, resampling, and scaling to UInt16 ranges Unity expects.

Prerequisites

• uv installed. Use Python 3.10–3.12 for now (triangle2 from cjio[export] has no CPython 3.13 wheels).
• Optional: Java 17+ if you want to experiment with the bundled citygml-tools utilities (not needed for heightmaps/orthophotos).

Environment setup (uv)

• Install deps (creates .venv if missing): uv sync. You can skip manual activation by prefixing commands with uv run ...; if you prefer activation, run uv venv && source .venv/bin/activate.
• uv run <cmd> executes with the project environment (e.g., uv run python geodata_to_unity.py --setup). Use --directory to target another path if needed; --offline disables network fetches.
• If wheels fail to resolve, ensure system GDAL is present (e.g., brew install gdal or apt-get install gdal-bin libgdal-dev), then rerun uv sync.
• Create the default directory tree and config: uv run python geodata_to_unity.py --setup (or bash scripts/setup_dirs.sh for directories only).

Repository Layout

• raw/ — working inputs (not versioned): raw/dgm1/, raw/dop20/jp2/, raw/citygml/lod1/, raw/citygml/lod2/.
• archive/ — offline storage for untouched downloads (e.g., zipped DOP/CityGML tiles, dop20 filelist).
• work/ — intermediates such as dgm.vrt and _tmp.tif files; safe to delete/regenerate.
• export_unity/height_png16/ — final 16-bit PNG heightmaps for Unity import.
• export_unity/tile_index.csv — manifest mapping tile IDs to world bounds and scaling ranges (global_min/global_max plus per-tile tile_min/tile_max), plus tile_key; built during heightmap export and required by orthophotos.
• export_unity/ortho_jpg/ — cropped orthophoto tiles aligned to the terrain grid (JPEG + worldfiles).
• geodata_to_unity.py — main CLI (uses geodata_pipeline/ library modules).
• scripts/ — helpers to create the directory tree and fetch DOP20 inputs.
• geodata_config.toml — generated config (see geodata_config.example.toml for defaults).
• AGENTS.md — contributor guide.

Quick Start

1. Activate the uv venv (source .venv/bin/activate) or prefix commands with uv run.
2. Initialize config + directories: uv run python geodata_to_unity.py --setup.
3. Export assets (builds VRTs automatically if missing):

   uv run python geodata_to_unity.py --export all
   # heightmaps only: uv run python geodata_to_unity.py --export heightmap
   # textures only:   uv run python geodata_to_unity.py --export textures
   # buildings:       uv run python geodata_to_unity.py --export buildings
   # trees (CSV+GLB): uv run python geodata_to_unity.py --export trees
   

4. Import the PNGs into Unity Terrains using tile_index.csv for placement and height scaling (0–65535, per-tile by default).

How the export works

• Heightmaps: the pipeline builds work/dgm.vrt from all raw/dgm1/*.tif, computes a global min/max once (legacy fallback), and warps each tile footprint to heightmap.out_res with srcNodata=-9999. Per-tile min/max are computed from the warped tile and used to scale PNGs to [0, 65535] by default (heightmap.use_tile_minmax=false keeps global scaling). export_unity/tile_index.csv records global_min/global_max, tile_min/tile_max, and tile_key = f"{floor((xmin + overlap_x) / tile_size_x)}_{floor((ymin + overlap_y) / tile_size_y)}" (defaults: tile_size_x=1000.0, tile_size_y=1000.0, overlap_x=0.5, overlap_y=0.5 in [tile_key]).
• Orthophotos: work/dop.vrt is built from raw/dop20/jp2/*.jp2; the manifest drives the cropping bounds. JPEG tiles are written to export_unity/ortho_jpg/ with matching .jgw worldfiles. If the manifest is missing, the orthophoto export aborts—run the heightmap export first or use --export all.
• Archives: --build-from-archive expands every *.zip under archive/* into the matching raw/* directories and copies archive/dop20/filelist.txt next to raw/dop20/ for the downloader.
• Cleanup: temporary _tmp.tif and GDAL aux XML files under work/ and raw/dgm1/ are removed at the end of the heightmap export; avoid storing non-GDAL metadata in those folders.

Key Commands

• Refresh VRT: gdalbuildvrt work/dgm.vrt raw/dgm1/*.tif
• Run export pipeline: uv run python geodata_to_unity.py --export all
• Inspect an output tile: gdalinfo export_unity/height_png16/<tile>.png | head
• Override config paths: use --config <path>, --raw-dgm1-path <dir>, --raw-dop20-path <dir>.
• Build raws from archives: uv run python geodata_to_unity.py --build-from-archive --export all (unzips archive/*; dop20 filelist stays in archive/dop20/ for the downloader).
• Rebuild VRTs after moving data: add --force-vrt.

Workflow Notes

• Heightmaps normalize per tile using tile_min/tile_max by default; set heightmap.use_tile_minmax=false to restore global scaling across tiles. Adjust heightmap.out_res or heightmap.resample in geodata_config.toml if your AOI or target resolution changes.
• tile_key config controls the tile grouping key in the manifest; defaults are tile_size_x=1000.0, tile_size_y=1000.0, overlap_x=0.5, overlap_y=0.5 with enabled=true.
• _tmp.tif files in work/ are transient; you can delete work/ to force a clean rebuild.
• Keep file names stable to avoid churn in Unity scenes; re-exports overwrite in place.
• Large raw datasets are intentionally excluded from version control—document download sources or scripts instead of committing data.
• Additional inputs: download helper lives in scripts/dlscript_dop20.sh and pulls JP2/J2W/XML orthophotos listed in archive/dop20/filelist.txt (one URL per line); archive/ can hold zipped 3D building tiles for future use.
• Handoff to Unity: copy/sync export_unity/height_png16/ and export_unity/tile_index.csv into DTrierFlood/Assets/GeoData/ before running the Unity-side importer. Keep heightmap.out_res aligned with the importer’s expected resolution (currently 1025).

Orthophotos (textures)

1. Ensure DOP assets are present in raw/dop20/jp2/, raw/dop20/j2w/, and raw/dop20/meta/; use scripts/dlscript_dop20.sh to fetch JP2/J2W/XML entries listed in archive/dop20/filelist.txt (one URL per line).
2. From GeoData/, run:

   uv run python geodata_to_unity.py --export textures
   

   This builds work/dop.vrt if missing and writes export_unity/ortho_jpg/<tile>.jpg + .jgw aligned to tile_index.csv.
   • If you see Computed -srcwin ... falls partially outside source raster extent warnings, the DOP coverage is slightly smaller than the tile footprint; edge pixels will be filled with NoData/zeros. Add adjacent JP2s or shrink the requested window if you need to avoid the warning.
   • The download script relies on a Linux/OpenSSL toolchain with system CA bundle at /etc/ssl/certs/ca-certificates.crt; it builds a trust chain by fetching the geobasis intermediate. macOS/Windows users should either provide a combined CA via CURL_CA_BUNDLE or download with a browser/wget and place files manually.
   • Place companion .j2w and .xml files under raw/dop20/j2w/ and raw/dop20/meta/ if available; they are not required for the VRT but help provenance.

Downloads (raw data)

• Run: uv run python geodata_to_unity.py --download (uses geodata_download.toml).
• Shows a progress bar while downloading and exits cleanly on Ctrl+C (exit code 130).

Buildings (automated exporter)

• Run: uv run python geodata_to_unity.py --export buildings
• What it does per tile:
  • Converts LoD2 CityGML → CityJSON (citygml-tools), triangulates with cjio, rebases to tile-local XY using tile_index.csv.
  • Merges all buildings into one GLB (1 mesh with roof/wall primitives), decimates to the configured triangle budget.
  • Roofs: planar UVs from tile-local XY, embedded per-tile DOP20 orthophoto as base color (unlit by default).
  • Walls: vertex colors sampled from the ortho as a fallback (neutral otherwise).
  • Coordinates: glTF-friendly (x=east, y=height, z=-north) so glTFast instantiates one GameObject with two submeshes.
• Requirements: LoD2 GMLs under raw/citygml/lod2/, per-tile orthos in export_unity/ortho_jpg/, tools on PATH (tools/citygml-tools-2.4.0/citygml-tools, cjio or override via CJIO env).
• Open items: richer wall coloring from BDOM/LPO facades, better simplification, footprint-aware ground snapping beyond the current clamp-to-ground.

Troubleshooting

• Empty raw directories cause VRT creation to fail fast (No sources available to build VRT); populate inputs or adjust --raw-* overrides.
• If you moved raw data or deleted work/, add --force-vrt to rebuild VRTs before exporting.
• Orthophoto export warnings like Computed -srcwin ... falls partially outside source raster extent indicate coverage gaps; add neighboring JP2s or accept the NoData edge fill.
• If GDAL Python bindings are missing, install system GDAL first and re-run uv sync so osgeo imports succeed.