GeoData/README.md at e83b1820008a5ecaa9d9f2fc20ccf383ac5510fc

Files

s0wlz (Matthias Puchstein) e83b182000 Handle citygml-tools directory outputs and doc cjio paths

2025-12-17 01:38:53 +01:00

8.5 KiB

Raw Blame History

GeoData Toolkit

This repository converts DGM1 elevation tiles into Unity-ready 16-bit PNG heightmaps and a placement manifest. It relies on GDAL for mosaicking, resampling, and scaling to UInt16 ranges Unity expects.

Prerequisites

uv installed. Use Python 3.10–3.12 for now (triangle2 from cjio[export] has no CPython 3.13 wheels).
Optional: Java 17+ if you want to experiment with the bundled citygml-tools utilities (not needed for heightmaps/orthophotos).

Environment setup (uv)

Install deps (creates .venv if missing): uv sync. You can skip manual activation by prefixing commands with uv run ...; if you prefer activation, run uv venv && source .venv/bin/activate.
uv run <cmd> executes with the project environment (e.g., uv run python geodata_to_unity.py --setup). Use --directory to target another path if needed; --offline disables network fetches.
If wheels fail to resolve, ensure system GDAL is present (e.g., brew install gdal or apt-get install gdal-bin libgdal-dev), then rerun uv sync.
Create the default directory tree and config: uv run python geodata_to_unity.py --setup (or bash scripts/setup_dirs.sh for directories only).

Repository Layout

raw/ — working inputs (not versioned): raw/dgm1/, raw/dop20/jp2/, raw/citygml/lod1/, raw/citygml/lod2/.
archive/ — offline storage for untouched downloads (e.g., zipped DOP/CityGML tiles, dop20 filelist).
work/ — intermediates such as dgm.vrt and _tmp.tif files; safe to delete/regenerate.
export_unity/height_png16/ — final 16-bit PNG heightmaps for Unity import.
export_unity/tile_index.csv — manifest mapping tile IDs to world bounds and global min/max used for scaling; built during heightmap export and required by orthophotos.
export_unity/ortho_jpg/ — cropped orthophoto tiles aligned to the terrain grid (JPEG + worldfiles).
geodata_to_unity.py — main CLI (uses geodata_pipeline/ library modules).
scripts/ — helpers to create the directory tree and fetch DOP20 inputs.
geodata_config.json — generated config (see geodata_config.example.json for defaults).
AGENTS.md — contributor guide.

Quick Start

Activate the uv venv (source .venv/bin/activate) or prefix commands with uv run.
Initialize config + directories: uv run python geodata_to_unity.py --setup.

Export assets (builds VRTs automatically if missing):

uv run python geodata_to_unity.py --export all
# heightmaps only: uv run python geodata_to_unity.py --export heightmap
# textures only:   uv run python geodata_to_unity.py --export textures

Import the PNGs into Unity Terrains using tile_index.csv for placement and consistent height scaling (0–65535).

How the export works

Heightmaps: the pipeline builds work/dgm.vrt from all raw/dgm1/*.tif, computes a global min/max once, and warps each tile footprint to heightmap.out_res with srcNodata=-9999 and dstNodata set to the global min. PNGs are scaled to [0, 65535] with worldfiles and listed in export_unity/tile_index.csv.
Orthophotos: work/dop.vrt is built from raw/dop20/jp2/*.jp2; the manifest drives the cropping bounds. JPEG tiles are written to export_unity/ortho_jpg/ with matching .jgw worldfiles. If the manifest is missing, the orthophoto export aborts—run the heightmap export first or use --export all.
Archives: --build-from-archive expands every *.zip under archive/* into the matching raw/* directories and copies archive/dop20/filelist.txt next to raw/dop20/ for the downloader.
Cleanup: temporary _tmp.tif and GDAL aux XML files under work/ and raw/dgm1/ are removed at the end of the heightmap export; avoid storing non-GDAL metadata in those folders.

Key Commands

Refresh VRT: gdalbuildvrt work/dgm.vrt raw/dgm1/*.tif
Run export pipeline: uv run python geodata_to_unity.py --export all
Inspect an output tile: gdalinfo export_unity/height_png16/<tile>.png | head
Override config paths: use --config <path>, --raw-dgm1-path <dir>, --raw-dop20-path <dir>.
Build raws from archives: uv run python geodata_to_unity.py --build-from-archive --export all (unzips archive/*; dop20 filelist stays in archive/dop20/ for the downloader).
Rebuild VRTs after moving data: add --force-vrt.

Workflow Notes

The pipeline computes a global min/max from the VRT to scale all tiles consistently; adjust heightmap.out_res or heightmap.resample in geodata_config.json if your AOI or target resolution changes.
_tmp.tif files in work/ are transient; you can delete work/ to force a clean rebuild.
Keep file names stable to avoid churn in Unity scenes; re-exports overwrite in place.
Large raw datasets are intentionally excluded from version control—document download sources or scripts instead of committing data.
Additional inputs: download helper lives in scripts/dlscript_dop20.sh and pulls JP2/J2W/XML orthophotos listed in archive/dop20/filelist.txt (one URL per line); archive/ can hold zipped 3D building tiles for future use.
Handoff to Unity: copy/sync export_unity/height_png16/ and export_unity/tile_index.csv into DTrierFlood/Assets/GeoData/ before running the Unity-side importer. Keep heightmap.out_res aligned with the importer’s expected resolution (currently 1025).

Orthophotos (textures)

Ensure DOP assets are present in raw/dop20/jp2/, raw/dop20/j2w/, and raw/dop20/meta/; use scripts/dlscript_dop20.sh to fetch JP2/J2W/XML entries listed in archive/dop20/filelist.txt (one URL per line).
From GeoData/, run:
```
uv run python geodata_to_unity.py --export textures
```
This builds work/dop.vrt if missing and writes export_unity/ortho_jpg/<tile>.jpg + .jgw aligned to tile_index.csv.
- If you see Computed -srcwin ... falls partially outside source raster extent warnings, the DOP coverage is slightly smaller than the tile footprint; edge pixels will be filled with NoData/zeros. Add adjacent JP2s or shrink the requested window if you need to avoid the warning.
- The download script relies on a Linux/OpenSSL toolchain with system CA bundle at /etc/ssl/certs/ca-certificates.crt; it builds a trust chain by fetching the geobasis intermediate. macOS/Windows users should either provide a combined CA via CURL_CA_BUNDLE or download with a browser/wget and place files manually.
- Place companion .j2w and .xml files under raw/dop20/j2w/ and raw/dop20/meta/ if available; they are not required for the VRT but help provenance.

Buildings

LoD2 CityGML tiles can be converted to GLB per tile while preserving roof/wall semantics:

CityGML → CityJSON:

for f in raw/citygml/lod2/*.gml; do
  base="$(basename "$f" .gml)"
  tools/citygml-tools-2.4.0/citygml-tools to-cityjson "$f" \
    -o "work/cityjson/${base}.city.json"
done

Clean + triangulate:

for f in work/cityjson/*.city.json; do
  base="$(basename "$f" .city.json)"
  input_json="$f/$base.json"  # citygml-tools writes a .json inside the .city.json folder
  uv run cjio "$input_json" upgrade triangulate vertices_clean \
    save "work/cityjson_tri/${base}.tri.city.json"
done

Split roof/wall semantics without reindexing vertices: uv run python scripts/split_semantics.py

Export GLBs (tile-local coords; place with export_unity/tile_index.csv):

for f in work/cityjson_tri/*.tri.city.json; do
  base="$(basename "$f" .tri.city.json)"
  uv run cjio "$f" export --format glb "export_unity/buildings_glb/${base}.glb"
done
for f in work/cityjson_split/*.roof.city.json; do
  base="$(basename "$f" .roof.city.json)"
  uv run cjio "$f" export --format glb "export_unity/buildings_glb_split/${base}_roof.glb"
done
for f in work/cityjson_split/*.wall.city.json; do
  base="$(basename "$f" .wall.city.json)"
  uv run cjio "$f" export --format glb "export_unity/buildings_glb_split/${base}_wall.glb"
done

Confirm intermediates/GLBs exist: uv run python scripts/verify_pipeline.py --mode both

Troubleshooting

Empty raw directories cause VRT creation to fail fast (No sources available to build VRT); populate inputs or adjust --raw-* overrides.
If you moved raw data or deleted work/, add --force-vrt to rebuild VRTs before exporting.
Orthophoto export warnings like Computed -srcwin ... falls partially outside source raster extent indicate coverage gaps; add neighboring JP2s or accept the NoData edge fill.
If GDAL Python bindings are missing, install system GDAL first and re-run uv sync so osgeo imports succeed.

8.5 KiB Raw Blame History Unescape Escape