122 lines
9.1 KiB
Markdown
122 lines
9.1 KiB
Markdown
## GeoData Toolkit
|
||
|
||
This repository converts DGM1 elevation tiles into Unity-ready 16-bit PNG heightmaps and a placement manifest. It relies on GDAL for mosaicking, resampling, and scaling to UInt16 ranges Unity expects.
|
||
|
||
### Prerequisites
|
||
- `uv` installed. Use Python 3.10–3.12 for now (`triangle2` from `cjio[export]` has no CPython 3.13 wheels).
|
||
- Optional: Java 17+ if you want to experiment with the bundled `citygml-tools` utilities (not needed for heightmaps/orthophotos).
|
||
|
||
### Environment setup (uv)
|
||
- Install deps (creates `.venv` if missing): `uv sync`. You can skip manual activation by prefixing commands with `uv run ...`; if you prefer activation, run `uv venv && source .venv/bin/activate`.
|
||
- `uv run <cmd>` executes with the project environment (e.g., `uv run python geodata_to_unity.py --setup`). Use `--directory` to target another path if needed; `--offline` disables network fetches.
|
||
- If wheels fail to resolve, ensure system GDAL is present (e.g., `brew install gdal` or `apt-get install gdal-bin libgdal-dev`), then rerun `uv sync`.
|
||
- Create the default directory tree and config: `uv run python geodata_to_unity.py --setup` (or `bash scripts/setup_dirs.sh` for directories only).
|
||
|
||
### Repository Layout
|
||
- `raw/` — working inputs (not versioned): `raw/dgm1/`, `raw/dop20/jp2/`, `raw/citygml/lod1/`, `raw/citygml/lod2/`.
|
||
- `archive/` — offline storage for untouched downloads (e.g., zipped DOP/CityGML tiles, dop20 filelist).
|
||
- `work/` — intermediates such as `dgm.vrt` and `_tmp.tif` files; safe to delete/regenerate.
|
||
- `export_unity/height_png16/` — final 16-bit PNG heightmaps for Unity import.
|
||
- `export_unity/tile_index.csv` — manifest mapping tile IDs to world bounds and global min/max used for scaling; built during heightmap export and required by orthophotos.
|
||
- `export_unity/ortho_jpg/` — cropped orthophoto tiles aligned to the terrain grid (JPEG + worldfiles).
|
||
- `geodata_to_unity.py` — main CLI (uses `geodata_pipeline/` library modules).
|
||
- `scripts/` — helpers to create the directory tree and fetch DOP20 inputs.
|
||
- `geodata_config.json` — generated config (see `geodata_config.example.json` for defaults).
|
||
- `AGENTS.md` — contributor guide.
|
||
|
||
### Quick Start
|
||
1. Activate the uv venv (`source .venv/bin/activate`) or prefix commands with `uv run`.
|
||
2. Initialize config + directories: `uv run python geodata_to_unity.py --setup`.
|
||
3. Export assets (builds VRTs automatically if missing):
|
||
```bash
|
||
uv run python geodata_to_unity.py --export all
|
||
# heightmaps only: uv run python geodata_to_unity.py --export heightmap
|
||
# textures only: uv run python geodata_to_unity.py --export textures
|
||
```
|
||
4. Import the PNGs into Unity Terrains using `tile_index.csv` for placement and consistent height scaling (0–65535).
|
||
|
||
### How the export works
|
||
- Heightmaps: the pipeline builds `work/dgm.vrt` from all `raw/dgm1/*.tif`, computes a global min/max once, and warps each tile footprint to `heightmap.out_res` with `srcNodata=-9999` and `dstNodata` set to the global min. PNGs are scaled to `[0, 65535]` with worldfiles and listed in `export_unity/tile_index.csv`.
|
||
- Orthophotos: `work/dop.vrt` is built from `raw/dop20/jp2/*.jp2`; the manifest drives the cropping bounds. JPEG tiles are written to `export_unity/ortho_jpg/` with matching `.jgw` worldfiles. If the manifest is missing, the orthophoto export aborts—run the heightmap export first or use `--export all`.
|
||
- Archives: `--build-from-archive` expands every `*.zip` under `archive/*` into the matching `raw/*` directories and copies `archive/dop20/filelist.txt` next to `raw/dop20/` for the downloader.
|
||
- Cleanup: temporary `_tmp.tif` and GDAL aux XML files under `work/` and `raw/dgm1/` are removed at the end of the heightmap export; avoid storing non-GDAL metadata in those folders.
|
||
|
||
### Key Commands
|
||
- Refresh VRT: `gdalbuildvrt work/dgm.vrt raw/dgm1/*.tif`
|
||
- Run export pipeline: `uv run python geodata_to_unity.py --export all`
|
||
- Inspect an output tile: `gdalinfo export_unity/height_png16/<tile>.png | head`
|
||
- Override config paths: use `--config <path>`, `--raw-dgm1-path <dir>`, `--raw-dop20-path <dir>`.
|
||
- Build raws from archives: `uv run python geodata_to_unity.py --build-from-archive --export all` (unzips `archive/*`; dop20 filelist stays in `archive/dop20/` for the downloader).
|
||
- Rebuild VRTs after moving data: add `--force-vrt`.
|
||
|
||
### Workflow Notes
|
||
- The pipeline computes a global min/max from the VRT to scale all tiles consistently; adjust `heightmap.out_res` or `heightmap.resample` in `geodata_config.json` if your AOI or target resolution changes.
|
||
- `_tmp.tif` files in `work/` are transient; you can delete `work/` to force a clean rebuild.
|
||
- Keep file names stable to avoid churn in Unity scenes; re-exports overwrite in place.
|
||
- Large raw datasets are intentionally excluded from version control—document download sources or scripts instead of committing data.
|
||
- Additional inputs: download helper lives in `scripts/dlscript_dop20.sh` and pulls JP2/J2W/XML orthophotos listed in `archive/dop20/filelist.txt` (one URL per line); `archive/` can hold zipped 3D building tiles for future use.
|
||
- Handoff to Unity: copy/sync `export_unity/height_png16/` and `export_unity/tile_index.csv` into `DTrierFlood/Assets/GeoData/` before running the Unity-side importer. Keep `heightmap.out_res` aligned with the importer’s expected resolution (currently 1025).
|
||
|
||
### Orthophotos (textures)
|
||
1. Ensure DOP assets are present in `raw/dop20/jp2/`, `raw/dop20/j2w/`, and `raw/dop20/meta/`; use `scripts/dlscript_dop20.sh` to fetch JP2/J2W/XML entries listed in `archive/dop20/filelist.txt` (one URL per line).
|
||
2. From `GeoData/`, run:
|
||
```bash
|
||
uv run python geodata_to_unity.py --export textures
|
||
```
|
||
This builds `work/dop.vrt` if missing and writes `export_unity/ortho_jpg/<tile>.jpg` + `.jgw` aligned to `tile_index.csv`.
|
||
- If you see `Computed -srcwin ... falls partially outside source raster extent` warnings, the DOP coverage is slightly smaller than the tile footprint; edge pixels will be filled with NoData/zeros. Add adjacent JP2s or shrink the requested window if you need to avoid the warning.
|
||
- The download script relies on a Linux/OpenSSL toolchain with system CA bundle at `/etc/ssl/certs/ca-certificates.crt`; it builds a trust chain by fetching the geobasis intermediate. macOS/Windows users should either provide a combined CA via `CURL_CA_BUNDLE` or download with a browser/wget and place files manually.
|
||
- Place companion `.j2w` and `.xml` files under `raw/dop20/j2w/` and `raw/dop20/meta/` if available; they are not required for the VRT but help provenance.
|
||
|
||
### Buildings
|
||
LoD2 CityGML tiles can be converted to GLB per tile while preserving roof/wall semantics:
|
||
|
||
1. CityGML → CityJSON:
|
||
```bash
|
||
for f in raw/citygml/lod2/*.gml; do
|
||
base="$(basename "$f" .gml)"
|
||
tools/citygml-tools-2.4.0/citygml-tools to-cityjson "$f" \
|
||
-o "work/cityjson/${base}.city.json"
|
||
done
|
||
```
|
||
If citygml-tools outputs directories named `<tile>.city.json/`, they contain `<tile>.json` inside.
|
||
If cjio later reports out-of-range vertex indices during triangulation, clean first:
|
||
```bash
|
||
mkdir -p work/cityjson_clean
|
||
uv run python scripts/clean_cityjson_vertices.py --input-dir work/cityjson --output-dir work/cityjson_clean
|
||
```
|
||
2. Clean + triangulate:
|
||
```bash
|
||
mkdir -p work/cityjson_tri
|
||
for f in work/cityjson_clean/*.city.json; do
|
||
base="$(basename "$f" .city.json)"
|
||
input_json="$f/$base.json" # citygml-tools writes a .json inside the .city.json folder
|
||
uv run cjio --ignore_duplicate_keys "$input_json" upgrade triangulate vertices_clean \
|
||
save "work/cityjson_tri/${base}.tri.city.json"
|
||
done
|
||
```
|
||
If cjio still complains about duplicate object IDs, inspect the source GML; `--ignore_duplicate_keys` tells cjio to keep the last occurrence.
|
||
3. Split roof/wall semantics without reindexing vertices: `uv run python scripts/split_semantics.py`
|
||
4. Export GLBs (tile-local coords; place with `export_unity/tile_index.csv`):
|
||
```bash
|
||
for f in work/cityjson_tri/*.tri.city.json; do
|
||
base="$(basename "$f" .tri.city.json)"
|
||
uv run cjio "$f" export glb "export_unity/buildings_glb/${base}.glb"
|
||
done
|
||
for f in work/cityjson_split/*.roof.city.json; do
|
||
base="$(basename "$f" .roof.city.json)"
|
||
uv run cjio "$f" export glb "export_unity/buildings_glb_split/${base}_roof.glb"
|
||
done
|
||
for f in work/cityjson_split/*.wall.city.json; do
|
||
base="$(basename "$f" .wall.city.json)"
|
||
uv run cjio "$f" export glb "export_unity/buildings_glb_split/${base}_wall.glb"
|
||
done
|
||
```
|
||
5. Confirm intermediates/GLBs exist: `uv run python scripts/verify_pipeline.py --mode both`
|
||
|
||
### Troubleshooting
|
||
- Empty raw directories cause VRT creation to fail fast (`No sources available to build VRT`); populate inputs or adjust `--raw-*` overrides.
|
||
- If you moved raw data or deleted `work/`, add `--force-vrt` to rebuild VRTs before exporting.
|
||
- Orthophoto export warnings like `Computed -srcwin ... falls partially outside source raster extent` indicate coverage gaps; add neighboring JP2s or accept the NoData edge fill.
|
||
- If GDAL Python bindings are missing, install system GDAL first and re-run `uv sync` so `osgeo` imports succeed.
|