From 5640e0fd412454c84dc9c194da147abbb168391c Mon Sep 17 00:00:00 2001 From: "s0wlz (Matthias Puchstein)" Date: Mon, 15 Dec 2025 23:35:06 +0100 Subject: [PATCH] Refresh docs and attribution --- AGENTS.md | 10 ++++++++++ ATTRIBUTION.md | 22 +++++++++++++++------- README.md | 19 ++++++++++++++++--- 3 files changed, 41 insertions(+), 10 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5d8436b..4deb51c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,6 +2,7 @@ ## Project Structure & Module Organization - `geodata_to_unity.py` is the main CLI; library code lives in `geodata_pipeline/` (`heightmaps.py`, `orthophotos.py`, `config.py`, `setup_helpers.py`). +- `export_heightmaps.py` / `export_ortho_tiles.py` are legacy entrypoints packaged for wheels; the supported entrypoint is `geodata_to_unity.py`. - Working inputs (ignored): `raw/dgm1/`, `raw/dop20/jp2/`, `raw/citygml/lod1/`, `raw/citygml/lod2/`. - Archives (ignored): `archive/dgm1/`, `archive/dop20/`, `archive/citygml/lod1/`, `archive/citygml/lod2/` (zip storage + dop20 filelist). - Config: `geodata_config.json` (generated) or `geodata_config.example.json` for defaults. @@ -20,6 +21,8 @@ - Expected warning: `Computed -srcwin ... falls partially outside source raster extent` means the DOP coverage is slightly smaller than the tile footprint; edge pixels will be filled with NoData/zeros. Add adjacent JP2s or shrink the requested window if you need to silence it. - Scripts accept CLI overrides (e.g., `--config`, `--raw-dgm1-path`, `--raw-dop20-path`, `--export`, `--build-from-archive`); run `uv run python geodata_to_unity.py -h` to see options. - DOP20 downloader assumes Linux/OpenSSL with system CA at `/etc/ssl/certs/ca-certificates.crt` to build a trust chain from the geobasis site. macOS/Windows users should either set `CURL_CA_BUNDLE` to a combined CA or download manually and place files in `raw/dop20/`. +- Orthophotos depend on a prebuilt manifest: run the heightmap export first (or `--export all`) so `export_unity/tile_index.csv` exists. +- VRTs are built from whatever is present in the raw directories; empty directories will fail fast. Use `--force-vrt` after moving data or deleting `work/`. ## Coding Style & Naming Conventions - Python scripts use 4-space indentation, early-exit error handling, and `SystemExit` for fatal issues; follow PEP 8 where practical. @@ -38,3 +41,10 @@ ## Security & Data Handling - Keep raw geodata local; avoid publishing source tiles or credentials. Document download sources/scripts instead of committing data. - Outputs may be large; prefer syncing `export_unity/` artifacts via project-specific channels rather than embedding in the repo. + +## Pipeline Behavior (for maintainers) +- Heightmaps: a VRT of all DGM1 tiles is warped per tile footprint, scaled once using the global min/max from the VRT to `[0, 65535]`, and written with worldfiles. Manifest rows include bounds, global min/max, and `out_res`. +- Orthophotos: `export_orthophotos` reuses the manifest for target windows and will abort if it is missing; JPEGs are resampled to `ortho.out_res` with worldfiles and default JPEG quality 90. +- Temporary files are written to `work/*_tmp.tif` and cleaned with broad `*.aux.xml` patterns in `work/` and the raw DGM1 directory—avoid placing non-GDAL aux files there. +- `materialize_archives` unzips every `*.zip` under `archive/*` into the matching raw folders and copies `archive/dop20/filelist.txt` next to `raw/dop20/` for the downloader. +- `geodata_config.example.json` includes `archives.dop20_filelist` for human reference; the dataclass ignores it, so keep the example in sync with actual CLI options rather than adding new unused keys. diff --git a/ATTRIBUTION.md b/ATTRIBUTION.md index bc1849a..6a1ade5 100644 --- a/ATTRIBUTION.md +++ b/ATTRIBUTION.md @@ -1,4 +1,3 @@ -```md # Attribution This repository contains code, documentation, and (optionally) processed geodata for an academic digital twin / flood-visualization prototype of the city of Trier (Germany). @@ -22,6 +21,7 @@ Please keep this file with any redistribution of the repository or any exported Notes: - The repository may include derived products (e.g., heightmaps, texture tiles, meshes) generated from LVermGeo datasets. These derivatives remain subject to the original attribution requirements. +- The download helper (`scripts/dlscript_dop20.sh`) builds a trust chain to `geobasis-rlp.de`; ensure CA handling complies with your environment’s security policies. --- @@ -57,17 +57,25 @@ Notes: --- +## Software and tooling + +- **GDAL** (core dependency, MIT/X-style license). Used for VRT creation, warping, scaling, and raster translation in the Python pipeline. +- **cjio** (Python dependency, MIT license). Included for CityJSON tooling; not all commands are wired into the main pipeline yet. +- **citygml-tools 2.4.0** (bundled under `citygml-tools-2.4.0/`, Apache License 2.0). Java CLI utilities for future CityGML/CityJSON conversion experiments. + +Retain upstream license files when redistributing these tools or building derivative artifacts that embed them. + +--- + ## Project authorship (this repository) All original code, scripts, and documentation authored by the project team are licensed under the repository’s main license (see `LICENSE`), unless stated otherwise in-file. --- ## How to update this file -When adding new datasets or services, add: -1) Dataset/service name -2) Provider +When adding new datasets, services, or bundled tools, include: +1) Dataset/service/tool name +2) Provider or upstream project 3) License name/version 4) Required attribution text -5) Date accessed -``` - +5) Date accessed/downloaded diff --git a/README.md b/README.md index e5ed9d7..d969404 100644 --- a/README.md +++ b/README.md @@ -9,8 +9,8 @@ This repository converts DGM1 elevation tiles into Unity-ready 16-bit PNG height - Raw inputs (`raw/dop20/`, `raw/citygml/lod1/`, `raw/citygml/lod2/`) are **kept out of git**; keep them locally or document how to fetch/regenerate. ### Environment setup (uv) -- Create a project venv: `uv venv && source .venv/bin/activate`. -- Install dependencies from `pyproject.toml`: `uv sync` (generates `uv.lock`). +- Install deps (creates `.venv` if missing): `uv sync`. You can skip manual activation by prefixing commands with `uv run ...`; if you prefer activation, run `uv venv && source .venv/bin/activate`. +- `uv run ` executes with the project environment (e.g., `uv run python geodata_to_unity.py --setup`). Use `--directory` to target another path if needed; `--offline` disables network fetches. - If wheels fail to resolve, ensure system GDAL is present (e.g., `brew install gdal` or `apt-get install gdal-bin libgdal-dev`), then rerun `uv sync`. - Create the default directory tree and config: `uv run python geodata_to_unity.py --setup` (or `bash scripts/setup_dirs.sh` for directories only). @@ -19,7 +19,7 @@ This repository converts DGM1 elevation tiles into Unity-ready 16-bit PNG height - `archive/` — offline storage for untouched downloads (e.g., zipped DOP/CityGML tiles, dop20 filelist). - `work/` — intermediates such as `dgm.vrt` and `_tmp.tif` files; safe to delete/regenerate. - `export_unity/height_png16/` — final 16-bit PNG heightmaps for Unity import. -- `export_unity/tile_index.csv` — manifest mapping tile IDs to world bounds and global min/max used for scaling. +- `export_unity/tile_index.csv` — manifest mapping tile IDs to world bounds and global min/max used for scaling; built during heightmap export and required by orthophotos. - `export_unity/ortho_jpg/` — cropped orthophoto tiles aligned to the terrain grid (JPEG + worldfiles). - `geodata_to_unity.py` — main CLI (uses `geodata_pipeline/` library modules). - `scripts/` — helpers to create the directory tree and fetch DOP20 inputs. @@ -37,6 +37,12 @@ This repository converts DGM1 elevation tiles into Unity-ready 16-bit PNG height ``` 4. Import the PNGs into Unity Terrains using `tile_index.csv` for placement and consistent height scaling (0–65535). +### How the export works +- Heightmaps: the pipeline builds `work/dgm.vrt` from all `raw/dgm1/*.tif`, computes a global min/max once, and warps each tile footprint to `heightmap.out_res` with `srcNodata=-9999` and `dstNodata` set to the global min. PNGs are scaled to `[0, 65535]` with worldfiles and listed in `export_unity/tile_index.csv`. +- Orthophotos: `work/dop.vrt` is built from `raw/dop20/jp2/*.jp2`; the manifest drives the cropping bounds. JPEG tiles are written to `export_unity/ortho_jpg/` with matching `.jgw` worldfiles. If the manifest is missing, the orthophoto export aborts—run the heightmap export first or use `--export all`. +- Archives: `--build-from-archive` expands every `*.zip` under `archive/*` into the matching `raw/*` directories and copies `archive/dop20/filelist.txt` next to `raw/dop20/` for the downloader. +- Cleanup: temporary `_tmp.tif` and GDAL aux XML files under `work/` and `raw/dgm1/` are removed at the end of the heightmap export; avoid storing non-GDAL metadata in those folders. + ### Key Commands - Refresh VRT: `gdalbuildvrt work/dgm.vrt raw/dgm1/*.tif` - Run export pipeline: `uv run python geodata_to_unity.py --export all` @@ -62,6 +68,13 @@ This repository converts DGM1 elevation tiles into Unity-ready 16-bit PNG height This builds `work/dop.vrt` if missing and writes `export_unity/ortho_jpg/.jpg` + `.jgw` aligned to `tile_index.csv`. - If you see `Computed -srcwin ... falls partially outside source raster extent` warnings, the DOP coverage is slightly smaller than the tile footprint; edge pixels will be filled with NoData/zeros. Add adjacent JP2s or shrink the requested window if you need to avoid the warning. - The download script relies on a Linux/OpenSSL toolchain with system CA bundle at `/etc/ssl/certs/ca-certificates.crt`; it builds a trust chain by fetching the geobasis intermediate. macOS/Windows users should either provide a combined CA via `CURL_CA_BUNDLE` or download with a browser/wget and place files manually. + - Place companion `.j2w` and `.xml` files under `raw/dop20/j2w/` and `raw/dop20/meta/` if available; they are not required for the VRT but help provenance. ### Buildings The building export pipeline is temporarily disabled while we choose a mesh conversion approach (GDAL lacks a native OBJ writer). CityGML LoD2 sources remain in `raw/citygml/lod2/` locally (ignored in git); consider CityGML→glTF/OBJ tools (e.g., citygml-tools + cityjson2gltf) for future integration. + +### Troubleshooting +- Empty raw directories cause VRT creation to fail fast (`No sources available to build VRT`); populate inputs or adjust `--raw-*` overrides. +- If you moved raw data or deleted `work/`, add `--force-vrt` to rebuild VRTs before exporting. +- Orthophoto export warnings like `Computed -srcwin ... falls partially outside source raster extent` indicate coverage gaps; add neighboring JP2s or accept the NoData edge fill. +- If GDAL Python bindings are missing, install system GDAL first and re-run `uv sync` so `osgeo` imports succeed.