DTTrierFlood/GeoData
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor to config-driven CLI and library pipeline

Browse Source
This commit is contained in:
s0wlz (Matthias Puchstein)
2025-12-15 22:28:21 +01:00
parent 9c441cde99
commit 1cea248a6a
14 changed files with 505 additions and 321 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
.gitignore vendored
View File

@@ -36,6 +36,7 @@ export_unity/buildings_obj/
# Keep the manifest by default (remove this line if you want to ignore it too)
!export_unity/tile_index.csv
geodata_config.json
# --- Raw downloaded source data (usually too big for git) ---
*.zip
@@ -45,10 +46,7 @@ raw/**
raw/dop20/**
!raw/dop20/filelist.txt
raw/dop20/jp2/**
!raw/dop/
raw/dop/**
raw/dop/jp2/**
archives/
archive/
# If you DO want to keep small sample tiles in git for CI/tests,
# comment out raw_dgm1/ and add a whitelist like:
@@ -61,10 +59,3 @@ tmp/
temp/
.cache/
*.gfs
raw_dop/**
!raw_dop/dlscript.sh
!raw_dop/filelist.txt
raw_dop/jp2/**
raw_dgm1/
raw_3dgeb_lod1/
raw_3dgeb_lod2/

22
AGENTS.md
View File

@@ -1,26 +1,28 @@
# Repository Guidelines
## Project Structure & Module Organization
- `export_heightmaps.py` is the main pipeline: builds `work/dgm.vrt`, scales heights, and writes `export_unity/height_png16/*.png` plus `export_unity/tile_index.csv`.
- `export_ortho_tiles.py` exports orthophotos into `export_unity/ortho_jpg/` using the terrain manifest.
- Preferred raw layout: `raw/dgm1/`, `raw/dop20/jp2/` (or `raw/dop/jp2/`), `raw/citygml/lod1/`, `raw/citygml/lod2/`; legacy paths (`raw_dgm1/`, `raw_dop/`, `raw_3dgeb_lod*/`) still work. `archives/` can hold untouched downloads/zips.
- `export_unity/` is safe to sync to Unity projects; treat it as generated output. `work/` holds intermediates and is disposable.
- `geodata_to_unity.py` is the main CLI; library code lives in `geodata_pipeline/` (`heightmaps.py`, `orthophotos.py`, `config.py`, `setup_helpers.py`).
- Working inputs (ignored): `raw/dgm1/`, `raw/dop20/jp2/`, `raw/citygml/lod1/`, `raw/citygml/lod2/`.
- Archives (ignored): `archive/dgm1/`, `archive/dop20/`, `archive/citygml/lod1/`, `archive/citygml/lod2/` (zip storage + dop20 filelist).
- Config: `geodata_config.json` (generated) or `geodata_config.example.json` for defaults.
- `export_unity/` is generated output (heightmaps, orthophotos, manifest). `work/` holds intermediates and is disposable.
## Build, Test, and Development Commands
- Create a venv: `uv venv && source .venv/bin/activate`. Install deps: `uv sync` (generates `uv.lock`).
- If wheels fail, install system GDAL first (e.g., `brew install gdal` or `apt-get install gdal-bin libgdal-dev`), then rerun `uv sync`.
- Prepare the directory tree (raw/archives/work/exports): `bash scripts/setup_dirs.sh`.
- Heightmap export (rebuilds VRT if absent): `uv run python export_heightmaps.py`.
- Orthophoto export: `uv run python export_ortho_tiles.py` (requires JP2s under `raw/dop20/jp2/`; alt `raw/dop/jp2/` and legacy `raw_dop/jp2/` still work). Use `bash scripts/dlscript_dop20.sh` to fetch JP2/J2W/XML listed in `raw/dop20/filelist.txt`.
- Refresh VRT manually if needed: `gdalbuildvrt work/dgm.vrt raw/dgm1/*.tif` (legacy: `raw_dgm1/*.tif`).
- Prepare the directory tree and config: `uv run python geodata_to_unity.py --setup` (or `bash scripts/setup_dirs.sh` for directories only).
- Heightmap export: `uv run python geodata_to_unity.py --export heightmap`.
- Orthophoto export: `uv run python geodata_to_unity.py --export textures` (requires JP2s under `raw/dop20/jp2/`; use `bash scripts/dlscript_dop20.sh` to fetch JP2/J2W/XML listed in `raw/dop20/filelist.txt`).
- Refresh VRT manually if needed: `gdalbuildvrt work/dgm.vrt raw/dgm1/*.tif`.
- Inspect a result: `gdalinfo export_unity/height_png16/<tile>.png | head` to sanity-check bounds and scaling.
- Populate raw data from archives: `uv run python geodata_to_unity.py --use-archive --export all` (unzips `archive/*` and copies dop20 filelist).
- Expected warning: `Computed -srcwin ... falls partially outside source raster extent` means the DOP coverage is slightly smaller than the tile footprint; edge pixels will be filled with NoData/zeros. Add adjacent JP2s or shrink the requested window if you need to silence it.
- Scripts accept CLI overrides (e.g., `--out-dir`, `--jpeg-quality`, `--resample`); run `uv run python <script> -h` to see options.
- Scripts accept CLI overrides (e.g., `--config`, `--raw-dgm1-path`, `--raw-dop20-path`, `--export`); run `uv run python geodata_to_unity.py -h` to see options.
## Coding Style & Naming Conventions
- Python scripts use 4-space indentation, early-exit error handling, and `SystemExit` for fatal issues; follow PEP 8 where practical.
- Keep tile IDs stable (base filename without extension); avoid renaming inputs to reduce churn downstream.
- Prefer explicit constants for tunables (`OUT_RES`, `RESAMPLE`, `TILE_SIZE_M`) and log with clear context.
- Prefer explicit config fields for tunables (`heightmap.out_res`, `heightmap.resample`, `ortho.out_res`, `ortho.jpeg_quality`) and log with clear context.
## Testing Guidelines
- No automated tests yet; rely on manual validation: run exports on a small tile subset, open outputs in `gdalinfo` or GIS viewer, and confirm `tile_index.csv` aligns with Unity expectations.

44
README.md
View File

@@ -5,57 +5,61 @@ This repository converts DGM1 elevation tiles into Unity-ready 16-bit PNG height
### Prerequisites
- GDAL installed with Python bindings (`osgeo` importable).
- Python 3.9+ available on PATH.
- DGM1 source tiles placed under `raw/dgm1/` (legacy `raw_dgm1/` still works) as `dgm1_<utm_zone>_<easting>_<northing>.tif` with matching `.tfw` files.
- Raw inputs (`raw/dop20/`, `raw/dop/`, `raw/citygml/lod1/`, `raw/citygml/lod2/`) are **kept out of git**; keep them locally or document how to fetch/regenerate. Legacy `raw_dop/` and `raw_3dgeb_lod*/` remain ignored/compatible.
- DGM1 source tiles placed under `raw/dgm1/` as `dgm1_<utm_zone>_<easting>_<northing>.tif` with matching `.tfw` files.
- Raw inputs (`raw/dop20/`, `raw/citygml/lod1/`, `raw/citygml/lod2/`) are **kept out of git**; keep them locally or document how to fetch/regenerate.
### Environment setup (uv)
- Create a project venv: `uv venv && source .venv/bin/activate`.
- Install dependencies from `pyproject.toml`: `uv sync` (generates `uv.lock`; warning-free with dependency-groups).
- Install dependencies from `pyproject.toml`: `uv sync` (generates `uv.lock`).
- If wheels fail to resolve, ensure system GDAL is present (e.g., `brew install gdal` or `apt-get install gdal-bin libgdal-dev`), then rerun `uv sync`.
- Create the default directory tree (inputs/archives/outputs): `bash scripts/setup_dirs.sh`.
- Create the default directory tree and config: `uv run python geodata_to_unity.py --setup` (or `bash scripts/setup_dirs.sh` for directories only).
### Repository Layout
- `raw/` — preferred working inputs (not versioned): `raw/dgm1/`, `raw/dop20/jp2/` (or `raw/dop/jp2/`), `raw/citygml/lod1/`, `raw/citygml/lod2/`. Legacy directories (`raw_dgm1/`, `raw_dop/`, `raw_3dgeb_lod1/`, `raw_3dgeb_lod2/`) are still honored by the scripts.
- `archives/` — optional offline storage for untouched downloads (e.g., zipped DOP/CityGML tiles) to keep raw inputs separated from working copies.
- `raw/` — working inputs (not versioned): `raw/dgm1/`, `raw/dop20/jp2/`, `raw/citygml/lod1/`, `raw/citygml/lod2/`.
- `archive/` — offline storage for untouched downloads (e.g., zipped DOP/CityGML tiles, dop20 filelist).
- `work/` — intermediates such as `dgm.vrt` and `_tmp.tif` files; safe to delete/regenerate.
- `export_unity/height_png16/` — final 16-bit PNG heightmaps for Unity import.
- `export_unity/tile_index.csv` — manifest mapping tile IDs to world bounds and global min/max used for scaling.
- `export_unity/ortho_jpg/` — cropped orthophoto tiles aligned to the terrain grid (JPEG + worldfiles).
- `export_heightmaps.py` — main export script.
- `export_ortho_tiles.py` — exports orthophoto tiles from DOP JP2 inputs using the terrain manifest.
- `geodata_to_unity.py` — main CLI (uses `geodata_pipeline/` library modules).
- `scripts/` — helpers to create the directory tree and fetch DOP20 inputs.
- `geodata_config.json` — generated config (see `geodata_config.example.json` for defaults).
- `AGENTS.md` — contributor guide.
### Quick Start
1. Activate the uv venv (`source .venv/bin/activate`) or prefix commands with `uv run`.
2. Ensure directory tree exists: `bash scripts/setup_dirs.sh`.
3. Export Unity heightmaps and manifest (builds `work/dgm.vrt` automatically if missing):
2. Initialize config + directories: `uv run python geodata_to_unity.py --setup`.
3. Export assets (builds VRTs automatically if missing):
```bash
uv run python export_heightmaps.py
uv run python geodata_to_unity.py --export all
# heightmaps only: uv run python geodata_to_unity.py --export heightmap
# textures only: uv run python geodata_to_unity.py --export textures
```
4. Import the PNGs into Unity Terrains using `tile_index.csv` for placement and consistent height scaling (0–65535).
### Key Commands
- Refresh VRT: `gdalbuildvrt work/dgm.vrt raw/dgm1/*.tif` (legacy: `raw_dgm1/*.tif`)
- Run export pipeline: `uv run python export_heightmaps.py`
- Refresh VRT: `gdalbuildvrt work/dgm.vrt raw/dgm1/*.tif`
- Run export pipeline: `uv run python geodata_to_unity.py --export all`
- Inspect an output tile: `gdalinfo export_unity/height_png16/<tile>.png | head`
- Override defaults (e.g., orthophoto out dir): `uv run python export_ortho_tiles.py --out-dir export_unity/ortho_jpg` (see `-h` on each script for tunables).
- Override config paths: use `--config <path>`, `--raw-dgm1-path <dir>`, `--raw-dop20-path <dir>`.
- Use archives to populate raws: `uv run python geodata_to_unity.py --use-archive --export all` (unzips `archive/*` and copies dop20 filelist).
### Workflow Notes
- The script computes a global min/max from the VRT to scale all tiles consistently; adjust `OUT_RES`, `RESAMPLE`, or `TILE_SIZE_M` in `export_heightmaps.py` if your AOI or target resolution changes.
- The pipeline computes a global min/max from the VRT to scale all tiles consistently; adjust `heightmap.out_res` or `heightmap.resample` in `geodata_config.json` if your AOI or target resolution changes.
- `_tmp.tif` files in `work/` are transient; you can delete `work/` to force a clean rebuild.
- Keep file names stable to avoid churn in Unity scenes; re-exports overwrite in place.
- Large raw datasets are intentionally excluded from version control—document download sources or scripts instead of committing data.
- Additional inputs: download helper lives in `raw/dop20/dlscript.sh` (fallback layouts still work) and pulls JP2/J2W/XML orthophotos listed in `filelist.txt` (one URL per line); `3dgeblod1/` and `3dgeblod2/` hold zipped 3D building tiles with object lists for future use.
- Handoff to Unity: copy/sync `export_unity/height_png16/` and `export_unity/tile_index.csv` into `DTrierFlood/Assets/GeoData/` before running the Unity-side importer. Keep `OUT_RES` aligned with the importer’s expected resolution (currently 1025).
- Additional inputs: download helper lives in `scripts/dlscript_dop20.sh` and pulls JP2/J2W/XML orthophotos listed in `raw/dop20/filelist.txt` (one URL per line); `archive/` can hold zipped 3D building tiles for future use.
- Handoff to Unity: copy/sync `export_unity/height_png16/` and `export_unity/tile_index.csv` into `DTrierFlood/Assets/GeoData/` before running the Unity-side importer. Keep `heightmap.out_res` aligned with the importer’s expected resolution (currently 1025).
### Orthophotos (textures)
1. Ensure DOP JP2s are present in `raw/dop20/jp2/` (alt: `raw/dop/jp2/`, legacy: `raw_dop/jp2/`); use `scripts/dlscript_dop20.sh` to fetch JP2/J2W/XML entries listed in `raw/dop20/filelist.txt` (one URL per line).
1. Ensure DOP JP2s are present in `raw/dop20/jp2/`; use `scripts/dlscript_dop20.sh` to fetch JP2/J2W/XML entries listed in `raw/dop20/filelist.txt` (one URL per line).
2. From `GeoData/`, run:
```bash
uv run python export_ortho_tiles.py
uv run python geodata_to_unity.py --export textures
```
This builds `work/dop.vrt` if missing and writes `export_unity/ortho_jpg/<tile>.jpg` + `.jgw` aligned to `tile_index.csv`.
- If you see `Computed -srcwin ... falls partially outside source raster extent` warnings, the DOP coverage is slightly smaller than the tile footprint; edge pixels will be filled with NoData/zeros. Add adjacent JP2s or shrink the requested window if you need to avoid the warning.
### Buildings
The building export pipeline is temporarily disabled while we choose a mesh conversion approach (GDAL lacks a native OBJ writer). CityGML LoD2 sources remain in `raw_3dgeb_lod2/` locally (ignored in git); consider CityGML→glTF/OBJ tools (e.g., citygml-tools + cityjson2gltf) for future integration.
The building export pipeline is temporarily disabled while we choose a mesh conversion approach (GDAL lacks a native OBJ writer). CityGML LoD2 sources remain in `raw/citygml/lod2/` locally (ignored in git); consider CityGML→glTF/OBJ tools (e.g., citygml-tools + cityjson2gltf) for future integration.

161
export_heightmaps.py
View File

@@ -1,164 +1,11 @@
#!/usr/bin/env python3
"""Export DGM1 tiles to Unity-ready 16-bit PNG heightmaps and a manifest."""
"""Compatibility wrapper: run heightmap export via geodata_to_unity."""
from __future__ import annotations
import argparse
import glob
import os
from typing import Iterable
import sys
from osgeo import gdal
from gdal_utils import (
build_vrt,
cleanup_aux_files,
ensure_dir,
ensure_parent,
open_dataset,
resolve_first_existing,
safe_remove,
)
gdal.UseExceptions()
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Export heightmaps and manifest from DGM tiles.")
parser.add_argument("--raw-dir", default="raw_dgm1", help="Directory containing input DGM GeoTIFFs.")
parser.add_argument(
"--raw-dir-new",
dest="raw_dir_new",
default="raw/dgm1",
help="Preferred directory containing input DGM GeoTIFFs (new layout).",
)
parser.add_argument("--vrt-path", default="work/dgm.vrt", help="Path to build/read the DGM VRT.")
parser.add_argument("--out-dir", default="export_unity/height_png16", help="Output directory for PNG heightmaps.")
parser.add_argument(
"--manifest-path",
default=os.path.join("export_unity", "tile_index.csv"),
help="Output CSV manifest path.",
)
parser.add_argument("--out-res", type=int, default=1025, help="Output resolution per tile (2^n + 1 for Unity).")
parser.add_argument("--resample", default="bilinear", help="GDAL resampling algorithm used during warp.")
parser.add_argument(
"--tile-size-m",
type=int,
default=1000,
help="Real-world tile size in meters (informational; input footprints drive bounds).",
)
parser.add_argument(
"--skip-cleanup",
action="store_true",
help="Leave temp GDAL files instead of deleting aux XML and tmp rasters.",
)
return parser.parse_args()
def build_patterns(raw_dir: str) -> Iterable[str]:
return [
os.path.join("work", "*_tmp.tif"),
os.path.join("work", "*_tmp.tif.aux.xml"),
os.path.join("work", "*.aux.xml"),
os.path.join(raw_dir, "*.aux.xml"),
]
def export_heightmaps(args: argparse.Namespace) -> int:
ensure_dir("work")
ensure_dir(args.out_dir)
ensure_parent(args.manifest_path)
raw_dir = (
resolve_first_existing([args.raw_dir_new, args.raw_dir], "DGM input directory")
if args.raw_dir_new
else args.raw_dir
)
tif_paths = sorted(glob.glob(os.path.join(raw_dir, "*.tif")))
build_vrt(args.vrt_path, tif_paths)
ds = open_dataset(args.vrt_path, f"Could not open {args.vrt_path} after attempting to build it.")
band = ds.GetRasterBand(1)
gmin, gmax = band.ComputeRasterMinMax(False)
print(f"GLOBAL_MIN={gmin}, GLOBAL_MAX={gmax}")
with open(args.manifest_path, "w", encoding="utf-8") as f:
f.write("tile_id,xmin,ymin,xmax,ymax,global_min,global_max,out_res\n")
skipped = 0
written = 0
for tif in tif_paths:
try:
tds = open_dataset(tif, f"Skipping unreadable {tif}")
except SystemExit as exc:
print(exc)
skipped += 1
continue
gt = tds.GetGeoTransform()
ulx, xres, _, uly, _, yres = gt # yres typically negative in north-up rasters
# Use the source tile footprint directly to avoid shifting during export.
xmax = ulx + xres * tds.RasterXSize
ymin = uly + yres * tds.RasterYSize
xmin = ulx
ymax = uly
base = os.path.splitext(os.path.basename(tif))[0]
tile_id = base # keep stable naming = easy re-export + reimport
tmp_path = os.path.join("work", f"{tile_id}_tmp.tif")
out_path = os.path.join(args.out_dir, f"{tile_id}.png")
warp_opts = gdal.WarpOptions(
outputBounds=(xmin, ymin, xmax, ymax),
width=args.out_res,
height=args.out_res,
resampleAlg=args.resample,
srcNodata=-9999,
dstNodata=gmin, # fill nodata with global min to avoid deep pits
)
try:
gdal.Warp(tmp_path, ds, options=warp_opts)
except RuntimeError as exc:
print(f"Warp failed for {tile_id}: {exc}")
skipped += 1
continue
# Scale to UInt16 (0..65535) using Strategy-B global min/max
trans_opts = gdal.TranslateOptions(
outputType=gdal.GDT_UInt16,
scaleParams=[(gmin, gmax, 0, 65535)],
format="PNG",
creationOptions=["WORLDFILE=YES"], # emit .wld so GIS tools place tiles correctly
)
try:
gdal.Translate(out_path, tmp_path, options=trans_opts)
except RuntimeError as exc:
print(f"Translate failed for {tile_id}: {exc}")
skipped += 1
continue
safe_remove(tmp_path)
safe_remove(f"{tmp_path}.aux.xml")
f.write(f"{tile_id},{xmin},{ymin},{xmax},{ymax},{gmin},{gmax},{args.out_res}\n")
print(f"Wrote {out_path}")
written += 1
print(f"Manifest: {args.manifest_path}")
print(f"Summary: wrote {written} tiles; skipped {skipped}.")
if not args.skip_cleanup:
removed = cleanup_aux_files(build_patterns(raw_dir))
print(f"Cleanup removed {removed} temporary files/sidecars.")
return 1 if skipped else 0
def main() -> None:
args = parse_args()
raise SystemExit(export_heightmaps(args))
from geodata_to_unity import main as geodata_main
if __name__ == "__main__":
main()
sys.exit(geodata_main(["--export", "heightmap"]))

107
export_ortho_tiles.py
View File

@@ -1,110 +1,11 @@
#!/usr/bin/env python3
"""Export orthophoto tiles aligned to the terrain grid."""
"""Compatibility wrapper: run orthophoto export via geodata_to_unity."""
from __future__ import annotations
import argparse
import csv
import glob
import os
import sys
from osgeo import gdal
from gdal_utils import build_vrt, ensure_dir, ensure_parent, open_dataset, resolve_first_existing
gdal.UseExceptions()
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Export orthophoto tiles aligned to the terrain grid.")
parser.add_argument("--raw-ortho-dir", default="raw_dop/jp2", help="Legacy directory containing JP2 orthophoto tiles.")
parser.add_argument(
"--raw-ortho-dir-new",
dest="raw_ortho_dir_new",
default="raw/dop20/jp2",
help="Preferred directory containing JP2 orthophoto tiles (new layout).",
)
parser.add_argument(
"--raw-ortho-dir-alt",
dest="raw_ortho_dir_alt",
default="raw/dop/jp2",
help="Alternate directory containing JP2 orthophoto tiles (intermediate layout).",
)
parser.add_argument("--vrt-path", default="work/dop.vrt", help="Path to build/read the orthophoto VRT.")
parser.add_argument("--tile-index", default="export_unity/tile_index.csv", help="Tile manifest from heightmap export.")
parser.add_argument("--out-dir", default="export_unity/ortho_jpg", help="Output directory for cropped orthophotos.")
parser.add_argument(
"--out-res",
type=int,
default=2048,
help="Output resolution per tile (default matches 1000 m tiles at ~0.5 m/px).",
)
parser.add_argument("--jpeg-quality", type=int, default=90, help="JPEG quality for exported tiles.")
return parser.parse_args()
def export_orthos(args: argparse.Namespace) -> int:
ensure_dir("work")
ensure_dir(args.out_dir)
ensure_parent(args.vrt_path)
candidates = [args.raw_ortho_dir_new, args.raw_ortho_dir_alt, args.raw_ortho_dir]
raw_ortho_dir = resolve_first_existing(candidates, "orthophoto input directory")
jp2_paths = sorted(glob.glob(os.path.join(raw_ortho_dir, "*.jp2")))
if not jp2_paths:
raise SystemExit(
f"No JP2 files found in {raw_ortho_dir}. Run scripts/dlscript_dop20.sh (or use the fallback layouts) first."
)
build_vrt(args.vrt_path, jp2_paths)
vrt_ds = open_dataset(args.vrt_path, f"Could not open VRT at {args.vrt_path}")
if not os.path.exists(args.tile_index):
raise SystemExit(f"Tile index missing: {args.tile_index}. Run export_heightmaps.py first.")
with open(args.tile_index, newline="", encoding="utf-8") as f:
reader = csv.DictReader(f)
written = 0
skipped = 0
for row in reader:
try:
tile_id = row["tile_id"]
xmin = float(row["xmin"])
ymin = float(row["ymin"])
xmax = float(row["xmax"])
ymax = float(row["ymax"])
except (KeyError, ValueError) as exc:
print(f"Skipping malformed row {row}: {exc}")
skipped += 1
continue
out_path = os.path.join(args.out_dir, f"{tile_id}.jpg")
opts = gdal.TranslateOptions(
format="JPEG",
width=args.out_res,
height=args.out_res,
projWin=(xmin, ymax, xmax, ymin), # xmin,xmax,ymax,ymin (upper-left origin)
creationOptions=[f"QUALITY={args.jpeg_quality}", "WORLDFILE=YES"],
)
try:
gdal.Translate(out_path, vrt_ds, options=opts)
except RuntimeError as exc:
print(f"Translate failed for {tile_id}: {exc}")
skipped += 1
continue
written += 1
print(f"Wrote {out_path}")
print(f"Summary: wrote {written} orthophoto tiles; skipped {skipped}.")
return 1 if skipped else 0
def main() -> None:
args = parse_args()
raise SystemExit(export_orthos(args))
from geodata_to_unity import main as geodata_main
if __name__ == "__main__":
main()
sys.exit(geodata_main(["--export", "textures"]))

34
geodata_config.example.json Normal file
View File

@@ -0,0 +1,34 @@
{
"raw": {
"dgm1_dir": "raw/dgm1",
"dop20_dir": "raw/dop20/jp2",
"citygml_lod1_dir": "raw/citygml/lod1",
"citygml_lod2_dir": "raw/citygml/lod2"
},
"archives": {
"dgm1_dir": "archive/dgm1",
"dop20_dir": "archive/dop20",
"dop20_filelist": "archive/dop20/filelist.txt",
"citygml_lod1_dir": "archive/citygml/lod1",
"citygml_lod2_dir": "archive/citygml/lod2"
},
"work": {
"work_dir": "work",
"heightmap_vrt": "work/dgm.vrt",
"ortho_vrt": "work/dop.vrt"
},
"export": {
"heightmap_dir": "export_unity/height_png16",
"ortho_dir": "export_unity/ortho_jpg",
"manifest_path": "export_unity/tile_index.csv"
},
"heightmap": {
"out_res": 1025,
"resample": "bilinear",
"tile_size_m": 1000
},
"ortho": {
"out_res": 2048,
"jpeg_quality": 90
}
}

1
geodata_pipeline/__init__.py Normal file
View File

@@ -0,0 +1 @@
# GeoData pipeline package

107
geodata_pipeline/config.py Normal file
View File

@@ -0,0 +1,107 @@
from __future__ import annotations
import json
import os
from dataclasses import asdict, dataclass, field, replace
from typing import Any, Dict
DEFAULT_CONFIG_PATH = "geodata_config.json"
@dataclass
class RawConfig:
dgm1_dir: str = "raw/dgm1"
dop20_dir: str = "raw/dop20/jp2"
citygml_lod1_dir: str = "raw/citygml/lod1"
citygml_lod2_dir: str = "raw/citygml/lod2"
@dataclass
class ArchiveConfig:
dgm1_dir: str = "archive/dgm1"
dop20_dir: str = "archive/dop20"
dop20_filelist: str = "archive/dop20/filelist.txt"
citygml_lod1_dir: str = "archive/citygml/lod1"
citygml_lod2_dir: str = "archive/citygml/lod2"
@dataclass
class WorkConfig:
work_dir: str = "work"
heightmap_vrt: str = "work/dgm.vrt"
ortho_vrt: str = "work/dop.vrt"
@dataclass
class ExportConfig:
heightmap_dir: str = "export_unity/height_png16"
ortho_dir: str = "export_unity/ortho_jpg"
manifest_path: str = "export_unity/tile_index.csv"
@dataclass
class HeightmapConfig:
out_res: int = 1025
resample: str = "bilinear"
tile_size_m: int = 1000
@dataclass
class OrthoConfig:
out_res: int = 2048
jpeg_quality: int = 90
@dataclass
class Config:
raw: RawConfig = field(default_factory=RawConfig)
archives: ArchiveConfig = field(default_factory=ArchiveConfig)
work: WorkConfig = field(default_factory=WorkConfig)
export: ExportConfig = field(default_factory=ExportConfig)
heightmap: HeightmapConfig = field(default_factory=HeightmapConfig)
ortho: OrthoConfig = field(default_factory=OrthoConfig)
@classmethod
def default(cls) -> "Config":
return cls()
@classmethod
def load(cls, path: str = DEFAULT_CONFIG_PATH) -> "Config":
with open(path, "r", encoding="utf-8") as f:
data = json.load(f)
return cls.from_dict(data)
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "Config":
return cls(
raw=RawConfig(**data["raw"]),
archives=ArchiveConfig(**data["archives"]),
work=WorkConfig(**data["work"]),
export=ExportConfig(**data["export"]),
heightmap=HeightmapConfig(**data["heightmap"]),
ortho=OrthoConfig(**data["ortho"]),
)
def to_dict(self) -> Dict[str, Any]:
return asdict(self)
def save(self, path: str = DEFAULT_CONFIG_PATH) -> None:
with open(path, "w", encoding="utf-8") as f:
json.dump(self.to_dict(), f, indent=2)
def with_overrides(self, raw_dgm1_path: str | None = None, raw_dop20_path: str | None = None) -> "Config":
cfg = self
if raw_dgm1_path:
cfg = replace(cfg, raw=replace(cfg.raw, dgm1_dir=raw_dgm1_path))
if raw_dop20_path:
cfg = replace(cfg, raw=replace(cfg.raw, dop20_dir=raw_dop20_path))
return cfg
def ensure_default_config(path: str = DEFAULT_CONFIG_PATH) -> Config:
if not os.path.exists(path):
cfg = Config.default()
cfg.save(path)
return cfg
return Config.load(path)

17
gdal_utils.py → geodata_pipeline/gdal_utils.py
View File

@@ -1,5 +1,3 @@
#!/usr/bin/env python3
"""Shared GDAL helpers for the GeoData exporters."""
from __future__ import annotations
import glob
@@ -12,19 +10,16 @@ gdal.UseExceptions()
def ensure_dir(path: str) -> None:
"""Create a directory path if missing."""
os.makedirs(path, exist_ok=True)
def ensure_parent(path: str) -> None:
"""Create the parent directory for a file path."""
parent = os.path.dirname(path)
if parent:
ensure_dir(parent)
def open_dataset(path: str, purpose: str):
"""Open a dataset and fail fast with context."""
try:
ds = gdal.Open(path)
except RuntimeError as exc:
@@ -35,7 +30,6 @@ def open_dataset(path: str, purpose: str):
def build_vrt(vrt_path: str, sources: Sequence[str]) -> bool:
"""Build a VRT if missing; returns True if built."""
if os.path.exists(vrt_path):
return False
if not sources:
@@ -50,7 +44,6 @@ def build_vrt(vrt_path: str, sources: Sequence[str]) -> bool:
def safe_remove(path: str) -> bool:
"""Remove a file if present; return True when deleted."""
try:
os.remove(path)
return True
@@ -62,19 +55,9 @@ def safe_remove(path: str) -> bool:
def cleanup_aux_files(patterns: Iterable[str]) -> int:
"""Clear GDAL sidecars and temp files matching glob patterns."""
removed = 0
for pattern in patterns:
for match in glob.glob(pattern):
if safe_remove(match):
removed += 1
return removed
def resolve_first_existing(paths: Sequence[str], label: str) -> str:
"""Return the first existing path in order; fail with a clear message."""
for path in paths:
if path and os.path.exists(path):
return path
tried = ", ".join(paths)
raise SystemExit(f"No existing path found for {label}. Checked: {tried}")

103
geodata_pipeline/heightmaps.py Normal file
View File

@@ -0,0 +1,103 @@
from __future__ import annotations
import glob
import os
from typing import Iterable
from osgeo import gdal
from .config import Config
from .gdal_utils import build_vrt, cleanup_aux_files, ensure_dir, ensure_parent, open_dataset, safe_remove
gdal.UseExceptions()
def _cleanup_patterns(raw_dir: str) -> Iterable[str]:
return [
os.path.join("work", "*_tmp.tif"),
os.path.join("work", "*_tmp.tif.aux.xml"),
os.path.join("work", "*.aux.xml"),
os.path.join(raw_dir, "*.aux.xml"),
]
def export_heightmaps(cfg: Config) -> int:
ensure_dir(cfg.work.work_dir)
ensure_dir(cfg.export.heightmap_dir)
ensure_parent(cfg.export.manifest_path)
tif_paths = sorted(glob.glob(os.path.join(cfg.raw.dgm1_dir, "*.tif")))
build_vrt(cfg.work.heightmap_vrt, tif_paths)
ds = open_dataset(cfg.work.heightmap_vrt, f"Could not open {cfg.work.heightmap_vrt} after attempting to build it.")
band = ds.GetRasterBand(1)
gmin, gmax = band.ComputeRasterMinMax(False)
print(f"GLOBAL_MIN={gmin}, GLOBAL_MAX={gmax}")
with open(cfg.export.manifest_path, "w", encoding="utf-8") as f:
f.write("tile_id,xmin,ymin,xmax,ymax,global_min,global_max,out_res\n")
skipped = 0
written = 0
for tif in tif_paths:
try:
tds = open_dataset(tif, f"Skipping unreadable {tif}")
except SystemExit as exc:
print(exc)
skipped += 1
continue
gt = tds.GetGeoTransform()
ulx, xres, _, uly, _, yres = gt
xmax = ulx + xres * tds.RasterXSize
ymin = uly + yres * tds.RasterYSize
xmin = ulx
ymax = uly
base = os.path.splitext(os.path.basename(tif))[0]
tile_id = base
tmp_path = os.path.join(cfg.work.work_dir, f"{tile_id}_tmp.tif")
out_path = os.path.join(cfg.export.heightmap_dir, f"{tile_id}.png")
warp_opts = gdal.WarpOptions(
outputBounds=(xmin, ymin, xmax, ymax),
width=cfg.heightmap.out_res,
height=cfg.heightmap.out_res,
resampleAlg=cfg.heightmap.resample,
srcNodata=-9999,
dstNodata=gmin,
)
try:
gdal.Warp(tmp_path, ds, options=warp_opts)
except RuntimeError as exc:
print(f"Warp failed for {tile_id}: {exc}")
skipped += 1
continue
trans_opts = gdal.TranslateOptions(
outputType=gdal.GDT_UInt16,
scaleParams=[(gmin, gmax, 0, 65535)],
format="PNG",
creationOptions=["WORLDFILE=YES"],
)
try:
gdal.Translate(out_path, tmp_path, options=trans_opts)
except RuntimeError as exc:
print(f"Translate failed for {tile_id}: {exc}")
skipped += 1
continue
safe_remove(tmp_path)
safe_remove(f"{tmp_path}.aux.xml")
f.write(f"{tile_id},{xmin},{ymin},{xmax},{ymax},{gmin},{gmax},{cfg.heightmap.out_res}\n")
print(f"Wrote {out_path}")
written += 1
print(f"Manifest: {cfg.export.manifest_path}")
print(f"Summary: wrote {written} tiles; skipped {skipped}.")
removed = cleanup_aux_files(_cleanup_patterns(cfg.raw.dgm1_dir))
print(f"Cleanup removed {removed} temporary files/sidecars.")
return 1 if skipped else 0

66
geodata_pipeline/orthophotos.py Normal file
View File

@@ -0,0 +1,66 @@
from __future__ import annotations
import csv
import glob
import os
from osgeo import gdal
from .config import Config
from .gdal_utils import build_vrt, ensure_dir, ensure_parent, open_dataset
gdal.UseExceptions()
def export_orthophotos(cfg: Config) -> int:
ensure_dir(cfg.work.work_dir)
ensure_dir(cfg.export.ortho_dir)
ensure_parent(cfg.work.ortho_vrt)
jp2_paths = sorted(glob.glob(os.path.join(cfg.raw.dop20_dir, "*.jp2")))
if not jp2_paths:
raise SystemExit(f"No JP2 files found in {cfg.raw.dop20_dir}. Run scripts/dlscript_dop20.sh first.")
build_vrt(cfg.work.ortho_vrt, jp2_paths)
vrt_ds = open_dataset(cfg.work.ortho_vrt, f"Could not open VRT at {cfg.work.ortho_vrt}")
if not os.path.exists(cfg.export.manifest_path):
raise SystemExit(f"Tile index missing: {cfg.export.manifest_path}. Run heightmap export first.")
with open(cfg.export.manifest_path, newline="", encoding="utf-8") as f:
reader = csv.DictReader(f)
written = 0
skipped = 0
for row in reader:
try:
tile_id = row["tile_id"]
xmin = float(row["xmin"])
ymin = float(row["ymin"])
xmax = float(row["xmax"])
ymax = float(row["ymax"])
except (KeyError, ValueError) as exc:
print(f"Skipping malformed row {row}: {exc}")
skipped += 1
continue
out_path = os.path.join(cfg.export.ortho_dir, f"{tile_id}.jpg")
opts = gdal.TranslateOptions(
format="JPEG",
width=cfg.ortho.out_res,
height=cfg.ortho.out_res,
projWin=(xmin, ymax, xmax, ymin),
creationOptions=[f"QUALITY={cfg.ortho.jpeg_quality}", "WORLDFILE=YES"],
)
try:
gdal.Translate(out_path, vrt_ds, options=opts)
except RuntimeError as exc:
print(f"Translate failed for {tile_id}: {exc}")
skipped += 1
continue
written += 1
print(f"Wrote {out_path}")
print(f"Summary: wrote {written} orthophoto tiles; skipped {skipped}.")
return 1 if skipped else 0

59
geodata_pipeline/setup_helpers.py Normal file
View File

@@ -0,0 +1,59 @@
from __future__ import annotations
import glob
import os
import shutil
import zipfile
from typing import Iterable
from .config import Config, ensure_default_config
def ensure_directories(cfg: Config) -> None:
for path in _paths_from_config(cfg):
os.makedirs(path, exist_ok=True)
def _paths_from_config(cfg: Config) -> Iterable[str]:
return [
cfg.raw.dgm1_dir,
cfg.raw.dop20_dir,
cfg.raw.citygml_lod1_dir,
cfg.raw.citygml_lod2_dir,
cfg.archives.dgm1_dir,
cfg.archives.dop20_dir,
cfg.archives.citygml_lod1_dir,
cfg.archives.citygml_lod2_dir,
cfg.work.work_dir,
cfg.export.heightmap_dir,
cfg.export.ortho_dir,
]
def materialize_archives(cfg: Config) -> None:
"""Best-effort expansion of archive zips into raw inputs."""
ensure_directories(cfg)
_unpack_all(cfg.archives.dgm1_dir, cfg.raw.dgm1_dir)
_unpack_all(cfg.archives.citygml_lod1_dir, cfg.raw.citygml_lod1_dir)
_unpack_all(cfg.archives.citygml_lod2_dir, cfg.raw.citygml_lod2_dir)
_unpack_all(cfg.archives.dop20_dir, cfg.raw.dop20_dir)
_copy_filelist(cfg.archives.dop20_filelist, os.path.join(os.path.dirname(cfg.raw.dop20_dir), "filelist.txt"))
def _unpack_all(archive_dir: str, dest_dir: str) -> None:
os.makedirs(dest_dir, exist_ok=True)
for zpath in glob.glob(os.path.join(archive_dir, "*.zip")):
print(f"Unpacking {zpath} -> {dest_dir}")
with zipfile.ZipFile(zpath, "r") as zf:
zf.extractall(dest_dir)
def _copy_filelist(src: str, dest: str) -> None:
if not os.path.exists(src):
return
os.makedirs(os.path.dirname(dest), exist_ok=True)
shutil.copy2(src, dest)
print(f"Copied filelist: {src} -> {dest}")
__all__ = ["ensure_directories", "materialize_archives", "ensure_default_config"]

83
geodata_to_unity.py Normal file
View File

@@ -0,0 +1,83 @@
#!/usr/bin/env python3
"""CLI entrypoint to export GeoData assets for Unity."""
from __future__ import annotations
import argparse
import os
import sys
from typing import Iterable
from geodata_pipeline.config import Config, DEFAULT_CONFIG_PATH, ensure_default_config
from geodata_pipeline.heightmaps import export_heightmaps
from geodata_pipeline.orthophotos import export_orthophotos
from geodata_pipeline.setup_helpers import ensure_directories, materialize_archives
def parse_args(argv: Iterable[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Export GeoData assets for Unity.")
parser.add_argument(
"--config",
default=DEFAULT_CONFIG_PATH,
help="Path to config JSON (created on --setup if missing).",
)
parser.add_argument(
"--export",
choices=["heightmap", "textures", "all"],
default="all",
help="Which assets to export.",
)
parser.add_argument("--raw-dgm1-path", dest="raw_dgm1_path", help="Override raw DGM1 directory.")
parser.add_argument("--raw-dop20-path", dest="raw_dop20_path", help="Override raw DOP20 JP2 directory.")
parser.add_argument(
"--use-archive",
action="store_true",
help="Populate raw inputs from archives (unzips zips, copies dop20 filelist).",
)
parser.add_argument(
"--setup",
action="store_true",
help="Create default directory structure and config if missing; still runs export unless --export is omitted.",
)
parser.add_argument(
"--no-export",
action="store_true",
help="Only run setup/archive prep without exporting.",
)
return parser.parse_args(argv)
def load_config(args: argparse.Namespace) -> Config:
if args.setup and not os.path.exists(args.config):
cfg = ensure_default_config(args.config)
else:
cfg = ensure_default_config(args.config) if os.path.exists(args.config) else Config.default()
if not os.path.exists(args.config):
cfg.save(args.config)
return cfg.with_overrides(raw_dgm1_path=args.raw_dgm1_path, raw_dop20_path=args.raw_dop20_path)
def main(argv: Iterable[str] | None = None) -> int:
args = parse_args(argv)
cfg = load_config(args)
if args.setup:
ensure_directories(cfg)
print(f"Directories ensured. Config at {args.config}.")
if args.use_archive:
materialize_archives(cfg)
if args.no_export:
return 0
exit_codes = []
if args.export in ("heightmap", "all"):
exit_codes.append(export_heightmaps(cfg))
if args.export in ("textures", "all"):
exit_codes.append(export_orthophotos(cfg))
return max(exit_codes) if exit_codes else 0
if __name__ == "__main__":
sys.exit(main())

9
pyproject.toml
View File

@@ -13,14 +13,17 @@ requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = []
force-include = { "export_heightmaps.py" = "export_heightmaps.py", "export_ortho_tiles.py" = "export_ortho_tiles.py", "gdal_utils.py" = "gdal_utils.py" }
packages = ["geodata_pipeline"]
force-include = { "geodata_to_unity.py" = "geodata_to_unity.py", "export_heightmaps.py" = "export_heightmaps.py", "export_ortho_tiles.py" = "export_ortho_tiles.py" }
[tool.hatch.build.targets.sdist]
include = [
"geodata_to_unity.py",
"geodata_pipeline/",
"export_heightmaps.py",
"export_ortho_tiles.py",
"gdal_utils.py",
"geodata_config.example.json",
"scripts/",
"README.md",
"AGENTS.md",
"pyproject.toml",