Compare commits

..

4 Commits

5 changed files with 287 additions and 144 deletions

View File

@@ -1,55 +0,0 @@
# Repository Guidelines
## Project Structure & Module Organization
- `geodata_to_unity.py` is the main CLI; library code lives in `geodata_pipeline/` (`heightmaps.py`, `orthophotos.py`, `config.py`, `setup_helpers.py`). Legacy wrapper scripts have been removed; use `geodata_to_unity.py` directly.
- Working inputs (ignored): `raw/dgm1/`, `raw/dop20/jp2/`, `raw/citygml/lod1/`, `raw/citygml/lod2/`.
- Archives (ignored): `archive/dgm1/`, `archive/dop20/`, `archive/citygml/lod1/`, `archive/citygml/lod2/` (zip storage + dop20 filelist).
- Config: `geodata_config.toml` (generated) or `geodata_config.example.toml` for defaults.
- `export_unity/` is generated output (heightmaps, orthophotos, manifest). `work/` holds intermediates and is disposable.
## Build, Test, and Development Commands
- Create a venv: `uv venv && source .venv/bin/activate`. Install deps: `uv sync` (generates `uv.lock`).
- If wheels fail, install system GDAL first (e.g., `brew install gdal` or `apt-get install gdal-bin libgdal-dev`), then rerun `uv sync`.
- Prepare the directory tree and config: `uv run python geodata_to_unity.py --setup` (or `bash scripts/setup_dirs.sh` for directories only).
- Heightmap export: `uv run python geodata_to_unity.py --export heightmap`.
- Orthophoto export: `uv run python geodata_to_unity.py --export textures` (requires JP2s under `raw/dop20/jp2/`; use `bash scripts/dlscript_dop20.sh` to fetch JP2/J2W/XML listed in `archive/dop20/filelist.txt`).
- Refresh VRT manually if needed: `gdalbuildvrt work/dgm.vrt raw/dgm1/*.tif`.
- Inspect a result: `gdalinfo export_unity/height_png16/<tile>.png | head` to sanity-check bounds and scaling.
- Populate raw data from archives: `uv run python geodata_to_unity.py --build-from-archive --export all` (unzips `archive/*`; dop20 filelist stays in archive for the downloader).
- Rebuild VRTs after moving data: add `--force-vrt`.
- Expected warning: `Computed -srcwin ... falls partially outside source raster extent` means the DOP coverage is slightly smaller than the tile footprint; edge pixels will be filled with NoData/zeros. Add adjacent JP2s or shrink the requested window if you need to silence it.
- Scripts accept CLI overrides (e.g., `--config`, `--raw-dgm1-path`, `--raw-dop20-path`, `--export`, `--build-from-archive`); run `uv run python geodata_to_unity.py -h` to see options.
- DOP20 downloader assumes Linux/OpenSSL with system CA at `/etc/ssl/certs/ca-certificates.crt` to build a trust chain from the geobasis site. macOS/Windows users should either set `CURL_CA_BUNDLE` to a combined CA or download manually and place files in `raw/dop20/`.
- Orthophotos depend on a prebuilt manifest: run the heightmap export first (or `--export all`) so `export_unity/tile_index.csv` exists.
- VRTs are built from whatever is present in the raw directories; empty directories will fail fast. Use `--force-vrt` after moving data or deleting `work/`.
## Coding Style & Naming Conventions
- Python scripts use 4-space indentation, early-exit error handling, and `SystemExit` for fatal issues; follow PEP 8 where practical.
- Keep tile IDs stable (base filename without extension); avoid renaming inputs to reduce churn downstream.
- Prefer explicit config fields for tunables (`heightmap.out_res`, `heightmap.resample`, `ortho.out_res`, `ortho.jpeg_quality`) and log with clear context.
## Testing Guidelines
- No automated tests yet; rely on manual validation: run exports on a small tile subset, open outputs in `gdalinfo` or GIS viewer, and confirm `tile_index.csv` aligns with Unity expectations.
- When changing scaling or resolution, compare before/after stats (min/max) and spot-check terrain in Unity.
## Current Pipelines (v1 WIP)
- **Heightmaps/Orthos**: unchanged; see README.
- **Buildings (new)**: `--export buildings` converts LoD2 CityGML → CityJSON (citygml-tools), triangulates (cjio), rebases to tile-local XY, merges per tile into one GLB (1 mesh, roof/wall primitives), decimates to budget, planar-UV roofs with embedded DOP20 tile texture (unlit), walls colored from ortho fallback, axes glTF-friendly (x=east, y=height, z=-north). Open: better wall coloring (BDOM/LPO), stronger simplification, footprint-aware ground snap (currently clamp to DTM).
- **Trees (new)**: `--export trees` uses DOM1DGM1 CHM + CityGML building mask (buffered) to find trees, roughness/confidence heuristic with optional LPO/BDOM boost, caps by count. Outputs per-tile CSV and chunked GLBs (4×4 by default) built from 16 procedural proxies; instancing toggle in config; shared proxy library emitted.
- **Tools**: expects `citygml-tools-2.4.0/citygml-tools` and `cjio` on PATH (override `CJIO` env). Orthos must exist for roofs/walls.
## Commit & Pull Request Guidelines
- Commit messages are short, imperative summaries (e.g., "Ignore generated orthophotos"). Group related changes per commit; commit `uv.lock` when dependency versions change.
- Before opening a PR: describe the change, list commands run, note data locations (not committed), and include any screenshots from Unity/GIS if visuals changed.
- Ensure raw datasets and large intermediates stay out of git; verify `.gitignore` still covers generated files after changes.
## Security & Data Handling
- Keep raw geodata local; avoid publishing source tiles or credentials. Document download sources/scripts instead of committing data.
- Outputs may be large; prefer syncing `export_unity/` artifacts via project-specific channels rather than embedding in the repo.
## Pipeline Behavior (for maintainers)
- Heightmaps: a VRT of all DGM1 tiles is warped per tile footprint, scaled once using the global min/max from the VRT to `[0, 65535]`, and written with worldfiles. Manifest rows include bounds, global min/max, and `out_res`.
- Orthophotos: `export_orthophotos` reuses the manifest for target windows and will abort if it is missing; JPEGs are resampled to `ortho.out_res` with worldfiles and default JPEG quality 90.
- Temporary files are written to `work/*_tmp.tif` and cleaned with broad `*.aux.xml` patterns in `work/` and the raw DGM1 directory—avoid placing non-GDAL aux files there.
- `materialize_archives` unzips every `*.zip` under `archive/*` into the matching raw folders and copies `archive/dop20/filelist.txt` next to `raw/dop20/` for the downloader.
- `geodata_config.example.toml` includes `archives.dop20_filelist` for human reference; the dataclass ignores it, so keep the example in sync with actual CLI options rather than adding new unused keys.

115
CITATION.md Normal file
View File

@@ -0,0 +1,115 @@
# Geodata and Tooling Citation
## 1) LVermGeo RLP / GeoShop RLP (terrain, ortho, buildings)
Source: GeoShop RLP - Digitale Gelaendemodelle (DGM)
Source URL: https://geoshop.rlp.de/digitale_gelaendemodelle/digitale_gelaendemodelle_dgm.html
GeoShop entry point: https://geoshop.rlp.de/
Open data portal: https://www.lvermgeo.rlp.de/geodaten-geoshop/open-data
License: Data Licence Germany - Attribution - Version 2.0 (dl-de/by-2-0)
License URL: https://www.govdata.de/dl-de/by-2-0
Accessed: 2026-01-19
Products used:
- DGM1 (Digitales Gelaendemodell, GeoTIFF)
- DOM1 (Digitales Oberflaechenmodell, GeoTIFF)
- DOP20 (Digitale Orthophotos, JP2)
- 3D-Gebaeudemodelle LOD2 (CityGML)
- BDOM20RGBI (Gebaeude-/Objektpunktwolke, LAZ)
- LPOLPG (LAS-Punktwolke, kombiniert; abgeleitete Teilmengen: LPG/LPO)
GeoBasis download patterns used in the pipeline:
- DGM1: `https://geobasis-rlp.de/data/dgm1/current/tif/dgm01_32_{x}_{y}_1_rp.tif`
- DOM1: `https://geobasis-rlp.de/data/dom1/current/tif/dom1_32_{x}_{y}_1_rp_2020.tif`
- DOP20: `https://geobasis-rlp.de/data/dop20rgb/current/jp2/dop20rgb_32_{x}_{y}_2_rp_2023.jp2`
- LoD2 CityGML: `https://geobasis-rlp.de/data/geb3dlo/current/gml/LoD2_32_{x}_{y}_2_RP.gml`
- BDOM20RGBI: `https://geobasis-rlp.de/data/bdom20rgbi/current/las/bdom20rgbi_32_{x}_{y}_2_rp.laz`
- LPOLPG: `https://geobasis-rlp.de/data/las/current/las/lpolpg_32_{x}_{y}_1_rp.laz`
Required attribution (verbatim):
(c) GeoBasis-DE / LVermGeoRP 2026, dl-de/by-2-0, www.lvermgeo.rlp.de
If modified, append:
[Data modified] (or in German: [Daten bearbeitet])
## 2) HydroRIVERS (river network for erosion workflow)
Dataset: HydroRIVERS v1.0 (Europe tile used in pipeline)
Download URL used by pipeline: https://data.hydrosheds.org/file/HydroRIVERS/HydroRIVERS_v10_eu_shp.zip
Project page: https://www.hydrosheds.org
License: Covered by the HydroSHEDS License Agreement (see https://www.hydrosheds.org)
Accessed: 2026-01-23
Recommended citation (from HydroRIVERS Technical Documentation v1.0):
Lehner, B., Grill G. (2013): Global river hydrography and network routing: baseline data and new approaches to study the world's large river systems. Hydrological Processes, 27(15): 2171-2186. Data is available at www.hydrosheds.org.
## 3) HydroLAKES (lake depth support for erosion workflow)
Dataset: HydroLAKES v1.0 polygon database
Download URL used by pipeline: https://data.hydrosheds.org/file/hydrolakes/HydroLAKES_polys_v10_shp.zip
Project page: https://www.hydrosheds.org
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
License URL: https://creativecommons.org/licenses/by/4.0/
Accessed: 2026-01-23
Recommended citation (from HydroLAKES Technical Documentation v1.0):
Messager, M.L., Lehner, B., Grill, G., Nedeva, I., Schmitt, O. (2016): Estimating the volume and age of water stored in global lakes using a geo-statistical approach. Nature Communications: 13603. doi: 10.1038/ncomms13603. Data is available at www.hydrosheds.org.
## 4) GeoData pipeline tools used
Core geospatial and conversion tooling used by `GeoData/geodata_to_unity.py` and `GeoData/geodata_pipeline/`:
- Python (project runtime; configured for 3.11-3.12)
- `uv` (environment and dependency management)
- GDAL / OGR / OSR (`osgeo`) including `gdalbuildvrt`
- PDAL
- citygml-tools 2.4.0
- cjio (`cjio[export,reproject]`)
- laspy (`laspy[lazrs]`)
- NumPy
- SciPy
- scikit-learn
- Shapely
- trimesh
- requests
- tomli-w
Reference implementation locations:
- Dependencies: `GeoData/pyproject.toml`
- Main CLI: `GeoData/geodata_to_unity.py`
- Pipeline modules: `GeoData/geodata_pipeline/`
## 5) Numerical methods and related implementations (BibTeX)
```bibtex
@article{saetra2012,
title={Shallow Water Simulations on Multiple GPUs},
author={S{\ae}tra, ML and Brodtkorb, AR},
year={2012}
}
@book{toro2009,
title={Riemann Solvers and Numerical Methods for Fluid Dynamics},
author={Toro, EF},
edition={3},
year={2009}
}
@article{kader2022,
title={A New Variant of Rusanov Scheme},
author={Kader, ...},
journal={IJAM},
year={2022}
}
@article{audusse2004,
title={A Fast and Stable Well-Balanced Scheme...},
author={Audusse, E and Bouchut, F and ...},
year={2004}
}
@github{bshishov2019,
title={UnityTerrainErosionGPU},
author={Bshishov, A},
url={https://github.com/bshishov/UnityTerrainErosionGPU}
}
```

133
README.md
View File

@@ -1,96 +1,61 @@
## GeoData Toolkit
# GeoData
This repository converts DGM1 elevation tiles into Unity-ready 16-bit PNG heightmaps and a placement manifest. It relies on GDAL for mosaicking, resampling, and scaling to UInt16 ranges Unity expects.
Toolkit zum Export von Geodaten in Unity-kompatible Assets (`export_unity/...`).
### Prerequisites
- `uv` installed. Use Python 3.103.12 for now (`triangle2` from `cjio[export]` has no CPython 3.13 wheels).
- Optional: Java 17+ if you want to experiment with the bundled `citygml-tools` utilities (not needed for heightmaps/orthophotos).
## 1) Wie startet man das Projekt?
### Environment setup (uv)
- Install deps (creates `.venv` if missing): `uv sync`. You can skip manual activation by prefixing commands with `uv run ...`; if you prefer activation, run `uv venv && source .venv/bin/activate`.
- `uv run <cmd>` executes with the project environment (e.g., `uv run python geodata_to_unity.py --setup`). Use `--directory` to target another path if needed; `--offline` disables network fetches.
- If wheels fail to resolve, ensure system GDAL is present (e.g., `brew install gdal` or `apt-get install gdal-bin libgdal-dev`), then rerun `uv sync`.
- Create the default directory tree and config: `uv run python geodata_to_unity.py --setup` (or `bash scripts/setup_dirs.sh` for directories only).
**Voraussetzungen**
- Python `>=3.11,<3.13`
- `uv`
- GDAL (inkl. `osgeo`)
- Optional fuer Gebaeude: `cjio`, `citygml-tools`
### Repository Layout
- `raw/` — working inputs (not versioned): `raw/dgm1/`, `raw/dop20/jp2/`, `raw/citygml/lod1/`, `raw/citygml/lod2/`.
- `archive/` — offline storage for untouched downloads (e.g., zipped DOP/CityGML tiles, dop20 filelist).
- `work/` — intermediates such as `dgm.vrt` and `_tmp.tif` files; safe to delete/regenerate.
- `export_unity/height_png16/` — final 16-bit PNG heightmaps for Unity import.
- `export_unity/tile_index.csv` — manifest mapping tile IDs to world bounds and scaling ranges (`global_min/global_max` plus per-tile `tile_min/tile_max`), plus `tile_key`; built during heightmap export and required by orthophotos.
- `export_unity/ortho_jpg/` — cropped orthophoto tiles aligned to the terrain grid (JPEG + worldfiles).
- `geodata_to_unity.py` — main CLI (uses `geodata_pipeline/` library modules).
- `scripts/` — helpers to create the directory tree and fetch DOP20 inputs.
- `geodata_config.toml` — generated config (see `geodata_config.example.toml` for defaults).
- `AGENTS.md` — contributor guide.
**Schnellstart**
```bash
cd GeoData
uv sync
uv run python geodata_to_unity.py --setup
uv run python geodata_to_unity.py --export all
```
### Quick Start
1. Activate the uv venv (`source .venv/bin/activate`) or prefix commands with `uv run`.
2. Initialize config + directories: `uv run python geodata_to_unity.py --setup`.
3. Export assets (builds VRTs automatically if missing):
```bash
uv run python geodata_to_unity.py --export all
# heightmaps only: uv run python geodata_to_unity.py --export heightmap
# textures only: uv run python geodata_to_unity.py --export textures
# buildings: uv run python geodata_to_unity.py --export buildings
# trees (CSV+GLB): uv run python geodata_to_unity.py --export trees
```
4. Import the PNGs into Unity Terrains using `tile_index.csv` for placement and height scaling (065535, per-tile by default).
**Alternative mit Archivdaten**
```bash
uv run python geodata_to_unity.py --build-from-archive --export all
```
### How the export works
- Heightmaps: the pipeline builds `work/dgm.vrt` from all `raw/dgm1/*.tif`, computes a global min/max once (legacy fallback), and warps each tile footprint to `heightmap.out_res` with `srcNodata=-9999`. Per-tile min/max are computed from the warped tile and used to scale PNGs to `[0, 65535]` by default (`heightmap.use_tile_minmax=false` keeps global scaling). `export_unity/tile_index.csv` records `global_min/global_max`, `tile_min/tile_max`, and `tile_key = f"{floor((xmin + overlap_x) / tile_size_x)}_{floor((ymin + overlap_y) / tile_size_y)}"` (defaults: `tile_size_x=1000.0`, `tile_size_y=1000.0`, `overlap_x=0.5`, `overlap_y=0.5` in `[tile_key]`).
- Orthophotos: `work/dop.vrt` is built from `raw/dop20/jp2/*.jp2`; the manifest drives the cropping bounds. JPEG tiles are written to `export_unity/ortho_jpg/` with matching `.jgw` worldfiles. If the manifest is missing, the orthophoto export aborts—run the heightmap export first or use `--export all`.
- Archives: `--build-from-archive` supports a monolithic ZIP (`archive/archive_raw.zip`) and expands every `*.zip` under `archive/*` into the matching `raw/*` directories; dataset zips overlay the monolithic data. It also copies `archive/dop20/filelist.txt` next to `raw/dop20/` for the downloader.
- Cleanup: temporary `_tmp.tif` and GDAL aux XML files under `work/` and `raw/dgm1/` are removed at the end of the heightmap export; avoid storing non-GDAL metadata in those folders.
## 2) Welche Funktionen umfasst das Projekt und wie werden diese genutzt?
### Key Commands
- Refresh VRT: `gdalbuildvrt work/dgm.vrt raw/dgm1/*.tif`
- Run export pipeline: `uv run python geodata_to_unity.py --export all`
- Inspect an output tile: `gdalinfo export_unity/height_png16/<tile>.png | head`
- Override config paths: use `--config <path>`, `--raw-dgm1-path <dir>`, `--raw-dop20-path <dir>`.
- Build raws from archives: `uv run python geodata_to_unity.py --build-from-archive --export all` (uses `archive/archive_raw.zip` when present, then overlays `archive/*/*.zip`).
- Deterministic submission rebuild: `uv run python geodata_to_unity.py --build-from-archive --clean-raw --validate --export all --force-vrt`.
- Rebuild VRTs after moving data: add `--force-vrt`.
```bash
# Hoehenmodelle (PNG16) + tile_index.csv
uv run python geodata_to_unity.py --export heightmap
### Workflow Notes
- Heightmaps normalize per tile using `tile_min/tile_max` by default; set `heightmap.use_tile_minmax=false` to restore global scaling across tiles. Adjust `heightmap.out_res` or `heightmap.resample` in `geodata_config.toml` if your AOI or target resolution changes.
- `tile_key` config controls the tile grouping key in the manifest; defaults are `tile_size_x=1000.0`, `tile_size_y=1000.0`, `overlap_x=0.5`, `overlap_y=0.5` with `enabled=true`.
- `_tmp.tif` files in `work/` are transient; you can delete `work/` to force a clean rebuild.
- Keep file names stable to avoid churn in Unity scenes; re-exports overwrite in place.
- Large raw datasets are intentionally excluded from version control—document download sources or scripts instead of committing data.
- Additional inputs: download helper lives in `scripts/dlscript_dop20.sh` and pulls JP2/J2W/XML orthophotos listed in `archive/dop20/filelist.txt` (one URL per line); `archive/` can hold zipped 3D building tiles for future use.
- `--clean-raw` only removes managed ingestion dirs (`raw/dgm1`, `raw/dop20/jp2`, `raw/dop20/j2w`, `raw/dop20/meta`, `raw/citygml/lod1`, `raw/citygml/lod2`) and intentionally keeps custom masks.
- `--validate` writes `work/archive_materialize_report.json` and fails only when core datasets are missing (`dgm1 tif`, `dop20 jp2`, `citygml lod2`); optional sidecar gaps are warnings.
- Handoff to Unity: copy/sync `export_unity/height_png16/` and `export_unity/tile_index.csv` into `DTrierFlood/Assets/GeoData/` before running the Unity-side importer. Keep `heightmap.out_res` aligned with the importers expected resolution (currently 1025).
# Orthofotos (JPG), nutzt tile_index.csv
uv run python geodata_to_unity.py --export textures
### Orthophotos (textures)
1. Ensure DOP assets are present in `raw/dop20/jp2/`, `raw/dop20/j2w/`, and `raw/dop20/meta/`; use `scripts/dlscript_dop20.sh` to fetch JP2/J2W/XML entries listed in `archive/dop20/filelist.txt` (one URL per line).
2. From `GeoData/`, run:
```bash
uv run python geodata_to_unity.py --export textures
```
This builds `work/dop.vrt` if missing and writes `export_unity/ortho_jpg/<tile>.jpg` + `.jgw` aligned to `tile_index.csv`.
- If you see `Computed -srcwin ... falls partially outside source raster extent` warnings, the DOP coverage is slightly smaller than the tile footprint; edge pixels will be filled with NoData/zeros. Add adjacent JP2s or shrink the requested window if you need to avoid the warning.
- The download script relies on a Linux/OpenSSL toolchain with system CA bundle at `/etc/ssl/certs/ca-certificates.crt`; it builds a trust chain by fetching the geobasis intermediate. macOS/Windows users should either provide a combined CA via `CURL_CA_BUNDLE` or download with a browser/wget and place files manually.
- Place companion `.j2w` and `.xml` files under `raw/dop20/j2w/` and `raw/dop20/meta/` if available; they are not required for the VRT but help provenance.
# Gebaeude (GLB) aus CityGML LoD2
uv run python geodata_to_unity.py --export buildings
### Downloads (raw data)
- Run: `uv run python geodata_to_unity.py --download` (uses `geodata_download.toml`).
- Shows a progress bar while downloading and exits cleanly on Ctrl+C (exit code 130).
# Baeume (CSV + GLB)
uv run python geodata_to_unity.py --export trees
```
### Buildings (automated exporter)
- Run: `uv run python geodata_to_unity.py --export buildings`
- What it does per tile:
- Converts LoD2 CityGML → CityJSON (citygml-tools), triangulates with cjio, rebases to tile-local XY using `tile_index.csv`.
- Merges all buildings into one GLB (1 mesh with roof/wall primitives), decimates to the configured triangle budget.
- Roofs: planar UVs from tile-local XY, embedded per-tile DOP20 orthophoto as base color (unlit by default).
- Walls: vertex colors sampled from the ortho as a fallback (neutral otherwise).
- Coordinates: glTF-friendly (x=east, y=height, z=-north) so glTFast instantiates one GameObject with two submeshes.
- Requirements: LoD2 GMLs under `raw/citygml/lod2/`, per-tile orthos in `export_unity/ortho_jpg/`, tools on PATH (`tools/citygml-tools-2.4.0/citygml-tools`, `cjio` or override via `CJIO` env).
- Open items: richer wall coloring from BDOM/LPO facades, better simplification, footprint-aware ground snapping beyond the current clamp-to-ground.
**Hauptausgaben**
- `export_unity/height_png16/` und `export_unity/tile_index.csv`
- `export_unity/ortho_jpg/`
- `export_unity/buildings_tiles/`
- `export_unity/trees*/`
### Troubleshooting
- Empty raw directories cause VRT creation to fail fast (`No sources available to build VRT`); populate inputs or adjust `--raw-*` overrides.
- If you moved raw data or deleted `work/`, add `--force-vrt` to rebuild VRTs before exporting.
- Orthophoto export warnings like `Computed -srcwin ... falls partially outside source raster extent` indicate coverage gaps; add neighboring JP2s or accept the NoData edge fill.
- If GDAL Python bindings are missing, install system GDAL first and re-run `uv sync` so `osgeo` imports succeed.
## 3) Welche externen Bibliotheken und Quellen wurden verwendet?
**Bibliotheken/Tools**
- Python
- uv
- GDAL
- cjio
- citygml-tools
**Externe Datenquellen (Kategorien)**
- DGM1 (GeoTIFF) in `raw/dgm1/`
- DOP20 (JP2) in `raw/dop20/jp2/`
- CityGML LoD2 in `raw/citygml/lod2/`
- Optional: DOM1, BDOM, LPO/LPG fuer Baum-/Masken-Workflows

View File

@@ -425,8 +425,18 @@ def _apply_water_mask_to_ortho(tile_id: str, mask: np.ndarray, cfg: Config) -> N
mask_res = np.clip(mask_res, 0.0, 1.0)
threshold = float(cfg.ortho.water_mask_threshold)
water_mask = mask_res >= threshold
trans_opts = gdal.TranslateOptions(
outputType=gdal.GDT_Byte,
format="JPEG",
creationOptions=[f"QUALITY={cfg.ortho.jpeg_quality}", "WORLDFILE=YES"],
)
if not np.any(water_mask):
try:
gdal.Translate(out_path, ds, options=trans_opts)
except RuntimeError as exc:
print(f"[river_erosion] Ortho write failed for {tile_id}: {exc}")
ds = None
print(f"[river_erosion] Wrote ortho {out_path} (copied; empty mask)")
return
mode = str(getattr(cfg.ortho, "water_color_mode", "median") or "median").lower()
@@ -455,11 +465,6 @@ def _apply_water_mask_to_ortho(tile_id: str, mask: np.ndarray, cfg: Config) -> N
for c in range(3):
out_ds.GetRasterBand(c + 1).WriteArray(rgb[c])
trans_opts = gdal.TranslateOptions(
outputType=gdal.GDT_Byte,
format="JPEG",
creationOptions=[f"QUALITY={cfg.ortho.jpeg_quality}", "WORLDFILE=YES"],
)
try:
gdal.Translate(out_path, out_ds, options=trans_opts)
except RuntimeError as exc:

View File

@@ -0,0 +1,113 @@
#!/usr/bin/env python3
"""Rebuild river-masked orthophotos from existing tile masks."""
from __future__ import annotations
import argparse
import csv
from pathlib import Path
import numpy as np
from osgeo import gdal
from geodata_pipeline.config import Config
from geodata_pipeline.river_erosion import _apply_water_mask_to_ortho
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(
description="Rebuild export_unity/ortho_jpg_river from work/river_masks and export_unity/ortho_jpg."
)
p.add_argument("--config", default="geodata_config.toml", help="Path to pipeline config TOML.")
p.add_argument("--manifest", default="", help="Optional manifest CSV override.")
p.add_argument("--mask-dir", default="", help="Optional mask directory override.")
p.add_argument("--output-dir", default="", help="Optional output ortho_jpg_river directory override.")
p.add_argument("--skip-existing", action="store_true", help="Skip tiles that already exist in output.")
p.add_argument("--mask-res", type=int, default=0, help="Fallback mask resolution when a tile mask is missing.")
return p.parse_args()
def load_tile_ids(manifest_path: Path) -> list[str]:
with manifest_path.open("r", encoding="utf-8", newline="") as f:
rows = list(csv.DictReader(f))
return [row["tile_id"].strip() for row in rows if row.get("tile_id")]
def load_mask(mask_path: Path) -> np.ndarray | None:
ds = gdal.Open(str(mask_path), gdal.GA_ReadOnly)
if ds is None:
return None
arr = ds.ReadAsArray()
ds = None
if arr is None:
return None
if arr.ndim == 3:
arr = arr[0]
arr = arr.astype(np.float32)
if arr.max() > 1.0:
arr /= 255.0
return np.clip(arr, 0.0, 1.0)
def main() -> int:
args = parse_args()
cfg = Config.load(args.config)
gdal.UseExceptions()
manifest_path = Path(args.manifest or cfg.export.manifest_path)
if not manifest_path.exists():
raise SystemExit(f"[rebuild_ortho_jpg_river] Missing manifest: {manifest_path}")
mask_dir = Path(args.mask_dir or Path(cfg.work.work_dir) / "river_masks")
output_dir = Path(args.output_dir or cfg.ortho.river_dir)
output_dir.mkdir(parents=True, exist_ok=True)
fallback_res = int(args.mask_res or cfg.heightmap.out_res)
if fallback_res <= 0:
fallback_res = 1025
tile_ids = load_tile_ids(manifest_path)
written = 0
skipped = 0
with_mask = 0
without_mask = 0
missing_source = 0
print(f"[rebuild_ortho_jpg_river] Tiles in manifest: {len(tile_ids)}")
print(f"[rebuild_ortho_jpg_river] Mask dir: {mask_dir}")
print(f"[rebuild_ortho_jpg_river] Output dir: {output_dir}")
for tile_id in tile_ids:
out_path = output_dir / f"{tile_id}.jpg"
if args.skip_existing and out_path.exists():
skipped += 1
continue
source_path = Path(cfg.export.ortho_dir) / f"{tile_id}.jpg"
if not source_path.exists():
missing_source += 1
print(f"[rebuild_ortho_jpg_river] Missing source ortho for {tile_id}: {source_path}")
continue
mask_path = mask_dir / f"{tile_id}_mask.png"
mask = load_mask(mask_path)
if mask is None:
without_mask += 1
mask = np.zeros((fallback_res, fallback_res), dtype=np.float32)
else:
with_mask += 1
_apply_water_mask_to_ortho(tile_id, mask, cfg)
if out_path.exists():
written += 1
print(
"[rebuild_ortho_jpg_river] Summary: "
f"written={written}, skipped={skipped}, with_mask={with_mask}, "
f"without_mask={without_mask}, missing_source={missing_source}"
)
return 0 if missing_source == 0 else 2
if __name__ == "__main__":
raise SystemExit(main())