cleanup

AGENTS.md

@@ -1,55 +0,0 @@
-# Repository Guidelines
-
-## Project Structure & Module Organization
-- `geodata_to_unity.py` is the main CLI; library code lives in `geodata_pipeline/` (`heightmaps.py`, `orthophotos.py`, `config.py`, `setup_helpers.py`). Legacy wrapper scripts have been removed; use `geodata_to_unity.py` directly.
-- Working inputs (ignored): `raw/dgm1/`, `raw/dop20/jp2/`, `raw/citygml/lod1/`, `raw/citygml/lod2/`.
-- Archives (ignored): `archive/dgm1/`, `archive/dop20/`, `archive/citygml/lod1/`, `archive/citygml/lod2/` (zip storage + dop20 filelist).
-- Config: `geodata_config.toml` (generated) or `geodata_config.example.toml` for defaults.
-- `export_unity/` is generated output (heightmaps, orthophotos, manifest). `work/` holds intermediates and is disposable.
-
-## Build, Test, and Development Commands
-- Create a venv: `uv venv && source .venv/bin/activate`. Install deps: `uv sync` (generates `uv.lock`).
-- If wheels fail, install system GDAL first (e.g., `brew install gdal` or `apt-get install gdal-bin libgdal-dev`), then rerun `uv sync`.
-- Prepare the directory tree and config: `uv run python geodata_to_unity.py --setup` (or `bash scripts/setup_dirs.sh` for directories only).
-- Heightmap export: `uv run python geodata_to_unity.py --export heightmap`.
-- Orthophoto export: `uv run python geodata_to_unity.py --export textures` (requires JP2s under `raw/dop20/jp2/`; use `bash scripts/dlscript_dop20.sh` to fetch JP2/J2W/XML listed in `archive/dop20/filelist.txt`).
-- Refresh VRT manually if needed: `gdalbuildvrt work/dgm.vrt raw/dgm1/*.tif`.
-- Inspect a result: `gdalinfo export_unity/height_png16/<tile>.png | head` to sanity-check bounds and scaling.
-- Populate raw data from archives: `uv run python geodata_to_unity.py --build-from-archive --export all` (unzips `archive/*`; dop20 filelist stays in archive for the downloader).
-- Rebuild VRTs after moving data: add `--force-vrt`.
-- Expected warning: `Computed -srcwin ... falls partially outside source raster extent` means the DOP coverage is slightly smaller than the tile footprint; edge pixels will be filled with NoData/zeros. Add adjacent JP2s or shrink the requested window if you need to silence it.
-- Scripts accept CLI overrides (e.g., `--config`, `--raw-dgm1-path`, `--raw-dop20-path`, `--export`, `--build-from-archive`); run `uv run python geodata_to_unity.py -h` to see options.
-- DOP20 downloader assumes Linux/OpenSSL with system CA at `/etc/ssl/certs/ca-certificates.crt` to build a trust chain from the geobasis site. macOS/Windows users should either set `CURL_CA_BUNDLE` to a combined CA or download manually and place files in `raw/dop20/`.
-- Orthophotos depend on a prebuilt manifest: run the heightmap export first (or `--export all`) so `export_unity/tile_index.csv` exists.
-- VRTs are built from whatever is present in the raw directories; empty directories will fail fast. Use `--force-vrt` after moving data or deleting `work/`.
-
-## Coding Style & Naming Conventions
-- Python scripts use 4-space indentation, early-exit error handling, and `SystemExit` for fatal issues; follow PEP 8 where practical.
-- Keep tile IDs stable (base filename without extension); avoid renaming inputs to reduce churn downstream.
-- Prefer explicit config fields for tunables (`heightmap.out_res`, `heightmap.resample`, `ortho.out_res`, `ortho.jpeg_quality`) and log with clear context.
-
-## Testing Guidelines
-- No automated tests yet; rely on manual validation: run exports on a small tile subset, open outputs in `gdalinfo` or GIS viewer, and confirm `tile_index.csv` aligns with Unity expectations.
-- When changing scaling or resolution, compare before/after stats (min/max) and spot-check terrain in Unity.
-
-## Current Pipelines (v1 WIP)
-- **Heightmaps/Orthos**: unchanged; see README.
-- **Buildings (new)**: `--export buildings` converts LoD2 CityGML → CityJSON (citygml-tools), triangulates (cjio), rebases to tile-local XY, merges per tile into one GLB (1 mesh, roof/wall primitives), decimates to budget, planar-UV roofs with embedded DOP20 tile texture (unlit), walls colored from ortho fallback, axes glTF-friendly (x=east, y=height, z=-north). Open: better wall coloring (BDOM/LPO), stronger simplification, footprint-aware ground snap (currently clamp to DTM).
-- **Trees (new)**: `--export trees` uses DOM1–DGM1 CHM + CityGML building mask (buffered) to find trees, roughness/confidence heuristic with optional LPO/BDOM boost, caps by count. Outputs per-tile CSV and chunked GLBs (4×4 by default) built from 16 procedural proxies; instancing toggle in config; shared proxy library emitted.
-- **Tools**: expects `citygml-tools-2.4.0/citygml-tools` and `cjio` on PATH (override `CJIO` env). Orthos must exist for roofs/walls.
-
-## Commit & Pull Request Guidelines
-- Commit messages are short, imperative summaries (e.g., "Ignore generated orthophotos"). Group related changes per commit; commit `uv.lock` when dependency versions change.
-- Before opening a PR: describe the change, list commands run, note data locations (not committed), and include any screenshots from Unity/GIS if visuals changed.
-- Ensure raw datasets and large intermediates stay out of git; verify `.gitignore` still covers generated files after changes.
-
-## Security & Data Handling
-- Keep raw geodata local; avoid publishing source tiles or credentials. Document download sources/scripts instead of committing data.
-- Outputs may be large; prefer syncing `export_unity/` artifacts via project-specific channels rather than embedding in the repo.
-
-## Pipeline Behavior (for maintainers)
-- Heightmaps: a VRT of all DGM1 tiles is warped per tile footprint, scaled once using the global min/max from the VRT to `[0, 65535]`, and written with worldfiles. Manifest rows include bounds, global min/max, and `out_res`.
-- Orthophotos: `export_orthophotos` reuses the manifest for target windows and will abort if it is missing; JPEGs are resampled to `ortho.out_res` with worldfiles and default JPEG quality 90.
-- Temporary files are written to `work/*_tmp.tif` and cleaned with broad `*.aux.xml` patterns in `work/` and the raw DGM1 directory—avoid placing non-GDAL aux files there.
-- `materialize_archives` unzips every `*.zip` under `archive/*` into the matching raw folders and copies `archive/dop20/filelist.txt` next to `raw/dop20/` for the downloader.
-- `geodata_config.example.toml` includes `archives.dop20_filelist` for human reference; the dataclass ignores it, so keep the example in sync with actual CLI options rather than adding new unused keys.

geodata_pipeline/river_erosion.py

@@ -425,8 +425,18 @@ def _apply_water_mask_to_ortho(tile_id: str, mask: np.ndarray, cfg: Config) -> N
     mask_res = np.clip(mask_res, 0.0, 1.0)
     threshold = float(cfg.ortho.water_mask_threshold)
     water_mask = mask_res >= threshold
+    trans_opts = gdal.TranslateOptions(
+        outputType=gdal.GDT_Byte,
+        format="JPEG",
+        creationOptions=[f"QUALITY={cfg.ortho.jpeg_quality}", "WORLDFILE=YES"],
+    )
     if not np.any(water_mask):
+        try:
+            gdal.Translate(out_path, ds, options=trans_opts)
+        except RuntimeError as exc:
+            print(f"[river_erosion] Ortho write failed for {tile_id}: {exc}")
         ds = None
+        print(f"[river_erosion] Wrote ortho {out_path} (copied; empty mask)")
         return
 
     mode = str(getattr(cfg.ortho, "water_color_mode", "median") or "median").lower()
@@ -455,11 +465,6 @@ def _apply_water_mask_to_ortho(tile_id: str, mask: np.ndarray, cfg: Config) -> N
     for c in range(3):
         out_ds.GetRasterBand(c + 1).WriteArray(rgb[c])
 
-    trans_opts = gdal.TranslateOptions(
-        outputType=gdal.GDT_Byte,
-        format="JPEG",
-        creationOptions=[f"QUALITY={cfg.ortho.jpeg_quality}", "WORLDFILE=YES"],
-    )
     try:
         gdal.Translate(out_path, out_ds, options=trans_opts)
     except RuntimeError as exc:

scripts/rebuild_ortho_jpg_river.py

@@ -0,0 +1,113 @@
+#!/usr/bin/env python3
+"""Rebuild river-masked orthophotos from existing tile masks."""
+
+from __future__ import annotations
+
+import argparse
+import csv
+from pathlib import Path
+
+import numpy as np
+from osgeo import gdal
+
+from geodata_pipeline.config import Config
+from geodata_pipeline.river_erosion import _apply_water_mask_to_ortho
+
+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(
+        description="Rebuild export_unity/ortho_jpg_river from work/river_masks and export_unity/ortho_jpg."
+    )
+    p.add_argument("--config", default="geodata_config.toml", help="Path to pipeline config TOML.")
+    p.add_argument("--manifest", default="", help="Optional manifest CSV override.")
+    p.add_argument("--mask-dir", default="", help="Optional mask directory override.")
+    p.add_argument("--output-dir", default="", help="Optional output ortho_jpg_river directory override.")
+    p.add_argument("--skip-existing", action="store_true", help="Skip tiles that already exist in output.")
+    p.add_argument("--mask-res", type=int, default=0, help="Fallback mask resolution when a tile mask is missing.")
+    return p.parse_args()
+
+
+def load_tile_ids(manifest_path: Path) -> list[str]:
+    with manifest_path.open("r", encoding="utf-8", newline="") as f:
+        rows = list(csv.DictReader(f))
+    return [row["tile_id"].strip() for row in rows if row.get("tile_id")]
+
+
+def load_mask(mask_path: Path) -> np.ndarray | None:
+    ds = gdal.Open(str(mask_path), gdal.GA_ReadOnly)
+    if ds is None:
+        return None
+    arr = ds.ReadAsArray()
+    ds = None
+    if arr is None:
+        return None
+    if arr.ndim == 3:
+        arr = arr[0]
+    arr = arr.astype(np.float32)
+    if arr.max() > 1.0:
+        arr /= 255.0
+    return np.clip(arr, 0.0, 1.0)
+
+
+def main() -> int:
+    args = parse_args()
+    cfg = Config.load(args.config)
+    gdal.UseExceptions()
+
+    manifest_path = Path(args.manifest or cfg.export.manifest_path)
+    if not manifest_path.exists():
+        raise SystemExit(f"[rebuild_ortho_jpg_river] Missing manifest: {manifest_path}")
+
+    mask_dir = Path(args.mask_dir or Path(cfg.work.work_dir) / "river_masks")
+    output_dir = Path(args.output_dir or cfg.ortho.river_dir)
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    fallback_res = int(args.mask_res or cfg.heightmap.out_res)
+    if fallback_res <= 0:
+        fallback_res = 1025
+
+    tile_ids = load_tile_ids(manifest_path)
+    written = 0
+    skipped = 0
+    with_mask = 0
+    without_mask = 0
+    missing_source = 0
+
+    print(f"[rebuild_ortho_jpg_river] Tiles in manifest: {len(tile_ids)}")
+    print(f"[rebuild_ortho_jpg_river] Mask dir: {mask_dir}")
+    print(f"[rebuild_ortho_jpg_river] Output dir: {output_dir}")
+
+    for tile_id in tile_ids:
+        out_path = output_dir / f"{tile_id}.jpg"
+        if args.skip_existing and out_path.exists():
+            skipped += 1
+            continue
+
+        source_path = Path(cfg.export.ortho_dir) / f"{tile_id}.jpg"
+        if not source_path.exists():
+            missing_source += 1
+            print(f"[rebuild_ortho_jpg_river] Missing source ortho for {tile_id}: {source_path}")
+            continue
+
+        mask_path = mask_dir / f"{tile_id}_mask.png"
+        mask = load_mask(mask_path)
+        if mask is None:
+            without_mask += 1
+            mask = np.zeros((fallback_res, fallback_res), dtype=np.float32)
+        else:
+            with_mask += 1
+
+        _apply_water_mask_to_ortho(tile_id, mask, cfg)
+        if out_path.exists():
+            written += 1
+
+    print(
+        "[rebuild_ortho_jpg_river] Summary: "
+        f"written={written}, skipped={skipped}, with_mask={with_mask}, "
+        f"without_mask={without_mask}, missing_source={missing_source}"
+    )
+    return 0 if missing_source == 0 else 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())