- Prompt now requires year verification before extracting any field
- Opening times and prices from prior years must be nulled with a hint
- imageURLReachable does a HEAD request (5s timeout) and strips the
image_url from research results when the resource returns 4xx/5xx
All Input fields used market?.xxx as initial value, so a Svelte re-render
triggered by researchResult=null would reset them back to the server-loaded
value, wiping every applied research suggestion.
Replace all research-applicable fields with $state variables and route all
apply calls through setField() instead of querySelector+dispatch. Country
name->code mapping added for LLM-returned values like "Deutschland" -> "DE".
writeReverseResult also updated to use setField.
Description wasn't being applied because querySelector-then-assign runs before
Svelte's reactive flush of researchResult=null, which resets the textarea to
its initial market.description value. Fix: reactive state + exported setter
(same pattern as setHours/setAdmission).
Also add markt_name to felder in both schemas and the prompt so the LLM can
suggest a name correction. Name suggestions are gated to extraktion=direkt
(high confidence only) and guarded on the frontend with setName().
The beschreibung field was schema-required but absent from ## Felder,
causing the LLM to always return null. Add explicit extraction instruction.
Also reword the opening line which said "Keine Beschreibungstexte" —
contradicting the field we actually want.
On apply, append "KI-Recherche: DD.MM.YYYY HH:MM" to admin_notes so
there's a permanent audit trail of when research was run.
Researcher emits {datum_von,von,bis} for opening hours and [{name,betrag,waehrung}]
for admission info — both incompatible with the form's {day,open,close} and
AdmissionInfo shapes. Normalize on apply; extend normalizeDayName to handle
ISO YYYY-MM-DD dates the LLM produces. ResearchPanel renders both LLM and
form-native formats with dedicated table/list views.
Gemini rejects requests that set both GoogleSearchRetrieval and
response_schema. The orchestrator already provides web content via
SearxNG + scraping, so grounding is unnecessary here.
factory.go: treat DB errors from GetGeminiAPIKey as "no key" and fall
back to the GEMINI_API_KEY env var instead of propagating the error
(which caused a panic/crash when migrations haven't been run yet).
gemini.go: ListModelNames returns a ProviderError when the client is
nil so that connected=false is reported correctly in GetAI instead of
the previous nil,nil→connected=true false positive.
+page.server.ts: catch fetch errors so a backend outage doesn't 500 the
whole page. +page.svelte: guard all data.ai access with {#if data.ai}
so the page renders an error banner instead of crashing on null access.
When a freshly-inserted discovered_market has a matched series, konfidenz
"hoch" (≥2 sources), and both start/end dates present, Accept() is called
inline with a nil reviewer (mapped to NULL reviewed_by) so the row goes
straight to accepted without manual review.
CrawlSummary gains auto_accepted counter; slog summary logs it.
MarkAccepted / Service.Accept now take *uuid.UUID for reviewer so nil
cleanly maps to NULL in the DB column (already nullable).
Replace the Mistral + Ollama AI stack with a single Google Gemini provider
backed by google.golang.org/genai. API key moves from env/Helm to the DB
(AES-256-GCM, key derived from JWT_SECRET via HKDF) so it can be rotated
via the admin UI without a pod restart.
New:
- pkg/crypto/secretbox — AES-256-GCM encrypt/decrypt for secrets at rest
- pkg/ai/gemini — GeminiProvider with grounding, structured output, usage
recording, and hot-reload (Reinitialize swaps client under mutex)
- pkg/ai/usage — UsageRecorder interface + UsageEvent struct
- domain/settings/store — DB-backed settings (model, grounding toggle, key)
- domain/settings/usage — UsageRepo implementing UsageRecorder; ai_usage table
- migrations 000021 (system_settings) + 000022 (ai_usage)
- settings API: GET /ai, POST /ai/key, POST /ai/model, POST /ai/grounding,
GET /ai/usage
- admin UI: 4-card settings page — provider status, model selector, grounding
toggle with quota, usage rollups + recent-calls table
Removed:
- pkg/ai/ollama, mistral_provider, ratelimiter (+ tests)
- Helm AI_API_KEY, AI_PROVIDER, AI_MODEL_COMPLEX, AI_AGENT_DISCOVERY,
AI_RATE_LIMIT_RPS env vars
Call sites set Grounded+CallType: research (true/"research"), enrich Pass B
(true/"enrich_b"), similarity (false/"similarity"). Integration test updated
to use a stub ai.Provider instead of a fake Ollama HTTP server.
Run CrawlEnrich + Nominatim geocoding in the background immediately
after a crawl discovers new rows. Manual triggers via the
/enrichment/crawl-all endpoint remain for backfills but are no longer
needed for fresh crawls.
Add /admin/settings/ai endpoint (GET status + available models, POST
model switch). OllamaProvider gains SetModel/Model/ListModels with a
RWMutex so the active model can be swapped at runtime without restart.
New /admin/einstellungen page shows provider, connection badge, and a
model dropdown that calls the API on submit.
Transform raw LLM felder output into FieldSuggestion[] for the UI panel.
Skip suggestions identical to current market values. Add beschreibung to
both schemas, the Go struct, and the transformation mapping so description
is extracted during research. Fix field labels (Land, Startdatum, Enddatum)
in ResearchPanel.
Ollama's llama.cpp grammar converter supports anyOf with primitive
null — use it for all nullable wert/hinweis fields instead of
type:string-only, so constrained decoding emits JSON null directly.
This also fixes the orchestrator test fixture which uses JSON null
for optional wert fields.
Adds pkg/search (SearxNG impl), domain/market/research (orchestrator + embedded
German prompt and JSON schema), and reinstates POST /markets/:id/research on
top of the new pipeline. Seeds URLs from crawler provenance; falls back to
search when fewer than two distinct seed domains are known.
Replaces the Mistral-only ai.Client with an ai.Provider interface backed by
Ollama and Mistral implementations. Migrates enrichment + similarity callers
to ai.Provider.Chat. Research endpoint returns 501 until commit 2 reinstates
it on the new orchestrator.
Complements the existing forward geocoder with Nominatim's /reverse
endpoint so the admin edit form can populate the address from
coordinates (useful when a crawl gave us lat/lng but no street,
e.g. after running crawl-enrich).
Backend:
- geocode.Reverse(ctx, lat, lng) hits Nominatim /reverse with
addressdetails=1 and accept-language=de, reuses the 1 rps mutex
already guarding forward calls. Falls through city → town →
village → municipality → hamlet for small places. Returns nil
when Nominatim has no match so callers can distinguish "no hit"
from "all-empty address."
- New DTOs ReverseGeocodeRequest/Response.
- GeocodeHandler.ReverseGeocode wired at POST /reverse-geocode
behind the same geocodeLimit middleware as /geocode.
Frontend:
- /api/reverse-geocode SvelteKit proxy mirrors /api/geocode.
- MarketForm gets a second button next to "Koordinaten aus Adresse
ermitteln" — "Adresse aus Koordinaten ermitteln". Writes non-empty
street/city/zip back into the form; empty result surfaces
"Keine Adresse gefunden."
Accepting a row triggered a background POST to
/admin/markets/<edition>/research. The intent was to "warm up" the
edit page, but the result was discarded (fire-and-forget), the edit
page only renders research from its own form action, and the
backend's 5-minute-per-market cooldown still got set — so the
operator's first manual "Mit KI recherchieren" click hit "Bitte
warte 5 Minuten zwischen Recherche-Aufrufen" instead.
Removes the auto-fire. Research runs on user click. If we want
prefetched suggestions later, that needs server-side caching + a
load-time fetch, not fire-and-forget.
Pain: a 1400+ row pending queue can't finish crawl-enrich inside the
old 10-minute cap (Nominatim's 1 rps means ~23m minimum). Operators
saw a scary red "Crawl-enrich fehlgeschlagen: context deadline
exceeded" banner even though the pipeline is resumable.
- Introduce enrichAllTimeout constant (45m) sized for ~2700 rows per
press; the original 10m assumed 600 rows worst-case.
- On context.DeadlineExceeded, translate to a user-facing message
("Zeitlimit erreicht nach N von M Zeilen. Erneut starten, um die
verbleibenden Zeilen zu bearbeiten.") instead of raw Go error.
- Always stash the summary in handler state, even on error, so the
UI can show partial progress (N/M processed) alongside the message.
- Service: populate DurationMs on early-return too, so the status
endpoint's duration reflects the partial run instead of zero.
Behavior unchanged when a run finishes cleanly; the queue remains
resumable across presses as before.
The accept action redirected to /admin/maerkte/<id>/edit, but the
route is /admin/maerkte/[id]/bearbeiten — every other admin link
uses the German segment. Reviewers hit a 404 after every Accept.
Adds a Vorschau button to the detail drawer header that opens a
full-width modal showing an approximate public /markt/[slug] layout
for the candidate row. Lets reviewers sanity-check the user-facing
result before clicking Accept.
- DiscoveryPreview.svelte: renders title, date range, venue/PLZ/city
location line, organizer, description, opening hours, website link
and a Leaflet map pin (if lat/lng present). Banner calls out which
fields (street, admission prices, title image) will come later from
the organizer so the preview's gaps are not mistaken for bugs.
- DetailDrawer.svelte: adds previewOpen state, an eye-icon Vorschau
button next to Accept/Reject, and an overlay at z-60 over the
drawer. Backdrop click or ✕ closes the preview without closing the
drawer.
MergePendingSources re-aggregates the jsonb array with
ORDER BY source_name for DB determinism, but the admin UI treats
index 0 as "Rang 1 = winning source." Legacy auto-merged rows were
therefore surfacing mittelalterkalender (alphabetically first) as
Rang 1 instead of the actual rank-1 source mittelaltermarkt_online.
- Export crawler.SourceRank (was unexported rankOf) so other packages
in the discovery domain can reference the canonical rank map.
- scanDiscoveredMarket: sort.SliceStable SourceContributions by rank
after unmarshal. Every read path now sees contributions in rank
order regardless of how they were persisted; legacy rows
self-correct on next read, no migration needed.
- Wrap $state initializers that read props (MarketForm, ResearchPanel,
maerkte +page) in untrack() so Svelte 5 stops warning about
state_referenced_locally. Intent stays "take an initial snapshot of
the prop" — the warning existed to make that intent explicit.
- Add enrichment/crawl-all-status/+server.ts proxy route; the admin
discovery page was polling this path and getting 404s in a tight
loop because the equivalent SvelteKit proxy only existed for the
plain /crawl-status endpoint.
Discovery drawer
- Wrap each section in a rounded card so boundaries are visible without
parsing the uppercase headers.
- Header: N Quellen and enrichment_status become consistent pills,
matching the existing konfidenz pill treatment.
- Enrichment: replace the inline "(llm)"/"(crawl)" trailing text with a
color-coded badge on the label side (purple = llm, sky = crawl).
- Empty enrichment state now tells the operator how to trigger it.
- Audit timestamp uses a local-time helper so the displayed time
matches the browser timezone (was UTC-as-local).
- Quellen list: prefix each URL with its hostname for scannability;
long URLs truncated with full URL in the title attribute.
ContributionsPanel
- Amber border/background now only on conflict rows; every row
previously got border-amber-100 unconditionally, which diluted the
conflict signal. Rang 1 badge flipped to emerald so it reads as a
positive "winner" marker, not a warning.
Discovery page
- Remove dead dateInputValue() function and the stale
a11y_click_events_have_key_events suppression — both flagged by
eslint after earlier refactors.
- Render crawl/enrich timestamps in the browser's local timezone via a
new fmtLocalStamp helper; the previous .slice(0,16).replace('T',' ')
treated the ISO UTC string as if it were local time.
gin panics at startup with:
':aid' in new path '/api/v1/admin/discovery/queue/:aid/similar/:bid/classify'
conflicts with existing wildcard ':id' in existing prefix
'/api/v1/admin/discovery/queue/:id'
Gin's trie requires identical parameter names at the same prefix position.
All sibling routes use :id; the tiebreak route was registered with :aid,
crashing the server on every deploy since e0b73ac. Prod has been running
the pre-tiebreak image (52f3e4c0) the whole time because every Helm
upgrade crash-looped and rolled back.
Rename :aid to :id in both the route and the handler's c.Param read.
:bid is in a different slot and stays.
- Extract readJSONFile + writeJSONAtomic in cache.go; category cache
reuses them (saveCategoryCache is one line, loadCategoryCache uses
the standard load-or-empty shape).
- Drop dead errMsg param from scoreCategoryResult (always "").
- Wrap writeCategoryReport errors with context for consistency.
- Wrap runSimilarityMode / runCategoryMode's 5 per-mode flags into an
evalConfig struct so params don't drift.
- Promote validModes to a package-level var.
- Remove redundant cache = new...() fallback after load* (both load
helpers already return a non-nil empty cache on error).
- Strip narrating / diff-referencing comments per CLAUDE.md; keep the
one genuine WHY on normalizeCategory (divergence from normalize.Name).
Net -54 lines across 4 files; go build + go vet + tests green.
Ship 2 MR 5b. Extends discovery-eval with a second mode that grades
MistralLLMEnricher's category output against labelled ground truth.
Accuracy + per-label confusion matrix so mix-ups between similar
categories (mittelaltermarkt vs ritterfest, weihnachtsmarkt vs
kirchweih) are visible at a glance.
Usage:
-mode similarity — existing MR 5 path, unchanged.
-mode category — new: scrapes quellen URLs, asks LLM for
{category, opening_hours, description},
scores category only.
Structure
- main.go: split into runSimilarityMode + runCategoryMode. Both
share ai.Client construction and the ctx timeout (bumped to 15min
for category mode since scraping adds I/O). Mode dispatched on
-mode flag; unknown modes exit 2.
- category.go: fixture / cache / run / metrics / report — parallel
to the similarity files, not shared because the data shapes differ
enough that generics would add more noise than they save. Cache
key is sha256(markt_name_lower|stadt_lower|year|model); separate
from SimilarityPairKey since that one takes two rows.
- fixtures/category.json: 10 hand-labelled DACH-market rows
exercising the categories we expect the LLM to produce —
mittelaltermarkt, weihnachtsmarkt, ritterfest, ritterturnier,
handwerkermarkt, schlossfest, kirchweih. Each row lists a quelle
URL the enricher will scrape live (first run only; cache takes
over after).
- normalizeCategory: strips casing + German umlauts + the -märkte
plural drift so a correctly-categorised row doesn't get scored
wrong for cosmetic LLM output variation.
Metrics: Accuracy + per-label confusion matrix. Confusion format is
`want → predictions` with `!` markers on off-diagonal predictions —
readable in a terminal, machine-parseable in the JSON report.
Mismatches are listed at the end with want/got pairs so operators
can spot prompt failures and patch either the prompt or the fixture.
Threshold gate reads accuracy (not F1) — category is multi-class,
precision/recall don't have a single-label meaning.
Tests: normalisation edge cases (casing, umlaut, plural, trimming),
scoring drift tolerance, metrics counts + confusion matrix shape,
errors excluded from confusion, cache round-trip + model scoping,
missing/corrupt file handling.
.gitignore adds .cat-eval-cache.json and cat-eval-report.json.
Follow-ups (MR 5c / later): opening_hours and description scoring.
Both need fuzzier matching (regex structure vs LLM judge) which is
its own design problem.
Ship 2 MR 8. Operator-productivity layer on top of the detail drawer:
j/k to walk rows, Enter to open, a/r to accept-reject the selection,
e/s to jump into the drawer with AI enrich / Similar already visible,
? for a help modal listing everything. Escape closes the drawer (or
the help modal if it's open).
Implementation
- selectedId $state drives a subtle indigo ring on the highlighted
row. Follows drawerId when the drawer opens so Esc → j leaves you
on the same row. Auto-resets to queue[0] if the selected row
scrolls off the page (pagination / refresh).
- Global <svelte:window onkeydown> listener. isTypingTarget() bails
out when focus is inside an input/textarea/select/contenteditable
so typing in the drawer's edit form doesn't trigger shortcuts.
Cmd/Ctrl/Alt combos also skipped so browser shortcuts stay intact.
- selectRelative() updates selectedId + scrolls the row into view
(block: 'nearest') so keyboard-driven scanning through a long
queue keeps the highlight visible.
- submitRowAction() builds + submits a hidden <form> for a/r so the
SvelteKit action pipeline (invalidations, form result propagation)
runs the same way a button click would.
Decisions baked in
- 'e' (AI enrich) and 's' (Similar) open the drawer rather than
firing the LLM call directly. LLM calls cost money; keeping the
UI explicit avoids hidden side effects from a misclick.
- Persistent '?' button bottom-right for discoverability — operators
shouldn't have to read docs to find the help.
- Modal uses click-outside-to-dismiss + Esc + ✕ button, all three.
No backend changes. Frontend-only.
Ship 2 MR 6. Consolidates every market-specific action that used to
expand into the queue table into a single side drawer. Queue rows
keep Accept/Reject for fast-path review; clicking anywhere else on a
row opens the drawer with the full context.
State via URL param ?drawer=<id>. F5 preserves the open row; links
like /admin/discovery?drawer=<uuid>&sort=konfidenz are shareable and
compose with existing pagination/sort state.
DetailDrawer.svelte (new) sections:
- Header: name, konfidenz, source count, Accept/Reject, close (✕)
- Identity: editable form (name, stadt, bundesland, start/end, website)
- Enrichment: full payload with per-field provenance tags + AI enrich
button; "Noch keine Enrichment-Daten" empty state
- Quellen: URL list (link-out)
- Quellen-Vergleich: per-source contribution diff (reuses
ContributionsPanel) — only rendered when >=2 sources
- Similar: candidates loaded lazily on drawer open; AI? tiebreak
button per candidate shows ✓ same / ✗ diff chips with LLM reason
- Audit: discovered_at, agent_status, hinweis
+page.svelte: removed the three inline <tr> panels (Similar,
Quellen-Vergleich, expanded) and their associated state (expandedId,
similarOpenId, quellenVergleichOpenId, similarLoading, similarEntries,
similarVerdicts, similarClassifying, toggleSimilar, classifySimilar,
toggleQuellenVergleich). Row actions collapsed from 5 buttons
(Accept/Reject/Similar/AI/Quellen-Vergleich) to 2 (Accept/Reject).
The chevron glyph stays as a visual affordance but is inert — the
whole row is clickable. Buttons/forms/links inside the row stop
propagation via a closest()-based guard so fast-path Accept/Reject
don't accidentally open the drawer.
No backend changes; the drawer consumes existing queue data +
existing endpoints (similar, similar/classify, enrich).
Follow-ups: MR 8 adds keyboard shortcuts that naturally compose with
the drawer (j/k navigation, Enter opens, Esc closes).
