# CLAUDE.md

Real estate investment analysis agent: **ReAct LLM agent** (GPT-4o) + **LLM-free geographic estimation** (DVF). Covers Nord/59 and Haute-Garonne/31. FastAPI REST API + CLI.

## Commands

```bash
uvicorn api:app --host 0.0.0.0 --port 8000 --reload   # API server
python main.py                                           # CLI loop
pytest                                                   # unit tests only (no tokens)
pytest --run-integration                                 # integration tests (OpenAI tokens)
pytest tests/test_integration.py -v -m integration
pytest tests/test_qualite.py -v -m integration -s
pytest tests/test_integration.py::test_name -v -m integration
python build_prix_evolution.py                           # rebuild prix_evolution table
python build_securite_lille.py
python build_securite_nord.py
python build_iris_data.py
python build_mutations_iris.py                           # DVF → IRIS spatial join
python build_flood_zones.py                              # GEORISQUES TRI → flood_zones table
python build_noise_zones.py                              # Overpass API → noise_zones table (OSM roads, LDEN estimate)
python build_poi.py                                      # Overpass API → poi table (Nord)
python build_pression_iris.py                            # SPS+BPS+NPI per IRIS → iris_pression table + data/2026/iris_pression.json
python build_nature_zones.py                             # Overpass API → nature_zones table + data/2026/iris_nature.json (park/forest coverage per IRIS)
```

## File map

| File | Purpose |
|------|---------|
| `api.py` | FastAPI server; lifespan() inits DB + DVF data; auth, SSE, all endpoints |
| `main.py` | CLI loop; .m4a Whisper transcription via audio.py; same agent pipeline |
| `config.py` | Constants only: MODEL, MODEL_LIGHT, DB_PATH, TEMPERATURE, MAX_TOKENS, OPENAI_API_KEY, system prompt strings |
| `audio.py` | trouver_enregistrement() scans .m4a; transcrire_fichier() → Whisper API |
| `agent/react.py` | react_loop(); 3-iteration max; ThreadPoolExecutor prefetch; dedup guard |
| `agent/memory.py` | deque(maxlen=10); store()/recall(n)/clear() |
| `core/llm.py` | OpenAI singleton; appeler_llm(), choisir_outil(), classifier_question(), appeler_llm_json(); retry + prompt caching |
| `core/langfuse_http.py` | HTTP ingestion to Langfuse; create_trace/update_trace/create_generation/create_span; daemon threads; silently no-ops if keys absent |
| `core/scorer.py` | LLM-as-Judge gpt-4o-mini; pertinence/fidelite/coherence 1–5; noter_async() → daemon thread |
| `core/monitoring.py` | In-memory FIFO last 1000 requests; enregistrer()/get_metrics(); cost + ROI vs manual baseline |
| `core/security.py` | valider_input() 6 injection patterns; valider_sql() blocks DDL/DML/stacked; raises InputSecurityError |
| `tools/database.py` | setup_db() rebuilds mutations+loyers+prix_evolution_iris+prix_evolution from CSVs; exports DB_SCHEMA |
| `tools/query_db.py` | SELECT-only SQL; auto-adds LIMIT 20; calls valider_sql() |
| `tools/get_loyer_data.py` | Filtered query on loyers (zone/type_habitat/nombre_pieces) |
| `tools/analyze_property.py` | prix_m2 = prix / surface |
| `tools/calculate_profitability.py` | rentabilite_brute_pct + rentabilite_nette_pct |
| `tools/compare_with_market.py` | statut (sous-évalué/prix marché/surévalué) + ecart_pct/ecart_euros |
| `tools/investment_score.py` | Weighted score 0–10 (rentabilite 40%, prix_marche 30%, demande 20%, risque 10%); returns score/niveau/details/poids |
| `tools/generate_report.py` | Final Markdown report from bien/analyse_marche/indicateurs_financiers/score_investissement |
| `tools/analysis.py` | LLM-free estimation; loads DVF 2021-2025; geocode BAN → Haversine → top-20 → market evolution → coefficients |
| `tools/iris_map.py` | Builds enriched GeoJSON from IGN WFS IRIS polygons + security scores + DVF prices |
| `build_mutations_iris.py` | DVF CSVs → spatial join IRIS → mutations + prix_evolution_iris tables |
| `build_prix_evolution.py` | mutations → prix_evolution (17 communes Lille métropole, commune-level) |
| `build_iris_data.py` | IGN WFS + security/prices → enriched GeoJSON |
| `build_securite_lille.py` | Security data for Lille métropole |
| `build_securite_nord.py` | Security data for département Nord |
| `build_flood_zones.py` | GEORISQUES WFS TRI → flood_zones table |
| `build_noise_zones.py` | Overpass API OSM roads → noise_zones table (LDEN estimate per road type) |
| `build_poi.py` | Overpass API → poi table (mairie, gare, metro, tram — Nord) |
| `build_pression_iris.py` | INSEE RP2022+Filosofi2021+DEFM2024+CAF2023+BPE2024 → SPS/BPS/NPI per IRIS → iris_pression table + data/2026/iris_pression.json |
| `build_nature_zones.py` | Overpass API OSM → park/forest/natural polygons → nature_zones table + data/2026/iris_nature.json (nature_pct per IRIS; target bbox covers Lille métropole) |
| `tests/test_integration.py` | A/B/S: classifier, tool selection, ReAct loop, memory, injection resistance |
| `tests/test_qualite.py` | LLM-as-Judge over tests/questions.json; ≥3.0/5 per question, ≥3.5/5 global |
| `tests/test_analysis.py` | tools/analysis.py tests |
| `tests/test_tools.py` | Unit tests for individual tools |
| `benchmarks/benchmark_cache.py` | OpenAI prompt caching hit rates + cost savings |
| `benchmarks/benchmark_integration.py` | End-to-end latency + token benchmarks |
| `conftest.py` | --run-integration flag; skips integration tests by default |

## Configuration (`config.py`)

```python
MODEL       = "gpt-4o"       # classificateur + agent expert
MODEL_LIGHT = "gpt-4o-mini"  # orchestrateur + scorer
TEMPERATURE = 0.3
MAX_TOKENS  = 12000
DB_PATH     = "data/immobilier.db"
```

System prompts: `SYSTEM_PROMPT` (expert), `SYSTEM_ORCHESTRATEUR`, `SYSTEM_CLASSIFICATEUR`.

## Environment variables

| Variable | Default | Purpose |
|----------|---------|---------|
| `OPENAI_API_KEY` | **required** | OpenAI API |
| `API_KEY` | `""` (disabled) | X-API-Key header enforcement on /ask + /analysis |
| `LANGFUSE_HOST` | `http://localhost:3000` | Langfuse server |
| `LANGFUSE_PUBLIC_KEY` | `""` (disabled) | Langfuse public key |
| `LANGFUSE_SECRET_KEY` | `""` (disabled) | Langfuse secret key |
| `ORS_API_KEY` | `""` (disabled) | OpenRouteService for car isochrones |
| `OTP_URL` | `http://localhost:8080` | OpenTripPlanner for transit isochrones |

## API endpoints (`api.py`)

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/` | No | Redirect → /docs |
| GET | `/dashboard` | No | Dashboard HTML |
| GET | `/health` | No | Service status + model ID |
| GET | `/metrics` | No | Latency, tokens, cost, ROI (last 1000 requests) |
| GET | `/logs/stream` | No | SSE real-time log streaming |
| POST | `/ask` | **YES** | ReAct agent Q&A (gpt-4o) |
| POST | `/analysis` | **YES** | LLM-free valuation (Nord-59) |
| GET | `/map` | No | Interactive security map HTML (Lille métropole) |
| GET | `/map-data` | No | IRIS GeoJSON (security + price) |
| POST | `/map-refresh` | No | Clear GeoJSON cache |
| GET | `/iris-evolution/{code_iris}` | No | IRIS price evolution by year (indexed, base 100) |
| GET | `/duration-map` | No | Travel time map HTML |
| GET | `/flood-map` | No | Flood zone map HTML |
| GET | `/flood-zones` | No | Point-in-polygon: is (lat,lng) in TRI flood zone (3 scenarios) |
| GET | `/flood-polygons` | No | Flood zone GeoJSON for bbox |
| GET | `/isochrone` | No | Isochrones: mode=car (ORS) or mode=transit (OTP) |
| GET | `/seveso-sites` | No | Seveso ICPE hazard sites from GEORISQUES |
| GET | `/flood-risk` | No | Flood risk assessment from GEORISQUES |

Auth: `X-API-Key: <value>` header required on /ask and /analysis when `API_KEY` env var is set.

## Request routing

```
user input → valider_input() → classifier_question() → 
  "analyse"      → react_loop() → tools → appeler_llm() → memory.store()
  "conversation" → appeler_llm() → memory.store()
  "hors_scope"   → hardcoded refusal
```

**Three LLM roles:**

| Role | Prompt constant | Model | Purpose |
|------|----------------|-------|---------|
| Classificateur | `SYSTEM_CLASSIFICATEUR` | gpt-4o | Routes to analyse/conversation/hors_scope |
| Orchestrateur | `SYSTEM_ORCHESTRATEUR` | gpt-4o-mini | Selects next tool each ReAct iteration |
| Agent expert | `SYSTEM_PROMPT` | gpt-4o | Final Markdown answer |

## ReAct loop (`agent/react.py`)

`react_loop(question, historique, max_iterations=3)`:
1. **Reason** — `choisir_outil()` → orchestrator LLM
2. **Act** — executes tool from `OUTILS` dict
3. **Observe** — appends to `contexte_court` (truncated, for orchestrator) and `contexte_final` (full, for answer)

Optimization: at iteration N, tool execution + orchestrator call for N+1 run in parallel via `ThreadPoolExecutor`. Prefetched decision reused if tool differs from current (dedup guard blocks same tool+param pair).

If `generate_report` called → return Markdown directly. Else → `appeler_llm()` with all collected data.

**OUTILS dict:**
```python
OUTILS = {
    "query_db":                query_db,
    "get_loyer_data":          get_loyer_data,
    "analyze_property":        analyze_property,
    "calculate_profitability": calculate_profitability,
    "compare_with_market":     compare_with_market,
    "investment_score":        investment_score,
    "generate_report":         generate_report,
}
```

## Tool signatures

All tool files follow the pattern: `_parse(parametre: str) -> dict` then main function `tool_name(parametre: str) -> dict`.

| Tool | Input JSON keys | Output keys |
|------|----------------|-------------|
| `analyze_property` | prix, surface, ville, type_bien | prix_m2, ... |
| `calculate_profitability` | prix, loyer_mensuel, charges_annuelles, taxe_fonciere, frais_gestion | rentabilite_brute_pct, rentabilite_nette_pct |
| `compare_with_market` | prix_m2_bien, prix_m2_marche | statut, ecart_pct, ecart_euros |
| `investment_score` | rentabilite, prix_marche, demande, risque | score (0-10), niveau, details, poids |
| `generate_report` | bien, analyse_marche, indicateurs_financiers, score_investissement | Markdown string |
| `get_loyer_data` | zone?, type_habitat?, nombre_pieces? | filtered loyers rows |
| `query_db` | SQL string (SELECT only) | rows list or dict |

`query_db` auto-adds `LIMIT 20` if absent. `tools/analysis.py:analyser_bien()` is async, called by `/analysis` endpoint only.

## Database schema (`data/immobilier.db`)

### mutations (rebuilt from data/DSV.csv at startup)
DVF 2025 transactions, Haute-Garonne (31). Filter: ventes only, surface >0, prix/m² ∈ [500–12000€], valeur ∈ [10k–5M€], valid GPS.

| Column | Type |
|--------|------|
| id | INTEGER PK |
| id_mutation, date_mutation, code_commune, nom_commune, code_departement | TEXT |
| annee | INTEGER |
| valeur_fonciere, prix_m2, surface_reelle_bati, latitude, longitude | REAL |
| adresse_numero, adresse_nom_voie, code_postal | TEXT |
| type_local | TEXT — "Appartement" \| "Maison" |
| code_iris, nom_iris | TEXT (from spatial join) |

### loyers (rebuilt from data/Base_OP_2025_Nationale.csv at startup)
Observatoire des Loyers 2025, Agglomération de Toulouse.

| Column | Type |
|--------|------|
| id | INTEGER PK |
| annee | INTEGER |
| agglomeration, zone, type_habitat, epoque_construction, anciennete_locataire, nombre_pieces | TEXT (nullable) |
| loyer_m2_moyen/median/q1/q3 | REAL |
| loyer_mensuel_moyen/median/q1/q3 | REAL |
| surface_moyenne | REAL |
| nombre_observations, nombre_logements | INTEGER |

zone: "Ville centre" \| "Périphérie". type_habitat: "Appartement" \| "Maison". nombre_pieces: "Appart 1P/2P/3P/4P+" etc.

### prix_evolution (built by build_prix_evolution.py — not rebuilt at startup)
17 communes Lille métropole. Commune-level annual price evolution.

| Column | Type |
|--------|------|
| id | INTEGER PK |
| annee | INTEGER |
| code_postal, nom_commune, type_local | TEXT |
| nb_transactions | INTEGER |
| prix_m2_moyen, prix_m2_median, evolution_m2_pct, prix_moyen, prix_median, surface_moyenne | REAL |

### prix_evolution_iris (built by build_mutations_iris.py)
IRIS-level annual prices. Minimum 3 transactions per (IRIS, type, year).

| Column | Type |
|--------|------|
| id | INTEGER PK |
| code_iris, nom_iris, nom_commune, type_local | TEXT |
| annee, nb_transactions | INTEGER |
| prix_m2_median, evolution_m2_pct | REAL (evolution NULL for first year) |

### flood_zones (built by build_flood_zones.py)
GEORISQUES WFS TRI débordement flood zones.

| Column | Type |
|--------|------|
| scenario | TEXT — "frequent" \| "moyen" \| "rare" |
| geom_type | TEXT — "Polygon" \| "MultiPolygon" |
| coordinates | TEXT (JSON-serialized) |
| bbox_min_lng, bbox_max_lng, bbox_min_lat, bbox_max_lat | REAL |

### iris_pression (built by build_pression_iris.py)
Selling/buying pressure scores per IRIS, Nord (59). Sources: RP2022, RP2018, Filosofi2021, DEFM2024, CAF2023, BPE2024, prix_evolution_iris, iris_nature.json (v4).

| Column | Type |
|--------|------|
| code_iris | TEXT PRIMARY KEY (9-digit) |
| senior_ratio, senior_ownership, vacancy_rate, pop_decline, poverty_rate | REAL — SPS raw indicators |
| income_level, prime_age_ratio, employment_rate, diploma_rate, pop_growth, greenery_pct, amenity_score, transaction_rate | REAL — BPS raw indicators (v4: +greenery_pct 10%) |
| *_norm columns | REAL — min-max normalised per indicator across Nord |
| sps, bps, npi | REAL — composite scores; NPI in [-1, +1] |
| built_at | TEXT (ISO timestamp) |

## Data files

| Path | Format | Content |
|------|--------|---------|
| `data/DSV.csv` | `\|`-sep, UTF-8 | DVF 2025 mutations, Haute-Garonne (31) |
| `data/Base_OP_2025_Nationale.csv` | `,`-sep, Latin-1 | Observatoire des Loyers 2025, Toulouse |
| `data/{2021..2025}/59.csv` | `,`-sep, UTF-8 | DVF individual transactions, Nord-59, geocoded |
| `data/2026/iris_nord.geojson` | GeoJSON | 1345 IRIS polygons from IGN WFS for Nord |
| `data/2026/iris_prix.json` | JSON | Per-IRIS avg price/m² (DVF 2021-2025 spatial join) |
| `data/2026/iris_securite.json` | JSON | Per-IRIS danger scores 1–10 |
| `data/2026/iris_pression.json` | JSON | Per-IRIS SPS/BPS/NPI scores + raw indicators (Nord 59) |
| `data/2026/iris_nature.json` | JSON | Per-IRIS nature coverage: nature_pct, nature_m2_approx, iris_m2_approx, categories (OSM parks/forests/natural) |
| `data/2026/securite_nord_complet.xlsx` | Excel | Commune-level security scores (fallback) |
| `data/2026/securite_lille_metro.xlsx` | Excel | Lille métropole security data |
| `data/immobilier.db` | SQLite | Main DB (5 tables) |

## LLM-free estimation algorithm (`tools/analysis.py`)

`analyser_bien(adresse, code_postal, ville, surface_m2, statut, type_bien, orientation, taille_terrain)`:
1. `_geocode_adresse(adresse)` → BAN API → (lat, lon)
2. Load DVF 2021–2025 transactions (cached at startup by `init_analysis()`)
3. `_haversine()` distance to each transaction; use parcel GPS if available, else commune centroid from API Geo
4. Take top-20 closest transactions
5. Compound annual price evolution from `prix_evolution` table (projecting historical prices to current year)
6. Weighted average: weight = 1/distance × recency_weight
7. Apply coefficients: surface (`get_surface_coefficient()`), orientation (N/S/E/W), terrain (`_coef_terrain()`), renovation (statut)

## Observability

- **Langfuse** (`core/langfuse_http.py`): trace per request → generations per LLM call → spans per tool. Background daemon threads; fully optional (silently no-ops without keys).
- **Scorer** (`core/scorer.py`): `noter_async(trace_id, question, reponse)` → daemon thread → `_evaluer()` gpt-4o-mini → pertinence/fidelite/coherence (1–5) → POST to Langfuse. Never blocks response.
- **Monitoring** (`core/monitoring.py`): `enregistrer(question, duree_ms, tokens_prompt, tokens_completion, erreur)` → FIFO deque maxlen=1000. `get_metrics()` → latency percentiles, token totals, cost (GPT-4o pricing), ROI vs manual analysis baseline.

## Security (`core/security.py`)

`valider_input(texte)` — checks length, control chars, then 6 regex patterns:
1. Prompt injection (ignore/forget/override instructions)
2. Data exfiltration (send/export/reveal data)
3. Context injection (you are now / act as)
4. Indirect injection (from document/URL/external)
5. SQL injection in free text
6. Unauthorized actions (delete/drop/modify)

`valider_sql(sql)` — blocks: non-SELECT statements, DDL (CREATE/DROP/ALTER), DML (INSERT/UPDATE/DELETE), dangerous comments (`--`, `/*`), stacked queries (`;`).

Both raise `InputSecurityError`. Pattern matching uses normalized Unicode-stripped text; original text passed to agent on success.

## Memory (`agent/memory.py`)

`deque(maxlen=10)` storing `{"role": "user"|"assistant", "content": "..."}`. Cleared at startup. API passes last 3–4 messages as `historique` to all LLM calls.

## Tests

| File | Scope | Markers |
|------|-------|---------|
| `tests/test_integration.py` | Classifier, tool selection, full ReAct loop, memory, injection resistance | `@pytest.mark.integration` |
| `tests/test_qualite.py` | LLM-as-Judge over `tests/questions.json`; thresholds ≥3.0/5 per Q, ≥3.5/5 global | `@pytest.mark.integration` |
| `tests/test_analysis.py` | tools/analysis.py | — |
| `tests/test_tools.py` | Individual tools | — |

Integration tests skipped by default. `--run-integration` flag in conftest.py enables them.

## External APIs used

| API | Purpose | Auth |
|-----|---------|------|
| OpenAI | LLM + Whisper | OPENAI_API_KEY |
| API BAN (adresse.data.gouv.fr) | Address geocoding | None |
| API Geo (geo.api.gouv.fr) | Commune centroids | None |
| IGN WFS (wxs.ign.fr) | IRIS polygons | None |
| GEORISQUES API | Flood risk + Seveso ICPE | None |
| OpenRouteService | Car isochrones | ORS_API_KEY |
| OpenTripPlanner | Transit isochrones | None (self-hosted OTP_URL) |
| Langfuse | Observability ingestion | LANGFUSE_* keys |