From 8879ecf779813cf6f96eadc2bbb51f6efef9fb0b Mon Sep 17 00:00:00 2001 From: Rohit Ghumare Date: Tue, 21 Apr 2026 19:59:36 +0100 Subject: [PATCH] =?UTF-8?q?fix(phase-4/25):=20CMER=20=E2=80=94=20text=5Fco?= =?UTF-8?q?nf=20is=20a=20probability=20in=20[0,1];=20flag=20directionality?= =?UTF-8?q?=20clarified;=20shape=20validation=20+=20PyTorch=20assumed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../outputs/skill-cmer-monitor.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/phases/04-computer-vision/25-vision-language-models/outputs/skill-cmer-monitor.md b/phases/04-computer-vision/25-vision-language-models/outputs/skill-cmer-monitor.md index 3bbe621b5..53e7383b3 100644 --- a/phases/04-computer-vision/25-vision-language-models/outputs/skill-cmer-monitor.md +++ b/phases/04-computer-vision/25-vision-language-models/outputs/skill-cmer-monitor.md @@ -20,7 +20,7 @@ Treat cross-modal alignment as a first-class production KPI. ## Inputs - `vlm_output`: generated text. -- `text_confidence`: average per-token log-prob converted to probability (or mean logit). +- `text_confidence`: mean per-token probability after softmax, in `[0, 1]`. Compute as `exp(mean(log_probs))`. Do not pass raw logits; raw logits are unbounded and `conf_threshold` assumes a probability. - `image_embedding`: CLIP-family embedding of the image (DINOv3, SigLIP, CLIP). - `text_embedding`: CLIP-family embedding of the generated text. - Optional `prompt_type`: label for grouping (vqa / ocr / captioning / agent). @@ -28,14 +28,20 @@ Treat cross-modal alignment as a first-class production KPI. ## Per-request computation ```python +import torch + def cmer_flag(image_emb, text_emb, text_conf, sim_thr=0.25, conf_thr=0.8): - image_emb = image_emb / image_emb.norm() - text_emb = text_emb / text_emb.norm() + if image_emb.shape != text_emb.shape: + raise ValueError(f"emb shape mismatch: {image_emb.shape} vs {text_emb.shape}") + image_emb = image_emb / (image_emb.norm() + 1e-8) + text_emb = text_emb / (text_emb.norm() + 1e-8) sim = float((image_emb * text_emb).sum()) flagged = (text_conf > conf_thr) and (sim < sim_thr) return {"sim": sim, "flagged": flagged} ``` +Embeddings are 1-D PyTorch tensors (`torch.float32`) from an independent CLIP-family encoder. If you use NumPy arrays, swap `.norm()` for `np.linalg.norm(...)` and cast the output accordingly. + Store `sim`, `text_conf`, `flagged`, `prompt_type`, `timestamp`, `model_version`, `request_id` to your monitoring pipeline (Prometheus, DataDog, OpenTelemetry). ## Aggregate metric @@ -72,4 +78,4 @@ Report per endpoint, per prompt_type, per model version. - Never compute CMER using the VLM's own embeddings; use an independent encoder (DINOv3, SigLIP, or CLIP-L/14). Otherwise you are measuring the model's self-consistency, not alignment. - Always log the raw `sim` value, not just the `flagged` bit; distribution shifts show up in the lower quartile before the flag rate changes. - Do not ship a VLM endpoint without CMER monitoring; hallucinations are the dominant production failure mode and silent without this metric. -- For sensitive domains (medical, legal, financial), drop `sim_threshold` to 0.35 or higher; a stricter gate is the right default. +- For sensitive domains (medical, legal, financial), raise `sim_threshold` to 0.35 or higher; the flag condition is `sim < sim_threshold`, so a higher threshold catches more outputs as potentially ungrounded — the right default for high-stakes use.