Bloobability v2 is two separately trained models behind one endpoint. You supply any policies you can write down and a conversation; for each one you get back a probability, a separate estimate of whether the conversation could answer it at all, and — in full mode — the messages on each side. Still a probability you can average, threshold, and trend.
1,435 / 1,542 synthetic known answers
v1: 0.549 · lower is better
4090 · 1 policy · 344 joint tokens
Any definition, one generic path
Measurement review: October 1, 2026. Benchmarks describe the P / HE package versions below; check the model_version returned by your service.
Overview
v2 uses two separately trained scorers. Full has about 1.89B parameters, shares conversation processing across policies, and returns message evidence. Compact uses about 436M parameters in its scoring path and returns numbers only. Full has stronger average quality on the measured development set; speed and errors depend on the workload.
bloo-full-1Reads the conversation once and scores every policy against it.
bloo-compact-1Reads each policy together with the conversation and scores it.
The models offer different quality and latency tradeoffs. Use full when its stronger measured average quality or evidence is useful, and compact when low latency matters most. Full does not call compact first: one model answers each request. The models can disagree, so compare rates within a fixed mode and model version.
Why it matters
You write the things you care about in plain language, send a conversation, and get a probability for each one. Below are some examples, end to end.
The walkthrough probabilities and evidence below are illustrative, not captured model outputs or benchmark results.
Conversation
customer
agentI'm sorry about that. Can I get your order number?
customer
agentI can start a return for the damaged item.
customer
agentDone. The refund will post in three to five business days.
{
"mode": "full",
"messages": [
{ "id": "m1", "speaker_id": "customer-1", "role": "customer",
"text": "My order was supposed to arrive Tuesday and it showed up Friday." },
{ "id": "m2", "speaker_id": "agent-1", "role": "agent",
"text": "I'm sorry about that. Can I get your order number?" },
{ "id": "m3", "speaker_id": "customer-1", "role": "customer",
"text": "It's 88421. When I opened the box the screen was cracked." },
{ "id": "m4", "speaker_id": "agent-1", "role": "agent",
"text": "I can start a return for the damaged item." },
{ "id": "m5", "speaker_id": "customer-1", "role": "customer",
"text": "Please do, and put the full amount back on my card." },
{ "id": "m6", "speaker_id": "agent-1", "role": "agent",
"text": "Done. The refund will post in three to five business days." }
],
"policies": [
{ "id": "delivery_delay",
"definition": "The order arrived later than the customer was promised." },
{ "id": "damaged_item",
"definition": "The item arrived damaged. A problem the customer only fears does not count." },
{ "id": "refund_request",
"definition": "The customer asks for their money back, in full or in part." }
]
}{
"mode": "full",
"model_version": "bloobability-v2-candidate-HE-4ad40b8b5dffeeba6001",
"policies": [
{
"id": "delivery_delay",
"probability": 0.95,
"answerable": 0.99,
"supporting_message_ids": ["m1"],
"contradicting_message_ids": []
},
{
"id": "damaged_item",
"probability": 0.97,
"answerable": 0.99,
"supporting_message_ids": [
"m3",
"m4"
],
"contradicting_message_ids": []
},
{
"id": "refund_request",
"probability": 0.96,
"answerable": 0.98,
"supporting_message_ids": ["m5"],
"contradicting_message_ids": []
}
]
}In this illustrative response, all three policies have a high probability — even though nobody in the thread used the words “delivery delay” or “damaged item.” Full mode always returns evidence, so each result also lists the messages that support it: late arrival in m1, cracked screen in m3 and m4, refund in m5. No contradicting message is selected in this example, so every contradicting_message_ids list is empty.
A support tool can now tag the ticket, start a return, and skip a person reading the thread. Across a week of chats, averaging comparable probabilities gives an estimated rate. Check calibration on that traffic first — a stable model and policy definition make changes easier to interpret.
None of these policies was registered anywhere. v2 has no roster: a definition written a minute ago is scored exactly like every other. There is no verdict in the response either — the probability is advice, and the threshold for acting on it is yours.
Architecture
Both models answer the same two questions for every policy: does it hold, and could this conversation tell? They differ in how they read. Because they are trained separately, switching between them can change a probability — the response always says which one answered, and in which version.
| Aspect | Full | Compact |
|---|---|---|
| Built on | Qwen3.5-2B | GLiNER2.5-Decide, a DeBERTa-v3-large encoder |
| Parameters | 1.881B backbone + 9.58M adapters and shared heads (1.891B total) | 436.3M in the scoring path; 50.4M unused extraction parameters also stored |
| Reads | The conversation once, shared by every policy | The conversation and one policy together, once per policy |
| Policies | Separate branches off one shared read | Each in its own pass |
| Evidence IDs | Always, as separate support and contradiction lists | None |
| Input limit | 32,768 conversation tokens, 512 per policy | 6,144 tokens per conversation-and-policy pair |
Compact jointly reads the full policy definition and conversation with a bidirectional encoder. A shared yes/no scorer supplies the decision logit; a separate head estimates answerability. Full uses a causal Qwen backbone with hybrid attention. It reuses the conversation prefix and isolates attention, recurrent and convolution state for each policy branch. Its additional policy-conditioned heads score supporting and contradicting messages. Neither model performs autoregressive decoding or a full-vocabulary output projection.
Performance
p50 is the median request time; p95 is the time that 95% of the recorded requests finished within. These are local RTX 4090 measurements, including input processing, inference, calibration and response serialization. Network and queue time are excluded.
Warm models, uncached inputs, one request at a time. BF16 backbone computation, pinned PyTorch 2.11.0 / CUDA 12.8 runtimes and synchronized GPU timing. The two screens below use different inputs and were run separately; their ratios are not a matched compact-versus-full benchmark.
| Aspect | p50 | p95 |
|---|---|---|
| 1 policy · 344 joint tokens | 13.44 ms | 13.84 ms |
| 1 policy · 874 joint tokens | 29.53 ms | 29.71 ms |
| 10 policies · 321–597 joint tokens each | 130.05 ms | 131.96 ms |
| 10 policies · 604–874 joint tokens each | 193.95 ms | 194.86 ms |
| 79 policies · 592–923 joint tokens each | 1,589.16 ms | 1,595.35 ms |
30 observations per workload, three interleaved blocks, three excluded warm-ups per block. Joint tokens include the conversation, definition and input structure. These are the unchanged P control measurements; experimental runtime optimizations are excluded. Three compact variants were resident during the study, so memory totals are not isolated-worker VRAM.
| Aspect | p50 | p95 |
|---|---|---|
| 1 policy · 403 conversation + 352 policy tokens | 71.62 ms | 72.10 ms |
| 10 varied policies · 630 conversation + 23–262 tokens per policy | 472.98 ms | 474.89 ms |
| 79 repeated policies · 403 conversation + 352 tokens per policy | 2,507.64 ms | 2,509.94 ms |
| 128 repeated policies · same token lengths | 4,030.95 ms | 4,033.74 ms |
40 observations per workload, four interleaved before/after-correction blocks, two excluded warm-ups per block; one resident full model. The 79- and 128-policy cells repeat a definition to test batching behavior. They are not a fixed policy roster or a representative quality evaluation.
These bounded screens do not measure maximum context, production load, p99, or sustained throughput under a latency limit. We have no qualified same-hardware optimized-4B comparison, so they establish no speedup over v1. Auto routing is a request-based choice, not a measured guarantee that full becomes faster at four policies.
Methodology
The selected checkpoints are compact P and full HE. Their latest training stages used 8,410 synthetic research records, following earlier task training. That is the latest-stage corpus, not the total lifetime training data of either foundation model.
128 updates and 14,904 training forwards, seed 29. The 433.9M-parameter encoder and 262K answerability head were trainable; the native classifier, unused extraction modules and normalization buffers stayed frozen. Supervised labels and equivalent-name pairs were combined with a penalty that retained earlier raw outputs on original training examples. Those earlier outputs were a behavior reference, not ground-truth confidence values.
Two matched 128-update stages used 14,016 training forwards each, seed 29. HE combines one stage's 9.58M adapter/shared-head parameters with five learned boundary-token embedding rows from the other. The remaining backbone stayed frozen. These components are folded into one model; full inference does not ensemble two checkpoints. The later batching correction changed no trained weights or calibration maps.
References include independently checked synthetic examples and executable labels for structured scenarios. Unanswerable cases supervise answerability; their decision labels are masked, never replaced with 0.5. Unobserved evidence is not treated as negative. Calibration uses a separate 1,975-record partition, with evidence selection and diagnostic operating rules chosen on 1,129 tuning records. Semantic relatives and conversation derivatives are grouped for splitting. Inspected development sets and training-related stress tests are not untouched release audits.
Inputs
Policies travel with every request rather than being baked into the model. A policy is a key you choose plus a plain-language definition — and the definition is the whole policy.
{
"mode": "full", // "full" | "compact" | "auto"
"messages": [
{
"id": "m1",
"role": "customer", // optional, any label
"speaker_id": "customer-1", // optional
"text": "The package arrived three days late. The screen is cracked."
// "timestamp": "2026-09-29T10:00:00Z" — optional, only if known
},
{
"id": "m2",
"role": "agent",
"speaker_id": "support-1",
"text": "I can send a replacement."
},
{
"id": "m3",
"role": "customer",
"speaker_id": "customer-1",
"text": "I want a full refund instead."
}
],
"policies": [
{
"id": "delivery_delay",
"definition": "The delivered order arrived later than promised."
},
{
"id": "refund_request",
"definition": "The customer currently asks for money back."
}
]
// "assessment_time": "2026-09-29T10:05:00Z" — optional "now"
}Neither model generates text. Probabilities come straight from numerical heads and ordinary code assembles the JSON, so instructions inside a message are just more conversation to judge — they cannot change the shape of the response. Requests carry 1–128 policies and 1–2,048 messages in up to 1 MiB. Leave out any optional field you do not know; null, unknown fields, and duplicate IDs are rejected, and a conversation too long for the model is an error, never silently cut.
Outputs
Every request returns one result per policy you sent, in order: a calibrated probability, an answerable estimate, and — in full mode — the messages on each side. The response also says which model and version answered. Click a key to read what it means.
mode: "full" — the same request, captured. Full always returns both evidence lists.
{
"mode": "full",
"model_version": "bloobability-v2-candidate-HE-4ad40b8b5dffeeba6001",
"policies": [
{
"id": "delivery_delay",
"probability": 0.9230962934191795,
"answerable": 0.9720439914005667,
"supporting_message_ids": ["m1"],
"contradicting_message_ids": []
},
{
"id": "refund_request",
"probability": 0.8689960565083739,
"answerable": 0.8619463781068893,
"supporting_message_ids": ["m3"],
"contradicting_message_ids": []
}
]
}{
"mode": "compact",
"model_version": "bloobability-v2-candidate-P-8210e74fcf18341e4340",
"policies": [
{
"id": "delivery_delay",
"probability": 0.9820334660201304,
"answerable": 0.9991865820600747
},
{
"id": "refund_request",
"probability": 0.2517289459942402,
"answerable": 0.9400763439032207
}
]
}// HTTP 400 — 413 over 1 MiB, 422 too long
// for the model, 429 at capacity, 503 unavailable
{
"error": {
"code": "invalid_request",
"message": "The request does not match the supported contract.",
"details": {}
}
}Click a card behind to bring it forward
Both numbers always come back, and neither overrides the other. A low answerable does not null or shrink the probability; it tells you the probability rests on insufficient information. The decision head was not supervised against invented labels on such cases. Choose and report a consistent treatment of them when calculating rates.
There is no verdict field and no threshold in the response. v2.0.0 ships no audited cutoff for acting without review, so it does not imply one. Use the probability as advice, or set your own threshold and own that decision.
Benchmarks
The main table uses a frozen 1,639-judgment comparison subset of the larger development corpus: 216 synthetic scenario lineages and 60 family tags. A separate 384-case entity/scope diagnostic has saved v1 API responses. These are inspected synthetic research references, not audited customer traffic or untouched final testing.
Full — development set, 1,542 known answers
| Correct at 0.51,435 of 1,542 known answers | 93.1% |
|---|---|
| AUROCRanking quality | 0.975 |
| AUPRC | 0.983 |
| Log lossLower is better | 0.193 |
| Brier scoreLower is better | 0.053 |
| Unanswerable cases flagged25 answerable ones wrongly flagged | 82 of 97 |
Where the two modes disagree
Compact — development set, 1,542 known answers
| Correct at 0.51,346 of 1,542 known answers | 87.3% |
|---|---|
| AUROCRanking quality | 0.943 |
| AUPRC | 0.963 |
| Log lossLower is better | 0.294 |
| Brier scoreLower is better | 0.091 |
| Unanswerable cases flagged6 answerable ones wrongly flagged | 70 of 97 |
“Correct” counts a known answer on the right side of a 0.5 probability. The API returns no verdict; this is only a way to read the numbers. An unanswerable case counts as flagged when answerable is below 0.5.
Log loss penalizes confident mistakes; Brier score measures squared probability error. Lower is better for both. AUROC and AUPRC measure how well positives rank above negatives, rather than whether the stated probability matches the observed frequency. Both modes made small average probability-score gains over their earlier v2 controls, but the paired lineage 95% intervals include no improvement. The current checkpoints are not uniformly better on every capability.
Scale of the development evaluation
384 entity and scope cases — conversations where who did what, and to which item, decides the answer — went to v1's live public API, and the same records were scored by both v2 modes. Full has lower log loss and Brier score than v1 on this set, with log loss 0.438 against 0.549. Compact does not: on this set its log loss and Brier score are worse than v1's. v1 has the lowest binned calibration error here. Better average probability loss is not the same as better calibration in every slice.
Entity and scope cases — 384 judgments, 320 with a known answer
| Aspect | v1.0.0, live API | Full | Compact |
|---|---|---|---|
| Log loss | 0.549 | 0.438 | 0.604 |
| Brier score | 0.188 | 0.139 | 0.208 |
| AUROC | 0.728 | 0.894 | 0.717 |
| Correct at probability ≥ 0.5, of 320 | 215 · 67.2% | 265 · 82.8% | 216 · 67.5% |
| Calibration error · 10-bin ECE, lower is better | 0.049 | 0.113 | 0.072 |
| Unanswerable flagged | No such output | 64 of 64 | 44 of 64 |
| Wrongly flagged | — | 11 of 320 | 0 of 320 |
All three models are compared on the same 320 known references. v1's 215 correct at 0.5 were counted from its saved probabilities; its actual returned verdict used a different rule and was correct on 217 of 320 (67.8%). There was no new live API call. v1 had no separate answerability output and received fewer metadata fields. This is a 16-lineage diagnostic, not a matched hardware comparison or proof of an overall win over v1.
| Aspect | Supporting | Contradicting |
|---|---|---|
| Precision · selected messages that are correct | 87.5% | 61.2% |
| Recall · annotated messages found | 89.4% (483 / 540) | 54.2% (52 / 96) |
| F1 · balance of precision and recall | 0.885 | 0.575 |
| Average precision · pooled ranking | 0.939 | 0.533 |
254 known-answer development records with exhaustive annotations: 2,148 eligible message labels per channel, including 540 supporting and 96 contradicting positives. Unknown cases are masked; positive-only annotations are excluded from this precision/F1 table. Selection thresholds were fixed on tuning data. Ranking average precision and thresholded selection are different measurements.
Full is right on 1,435 of 1,542 known development answers and catches 82 of the 97 deliberately unanswerable ones. Its supporting-message evidence has 87.5% precision and 89.4% recall — when it says a message supports a policy, it is usually right on the annotated subset. Compact handled all 160 known scope-boundary cases in this development slice; that is not a guarantee for new scope definitions.
Compact is weakest when details decide the answer: it gets 142 of 208 on an earlier challenge set, and renaming tasks and stations in 320 entity cases flipped 64 of its answers. These are names in the text, not opaque API IDs. Full flags 25 of 1,542 answerable cases as unanswerable and finds only 54% of contradicting messages. Compact misses 27 of the 97 unknown development cases and 20 of 64 in the entity/scope diagnostic at the same 0.5 answerability cutoff. A high decision probability alone does not establish that the context is sufficient.
Calibration
A calibrated 0.80 means that the policy holds in about 80% of comparable cases assigned that probability. It is not an 80th percentile or a measure of confidence in whichever yes/no answer won. Calibration is an empirical property that can change across populations and time.
| Bin | Stated probability | Observed rate | Decisions |
|---|---|---|---|
| 0.0 – 0.1 | 0.0471 | 0.0225 | 444 |
| 0.1 – 0.2 | 0.1384 | 0.1379 | 58 |
| 0.2 – 0.3 | 0.2455 | 0.1852 | 27 |
| 0.3 – 0.4 | 0.3480 | 0.5000 | 24 |
| 0.4 – 0.5 | 0.4432 | 0.5263 | 19 |
| 0.5 – 0.6 | 0.5608 | 0.6364 | 22 |
| 0.6 – 0.7 | 0.6548 | 0.5833 | 24 |
| 0.7 – 0.8 | 0.7567 | 0.7059 | 34 |
| 0.8 – 0.9 | 0.8600 | 0.8039 | 51 |
| 0.9 – 1.0 | 0.9698 | 0.9714 | 839 |
Calibration quality
| Expected calibration errorFull · compact, average gap between stated and observed | 0.018 · 0.019 |
|---|---|
| Brier scoreLower is better | 0.053 · 0.091 |
| Log lossLower is better | 0.193 · 0.294 |
| Calibration partition1,814 known decision labels; separate answerability fits | 1,975 |
Each mode has one decision map and a separate answerability map, selected using grouped validation within the separate 1,975-record calibration partition and frozen with the model version. Decision fitting uses its 1,814 known labels; answerability uses all records. Full also has separate support and contradiction maps, with evidence thresholds chosen on tuning data. There are no policy-specific offsets or cohort normalization. Identical calibration code does not guarantee identical observed reliability for every family or future period.
| Aspect | Full | Compact |
|---|---|---|
| Log loss · raw → calibrated | 0.363 → 0.193 | 0.523 → 0.294 |
| Brier · raw → calibrated | 0.061 → 0.053 | 0.107 → 0.091 |
| Mean probability · observed positive rate 61.8% | 62.5% | 61.4% |
The same 1,542 known development judgments, before and after the frozen decision map. Both maps reduce excessive confidence without changing the ranking. The small overall ECE values do not imply a worst-case error bound: full's ECE rises from 0.018 here to 0.113 on entity/scope cases. Keep model, policy wording, population and treatment of unanswerable cases consistent when comparing rates.
Generalisation
v1 fitted 79 production policies individually and scored everything else on a shared map. v2 drops the roster entirely: the policy key never reaches either model, and every definition takes the same path.
Correct at 0.5, by evaluation set — all inspected during development; hard-wording cases are related to training data
| Aspect | Full | Compact |
|---|---|---|
| Development, 1,542 | 1,435 · 93.1% | 1,346 · 87.3% |
| Who did what, 320 | 265 · 82.8% | 216 · 67.5% |
| Earlier challenge set, 208 | 165 · 79.3% | 142 · 68.3% |
| Scope boundaries, 160 | 158 · 98.8% | 160 · 100% |
| Hard wording, 140 | 131 · 93.6% | 126 · 90.0% |
Specifications
Two models with separate runtimes behind one endpoint. Neither generates text, so there is no sampling or generation-temperature setting. Calibration uses fitted numerical scaling. The model_version identifies the build; other hardware or numerical paths need their own qualification.
The model-size figures describe the scoring graph, not a VRAM guarantee. Compact's package also contains unused extraction parameters. Input caps are execution limits, not proof of quality at maximum length.
Download benchmark summary, model versions and source hashes