Two models that read a conversation and score the policies you name.
Bloobability (blooio-probability) is a schema-conditioned matcher paired with a calibrated gauge. You supply the policies you care about and a conversation; you get back which policies apply, the exact messages that support each one, and a probability you can average, threshold, and trend.
- 0.9893
- Matcher micro F1
- 1.87%
- Expected calibration error
- 0.8737
- Median per-policy AUC
- 79
- Hyper-trained policies
Held-out evaluation
Gauge probabilities
Ranking quality
Individually fitted
Overview
One base model, two jobs
Bloobability is a collection of multiple models that solve the same problem in different ways, accessible through the same endpoint. Both models are adapters on the same Blooio 3.98B-parameter trained model, so a single process serves them together. They answer different questions, so each request runs only the pass it actually needs.
Matcher
bloo-match-1Reads the entire conversation against every supplied policy at once, and returns the matches with citations.
Returns
- The subset of your policies that apply
- A ranking score for each match
- Message IDs that support each match
- An explicit abstain when nothing applies
Best for: Triage, routing, tagging, and surfacing the exact evidence behind a decision.
Gauge
bloo-gauge-1Judges one policy at a time and returns a calibrated probability.
Returns
- A calibrated probability from 0 to 1
- A discrete yes/no verdict
- The calibration tier backing the number
- An accept/defer decision on the measured roster
Best for: Rates, trends, error budgets, and any threshold you intend to automate against.
Why it matters
What These Models Make Possible
You name the things you care about, send a conversation, and get a yes-or-no plus a probability for each one. Below are some examples, end to end.
Conversation
customer
agentI'm sorry about that. Can I get your order number?
customer
agentI can start a return for the damaged item.
customer
agentDone. The refund will post in three to five business days.
{
"reference_id": "order-88421",
"include_evidence": true,
"policies": [
{ "id": "delivery_delay", "description": "The order arrived later than promised." },
{ "id": "damaged_item", "description": "The item arrived damaged." },
{ "id": "refund_request", "description": "The customer asks for money back." }
],
"conversation": [
{ "id": "m1", "role": "customer",
"content": "My order was supposed to arrive Tuesday and it showed up Friday." },
{ "id": "m2", "role": "agent",
"content": "I'm sorry about that. Can I get your order number?" },
{ "id": "m3", "role": "customer",
"content": "It's 88421. When I opened the box the screen was cracked." },
{ "id": "m4", "role": "agent",
"content": "I can start a return for the damaged item." },
{ "id": "m5", "role": "customer",
"content": "Please do, and put the full amount back on my card." },
{ "id": "m6", "role": "agent",
"content": "Done. The refund will post in three to five business days." }
]
}{
"reference_id": "order-88421",
"policies": [
{
"id": "delivery_delay",
"calibrated_probability": 0.94,
"verdict": "yes",
"calibration_tier": "pooled_unseen",
"evidence_ids": ["m1"]
},
{
"id": "damaged_item",
"calibrated_probability": 0.96,
"verdict": "yes",
"calibration_tier": "pooled_unseen",
"evidence_ids": ["m3", "m4"]
},
{
"id": "refund_request",
"calibrated_probability": 0.98,
"verdict": "yes",
"calibration_tier": "pooled_unseen",
"evidence_ids": ["m5", "m6"]
}
]
}All three policies came back yes, each with a high probability — even though nobody in the thread used the words “delivery delay” or “damaged item.” include_evidence was on, so each match also cites the message IDs that support it: late arrival in m1, cracked screen in m3, refund in m5. reference_id comes back unchanged — it is just your own label for the call.
A support tool can now tag the ticket, start a return, and skip a person reading the thread. Across a week of chats, the same probabilities average into a rate. That is the useful part — one conversation becomes a decision, and many conversations become a number you can watch.
These policies are not on the measured roster, so they are labeled pooled_unseen. The number is still a probability. The tier just tells you which scorecard produced it.
Architecture
Two models, two different questions
Both models read the entire conversation. What separates them is how much of the policy set each one weighs at once — and what comes back. They are looking at genuinely different things, and reading one as if it were the other is the easiest way to misinterpret a result.
| Aspect | Matcher | Gauge |
|---|---|---|
| Reads | The entire conversation | The entire conversation |
| Considers | Every supplied policy together, in a single pass | Exactly one policy, in isolation |
| Answers | Which of these policies apply, and where? | How likely is it that this one policy holds? |
| Returns | Matching policies, a ranking score, evidence message IDs | A yes/no verdict and a calibrated probability |
| Evidence IDs | Yes | No — its output contract has no evidence field |
| When it runs | Only when you ask for evidence | Every request |
Inputs
Policies and a conversation
Policies travel with every request rather than being baked into the model. A policy is an identifier you choose plus a plain-language description of what should count.
Policy fields
- id
- Your stable identifier, echoed back on every policy in the response. The model reads it alongside the description, so a name that describes what you are looking for works better than an opaque key. It is also the calibration key: match one of the 79 roster names and the policy is scored on the calibration fitted for that name — the
rostertier — while any other id is scored by the shared pooled map (pooled_unseen). Because the name pulls in that calibration, keep it honest to what the description means. - description
- A plain-language definition of what should count. This is what the model actually judges against — the gauge reads it and its yes/no call is what becomes the calibrated probability. For a roster name, keep the wording true to what the name implies, so the calibration you inherit matches the thing you are actually scoring.
Conversation fields
- id
- A unique message identifier from your application. This is the vocabulary the model cites evidence in — matches come back as your own IDs rather than copied text, so each citation maps straight back to the message in your records.
- role
- Who sent the message — customer, agent, or system. The model reads it as part of the conversation, so it can weigh who said what: a promise made by the agent reads differently from the same words as a customer request.
- content
- The message text — the actual evidence both models read. Everything either model knows about the conversation comes from these strings, in the order you send them.
Optional fields
- reference_id
- Your own reference for the call, echoed back verbatim so you can reconcile a response with your records. It never reaches either model, so it cannot change a result, and if you omit it it is simply absent from the response. It is not an idempotency key, though: the endpoint is stateless, so reusing one re-runs inference rather than returning a cached result.
- include_evidence
- Off by default. Set it to true to run the matcher alongside the gauge and attach the message IDs behind each match. It runs concurrently, so it costs a second inference rather than extra wall-clock time. Those IDs are the matcher reading the conversation on its own, separate from the gauge score — so a policy can come back with a high probability and still an empty evidence list. The probability comes from the gauge; the citations come from the matcher.
{
"reference_id": "order-88421", // optional
"include_evidence": true, // optional
"policies": [
{ "id": "delivery_delay", "description": "The order arrived later than promised." },
{ "id": "damaged_item", "description": "The item arrived damaged." },
{ "id": "refund_request", "description": "The customer asks for money back." }
],
"conversation": [
{ "id": "m1", "role": "customer",
"content": "The package was three days late and the screen is cracked." },
{ "id": "m2", "role": "customer",
"content": "I would like a full refund." }
]
}The conversation is placed inside an explicitly untrusted evidence block and the policies in a trusted schema block, so instructions appearing inside message text are treated as evidence rather than as commands.
Outputs
Two shapes, because they answer two questions
The matcher returns the applicable policies and the message IDs supporting each. The gauge returns a calibrated probability per policy and no evidence IDs at all. Both are schema-constrained JSON. Click a key in the response to read what it means.
Returned only when include_evidence is on.
{
"abstained": false,
"labels": [
{ "id": "delivery_delay", "score": 0.91, "evidence_ids": ["m1"] },
{ "id": "damaged_item", "score": 0.96, "evidence_ids": ["m1"] },
{ "id": "refund_request", "score": 0.95, "evidence_ids": ["m2"] }
]
}{
"abstained": true,
"labels": [],
"reason": "No supplied label is supported."
}{
"policies": [
{
"id": "refund_request",
"calibrated_probability": 0.977,
"verdict": "yes",
"calibration_tier": "roster",
"accepted": true
}
]
}Click a card behind to bring it forward
A default request returns only the gauge shape, so it carries no evidence IDs. When the matcher pass is enabled, evidence arrives as message IDs — m1, m2 — rather than copied text, so your application maps them straight back to the original messages.
A high probability with no evidence_ids is still a complete answer. The number is the gauge's confidence that the policy holds; citations are a separate, optional pass. They are absent on a default request because that pass did not run, and they can come back empty even when you ask for them — the matcher may not point at a message even when the gauge is sure. Treat the probability as the decision, and the IDs, when present, as pointers back into the thread.
Benchmarks
Measured results
Matcher figures come from its held-out evaluation. Gauge figures come from a held-out test split spanning multiple organizations, so the numbers reflect conversations from businesses the models were not fitted on.
Matcher — held-out evaluation
| Micro F1 | 0.9893 |
|---|---|
| Micro precision | 0.9965 |
| Micro recall | 0.9823 |
| Macro F1 | 0.9747 |
| Exact label-set match | 0.9333 |
| JSON-schema validity | 0.9988 |
Gauge — held-out test split
| Median per-policy AUC | 0.8737 |
|---|---|
| Routing coverage | 62.34% |
| Observed risk among accepted decisions | 1.33% |
| 95% risk upper bound | 1.67% |
Policies you can trend
- 31qualified
- 13borderline
- 7held
- of 51 measured
Scale of the gauge evaluation
- 11,981
- Policy decisions scored
- 157
- Conversations
- 98.67%
- Correct among accepted
- 7,469
- Decisions accepted
Accuracy
A micro F1 of 0.9893 with precision at 0.9965 means the matcher rarely asserts a policy that does not hold, and an exact label-set match of 0.9333 means it reproduces the entire correct set of policies — not merely some of them — on most conversations.
Separation and risk
A median per-policy AUC of 0.8737 reflects how well the gauge orders true cases above false ones within each policy. Among decisions clearing the frozen thresholds, observed error was 1.33%, with a 95% upper bound of 1.67% — the figure to use when sizing an automation error budget.
Calibration
Probabilities that mean what they say
A calibrated model's stated confidence matches its observed accuracy. That is what makes a cohort average a rate, and the difference between two cohorts a trend.
| Bin | Stated probability | Observed rate | Decisions |
|---|---|---|---|
| 0.0 – 0.1 | 0.0178 | 0.0092 | 4,549 |
| 0.1 – 0.2 | 0.1431 | 0.0881 | 386 |
| 0.2 – 0.3 | 0.2449 | 0.2541 | 244 |
| 0.3 – 0.4 | 0.3419 | 0.3351 | 188 |
| 0.4 – 0.5 | 0.4497 | 0.4930 | 142 |
| 0.5 – 0.6 | 0.5483 | 0.6429 | 140 |
| 0.6 – 0.7 | 0.6498 | 0.5833 | 120 |
| 0.7 – 0.8 | 0.7609 | 0.7513 | 189 |
| 0.8 – 0.9 | 0.8479 | 0.9012 | 253 |
| 0.9 – 1.0 | 0.9491 | 0.9831 | 652 |
Calibration quality
| Expected calibration errorAverage gap between stated and observed probability | 0.0187 |
|---|---|
| Brier scoreLower is better | 0.0480 |
| Negative log-likelihoodLower is better | 0.1631 |
| Area under risk-coverageLower is better | 0.0112 |
One scale, not a moving curve
Every probability comes from one scoring scale, applied identically. A 0.80 means the same thing for one cohort as another, and next quarter as today — nothing is graded on its own curve.
Specialisation
Hyper-trained on 79 policies
Beyond the shared calibration map, 79 production policies each carry their own individually fitted offset — complete coverage of the roster the models were tuned and measured against.
The 79 roster policies
79 / 79 fitted
Specifications
Technical details
Both adapters share one base checkpoint and one deterministic decode, which is what allows a single process to serve them and makes results reproducible.
- Base model
- nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16
- Trained model
- bloobability-1-4b
- Parameters
- 3.98B (merged matcher)
- Precision
- BF16
- Context length
- 32,768 tokens
- Matcher LoRA
- bloo-match-1
- Gauge LoRA
- bloo-gauge-1
- Output mode
- JSON-schema constrained
- Decoding
- Temperature 0.0, top-p 1.0, fixed seed
- Max output tokens
- 1,024
- Calibration
- Frozen two-tier (per-label + pooled)
- Language
- English customer-support conversations