Bloobability (blooio-probability) is a schema-conditioned matcher paired with a calibrated gauge. You supply the policies you care about and a conversation; you get back which policies apply, the exact messages that support each one, and a probability you can average, threshold, and trend.
Held-out evaluation
Gauge probabilities
Ranking quality
Individually fitted
Overview
Bloobability is a collection of multiple models that solve the same problem in different ways, accessible through the same endpoint. Both models are adapters on the same Blooio 3.98B-parameter trained model, so a single process serves them together. They answer different questions, so each request runs only the pass it actually needs.
bloo-gauge-1Judges each policy on its own and returns a probability.
bloo-match-1Matches every policy against the conversation.
Why it matters
You name the things you care about, send a conversation, and get a yes-or-no plus a probability for each one. Below are some examples, end to end.
Conversation
customer
agentI'm sorry about that. Can I get your order number?
customer
agentI can start a return for the damaged item.
customer
agentDone. The refund will post in three to five business days.
{
"reference_id": "order-88421",
"include_evidence": true,
"policies": [
{ "id": "delivery_delay", "description": "The order arrived later than promised." },
{ "id": "damaged_item", "description": "The item arrived damaged." },
{ "id": "refund_request", "description": "The customer asks for money back." }
],
"conversation": [
{ "id": "m1", "role": "customer",
"content": "My order was supposed to arrive Tuesday and it showed up Friday." },
{ "id": "m2", "role": "agent",
"content": "I'm sorry about that. Can I get your order number?" },
{ "id": "m3", "role": "customer",
"content": "It's 88421. When I opened the box the screen was cracked." },
{ "id": "m4", "role": "agent",
"content": "I can start a return for the damaged item." },
{ "id": "m5", "role": "customer",
"content": "Please do, and put the full amount back on my card." },
{ "id": "m6", "role": "agent",
"content": "Done. The refund will post in three to five business days." }
]
}{
"reference_id": "order-88421",
"policies": [
{
"id": "delivery_delay",
"calibrated_probability": 0.94,
"verdict": "yes",
"calibration_tier": "pooled_unseen",
"evidence_ids": ["m1"]
},
{
"id": "damaged_item",
"calibrated_probability": 0.96,
"verdict": "yes",
"calibration_tier": "pooled_unseen",
"evidence_ids": ["m3", "m4"]
},
{
"id": "refund_request",
"calibrated_probability": 0.98,
"verdict": "yes",
"calibration_tier": "pooled_unseen",
"evidence_ids": ["m5", "m6"]
}
]
}All three policies came back yes, each with a high probability — even though nobody in the thread used the words “delivery delay” or “damaged item.” include_evidence was on, so each match also cites the message IDs that support it: late arrival in m1, cracked screen in m3, refund in m5. reference_id comes back unchanged — it is just your own label for the call.
A support tool can now tag the ticket, start a return, and skip a person reading the thread. Across a week of chats, the same probabilities average into a rate. That is the useful part — one conversation becomes a decision, and many conversations become a number you can watch.
These policies are not on the measured roster, so they are labeled pooled_unseen. The number is still a probability. The tier just tells you which scorecard produced it.
Architecture
Both models read the entire conversation. What separates them is how much of the policy set each one weighs at once — and what comes back. They are looking at genuinely different things, and reading one as if it were the other is the easiest way to misinterpret a result.
| Aspect | Gauge | Matcher |
|---|---|---|
| Reads | The entire conversation | The entire conversation |
| Considers | Exactly one policy, in isolation | Every supplied policy together, in a single pass |
| Answers | How likely is it that this one policy holds? | Which of these policies apply, and where? |
| Returns | A yes/no verdict and a calibrated probability | Matching policies, a ranking score, evidence message IDs |
| Evidence IDs | No — its output contract has no evidence field | Yes |
| When it runs | Every request | Only when you ask for evidence |
Inputs
Policies travel with every request rather than being baked into the model. A policy is an identifier you choose plus a plain-language description of what should count.
roster tier — while any other id is scored by the shared pooled map (pooled_unseen). Because the name pulls in that calibration, keep it honest to what the description means.{
"reference_id": "order-88421", // optional
"include_evidence": true, // optional
"policies": [
{ "id": "delivery_delay", "description": "The order arrived later than promised." },
{ "id": "damaged_item", "description": "The item arrived damaged." },
{ "id": "refund_request", "description": "The customer asks for money back." }
],
"conversation": [
{ "id": "m1", "role": "customer",
"content": "The package was three days late and the screen is cracked." },
{ "id": "m2", "role": "customer",
"content": "I would like a full refund." }
]
}The conversation is placed inside an explicitly untrusted evidence block and the policies in a trusted schema block, so instructions appearing inside message text are treated as evidence rather than as commands.
Outputs
Every request returns the same object: one entry per policy you sent, carrying a calibrated probability, a verdict, the calibration tier behind it, and — on roster policies — an accept decision. The matcher runs only when you ask for evidence, and only its message IDs surface. Click a key to read what it means.
Two roster policies — one accepted, one deferred.
{
"reference_id": "order-88421",
"policies": [
{
"id": "customer_request",
"calibrated_probability": 0.9762811750604389,
"verdict": "yes",
"calibration_tier": "roster",
"accepted": true
},
{
"id": "logistics_status",
"calibrated_probability": 0.32163115115645224,
"verdict": "no",
"calibration_tier": "roster",
"accepted": false
}
]
}{
"labels": [
{
"id": "refund_request",
"score": 0.864865,
"evidence_ids": ["m5"]
}
],
"abstained": false,
"reason": "The customer asks for money back."
}{
"labels": [],
"abstained": true,
"reason": "No supplied label is supported."
}Click a card behind to bring it forward
A default request runs the gauge only, so it carries no evidence IDs. Setting include_evidence adds the matcher pass, and evidence arrives as message IDs — m1, m5 — rather than copied text, so your application maps them straight back to the original messages.
A high probability with no evidence_ids is still a complete answer. The number is the gauge's confidence that the policy holds; citations are a separate, optional pass. They are absent on a default request because that pass did not run, and they can come back empty even when you ask for them — the matcher may not point at a message even when the gauge is sure. Treat the probability as the decision, and the IDs, when present, as pointers back into the thread.
Benchmarks
Gauge figures come from a held-out test split spanning multiple organizations, so the numbers reflect conversations from businesses the models were not fitted on. Matcher figures come from its held-out evaluation.
Gauge — held-out test split
| Median per-policy AUC | 0.8737 |
|---|---|
| Routing coverage | 62.34% |
| Observed risk among accepted decisions | 1.33% |
| 95% risk upper bound | 1.67% |
Policies you can trend
Matcher — held-out evaluation
| Micro F1 | 0.9893 |
|---|---|
| Micro precision | 0.9965 |
| Micro recall | 0.9823 |
| Macro F1 | 0.9747 |
| Exact label-set match | 0.9333 |
| JSON-schema validity | 0.9988 |
Scale of the gauge evaluation
A median per-policy AUC of 0.8737 reflects how well the gauge orders true cases above false ones within each policy. Among decisions clearing the frozen thresholds, observed error was 1.33%, with a 95% upper bound of 1.67% — the figure to use when sizing an automation error budget.
A micro F1 of 0.9893 with precision at 0.9965 means the matcher rarely asserts a policy that does not hold, and an exact label-set match of 0.9333 means it reproduces the entire correct set of policies — not merely some of them — on most conversations.
Calibration
A calibrated model's stated confidence matches its observed accuracy. That is what makes a cohort average a rate, and the difference between two cohorts a trend.
| Bin | Stated probability | Observed rate | Decisions |
|---|---|---|---|
| 0.0 – 0.1 | 0.0178 | 0.0092 | 4,549 |
| 0.1 – 0.2 | 0.1431 | 0.0881 | 386 |
| 0.2 – 0.3 | 0.2449 | 0.2541 | 244 |
| 0.3 – 0.4 | 0.3419 | 0.3351 | 188 |
| 0.4 – 0.5 | 0.4497 | 0.4930 | 142 |
| 0.5 – 0.6 | 0.5483 | 0.6429 | 140 |
| 0.6 – 0.7 | 0.6498 | 0.5833 | 120 |
| 0.7 – 0.8 | 0.7609 | 0.7513 | 189 |
| 0.8 – 0.9 | 0.8479 | 0.9012 | 253 |
| 0.9 – 1.0 | 0.9491 | 0.9831 | 652 |
Calibration quality
| Expected calibration errorAverage gap between stated and observed probability | 0.0187 |
|---|---|
| Brier scoreLower is better | 0.0480 |
| Negative log-likelihoodLower is better | 0.1631 |
| Area under risk-coverageLower is better | 0.0112 |
Every probability comes from one scoring scale, applied identically. A 0.80 means the same thing for one cohort as another, and next quarter as today — nothing is graded on its own curve.
Specialisation
Beyond the shared calibration map, 79 production policies each carry their own individually fitted offset — complete coverage of the roster the models were tuned and measured against.
The 79 roster policies
79 / 79 fitted
Specifications
Both adapters share one base checkpoint and one greedy decode, which is what allows a single process to serve them. Verdicts, tiers, and accept decisions are stable across repeated calls; the probability itself carries full float precision and shifts by around 1e-4, so round it before you compare or display it.