Bloobability v2 is two separately trained models, and you pick one on every request. You supply any policies you can write down and a conversation; for each one you get back a probability and a separate estimate of whether the conversation could answer it at all. v2.1.0 brings a new, stronger full tier. Instant is built for speed, the full tier for everything harder.
RTX 4090 · 1 policy · 2 messages · served
227 of 234 · Instant: 91.0%
31 of 34 · wrongly flags 4 of 234 answerable
Any definition, one generic path
Overview
Every request names a tier. The full tier is a 4.34B model built on Qwen3.5-4B and is the more capable one. Instant is a 0.76B model built on Qwen3.5-0.8B and is 3.3 to 11.7 times faster on the measured workloads. Both return the same two numbers per policy.
fullThe more capable tier. Use it when unsure.
instantThe fast tier, for short, explicit questions.
Choose by how hard each question is. One policy that turns on a reversal or a written rule can need the full tier, while several simple policies whose facts are stated run well on instant. When you are unsure, or when a conversation might be missing the fact a policy depends on, use the full tier. It reads the whole conversation once for every policy, so many policies over a long thread are its slowest requests. Nothing picks a tier for you, and a request never falls back to the other one. The tiers can disagree, so compare rates within one tier and model version.
What changed
v2.1.0 replaces the full tier with a new model trained on the same Qwen3.5-4B base. Instant is unchanged. The request, the response and the limits are the same, but both model_version strings are new, so start fresh baselines for anything you trend.
| Aspect | v2.0.0 full | v2.1.0 full |
|---|---|---|
| Correct at 0.5, of 900 public questions | 799 · 88.8% | 815 · 90.6% |
| Log loss · lower is better | 0.330 | 0.263 |
| Brier score · lower is better | 0.094 | 0.073 |
| Calibration error · lower is better | 0.049 | 0.040 |
| AUROC · ranking quality | 0.929 | 0.955 |
| Wrong among answers ≥ 90% confident | 47 of 666 | 32 of 710 |
| Narrow screen: missing values flagged, of 60 | 60 | 60 |
| Narrow screen: known answers correct, of 180 | 180 | 178 |
| Narrow screen: answerable wrongly flagged, of 180 | 2 | 0 |
On the same 900 public questions the v2.0.0 page reported, the new full tier gets 815 right where the old one got 799, with lower log loss and fewer confident mistakes. It still flags all 60 missing values on the narrow screen and no longer flags any answerable case there, but it misses 2 of the 180 known answers the old tier got all of.
The new tier reads each policy before the conversation and gives every policy its own pass over the thread. That is what makes it better with written rules and changed definitions, and it also makes many-policy requests slower: six policies on a two-message thread now take 153 ms, where v2.0.0 took 84 ms for six on a four-message thread.
Instant is the same model with the same calibration maps. In our serving check its outputs were bit-identical to v2.0.0's; only its model_version string changed, because it ships in a new serving package.
Why it matters
You write the things you care about in plain language, send a conversation, and get two numbers back for each one: how likely it holds, and whether the conversation could tell. Below are three examples, end to end, each sent to the tier that fits it.
Each example shows the request and what you can build on the answer. Real outputs from both tiers are in the next section.
Conversation
customer
agentI'm sorry about that. Can I get your order number?
customer
agentI can start a return for the damaged item.
customer
agentDone. The refund will post in three to five business days.
{
"tier": "full",
"messages": [
{ "id": "m1", "speaker_id": "customer-1", "role": "customer",
"text": "My order was supposed to arrive Tuesday and it showed up Friday." },
{ "id": "m2", "speaker_id": "agent-1", "role": "agent",
"text": "I'm sorry about that. Can I get your order number?" },
{ "id": "m3", "speaker_id": "customer-1", "role": "customer",
"text": "It's 88421. When I opened the box the screen was cracked." },
{ "id": "m4", "speaker_id": "agent-1", "role": "agent",
"text": "I can start a return for the damaged item." },
{ "id": "m5", "speaker_id": "customer-1", "role": "customer",
"text": "Please do, and put the full amount back on my card." },
{ "id": "m6", "speaker_id": "agent-1", "role": "agent",
"text": "Done. The refund will post in three to five business days." }
],
"policies": [
{ "id": "delivery_delay",
"definition": "The order arrived later than the customer was promised." },
{ "id": "damaged_item",
"definition": "The item arrived damaged. A problem the customer only fears does not count." },
{ "id": "refund_request",
"definition": "The customer asks for their money back, in full or in part." }
]
}Three policies, written the way your team talks, go to the full tier. Each comes back as a probability that it holds and an answerable score for whether the thread could tell, even though nobody used the words “delivery delay” or “damaged item.” The late arrival has to be read from Tuesday and Friday.
A support tool can tag the ticket, start the return, and skip a person reading the thread. Anything that moves money belongs on the full tier: it gets 97.0% of fresh decisions right, against instant's 91.0%, and 94.7% of decisions under supplied rules, against 48.3%.
Across a week of chats, the same probabilities average into a rate. That is the useful part: one conversation becomes a decision, and many conversations become a number you can watch.
Side by side
The same request sent to each tier: a customer states one fact, and four policies ask about it. Both tiers get the stated fact right. Only the full tier notices when a policy asks about something the conversation never said.
Conversation
customer
agentThe supplied value is recorded. No other attributes are provided.
{
"tier": "full",
"messages": [
{ "id": "fact", "speaker_id": "participant", "role": "customer",
"text": "The packing colour for reference case 0 is blue." },
{ "id": "ack", "speaker_id": "staff", "role": "agent",
"text": "The supplied value is recorded. No other attributes are provided." }
],
"policies": [
{ "id": "colour_blue",
"definition": "The packing colour is blue." },
{ "id": "colour_green",
"definition": "The packing colour is green." },
{ "id": "value_equals_blue",
"definition": "The supplied value of the packing colour equals blue." },
{ "id": "balance_over_100",
"definition": "The required account balance is greater than 100 dollars. The balance is necessary and was not given." }
]
}{
"tier": "full",
"model_version": "bloobability-v2-full-040b3c1d4f9151a4",
"policies": [
{
"id": "colour_blue",
"probability": 0.9789855984058057,
"answerable": 0.9925430186826678
},
{
"id": "colour_green",
"probability": 0.03931965613094063,
"answerable": 0.992896689509792
},
{
"id": "value_equals_blue",
"probability": 0.9804569759039311,
"answerable": 0.99236209781565
},
{
"id": "balance_over_100",
"probability": 0.06696203386124372,
"answerable": 0.0211137142537221
}
]
}The full tier gets all three stated-fact policies right, and is sharp about them: 0.979 for blue and 0.039 for green, with 0.980 for the reworded “equals blue.”
The fourth policy asks about an account balance nobody mentioned. The full tier's answerable drops to 0.021: the conversation cannot answer it, so the 0.067 probability rests on nothing. What to do with that is your call.
These are real responses from the serving build named in model_version. Each policy was sent in its own request under an opaque ID; they are shown together here under readable IDs. It is one deliberately easy row, and the fourth definition even says the balance “was not given,” so read it as an illustration, not a benchmark.
Architecture
Both tiers answer the same two questions for every policy: does it hold, and could this conversation tell? They share a base family and a response, but read differently: instant shares one pass over the conversation across all policies, while the full tier reads each policy first and then the conversation. Because they are trained separately, switching tiers can change a probability.
| Aspect | Full | Instant |
|---|---|---|
| Built on | Qwen/Qwen3.5-4B-Base | Qwen/Qwen3.5-0.8B-Base |
| Parameters | 4.206B frozen base + 130.2M trained (4.34B total) | 752.4M frozen base + 11.6M trained (0.76B total) |
| Trained by Blooio | 125.9M LoRA · 1.31M decision pointer · 661K answerability head · 2.38M policy reader | 10.8M LoRA · 525K decision pointer · 265K answerability head |
| Reads | Each policy first, then the whole conversation, once per policy | The conversation once, then each policy on its own private branch |
| A low answerable | Means the fact is likely missing; route the policy to a person or ask | Rarely happens even when a fact is missing; send those policies to full |
Both tiers are built on Qwen3.5 base models: full on Qwen3.5-4B-Base and instant on Qwen3.5-0.8B-Base, each a hybrid decoder that mixes Gated DeltaNet and full-attention layers. Instant reads the conversation once, and each policy then continues from a private copy of the attention, recurrent and convolution state. The full tier puts the policy first, so it knows what it is looking for while it reads, and runs a separate pass over the conversation for every policy. A two-hop attention reader then turns that pass into the decision. In both tiers a separately trained head estimates answerability, and neither generates text or runs a full-vocabulary output projection.
Calibration
A calibrated 0.80 means the policy holds in about 80% of comparable cases assigned that probability. It is not an 80th percentile or a measure of confidence in whichever answer won. Calibration is an empirical property that can change across populations and time, and this section shows where it slips.
| Bin | Stated probability | Observed rate | Decisions |
|---|---|---|---|
| 0.0 – 0.1 | 0.0651 | 0.0189 | 317 |
| 0.1 – 0.2 | 0.1335 | 0.1585 | 82 |
| 0.2 – 0.3 | 0.2569 | 0.3750 | 16 |
| 0.3 – 0.4 | 0.3406 | 0.4167 | 12 |
| 0.4 – 0.5 | 0.4497 | 0.6250 | 8 |
| 0.5 – 0.6 | 0.5500 | 0.4000 | 10 |
| 0.6 – 0.7 | 0.6641 | 0.3750 | 8 |
| 0.7 – 0.8 | 0.7504 | 0.7333 | 15 |
| 0.8 – 0.9 | 0.8597 | 0.7692 | 39 |
| 0.9 – 1.0 | 0.9531 | 0.9338 | 393 |
Calibration quality · 900 public questions and the fresh panel
| Expected calibration errorFull · instant, average gap between stated and observed | 0.040 · 0.130 |
|---|---|
| Brier scoreLower is better | 0.073 · 0.176 |
| Log lossLower is better | 0.263 · 0.574 |
| Public questionsHANS, BoolQ, SNLI · the same 900 as v2.0.0 | 900 |
| Fresh panel calibration error234 known synthetic decisions | 0.025 · 0.060 |
Each tier has one decision map and a separate answerability map, fitted on calibration data kept apart from training and evaluation, then frozen. There are no policy-specific offsets or cohort normalization. Identical calibration code does not guarantee identical reliability for every policy or future period.
Because the maps were fitted mostly on synthetic conversations, check calibration on your own traffic before treating averaged probabilities as exact rates.
Benchmarks
Four kinds of evidence, all with the served calibration maps: a fresh synthetic panel frozen before scoring, ten public datasets, screens built to test missing information, and the capability panels the tiers were developed on. None of it is audited customer traffic.
| Aspect | Full | Instant |
|---|---|---|
| Known decisions correct, of 234 | 227 · 97.0% | 213 · 91.0% |
| Simple policies, of 79 | 79 | 76 |
| Complex policies, of 117 | 111 | 103 |
| Complex, with the definitions changed, of 117 | 106 · 90.6% | 82 · 70.1% |
| Log loss · lower is better | 0.095 | 0.252 |
| Missing facts flagged, of 34 | 31 | 0 |
| Answerable cases wrongly flagged, of 234 | 4 | 0 |
268 synthetic conversations written and checked by independent AI models, with references frozen before either tier was scored: 234 with a known answer and 34 where the fact a policy needs is missing. The changed-definition row judges the same complex conversations against different definitions. Not customer data, and not labeled by people.
| Aspect | Full · correct, log loss | Instant · correct, log loss |
|---|---|---|
| ANLI · 780 | 77.1% · 0.545 | 60.6% · 0.795 |
| BoolQ · 1,320 | 90.1% · 0.280 | 79.8% · 0.471 |
| CB · 100 | 90.0% · 0.241 | 90.0% · 0.305 |
| HANS · 1,740 | 88.3% · 0.331 | 64.3% · 0.926 |
| MNLI · 254 | 90.2% · 0.252 | 81.5% · 0.404 |
| PAWS · 646 | 91.3% · 0.239 | 83.1% · 0.403 |
| QNLI · 80 | 93.8% · 0.181 | 87.5% · 0.371 |
| RTE · 100 | 87.0% · 0.331 | 78.0% · 0.552 |
| SNLI · 1,192 | 94.0% · 0.184 | 88.5% · 0.301 |
| Tweet offensive · 80 | 85.0% · 0.395 | 50.0% · 0.746 |
6,292 rows as yes-or-no questions. These panels were inspected during development, so they are not untouched tests, and they are not official leaderboard entries; overlap with the base models' pretraining is unknown. Averaging log loss with each source weighted equally gives 0.298 for full and 0.527 for instant.
Log loss penalizes confident mistakes; lower is better. “Correct” counts a known answer on the right side of 0.5; the API returns no verdict, so this is only a way to read the numbers. The panels overlap, so do not add them up as independent trials.
| Aspect | Full | Instant |
|---|---|---|
| Narrow screen · 60 missing values | 60 | 0 |
| Fresh panel · 34 missing facts | 31 | 0 |
| Supplied rules · 264 unanswerable | 264 | 71 |
| Independent screen · 70 unanswerable | 48 | 1 |
| Answerable wrongly flagged · fresh 234 | 4 | 0 |
A case counts as flagged when answerable falls below 0.5. The narrow screen is 240 short, intentionally easy rows from one repeated template; the supplied-rule and independent screens ask policies whose rules need a fact the conversation leaves out.
| Aspect | Full | Instant |
|---|---|---|
| Supplied rules · 872 | 826 | 421 |
| Entity and scheduling · 320 | 304 | 202 |
| Source authority · 220 | 220 | 203 |
| Policy scope · 160 | 159 | 156 |
| Difficult wording · 144 | 136 | 126 |
| Reversals · 32, all negative | 32 | 12 |
| Short implicit reversals · 36 | 30 | 35 |
Synthetic research references inspected during development, not untouched final tests. The 32 reversal examples are all negatives, so 32 of 32 is retention evidence, not a guarantee for every reversal.
The full tier is more accurate than instant on nine of the ten public datasets and ties on the tenth, with lower log loss on all ten. Its biggest leads are on rules you supply, 826 of 872 against 421, and on definitions that change under the same conversation, 106 of 117 against 82. It flags missing information: all 264 unanswerable rule cases, 60 of 60 on the narrow screen and 31 of 34 on the fresh panel. Instant matches it on easy stated facts.
Full is generally stronger, not universally better: instant got 35 of 36 short implicit reversals where full got 30. Full flagged 48 of 70 unanswerable cases on an independent screen, so a high answerable is not a guarantee. Instant flagged none of the missing facts on the fresh panel and got 12 of 32 reversals right. Both tiers are weaker on public questions than on the synthetic panels they were developed on, which is the gap to expect on unfamiliar traffic.
Performance
p50 is the median request time; p95 is the time that 95% of recorded requests finished within. Both tiers ran the same conversations and policies through the same serving gateway on the same RTX 4090, so these numbers compare directly.
| Aspect | Instant p50 / p95 ms | Full p50 / p95 ms |
|---|---|---|
| Short · 2 messages · 1 policy | 7.78 / 7.95 | 25.49 / 25.62 |
| Short · 6 policies | 26.01 / 26.12 | 152.78 / 153.02 |
| Medium · 40 messages · 1 policy | 30.87 / 31.39 | 105.96 / 106.47 |
| Medium · 6 policies | 53.77 / 54.19 | 629.04 / 630.63 |
| Long · 125 messages · 1 policy | 82.15 / 82.76 | 286.39 / 287.27 |
Instant is 3.3 to 11.7 times faster than the full tier across these cells. The gap is widest with many policies, because the full tier reads the whole conversation once per policy: six policies on 40 messages take 629 ms on full and 54 ms on instant. During this run instant peaked at 3.88 GiB of GPU memory and the full tier at 14.17 GiB. Both tiers stay loaded on one GPU and take turns on it, so under mixed traffic a request can also wait for the other tier's work. This screen does not measure production load, sustained throughput or a p99 guarantee, and the latency you see through the API will be higher.
120 observations per cell across three blocks with the tier order alternated, five warm-ups per cell, one request at a time, warm models, no cached inputs. Requests went to the serving gateway over loopback HTTP on the GPU host, so times include authentication, validation, inference, calibration and serialization; they exclude the internet and the public API in front of it.
Methodology
Both tiers are built on pretrained Qwen3.5 base models, with Blooio's task training on top: LoRA adapters on a frozen base, the decision pointer, a new answerability head and, on full, a policy reader. These are adaptations of pretrained models, not new foundation models.
Full was trained in two stages on one RTX 4090. The first adapted the 4.21B base to the task with 127.9M trained parameters: LoRA adapters, the decision pointer and the answerability head, over 512 updates and 22,528 forwards in about 100 minutes. The second froze all of that and trained a 2.38M-parameter policy reader: 1,024 updates, 49,152 forwards and 16.6M tokens from 21,864 training records, in 29 minutes. That stage kept the earlier broad public and synthetic training and added 790 verified training variants, trained in pairs to answer consistently. Seed 29 throughout.
Unchanged from v2.0.0. Instant went through project adaptation, semantic coverage, wording-diversity and natural-text stages, then a final joint-rule curriculum: 256 updates, 4,096 forwards, seed 29, 508 seconds on one RTX 4090. 11.6M parameters are trained; the 752M base stays frozen.
References are synthetic and executable scenario labels, annotations verified by independent AI models rather than by people, and native labels from public training sets. No private customer data was used. Unanswerable cases supervise answerability; their decision labels are masked, never set to 0.5. Each tier's calibration maps are fitted on their own data, kept apart from training and the inspected evaluations. One seed per model limits how far these results generalize.
Inputs
Policies travel with every request rather than being baked into the model. A policy is a key you choose plus a plain-language definition, and the definition is the whole policy.
speaker_id loses that link. Refer to people by role or by what they said.{
"tier": "full", // "full" | "instant"
"messages": [
{
"id": "m1",
"role": "customer", // optional, any label
"speaker_id": "customer-1", // optional
"text": "The package arrived three days late. The screen is cracked."
// "timestamp": "2026-10-07T10:00:00Z" — optional, only if known
},
{
"id": "m2",
"role": "agent",
"speaker_id": "support-1",
"text": "I can send a replacement."
},
{
"id": "m3",
"role": "customer",
"speaker_id": "customer-1",
"text": "Actually, I want a full refund instead."
}
],
"policies": [
{
"id": "delivery_delay",
"definition": "The delivered order arrived later than promised."
},
{
"id": "refund_request",
"definition": "The customer currently asks for money back."
}
]
// "assessment_time": "2026-10-07T10:05:00Z" — optional "now"
}Neither tier generates text. Probabilities come straight from numerical heads and ordinary code assembles the JSON, so instructions inside a message are just more conversation to judge; they cannot change the shape of the response. Requests carry 1–128 policies and 1–2,048 messages in up to 1 MiB. Leave out any optional field you do not know: null, unknown fields, blank strings, duplicate keys and duplicate IDs are rejected, and a conversation too long for the tier is an error, never silently cut.
Outputs
Every request returns one result per policy you sent, in order: a calibrated probability and an answerable estimate. The response also says which tier and version answered. Click a key to read what it means.
full, served. The customer said the colour is blue, and the policy asks whether it is blue.
{
"tier": "full",
"model_version": "bloobability-v2-full-040b3c1d4f9151a4",
"policies": [
{
"id": "colour_blue",
"probability": 0.9789855984058057,
"answerable": 0.9925430186826678
},
{
"id": "colour_green",
"probability": 0.03931965613094063,
"answerable": 0.992896689509792
},
{
"id": "value_equals_blue",
"probability": 0.9804569759039311,
"answerable": 0.99236209781565
},
{
"id": "balance_over_100",
"probability": 0.06696203386124372,
"answerable": 0.0211137142537221
}
]
}{
"tier": "instant",
"model_version": "bloobability-v2-instant-720537bfb0685920",
"policies": [
{
"id": "colour_blue",
"probability": 0.9349263035399773,
"answerable": 0.9835953524610266
},
{
"id": "colour_green",
"probability": 0.11247256078088064,
"answerable": 0.9857958779068584
},
{
"id": "value_equals_blue",
"probability": 0.9351385093028484,
"answerable": 0.9841084243378682
},
{
"id": "balance_over_100",
"probability": 0.05622780832011852,
"answerable": 0.8443138137744634
}
]
}// HTTP 400. Also: 413 over 1 MiB, 422 too long
// for the tier, 429 at capacity, 503 unavailable
{
"error": {
"code": "invalid_request",
"message": "The request does not match the supported contract.",
"details": { "reason": "invalid_tier", "path": "/tier" }
}
}Click a card behind to bring it forward
Both numbers always come back, and neither overrides the other. A low answerable does not null or shrink the probability; it tells you the probability rests on insufficient information. On instant, a high answerable is weak reassurance. Choose and report a consistent treatment of low-answerable cases when calculating rates.
There is no verdict field and no threshold in the response. v2.1.0 ships no audited cutoff for acting without review, and 0.5 is only the split these benchmarks are read at. Use the probability as advice, or set your own threshold and own that decision.
Limitations
v2.1.0 is a usable release, not proof that every research gate we have set passed or that your traffic will match these panels. Instant is offered only for short, explicit questions whose facts are stated. These are the weak spots we measured.
| Aspect | Full | Instant |
|---|---|---|
| Punctuation-only changes · 520 pairs | 12 crossed 0.5, by up to 81 points | 22 crossed 0.5, by up to 60 points |
| Short implicit reversals · 36 | 30 correct | 35 correct |
| ConjNLI · 623 | 513 correct · log loss 0.502 | 395 correct · log loss 0.923 |
The punctuation pairs are 520 verified ConjNLI examples that differ only in punctuation; a crossing is a pair whose two probabilities fall on opposite sides of 0.5. ConjNLI asks about sentences joined by and, or and but, and both tiers find it hard.
Specifications
Two models with separate runtimes behind one endpoint. Neither generates text, so there is no sampling or generation-temperature setting. Calibration uses fitted numerical scaling. The model_version identifies the build; other hardware or numerical paths need their own qualification.
Parameter counts describe the trained model, not a VRAM guarantee. Request limits are transport limits, not proof of quality at maximum length or of a fit within either tier's context.
Download the benchmark summary, tier identities and source hashes