Source: TypeSafe’s cookbooks and demos at docs.typesafe.ai/cookbooks, 25 cookbooks plus the smart-home demo, fetched 2026-11-01 and saved one per file as ai-research/typesafe-docs-cookbook-*.md and ai-research/typesafe-docs-demo-smart-home.md (full page text, code included).
Project: TypeSafe (Jev) · URL: https://docs.typesafe.ai/cookbooks · Version: mostly jev-1.13; the SDE cascade’s 100-prompt chart used jev-1.12; Box recipes need jev-1.14
TypeSafe’s end-to-end recipes for Jev: complete Python notebooks that solve one problem each, with cached results so the published numbers re-render without API spend. They are the best evidence of how TypeSafe expects Jev to be used: a cheap method finds candidates, Jev judges them, and plain code owns thresholds and control flow. They are also TypeSafe’s own tests, mostly on small samples (40 queries, 60 filings, 8 citations, 4 hierarchies), so read the numbers as worked examples, not benchmarks. The API itself is covered in Jev API Reference.
Key Takeaways
- The recurring shape: a cheap step (BM25, a regex, a mini LLM, an OCR engine) proposes candidates. Jev scores or picks among them, and code applies thresholds. Jev almost never produces the value itself.
- Measured wins, all in TypeSafe’s own tests:
- Legal re-ranking took top-1 accuracy from 5% to 18% and top-10 from 38% to 62% (40 queries).
- Skill selection halved wrong skill loads, from 15.7% to 7.9%.
- Wine-score prediction fell from RMSE 2.47 (word counts) to 1.77 (Jev features).
- A cheap-model + Jev-verifier cascade matched about 96% of a reasoning model’s quality at about 44% of its cost (0.79 vs 0.82 judge score; about 0.045). ^[inferred] The ratios are computed from the cookbook’s figures.
- Batching 13 questions into one call was 12.2× cheaper and 10.0× faster than 13 separate calls.
- Speed and cost against LLMs in the self-consistency cookbooks: Jev took about 114 ms and $0.000046 per call. Claude Haiku 4.5 took about 3.9 s and cost 76× more, and reasoning models cost 600–900× more. Those cookbooks measure repeatability, not accuracy, and TypeSafe says so.
- Pair every “where” with an “is it there”. A Box always returns a box and a Choice always picks an option, so recipes add a Noul (“is there a mug?”, “does the document contain an answer?”) and ignore the pick when it’s low.
- Two thresholds, three outcomes. Nearly every recipe sends a middle band to a person: Nouls between 0.3 and 0.7, a 0.6 top-probability floor, review below 0.25, and so on. The bands differ by risk.
- Confidence can change what you report, not just whether you act. With SEC filings, Jev’s industry group was right 90% of the time when confidence was at least 0.9, and 40% when it was lower. Reporting the low-confidence ones at the broader division level raised them to 70%.
- Use a second request only when it truly depends on the first. Examples: rerank the top three skills with their full text, or classify the blocks that a first pass created.
Search and Retrieval
| Recipe | How it works | Result (TypeSafe’s test) |
|---|---|---|
| Re-ranking | BM25 builds a 30-passage shortlist per query; one TypeSafe question per query–candidate pair; sort by the probability | 40 CLERC legal queries. The shortlist contained the right passage 100% of the time. Top-1 5% → 18%, top-5 15% → 35%, top-10 38% → 62% |
| Line-by-line search | Tag every line of a document with an ID; one Choice over 218 line IDs ranks the lines, and one Noul (exists) checks whether the document answers at all, in one request | GitHub ToS: an arbitration query ranked the closest line at 0.86, but exists was 0.14, so the answer isn’t in the document |
| Classifying RAG passages | Four Nouls per retrieved passage (prompt injection, contradicts the query’s premise, relevant, contains answer evidence), routed in a fixed order | 72 passages across 6 queries. At least two-thirds of every query’s passages were excluded. Thresholds: injection > 0.70 excludes first; premise contradiction > 0.70 goes to a “conflicting evidence” block; evidence > 0.55 is included |
| Skill suggestion | Request 1 scores the agent’s own reach and ranks Nous Research’s Hermes catalog (194 skills). Request 2 rechecks the top 3 using their full text | Wrong skill loaded 15.7% → 7.9%; a skill loaded when none fit 14.0% → 5.2%. An agent handed the right answer scores 3.2% / 4.1% |
Classification
- Hierarchical classification. A Choice at each tree level, using beam search (K=3) over the Choice probabilities. Paths are scored by geometric mean,
product(p) ** (1/decisions). On four hierarchies (CPC patents, Shopify products, MeSH, a codebase), beam search found 4 of 4 expected leaves and greedy search 2 of 4. Greedy put a poultry-perch patent under “not otherwise provided for” and a cat window bed under “Pet Chairs”. - Classification using confidence. 60 SEC 10-K business descriptions, one Choice across 75 SIC industry groups. Groups answered with confidence ≥ 0.9 were right about 90% of the time; below 0.9, 40%. Reporting those at the division level made them 70% right.
- Entity alignment. For 450 candidate pairs from two beer catalogues, one Score (same product?) plus three Nouls that show which fields disagree. 360 pairs were left unlinked, 40 merged, and 50 sent to a curator. The cut points depend on how the middle level is worded, not on tuning.
- Twitch chat filter. Three questions per message: a category Choice, a toxic Noul and a spoiler Noul. Code rules on the category hide insults, spam and unrequested tips and highlight questions, and the spoiler Noul blurs spoilers. Median latency was about 0.1 s, so a line shows in chat before its verdict arrives. The cookbook warns that non-English chat is less accurate, and that the spoiler check can’t know how far the streamer has played.
Extraction
| Recipe | Pattern | Detail |
|---|---|---|
| Date extraction | Choices pick date parts (month, day, year, absolute/relative), each with a “not stated” option; code assembles and validates the date | Six dates from four documents, including “next Thursday” and a deadline with no year. Code fills in the year. Dates below 0.60 confidence, or ones that can’t be assembled, go to a person |
| Pre-parsed value extraction | A regex finds candidate emails, phone numbers and amounts; a Choice picks the one asked for (“the address for the receipt”); the span is copied verbatim and normalized (e.g., E.164) | Limit: 255 options per Choice, so narrow by section first. Names have no regex, so candidates must come from a roster, an NER tool or an LLM |
| Function calling | Closed-set arguments in the function signatures become Choice questions; a “stated” Noul stops defaults being invented | 14 trading commands. “Plot rolling correlation between nvda and spy for the past month” filled four arguments correctly. Call confidence is the minimum of its judgments, not their product |
| Shopping cart | A loop of Choice questions over a shrinking option list finds every item in a café order; one batched request per item fills its fields | Handles an unknown number of items |
| Structure recovery | Pass 1: a Noul per line break (“did this split a sentence?”). Pass 2: a Choice per block (heading, list, code, callout) | The join threshold depends on the previous line’s punctuation: 0.2 after a dangling line, 0.5 after ./:. Blocks with type probability under 0.55 are underlined for review |
| SDE cascade | A mini LLM extracts; Jev scores each field; any one flag over the threshold escalates the item to a reasoning model | On 100 ScrapeGraphAI prompts (an internal TypeSafe run with jev-1.12), a cascade point at about 0.045. The cascade frontier sits above and left of every single model |
Verification and Guardrails
- Double-checking citations. An exact string match first, which catches fabricated quotes. Then one Choice per citation (verified, contradicted, unsupported) against the quote’s section, with auto-accept at confidence 0.8. On 8 citations against RFC 7519: four verified (≥ 0.93), one fabricated, one contradicted (0.99; the RFC says the claim is OPTIONAL), and two unsupported and sent to a human. One of those quoted the source exactly, which is why the string match alone isn’t enough.
- Guardrails for LLMs. One request per side screens every message going into and out of an LLM app. Five Nouls cover instruction override, “stop refusing”, harm or crime, diagnosis or dosage, and self-harm, plus a severity Score. A policy is just a set of thresholds: strict reviews at ≥ 0.35 and blocks at ≥ 0.70, while permissive reviews at ≥ 0.70 and acts at ≥ 0.85. The same jailbreak is blocked under strict and sent for review under permissive.
- OCR confidence and bounding boxes (image beta). Jev boxes each OCR’d invoice field and scores the value against the full page and against a zoomed crop. A field is flagged if either score is below 0.25. Neither check catches all three misreads on its own: the crop check caught
AN400074(0.18) and the page check caught16/09/2028(0.17). All correct values passed. - Reading a scanned vaccination record (image beta). A Noul plus a Box finds each vaccine, a Choice with an
unreadableoption reads each dose’s year, and a separate text-only request checks whether the dose covers the trip. Asking the date question directly on the scan let an out-of-date typhoid dose pass at 0.50, because one answer then covers two steps.
Images and Real-Time Apps
- Feeder-cam visit log (image beta). One frame a second, five questions per frame, including a Box for the bird. Code turns the answers into visits per species. It missed one woodpecker that shared a frame with a cardinal, and once put a “male cardinal” box on a woodpecker. Limits it states: one box per question; birds of the same species can’t be told apart by name (use a detector such as SAM 3, then have Jev name each crop); frames must be at least 280 × 280 pixels.
- Intelliprompter. Ticks off a presenter’s notes as they speak. The live transcript is the state, each unchecked note becomes a 5-level Score, and a note stays ticked once it scores ≥ 2.5. The threshold and the “stays ticked” rule live in code.
- Smart home assistant demo. Category, room, device and action are asked as Choices in one call. General chat falls back to an LLM, and Jev’s fast first answer “adds negligible latency”.
- Simple memory system. A gate Noul asks whether a ticket states a fact worth remembering; a wrapper adds saved memories to later calls. With memory, 6 of 6 follow-up tickets reached the right team, against 3 of 6 without. The same message (“slow dashboards”) routed differently for a self-hosted Enterprise customer than for a cloud Team customer.
Evaluation and Feature Engineering
- Parallel questions. 13 questions (8 Noul, 2 Choice, 3 Score) about the 54,000-character GDPR Wikipedia article, asked 5 times each way. One batched call gave the same answers as 13 single calls at 12.2× lower cost and 10.0× faster. The cost gap remains even if the singles run concurrently, because the article is re-sent 13 times.
- Self-consistency: choices. 8 moderation Choices on one borderline post, run 15 times. Jev repeated its plurality label 90.8% of the time, against 87.5–100% for the LLM settings. Its probabilities varied less than five of the six LLM settings, and less than all but Haiku at temperature 0. Adding a 0.60 top-probability floor (below it the answer is “uncertain”) raised agreement to 99.2%, with 74.2% of answers labelled automatically.
- Self-consistency: nouls. 14 claim-triage Nouls, run 15 times. Jev’s mean per-question standard deviation was 0.0102, the lowest of every condition. But one answer still ranged from 0.43 to 0.53, crossing a 0.5 threshold. The cookbook’s fix is to make 0.30–0.70 an explicit “uncertain” outcome.
- Cost and latency in those two runs: Jev took about 111–114 ms and $0.000043–0.000046 per call. Haiku 4.5 took 1.0–3.9 s (9–34× slower) and cost 33–76× more. GPT-5.4-mini was 7–20× slower and 20–50× costlier. GPT-5.5 and Opus 4.8 with reasoning were 90–125× slower and 600–900× costlier.
- Autoresearch feature discovery. An LLM proposes TypeSafe questions over wine tasting notes. Each answer becomes a numeric feature for a CatBoost model predicting the critic’s score, and the model’s errors guide the next round. Held-out RMSE on 800 reviews:
| How the note becomes a score | RMSE |
|---|---|
| Predict the training average | 3.09 |
| CatBoost on word counts | 2.47 |
| Ask TypeSafe for the score directly | 2.15 |
| 18 questions from one proposal, no loop | 1.87 |
| 38 questions after five loop rounds | 1.77 |
Try It
- Check citations in AI-drafted content. Run the citation-check pattern on a drafted article: exact-match each quote, then one Choice per claim against its source section. Fabricated and contradicted citations get caught before publishing.
- Guard a client-facing chatbot. Copy the guardrail setup (five Nouls plus a severity Score per side) and write your own strict and permissive threshold sets. Medical-advice (“diagnosis or dosage”) and self-harm checks are already in the battery.
- Route inbound leads with memory. Use the simple-memory gate so a lead’s earlier “we have three locations” changes how their next message is routed. Keep the memories in a database keyed by contact ID.
- Pull phone numbers and emails from form text. Use the pre-parsed pattern: a regex finds the candidates, Jev picks the one asked for, and code normalizes it. Don’t ask Jev to write the value.
- Rerank before an expensive model reads anything. BM25 or vector search, then one Jev question per candidate. Measure top-1 and top-10 on 20–40 of your own queries before adopting it.
Open Questions
- Nearly all results are TypeSafe’s own, on small samples. The only independent tests in this wiki are in the main Jev article (Artificial Analysis; an arXiv replication study; a Substack calibration test).
- The cookbooks don’t say whether the 1.13 results hold for
jev-1.14. The SDE cascade chart used the olderjev-1.12. - The two image cookbooks run on a beta feature that TypeSafe says is not production-ready.
Related
- Jev API Reference — TypeSafe’s Official Docs — the question types, confidence formulas and limits these recipes rely on
- Jev (TypeSafe AI) — use cases, creator tests, funding, independent benchmarks
- Pairing Jev with Claude Code (Simon Scrapes) — a support-triage build in practice
- RAG and Vector Retrieval for Agents — where re-ranking and passage filtering fit
- Cheap-Worker Routing — the cascade pattern across vendors