Jev Implementation Training Course

This course is intended for developers and technical decision-makers who want to incorporate AI judgements into software while retaining clear control over application behaviour.

It will be particularly useful for:

• Backend and full-stack developers building classification, routing, scoring or moderation into applications

• AI and machine learning engineers evaluating alternatives to generative-model calls or agent-based workflows

• Technical leads and solution architects deciding where typed decision models fit within an application

• Platform, quality and security engineers responsible for testing and monitoring AI-enabled components

The course concentrates on implementation and evaluation rather than on training machine learning models.

Delegates should be able to:

• Write Python functions and work with dictionaries or dataclasses

• Use a Python testing framework such as pytest

• Read and work with JSON

• Understand the basic purpose of an API call

Some previous experience of calling an LLM or another hosted AI API will be helpful. No machine learning background is required.

This Jev course is available for private / custom delivery for your team - as an in-house face-to-face workshop at your location of choice, or as online instructor-led training via MS Teams (or your own preferred platform).

Get in touch to find out how we can deliver tailored training which focuses on your project requirements and learning goals.

Introducing Jev and typed AI decisions

• How Jev differs from generative language models

• Application state and typed probabilistic answers

• Choice, Score and Noul question types

• Probabilities, confidence and structured responses

• Current model and context limitations

• Making an initial request using the Python SDK

Practical exercise: Send application state with several question types and inspect the returned answers, confidence, model version and token usage.

Choosing where Jev fits

• Using ordinary code for explicit rules and calculations

• Using Jev for bounded judgements over unstructured information

• When an LLM or human review is more appropriate

• Keeping control flow and side effects in deterministic code

• Suitable applications, including routing, classification, scoring and verification

• Evaluating performance and cost claims against representative data

Practical exercise: Assess a series of application decisions and choose whether each belongs in ordinary code, Jev, an LLM or human review.

Designing a decision component

• Selecting the relevant application state

• Designing narrow, independently testable questions

• Using contrastive criteria to clarify decisions

• Combining answers with deterministic rules or weighted calculations

• Short-circuiting cases that do not require a model call

• Patterns for routing, scoring and parallel evaluation

Practical exercise: Build the first version of a customer-support triage function, combining topic, spam-risk and customer-frustration judgements.

Working with probabilities and confidence

• Reading distributions and confidence values

• Calculating alternative confidence measures

• Creating act, confirm and escalate bands

• Adjusting thresholds to reflect the consequences of an incorrect decision

• Understanding why a structurally valid answer may still be wrong

Practical exercise: Apply confidence bands to a batch of results and identify which cases can be handled automatically and which require review.

Testing accuracy, calibration and stability

• Building a labelled test set

• Measuring accuracy against human-defined labels

• Comparing predicted probability with observed accuracy

• Choosing thresholds using the cost of errors and human review

• Testing option order, repetition, negation, numbers and dates

• Working around known weaknesses using deterministic code

Practical exercise: Evaluate the triage component against labelled fictional tickets, choose operating thresholds and measure stability across repeated and reordered requests.

Versions, regression testing and operational behaviour

• Pinning model versions rather than relying on a moving alias

• Recording the model version associated with each decision

• Converting labelled examples into pytest regression tests

• Comparing behaviour following a model change

• Measuring latency and API cost

• Handling rate limits, overload and exhausted retries

• Limiting and filtering the state sent to the service

Practical exercise: Create a regression suite, compare two model versions and measure request latency.

Threat modelling and adversarial testing

• Treating model input as untrusted data

• How instructions and misleading claims inside application data can affect decisions

• What a closed answer space does and does not protect

• Building an attack set

• Measuring attack success by question and attack type

• Keeping adversarial cases in the regression suite

Practical exercise: Attack the triage component with misleading, contradictory and instruction-bearing input, then identify its weakest questions.

Defence in depth and safe failure

• Reducing the amount of attacker-controlled state

• Separating untrusted content into clearly named fields

• Applying deterministic validation that the model cannot override

• Escalating uncertain and high-impact decisions

• Selecting safe defaults for timeouts, unavailable services and low-confidence answers

• Logging sufficient information to reconstruct a decision

• Preparing a release checklist

Practical exercise: Add layered defences, rerun the attack set and complete a release checklist for the finished component.

Course outcome

By the end of the day, you will have created a Python decision component that uses Jev to classify and assess fictional support tickets.

You will have:

• Tested it against labelled data

• Selected confidence thresholds

• Measured its stability

• Converted its tests into a regression suite

• Challenged it with adversarial input

• Added safe fallback behaviour

• Documented the checks required before release

What is Jev?

Jev is a decision model from TypeSafe AI designed to return structured, probabilistic answers rather than generate free-form text. You provide application state and one or more typed questions, and Jev returns answers that your software can evaluate and act upon. This course explores how to use that capability while keeping application control, safeguards and side effects in deterministic code.

How is Jev different from a large language model?

Large language models are particularly useful for tasks such as generating, transforming and reasoning over text. Jev is intended for narrower decision-shaped tasks such as classification, routing, scoring and verification. During the course, you will examine where Jev is appropriate and where ordinary application code, an LLM or human judgement is a better choice.

Do I need machine learning experience to attend?

No. The course is aimed primarily at software developers and engineers rather than data scientists. You should be comfortable writing Python, working with data structures such as dictionaries or dataclasses, and using a testing framework such as pytest. Some previous experience of calling an AI or LLM API will be helpful, but you do not need experience of building or training machine learning models.

Is this a hands-on course?

Yes. The course is practical throughout. You will build a fictional customer-support triage application in Python and progressively add decision logic, confidence thresholds, tests and safeguards. By the end of the day, you will have taken the same component through implementation, evaluation, adversarial testing and preparation for release.

What software and access will I need?

You will need a laptop with Python 3.10 or later, a code editor and permission to install Python packages. The exercises use the typesafe-sdk package and require access to the TypeSafe API, so you will also need the appropriate API credentials and outbound HTTPS access. Practice datasets and starter materials are provided for the course.

I would leave this one provisional until Olly confirms how Framework wants to handle API accounts and credentials.

Will we use our own company or customer data?

No. The course exercises use fictional and synthetic data, including labelled examples and deliberately hostile test cases. You should not bring real customer, personal or commercially sensitive data into the practical exercises.

Does Jev replace conventional application logic?

No, and understanding that boundary is an important part of the course. The approach taught here keeps deterministic application code responsible for things such as explicit rules, calculations, permissions, control flow and side effects. Jev is introduced where a probabilistic judgement is useful, with its answer checked before the application decides what happens next.

How do we know whether a Jev decision is reliable enough to use?

That is one of the main themes of the course. You will build a labelled test set, measure accuracy and calibration, examine confidence and probability values, and choose thresholds according to the consequences of making an incorrect decision. You will also test stability across repeated requests and model changes rather than assuming that a valid response is necessarily a correct one.

Does the course cover AI security and prompt injection?

Yes. The later part of the course treats input to the model as potentially untrusted. You will test the application against misleading content, contradictory information and injected instructions, measure the effect on its decisions, and then add defensive layers before running the tests again. The emphasis is on defence in depth rather than assuming any individual model or guardrail can provide complete protection.

What will I have built by the end of the course?

You will have built a Jev-based triage function in Python and tested it using labelled and adversarial data. You will have added confidence-based routing, regression tests, safe fallback behaviour and layered defences, and worked through a release checklist for putting a Jev-based decision component into an application.

1. TypeSafe AI: Jev documentation
   Covers Jev’s System One approach, application state, typed questions, Choice/Score/Noul primitives, probabilities and confidence.

2. TypeSafe AI: Jev Quick Start
   A particularly useful practical companion to the course, covering API-key setup, the SDK/API, question types and a first Jev request.

3. OWASP: Prompt Injection
   OWASP covers both direct and indirect injection and the risks posed when untrusted content can influence model behaviour.

4. OWASP: LLM Prompt Injection Prevention Cheat Sheet
   Covering input handling, external-content attacks and defensive controls; a great resource for developers after the course.

5. NIST: AI Risk Management Framework
   NIST positions the AI RMF around trustworthiness throughout the design, development, use and evaluation of AI systems.