See how it works

Now brewing

Checkpoint and attack test for AI agents

Kettle Labs is an independent AI lab building a checkpoint that holds an agent's risky actions for a human yes, and an attack test that counts what gets through.

See how it worksRead the test
Attacks in first test
23
Model went along
30%
Ran without a yes
0%

The problem: agents act on text that strangers wrote

Web pages, emails, files and invites all reach the model as text. Some of it is written to give orders. A model cannot always tell its owner's request from one planted in a file.

Text arrives from anywhere.

The model picks the next action.

The action lands in a real inbox, calendar or bank account.

See what the test found

One report for every attack the test runs

Now brewing

The test plants believable trick text where an agent reads. Its report counts attacks run, attacks the model went along with, actions that got through, and normal jobs still finished.

See the checkpoint

Attacks will fool the model.

In the lab's first test on a working agent, the model went along with 30% of 23 believable attacks, and none of those actions happened without human approval.

The checkpoint holds risky actions for a yes

Now brewing

The model can only ask. Plain code then checks who is asking, what the action touches and whether a person has to agree. Risky actions wait for a yes on your phone.

Code writes the approval message.

A yes covers one exact action, then expires.

Some actions stay locked whatever the message says.

See the lab's rules

Approval needed
Waiting
The agent wants to send your reply to the supplier.
Send an emailAsked for by you, in chat 1 email

Superintelligence, supervised.

The smarter the agent, the more the checkpoint matters. Both parts are plain code, free and open source.

The lab's four rules. Behind the checkpoint and the test.

Keeps the checkpoint in code

Plain code with fixed rules, not a second model. A persuasive message has nothing to argue with.

Fits the checkpoint to any model

It sits between a model and its tools. Change the model and the rules stay the same.

Model in use
Same rules
  • Hosted modelRuns in the cloud
  • Newer modelSwap in when it ships
  • Local modelRuns on your machine

Opens the checkpoint's source

Both parts will be free and open source. Anyone can read the rules their agent runs under.

rules.md, open source
Rules the agent runs under
Sending an email needs a yes.
Deleting a backup never runs.
Sharing a file needs a yes.
Paying needs a second check.
The model cannot edit rules.
Every action is logged: asked, answered, ran.
Reading a file runs. Searching the web runs.

Publishes every test failure

Every report prints what the model went along with. The first says 30%.

  • 30% went alongThe model asked for the planted action.
  • 0% got throughEach one was held for a yes.
  • 23 attacks runFirst lab test

Questions

Short answers on what exists today.

Release list
0 of 5 out
  • Checkpoint code
  • Rule format
  • Test runner
  • Report format
  • Open source licence

Can I install the checkpoint today?

Not yet. Both parts are still being built, and neither has a release.

Does the checkpoint stop every attack?

No. A convincing request can still talk a person into a yes, and approvals only work while they are few. It also covers only the actions that pass through it. The lab's own attacks are a minimum.

What will the checkpoint and test cost?

Nothing. Both parts will be free and open source when they are ready.

Why is the checkpoint not a second AI?

A guard model reads the same text and is fooled the same way. Fixed rules are not, and they stay the same under any model that calls tools.

Who builds the checkpoint and the test?

One engineer, James Cronin. Kettle Labs is an independent lab with nothing for sale.