Can AI predict what I would decide?

Partly. In our test on a public dataset of 2,058 people, a small model picked a held-out person’s answer 60% of the time, against 54% for simply picking the most common answer. When it was 90% sure or more it was right 98% of the time, but that covered only about one answer in seven.

Does Cachejay decide for me?

It answers an agent’s question only when something you already said, decided or corrected plainly settles it, and it quotes where. Anything else goes to you. It never approves spending, sending something in your name or deleting.

What does Cachejay do with its guesses?

When a question goes to you, Cachejay also keeps its own guess at your answer and how sure it was, and checks it against what you answer. The guess is never shown to the agent, and it isn’t used to train AI models. It’s how we measure how often Cachejay would be right.

Why not let the AI just guess when it’s confident?

Because confidence and being right come apart exactly when it matters: when the rule it’s confident about is out of date. In our test the wrong answer we’d least want, a confident one, came from an old rule the person had since changed.

Should an AI agent decide for you? What we measured

The question

Agents now run for an hour at a time. When one hits a fork (monthly or yearly, this client or that one, ship now or wait), the person who knows is in a meeting. Most of these questions have an answer somewhere. Some don’t, and then the choice is: wait, guess, or have something that knows you predict what you’d say.

Predicting is tempting. It’s also the one that can go wrong in your name. So before building anything that answers for people, we wanted numbers: how often does a prediction match what the person actually says, and does the model know when it doesn’t know?

Predicting 2,000 strangers

Twin-2K-500 is a public research dataset: 2,058 people in the US, each answering about 500 questions across four survey waves (personality, thinking tests, economic choices, prices). We hid a fifth of each person’s answers and tried to predict them from the rest, for 300 people the model had never seen: 12,000 answers in all.

Guess at random: 36% right.
Pick what most people picked: 54%.
A small model that reads the person’s other answers: 60%.

Knowing the person helps, but less than you’d hope: six points over the crowd. That matches what the dataset’s authors found: models capture how people differ from each other better than they predict any one person.

The more useful result was the model’s confidence. After calibration, when it said 90% or more, it was right 98% of the time. That covered about one answer in seven. Below that, it was right about as often as it said. So a model can tell you which of its guesses to trust; the trustworthy ones are just a minority.

The model was deliberately small: a sentence-embedding model and a logistic regression, trained in about two minutes on a laptop. Predicting a choice doesn’t need a large model; knowing which predictions to act on is the hard part.

Predicting one of us

Survey questions aren’t work decisions, so we also ran it on one person on the team: twenty decisions they might face in a week (deploy now or wait for CI, rules or a model call, one agent at a time or several), predicted blind from what Cachejay had kept about them, before they answered. They’ve answered nine so far, so treat this as an anecdote, not a result.

It got six of nine. The three it missed are the interesting part: each followed a rule that was written down in their memory and no longer held, or held only in some situations. One miss came with 92% confidence. Asking the model five times didn’t help: it read the same old rule every time and agreed with itself.

On the questions where it had little to go on, its confidence did drop, to between 57% and 75%. So it knew when it had nothing. It didn’t know when what it had was stale.

What we took from it

Answer from the person’s own words, not from a prediction. A quoted decision with a date can be checked. A prediction can’t.
Confidence isn’t enough when memory is old. The worst miss was confident. Rules need dates, and an old rule should be asked about again, not applied.
Put the rest to the person, and learn from the answer. Every answer is a new, dated fact, and a check on how good the guess would have been.
Measure before trusting. Whether a prediction can be acted on is a number you can only get from the person’s real answers, over time.

What Cachejay does

When an agent asks Cachejay something (ask_user), Cachejay answers only when something you already said, decided or corrected plainly settles it, and it quotes where. When the only source is a note an agent wrote about you, the kind of rule our test got wrong, it doesn’t apply it: the agent hears it as likely, and you get “Still true?” with that answer as one click. Otherwise the question goes on your to-do list with a box to answer, from your phone if you like: answering agents from your phone.

When a question goes to you, Cachejay also writes down its own guess at your answer and how sure it is, and checks it against what you say. The agent never sees the guess, and it isn’t used to train AI models. It’s how we’ll know whether, and where, an answer for you could be trusted. Until the numbers say so, it comes to you. Why agents stall in the first place: when your AI agent needs you and you’re not there.

Sources: Twin-2K-500 (arXiv 2505.17479) · the dataset on Hugging Face. Checked October 2026. Something out of date? Email hello@cachejay.com.