Lance Liang

AI in workflows

Rule Agent

An AI agent drafts clinical trial rules. I designed how study designers check them.

Product
Oracle Clinical One, a clinical trial platform · 2026
Role
Sole designer, with PM, AI engineering, design manager
Speed
3 coded prototypes in 3 weeks
Outcome
My first prototype became the Phase 1 requirements, approved for Q4 2026
On the left, at night, an illustrated study designer sits at her desk, chin in hand, frowning at a wall of JavaScript the AI wrote for a rule, her printed spec untouched beside her. On the right, the same study designer holds her spec and points at the Rule Logic card in the Rule Agent prototype, which explains each part of the rule next to the code lines it comes from.

Context

Complex trial rules are written in code.

Study designers know what a rule should do. Most can’t write the code.

The challenge

AI can write the code. But who checks it?

Rule Agent drafts a rule from her spec. She has to approve it, but can’t read what it wrote.

What I did

I designed how she checks a rule before she approves it.

The AI says the rule back, asks instead of guessing, and shows what the rule does. I built it as coded prototypes.

Tap any image to open it at full size.

Background

A rule protects patients. It takes two people to build.

In a trial, rules watch patient data and act on it.

In a hospital exam room, a research coordinator fills in a visit form on a tablet, the patient across the table.Research coordinator

A research coordinator enters the patient’s answers at each visit.

Over the coordinator's shoulder: answers on the tablet light up and an alert card pops up.

A rule spots a risky combination and emails the site team.

The coordinator shows the alert on her tablet to the investigator at his desk; a patient chart and a handheld lung function test device sit beside him.

The site team reviews it. The patient may need a lung function test.

Study designers write the spec. Getting it into the trial takes a programmer.

Rules are set up before the study starts, by a different person. Complex rules, the ones that combine answers across forms or visits, have to be written in JavaScript.

The study designer at her desk, the trial protocol open on her monitor, writes a rule spec on paper; the spec ends with a small orange bell, the alert the rule sends.Study designer

The study designer writes the spec: when the site team should be alerted.

The study designer holds her spec, marked with the orange bell, beside an empty code editor; an arrow with a question mark points from the spec to the code.

But turning it into code is a different skill.

The study designer walks into a programmer's office and hands him her spec.

So she hands it to a programmer to write the code.

The problem

The hard part started after the code was written

Engineering wanted to cut that dependency, and brought us a demo. It was a good one. A real model, connected to a clinical study. Describe the rule in plain language, and working rule code comes back. The pitch was simple. Our users don’t know how to code, so now they don’t have to.

Engineering’s demo: one sentence in, a whole rule in code.

My first thought was: don’t have to do what, exactly?

The code still lands on someone’s screen, and someone still has to accept it. Are we expecting a person who has never written a line of JavaScript to read that and say yes? Code that runs is not code you can trust.

Before she approves it, she has to know what the rule will actually do.

Did it get what I meant?Does it email again every time the form is saved?

What did it assume?Which grades count as “Moderate or Severe”?

Does it handle the tricky cases?If an answer is corrected, is the alert taken back?

The study designer, chin in hand, thinking.

None of these can be read off the code.

The AI took over writing code. That was never her job. Her job is making sure the rule does exactly what her spec means.

Today, that job is slow.

The only way to find out is to test it by hand.

  1. Test patient
  2. Earlier visits
  3. Enter answers
  4. Check result

Wrong? Fix the rule, start again

The study designer at her computer, working through a test patient's forms by hand.

She only tests the cases she thinks of. Before the rule goes live, a second person tests every scenario again, and any fix sends it back to the start. At one sponsor, each round of this takes a week or two, and a study usually needs about three.

Customers said the same. Our customer advisory board put it plainly. The hard part isn’t writing the first rule. It’s knowing whether it’s right.

The insight

She has always approved code she didn’t write

Every rule today goes from the study designer’s spec to a programmer. She knows the study. He writes the code, but doesn’t know the study. She never reads his code, and she still signs off on it. So what makes that hand-off trustworthy?

Before she signs off, the programmer does three things.

The programmer holds her spec and says the rule back to her.

“So this rule flags high breathing risk at Screening, and emails the site team. Right?”

Did it get what I meant?He says the rule back.

The programmer points at one line of her spec and asks her about it.

“Your spec says ‘Moderate or Severe’. On this form, is that Grade 2 and up?”

What did it assume?He asks instead of guessing.

The programmer shows her a list of test results on his monitor.

“I ran the tricky ones: a score of exactly 7, a missing answer. Here’s what each did.”

What about cases I didn’t think of?He runs the tricky cases.

Each habit answers one of her questions, and tells her what the rule will do. The demo skipped all three. It went from one sentence to finished code, with nothing for her to check on the way.

How might we help her trust the AI the way she trusts a programmer?

So I designed the AI to do the same three things. It says the rule back before it drafts anything. It asks wherever the spec and the form don’t match. And it shows what the rule does, case by case, before she approves it.

What I designed

The AI shows its work at three moments, before she approves

In the design prototype, she starts from the spec she already approved. The three moments below come in the order she meets them.

It says the rule back

It says the intent first, then each part beside the spec line it came from.

Review Behavior in the design prototype, with the read-back highlighted. Intent: Catch high breathing-risk subjects at Screening, so the site team can review them. Below, the rule's parts, Runs when, Needs, Calculates, Checks and Action, each with the AI's reading beside the spec text it came from.

The intent comes first, because if it is wrong, nothing after it matters.

Then come the same parts, in the same order: when it runs, what it needs, what it calculates, what it checks, what it does. That is how study designers already write a spec, and each part sits beside the line it came from. A summary would read better, but you can’t check a paragraph part by part.

If it misread the threshold or attached the rule to the wrong event, she finds it here, in a sentence she can argue with. You cannot argue with a line of JavaScript you did not write.

She doesn’t have to trust the AI’s reading. She can check it against her own spec.

It asks instead of guessing

A correct read-back can still point at the wrong data. Her spec talks in her words. The study has its own visits, forms and answer choices, and the AI matches one to the other.

Six matches were clear. The seventh stopped the work. Her spec says “Moderate or Severe”, and the form grades breathing from 0 to 4. The AI’s best reading was Grade 2 and up. Probably right.

It asks one question, in the form’s own choices, with a recommendation she can confirm.

Match Study Objects in the design prototype. Shortness of Breath needs review: Which grades count as Moderate or Severe? Grades 2, 3 and 4 are preselected, with the note Grade 2 and above is the GOLD cutoff, and a Confirm match button. Create Draft Rule at the bottom is disabled.

Probably is not something you build a study on. And the model can’t settle it by re-reading the sentence. Only someone who knows the study can. So the uncertain match becomes one question, in the form’s own choices, with a cited suggestion, and the draft can’t be created until she confirms it.

Flag only what needs her. The six clear matches just say Matched. Flag everything, and she learns to click through.

Blocking the draft still has a cost. It slows down the person who came here to move faster, and it depends on the AI flagging the right matches. A false flag wastes her time. A missed one passes silently. I took the trade because the two failures are not equal.

A blocked draft is visible. A wrong match is invisible.

Test it before trusting it

Reading what the AI understood still doesn’t show how the rule behaves. For that she has to run it. So the saved draft opens in a test workspace, with tests the AI has already suggested.

It suggests what to test, and says what each test proves.

The Test Workspace in the design prototype, with the What this proves column highlighted: nine suggested scenarios, such as A score of exactly 7 is enough to send the email, No email until all four answers are in, and While testing, the email goes only to the Test to address, never to the site.

What this proves is the column I cared about most. Not just a high-risk subject: a score of exactly 7, a missing answer, a form saved twice. Inputs and expected values can’t show what a set of tests left out. A plain sentence about each test’s purpose can.

It also keeps the AI honest about its own authority. Every expected result in that table is the AI’s claim, not a verdict, drawn from the same reading of the spec that might have been wrong two sections ago.

Any test opens in a real visit with its answers filled in. She saves, and sees what happened.

Step 1 of 4. The test workspace. The Try in visit button on the Score of exactly 7 test is highlighted.Step 2 of 4. A test Screening visit. The Rule Questions panel, listing the four answers the rule reads, and the Save button are highlighted.Step 3 of 4. After saving, the banner reads Email sent to the Test to address, Expected: one email sent, with View email.Step 4 of 4. The Email Preview drawer: sent to rule-testing@exampleclinical.com; Subject TEST-001 scored 7 on the Screening respiratory risk score, threshold 7.

She can also open a visit and try anything she likes. Two routes are more to learn than one, and I considered keeping only the suggested tests.

I kept both because each is weak exactly where the other is strong. The suggested set is repeatable but can never prove it is complete. A free visit can try anything but proves nothing twice.

She approves what she has seen it do.

Phase 1

Smaller scope, same goal

I designed the full version first. Phase 1 had to ship in Q4 and fit how rules are already made, so it keeps the existing setup:

What couldn’t shrink was the goal. Before she approves a rule, she still has to know what it will do. So all three moments had to stay, in a smaller form.

The three moments, in the existing editor

Before the draft, it says the rule back and asks. After, it explains the code and lists what to test.

Before the draft

Phase 1 prototype before the draft. Highlighted in the Rule Agent chat: the read-back, with the intent and the rule's parts; and the one question, with the suggested match Grade 2, 3 and 4 and a Confirm match button.1 · It says the rule back2 · It asks instead of guessing

After the draft

Phase 1 prototype after the draft. Highlighted: the Explained view of the rule logic, each part tied to its lines of code; and the five test cases the AI lists for Testing mode, each with what it proves.1 · …and again, explained3 · Test it before trusting it

It says the rule back in the chat, before the draft. After the draft, the logic is code, so Phase 1 adds an Explained view: the same parts again, each tied to the lines of code it explains. She checks the rule before the code and after it, never in it.

It asks instead of guessing, almost as designed. Where the spec and the form don’t match, the AI asks one question in the chat, with a suggested answer and its reason, and the draft waits until she confirms.

Test it before trusting it is what shrinks the most. There was no room for a test workspace, so the AI lists what to test, edge cases included, each with what it proves. She runs them in Testing mode, not the AI. The model that wrote the rule can’t be its only checker.

She still checks the rule before she approves it, and never by reading its code.

Keeping the parts she could already read

The same goal settled one more decision: what stays out of the code. In review, a cleaner idea came up. Put everything in code, with data references, condition and actions in one block that the AI writes end to end. It demos well, and it is one block instead of a page of fields.

The same rule, drafted two ways. Only one she can read.

All code

Phase 1

CodeExplainedCodeExplainedCodeFields

I argued against it, and the argument is in those two screens. A study designer could already select data, choose an action, name recipients and write the message, and check all of it without reading code.

Putting them in code would make the only readable parts of a rule unreadable, for the people the AI is meant to help.

Those controls belong to the existing editor. I didn’t design them. My part was the case for keeping them once an AI started filling them in. The team agreed. The editor stays, and the AI becomes an optional way into it.

Beyond the screens

Four questions every AI feature has to answer

Designing an AI feature takes more than the screens. These are the questions a team asks, in order, and what I made for each. One is worth a closer look.

  1. DefineWhat will it do?
  2. ShapeWhat will people see?
    • The Phase 1 prototype
    • How it fixes a failed test
  3. ProveDoes it work?
    • Concept feedback from customers
    • What good behavior looks like
  4. GovernCan we run it safely?
    • A history of every change to a rule Proposed
    • What it says at its limits Not yet

Define: when it speaks, and when it doesn’t

The brief had one line about this: the agent asks clarifying questions when required. When, about what, and how was mine to define. I wrote it down as a set of checkpoints between her spec and the draft.

It stops only when the answer is hers, and each answer is one tap.

Everything else it checks quietly. That document became the Phase 1 requirements, and engineering used it to set what the agent supports and what it says when it can’t.

Narrating every check it runs would look like transparency and work like noise.

Where it stands

What Phase 1 left out is what customers asked for next

  1. First prototypeSet the requirementsA week later, the read-back and the asking were Phase 1 requirements.
  2. June 2026Two customers confirmed the problem
  3. July 2026Approved for build
  4. Q4 2026Phase 1 planned to ship
  5. Next phaseWhat customers asked forHelp knowing what to test: the part Phase 1 left out.

How we would know it works. Two numbers: how many rules a study designer finishes without a programmer’s help, which is why the agent exists, and how few rule errors surface after a study goes live.

Closing

What I take from it

Here the AI did the writing. The design was everything around it: what she could check, in words she already uses, and when the AI stopped to wait for her.

AI can do the work. Design makes sure people can check it.

The contribution I would defend is not a screen. It is that at every stage, including the smallest release, she had something specific to disagree with. A sentence, a match, a decision, a result. Not a finished artifact to approve.

What I’d do differently: write good behavior as test cases. On this project, good behavior lived in the behavior spec as rules, which engineering then had to interpret. Next time I’d write it as cases, each an input and what the agent should do.

Engineering can run them as evals on every change, and counting failures by kind shows whether a change made the agent better or worse. The set grows as real failures come in: each one becomes a new case.

Four ways the agent can fail, and a case for each.

FailureIt fails whenA case it must pass
MisreadsThe read-back says something her spec doesn’t.“7 or higher” is read back as “7 or higher”, not “more than 7”.
GuessesIt makes a choice that was hers.“Moderate or Severe” on a 0–4 form: it asks which grades count.
Asks too muchIt asks about something it could handle.No recipients given: it flags the field instead of asking.
OverstepsIt does more than it can, or more than she agreed to.“Show the follow-up form”: it says it can’t, and offers a query.
The four kinds line up with Microsoft’s Guidelines for Human-AI Interaction, such as “scope services when in doubt” (G10), and with the error audit in Google PAIR’s People + AI Guidebook.
Next caseAll work
Regulated enterprise softwareFocused ViewThe patient was ready. The form wasn’t. →