How should an AI agent do machine learning for someone who can't check its work?

Three ways an agent can do machine learning for you, and what I built to test the third one.

I want you to picture a Business Analytics class in a university now. The class is handed a Customer dataset and asked to build and evaluate a predictive model for churn. The first thing you’ll notice is that a growing share of students are looking at the data anymore.

Their first instinct is to upload the file to an LLM and ask it to run the analysis. Unfortunately, in some instances, they just remove the em-dashes and hand in the report. Some of them cannot yet critically read a confusion matrix, which is precisely the skill the assignment was meant to build.

This story is not unique. The details change by department. A literature class hands out a novel, an engineering class asks for a load calculation. The scenario changes, but the shape does not: the university work meant to build a skill gets outsourced to AI that already has it.

The irony is that everyone outside the university wants these students to use AI. Job descriptions ask for it. And honestly, using it this way is the most convenient option. I’d be lying if I said I’ve never taken the shortcut. But I don’t think this is only an academic problem. Scroll LinkedIn for ten minutes, and you’ll find someone venting about a colleague who forwarded a report, a message, or a whole strategy document straight out of Claude, unread.

So why is it so hard to read the thing before handing it to another human?

Let me reference Brandolini’s law, the bullshit asymmetry principle:

“The amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce it.”

It was coined in reference to internet arguments, but it holds true here.

Reading a report properly, checking whether the numbers make sense, and whether the conclusion follows, is slower than generating it. It always was. The difference is that we’ve now built a tool that is extraordinarily efficient at the “producing” half. The “refuting” half has not moved. Maybe that’s the actual work ahead of us: not better generators, but better ways to check what they hand back.

I see this from close range. My Master’s dissertation in AI at the University of Waikato examines how LLM-based interfaces should present machine learning to people who are not yet, or may never need to be, ML experts.

Getting an AI to do ML automatically is established. AutoML has been around for years, and yes, an agent can train a model. What I keep getting stuck on is the mechanism. Through what process should it do that when the person asking has no realistic way to check the work?

I think this generalizes to workplaces right now. People are shipping apps, websites, and analyses they could not have built on their own, in fields where they are not experts. When it works, it looks like a win, and most of the time it often is. But the issue comes up when it fails. Their name is still on it, and the problem is that even if they created the work, they are the least equipped person in the room to explain why it failed. They do not need to become experts. But they need some way to refute the bullshit generated by AI. A tool to do work that makes it easy to check the work, wherein you are not yet an expert in doing the work.

Back to the mechanism question for ML. When an agent does ML on someone’s behalf, there are really only three ways it can happen that I have observed. And those fail in very different places.

Three ways an agent can do ML

1) Advisory: the human executes

How it works:
This is when you use ChatGPT or Claude to get advice on how to work on the ML task. They never touch the data. You describe the dataset, ask which models might suit it, talk through hyperparameters, and get back an explanation and code. Then you run it yourself, bring the output back, and ask what it means. Back and forth, until something works.

Pros:
This is the best collaboration of AI and humans among the three methods. Assuming that you did not just copy and paste everything from the conversation, and you thoroughly read it and analyzed it. Nothing enters the report that you did not run and look at yourself.

Cons:
It only works if you can already set up the environment, run the code, and read an error message, which is exactly the assumption that fails for the students in the analytics class who are not familiar with Machine Learning. And it requires effort at every step, while the other two options require almost none.

2) Code-mediated: the agent writes and runs the analysis

How it works:
The agent writes a script, runs it against your actual data, reads the output, and iterates without you. Claude Code and Codex work this way today. For heavier jobs, it can edit code locally, dispatch the run to Azure or AWS, pull the results back, and carry on.

Pros:
Anything that can be done in code is available, and the whole loop running models and getting results closes without a human in it.

Con:
Every run is a fresh “implementation.” So this means that the same request on different days can take different paths and use different library versions with different defaults. Auditing requires expertise that the student is still developing. Failures of applied ML, like data leakage or silent data loss, where a preprocessing step quietly drops the rows that were hardest to handle, can happen. And these types of errors are really hard to detect because it still gives you a result.

3) Tool-mediated: the agent works through a fixed set of operations

How it works:
The agents can choose specific actions (vocabulary) of a specific tool. It can pick and combine them. It cannot invent one.

Pros:
What happens is that it becomes more deterministic. Same inputs, same library version, same outputs. So if the model produces “87.3% Accuracy”, it's much easier to trace since the agent just has a limited vocabulary to use.

Cons:
If nobody built the tool, the agent cannot improvise around it. The agent is limited to the tool it’s provided.

The problem

Here is the problem. In the first two methods, the agent generates the code, and because AI is non-deterministic, there is ample surface area for it to get things wrong. The standard advice would be to put a human in the loop. But that advice assumes the human can check the work, and here they cannot do so. By limiting the tools the AI can use, it reduces what has to be checked to something a non-expert can inspect.

That said, tool mediation is not the automatic winner of the three methods. It only exists if someone builds the tools, and the fixed vocabulary is a real cost. Nobody can say whether that trade is worth it until the thing exists and people test it.

So I built WEKA-MCP: WEKA, driven by an agent through a fixed set of tools. This is a part of my dissertation which “tests” the third method.

WEKA-MCP

WEKA-MCP exposes WEKA, the open-source machine learning workbench developed at the University of Waikato, to any Model Context Protocol client through 46 MCP tools, each defined by a JSON Schema contract. Claude Code, Codex, or any other MCP-compatible client can upload datasets, explore them, train and evaluate models, cluster, mine association rules, and run diagnostics, with every operation executed by the real WEKA library.

The video is a demo showing a full session: inspecting a dataset, training a model, and reading back the results.

WEKA is a good fit for this. It has been used in ML teaching and research since the 1990s; it is deterministic and versioned, and its functionality is already broken into discrete, parameterized pieces: filters, classifiers, clusterers, evaluators. That structure maps almost one-to-one onto a set of tools.

The architecture

The system is two services with a strict chain of responsibility:

weka-mcp is a TypeScript/Node MCP server. It exposes the 46 tools, each with a JSON Schema input contract, and validates every call against its schema before anything is forwarded. weka-api is a Java 17 service wrapping WEKA 3.9.6 behind REST. That is where the machine learning happens, and where trained models and datasets persist to disk, so an agent can train a model in one conversation and evaluate it in another.

The single most important design rule in the project is that weka-mcp contains no ML logic at all. It is a thin, typed forwarder and nothing more

Does this prove that the third method is better?

Building WEKA-MCP demonstrates that tool-mediated machine learning is feasible: a complete analytical workflow, from upload to calibration plot, can run through a closed, JSON Schema-validated vocabulary.

It does not yet show that tool mediation works better for ML non-experts than simply letting a code execution agent decide what to do. What could happen is that an agent with more freedom may be able to handle situations that a fixed set of tools cannot, and that flexibility may matter more in practice than having tighter control.

That is what my dissertation is about. When it is done, the findings will get their own blog post here.

In the meantime, generating the “bullshit” part of the tool I’m building is done and open. If you build on it, break it, or connect it to a client I have not tested, I would genuinely like to hear about it.

The repository is at github.com/iamademar/weka-mcp.