Expertise alignment for AI apps

Why "explain it like I'm ten" wasn't quite the right fix for my prompts and UI copy

While I was building Glamli (my Master’s in AI Dissertation), I got feedback from my adviser regarding the copy used in the app.

So what happened was I used plain language when it was easy. For example: “Columns to learn from” instead of “input features”? But for terms where I couldn’t find a natural plain-language alternative like “one-hot encoding”, I just... used the term. 🤷 No explanation. I guess I was working on the app for so long, it slipped my eye.

Glamli (Guided LLM-Augmented Machine Learning Interface) for my dissertation. It’s a guided tool that helps non-experts build machine learning models. Basically, it’s WEKA through an MCP server I built called weka-mcp.

The problem it caused is that a person who is not familiar with Machine Learning would work one page easily, then hit a wall of Machine Learning jargon on the next page.

This problem was easily spotted by my advisers. As one of my advisers put it simply: When using an LLM, it’s easy to ask it to explain things to you like you’re ten, so why not apply the same idea to the app?

That one comment sent me down a research rabbit hole, and this post is what came out of it. Here’s the breakdown:

• The research question: applying the “explain it to me like I’m ten” trick to a whole app sounded simple, but Glamli’s users won’t all be beginners. What should the tone actually be?

• The paper: Discussed a Microsoft Research study and the findings that mattered for Glamli: tone mismatch hurts most on complex tasks, and users engage most when the AI speaks at their level.

• The implementation: how Glamli adapts its language to two independent things, your Machine Learning expertise and your dataset familiarity, in both the static UI and the LLM prompts, with real code.

• The open question: the paper hints users may actually prefer being talked slightly *up* to. Whether that holds when users pick their own level is what my user tests have to answer.

---

The research question

If someone who already knows machine learning and Glamli explains on each page what a feature is in Machine Learning, i.e., “think of columns like the headers in your spreadsheet” on every page, that becomes annoying really fast.

So the question is: what should the tone be for a task as complicated as building a machine learning model?

Has anyone actually studied this? Apparently, yes. I stumbled on a paper from Microsoft Research and the University of Maryland: ”Speaking the Right Language: The Impact of Expertise Alignment in User-AI Interactions” (Palta, Chandrasekaran, Rudinger and Counts, 2025). It’s the closest thing I found to an answer.

The paper

The paper analyzed 25,033 Copilot conversations from June 2024. The conversations were across domains including programming, business, writing, mathematics, data analysis, biology, and machine learning.

Using GPT-4-Turbo, they assigned three separate expertise labels to each conversation:

• User expertise: inferred from the user’s messages.

• Agent expertise: inferred from the AI’s responses.

• Gauged user expertise: the expertise level the AI appeared to assume the user had, inferred from how it responded.

Each label sits on a five-level scale: Novice, Beginner, Intermediate, Proficient or Expert. They then compared these labels against three outcome measures: satisfaction (an automatically estimated SAT score from -100 to 100), task complexity (low versus high), and engagement (how many words the user wrote across the conversation).

Worth flagging: this is all inferred by an LLM from logs, not from asking real users how they felt. The authors are upfront that the results are correlational. Still, 25,000 conversations is a lot of signal, and two findings jumped out as directly relevant to Glamli.

Figure 8 from the paper (the regression plots showing expertise misalignment vs SAT score for low- and high-complexity tasks). It’s the single most Glamli-relevant chart.

The paper measured a lot of things, but two findings were relevant to Glamli:

Finding 1: getting the tone wrong hurts more when the task is harder.
The expertise gap between user and AI did not matter for simple tasks. However, for complex tasks, the effect mattered more.

Building a machine learning model is a highly complex task, especially for a non-expert. This means that the tone problem my advisers have flagged is actually a really critical issue for the user experience of Glamli.

Finding 2: what made users happy wasn’t quite what kept them talking
Users wrote more and kept the conversation going when the AI communicated at a level similar to their own. Expert users engaged more with expert responses. Novice users engaged more with novice responses. This finding resonates with Glamli. To keep the user working on the ML problem in Glamli, this means the responses and copy should match the user’s expertise.

Applying the findings to Glamli

I started mapping out how to apply these findings on Glamli. I realized there isn’t one kind of “expertise.” There are actually two in Glamli, and they are independent of each other.

1) Machine Learning expertise: how well you know machine learning

ML expertise means your familiarity with machine learning itself. This expertise dictates the wording used across the app, mostly in labels, headers, and subheaders. Glamli asks for it during sign-up. Here are the three options, straight from the sign-up page:

const ML_EXPERTISE_OPTIONS: { value: MlExpertise; label: string }[] = [
  { value: "beginner",
    label: "I'm new to machine learning. Explain terms and tell me why each step matters." },
  { value: "intermediate",
    label: "I know the basics. Explain only the parts that affect my decision." },
  { value: "advanced",
    label: "I'm comfortable with metrics, models, preprocessing, and evaluation." },
];

Notice the options describe the behavior the user wants, not a skill grade. I’m guessing that nobody wants to tick a box that says “I am a novice”.

The sign-up page showing the “How familiar are you with machine learning?” question.

That answer then directs the frontend on the copy to use. The clearest example can be seen on the domain page, where the same field is labeled three different ways:

export const INPUT_SECTION_LABEL: Record<MlExpertise, string> = {
  beginner: "Columns to learn from",
  intermediate: "Input columns",
  advanced: "Input features",
};

“Columns to learn from” vs “Input columns” vs “Input features”

The reasoning per level:

• Beginner: “Columns to learn from”. Columns are what you actually see when you open a CSV. The ‘column’ label meets your mental model.

• Intermediate: “Input columns”. Here I’m assuming the user knows models need inputs. Honestly, I’m not certain it’s the right call, but hey, finding out what makes sense and what doesn’t is what the dissertation is for. 😅

• Advanced: “Input features”. The actual Machine Learning term. Since the user already knows ML, we use the actual term.

2) Dataset familiarity: how well you know your own data

Why is this separate? Because a user could be an expert in Machine Learning but have no idea what the dataset. A data scientist handed a business dataset for the first time doesn’t need “what is a target variable” explained, but might genuinely be confused with default used as “a failure to pay a debt” instead of “automatic setting”.

So Glamli asks about the dataset separately, right after upload, and it asks per dataset rather than per user:

const OVERALL_OPTIONS: { value: OverallFamiliarity; label: string }[] = [
  { value: "unfamiliar", label: "I'm not familiar with this dataset" },
  { value: "somewhat_familiar", label: "I know some, but want help with unclear terms" },
  { value: "familiar", label: "I understand most of the columns" },
];

There’s also an optional free-text note that gets passed along as extra context.

Dataset familiarity mostly steers the LLM-generated text rather than static labels. The answer becomes a tone instruction appended to every agent that narrates the dataset:

FAMILIARITY_TONE: dict[str, str] = {
    "unfamiliar": ("The user says they are not familiar with this dataset — define terms in "
        "plain language and don't assume they know what the columns mean."),
    "somewhat_familiar": ("The user says they know some of this dataset but want help with unclear "
        "terms — clarify anything domain-specific or ambiguous."),
    "familiar": ("The user says they understand most of this dataset's columns — skip basic "
        "definitions and focus on what's genuinely useful to point out."),
}

The open question: is talking up safer than talking down in AI apps?

On the paper, the there was an asymmetery in Satisfaction vs AI expertise.

• When the AI responded below the user’s expertise level, satisfaction tended to fall, and it kept falling as the gap grew.

• When the AI responded above the user’s level, satisfaction generally stayed positive, and often improved, as long as the response was still useful and understandable.

The difference in the paper is that the AI silently guesses your level from your messages, and sometimes guesses low.

Glamli doesn’t guess. It asks you upfront what you think your level is, and then commits to it.

So what happens when users choose their own level? A self-declared beginner getting beginner copy looks aligned, but the paper hints they'd prefer a slight step up.

But the paper suggests that people engaged most with an AI at their own level, so stepping up might trade retention for satisfaction.

I’ve searched, but I couldn't find any study that answers these open questions.

Hopefully these are the things I’ll find out once my dissertation is done. 😅 Fingers crossed 🤞