But there’s a whole other side of AI that almost nobody talks about: using it to predict things. How many cups of coffee will you sell next week? Is it a good day to sell coffee? ☕️ Which customers are about to stop buying? Whether you should stock up on inventory or hold off. These are the kinds of decisions that business owners make on gut instinct every day, and AI can actually help with them.

The tools already exist. They’re called AutoML platforms. Basically, you plug in your business data, and the system builds a model that predicts sales, demand, churn, whatever matters to your business.

Not familiar? You’re not alone.

Researchers sat 16 regular people in front of one of these tools and tasked them to build a Machine Learning model. Not a single one could figure it out. These tools were built for data scientists, not the people who actually need them.

The irony is that building an ML model might feel foreign, but chances are you’re already using this exact technology every single day. You just don’t call it that.

When Spotify queues up a Discover Weekly playlist that somehow nails your mood. That’s a machine learning model trained on billions of listening patterns, predicting what you’ll want to hear next. When YouTube serves you an ad for running shoes, the week you started watching marathon training videos. That’s a prediction model matching your behavior to an advertiser’s audience. Even this post showing up in your feed right now? LinkedIn recently rebuilt its feed algorithm using large language models and transformer architectures, designed to understand what a post is actually about and match it to professional interests.

All of these are just Machine Learning (ML) models. Data goes in, patterns come out, and the system takes its best guess at what you’ll do next. The exciting part is that you can use the same tech to predict next week’s sales, spot customers who are about to churn, or figure out how many tomatoes to order for your sandwich shop using your own data to maximize your profit.

The tools to build your own ML models with machine learning already exist. Platforms like Azure ML, Google Vertex AI, and Amazon SageMaker all offer what the industry calls Automated Machine Learning. Think of it like this: if you want to forecast next month’s sales numbers or identify which customers are likely to stop using your service, these platforms let you do exactly that with your own data. The goal of these platforms is to let you avoid hiring a data scientist. Just upload your spreadsheet, press a button, and the system figures out the best ML model for you.

What actually happens is you upload your Excel file and immediately hit a wall. The system asks for your “target variable,” basically, what are you trying to predict? Maybe you are next week’s tomato sales. Then it wants to know if this is “regression” or “classification.” Translation: are you predicting a number (like sales), or a category (like yes/no)? Next up: pick an “evaluation metric.” RMSE, MAE, R²—just different ways of measuring how close the predictions are to reality. You came in hoping to finally get a handle on your produce orders, and now you’re staring at a form that looks like it was built for a stats professor.

What actually happens is you upload your Excel file and immediately hit a wall. The system starts asking questions that sound like they belong in a university lecture. What is your “target variable,” “regression or classification,” “evaluation metric?”

If you’ve ever opened one of these platforms, seen that wall of options, and quietly closed the tab. Trust me, you’re not alone. A research team at Williams College ran a study where they sat 16 non-technical folks in front of these AutoML tools. No one could tell the system what they actually wanted it to do. Zero for sixteen.

But what’s actually interesting is how the researchers tried to fix it.

The fix: make people do more, not less

The researchers built a system called CoAutoML, and its most interesting feature is the counterintuitive way we design systems and giving shortcuts to users, i.e. Three-click rule. In CoAutoML, they actually gave more tasks to users.

Think of it like this: before CoAutoML builds anything, it asks you to write down a few real examples of what could happen. Imagine Spotify asking you to fill out a quick form before it builds your playlist.

Instead of the system getting that info directly from your history, the system gets your expertise upfront. Context in, recommendation out.

Does not make sense to manually do that for your Spotify, right? But for building an ML model, it works really well.

Test-Driven Machine Learning

This idea is concept Test-Driven Machine Learning. Test-Driven ML is the practice of defining test cases before model building. No one had actually built it into a working tool (to the best of my knowledge) until CoAutoML was built to take the concept in front of real users.

What happened when real people used it

The results were striking.

Participants started using tactile, embodied language to describe the experience. One said it felt like “being hands-on with the data… like touching the data.” Another said that the standard evaluation metrics meant nothing to them, but seeing their own expected output next to the model’s predicted output made everything clear. Seeing that they’d predicted 3.6 and the model predicted 3.58 gave them an intuitive grasp of model performance that no accuracy score ever could.

Even better, people started poking around. They’d try out extreme values just to see what happened. They’d spot patterns in their own data they’d never noticed before. Something they said they never would have caught if the system just did everything behind the scenes. If there were just a button that did it all, they never would have seen it.

The system didn’t teach these people machine learning. It gave them a way to express what they already knew, which was their domain expertise, in a format the machine could use.

By the end, participants reported high confidence in their ability to do ML tasks independently using the tool. Not because they suddenly understood gradient boosting, but because the interface matched the way they already thought about the problem: data goes in, a prediction comes out, and I can check whether it makes sense.

What this means if you’re building AI agents for data work

This is the part that matters if you’re thinking about agentic data teams, which, honestly, is what I spend most of my time on these days.

Most people in the AutoML and AI agent world assume the goal is total automation. Remove the human. Make it seamless. But this research points to something different: the best systems aren’t the ones that do everything for you. They’re the ones that ask you the right questions at the right time, in words you actually understand.

For SMEs without data teams, this distinction is critical. When you’re building AI agents that handle data pipelines, analysis, and reporting for a small company, you can’t assume the person on the other end knows what a “target variable” is. But you also can’t just hide everything behind a black box, because then they won’t trust the output or act on it.

The CoAutoML research points to a middle path: give people a role that matches how they already think. Don’t ask them to be data scientists. Let them say what they know in their own words, and let the system handle the translation into whatever the ML pipeline needs. Ask them to be domain experts, which they already are!

Starkova, V. & Howley, I. (2026). CoAutoML: User Interface Framework for Machine Learning Novices using LLM-based AutoML and Test-Driven Machine Teaching. IUI '26, Paphos, Cyprus. doi.org/10.1145/3742413.3789153

Now imagine that happening inside an automated data pipeline run by AI agents. At 2am. With no one watching.

This is the main issue in building AI agents that actually do useful work for companies. The same flexibility that makes LLMs impressive is exactly what makes them dangerous when you need reliability.

A research group at EPFL built a system called AmadeusGPT in 2023. It analyzes animal behavior in neuroscience labs using natural language. On the surface, it has nothing to do with business. But it shows something important: AI agents become more reliable when you give them less autonomy, not more.

What the researchers actually built

The EPFL team faced a problem that should sound familiar. Labs generate piles of video data, such as mice running through mazes or horses on treadmills. Analyzing that data means stitching together pretrained models like DeepLabCut for animal trs Segment Anything for object segmentation, etc. These are’ etc. These are work that traditionally requires coding scripts, training models, or integrating multiple codebases. The problem is that people who understand biology often can’t code, and the people who can code often don’t understand the biology.

Sound familiar? Swap “biology” for “your business domain” and “video data” for “your sales pipeline,” and you’ve got the exact problem most SMEs face with their data.

Their solution was to put GPT-3.5 in charge of writing and executing the analysis code based on plain language questions from researchers. A scientist could type “When is the mouse on the treadmill?” and the system would determine which models to run, generate the Python code, execute it, and return the answer.

But here’s the part that matters: they didn’t just point ChatGPT at the problem and hope for the best. They built a system of constraints and supporting components around it. And those constraints are what made it work.

The orchestration lesson

The researchers identified four reasons why a raw LLM can’t reliably do this kind of work: (1) it doesn’t know your private tools and APIs, (2) it hallucinates functions when given too much implementation detail, (3) it runs out of context window on complex tasks.

Now, to be fair: context windows have grown 30x since this paper. GPT-3.5 had a 4,096 token limit; today we’re talking 128K–200K+. That specific constraint is fading. But the other three problems haven’t gone anywhere. The (2) model still hallucinates functions, (1) still doesn’t know your private tools. And ironically, (3) bigger context windows introduce their own issues. Models get distracted by irrelevant context, and costs scale with the number of tokens. The architectural lesson isn’t about token limits. It’s about how constraint and orchestration outperformed raw capability. That finding becomes even more relevant as models become more powerful.

Their fix was architectural, not just a better prompt. And the single most important design decision was this: they hid the implementation and only showed the interface.

Instead of feeding GPT the full source code of their analysis tools (hundreds of lines of Python per function) they gave it only the function documentation. What each function does, what it accepts, what it returns. Nothing else. It’s like giving a new employee a clear job description instead of dumping the entire company codebase on their desk.

The architecture

Here’s how they did it.

Step 1: Register only the interface, never the implementation.

A Python decorator (@register_core_api) captures only three things about each function: its name, its parameter types, and its docstring. The actual logic is discarded at the point of registration.

python

CORE_API_REGISTRY[func.__name__] = {    
  “name”: func.__name__,
  “parameters”: inputs,        # type annotations only
  “description”: func.__doc__,  # docstring only
}

Step 2: Format those registrations as documentation for GPT.

A separate function (get_core_api_docs()) converts the registry into a clean, readable spec block. They are just function signatures and descriptions. No source code. Just the “job description.”

Step 3: Inject that documentation into the system prompt.

The code generator prompt explicitly tells GPT: “This block contains information about the core APIs... They do not contain implementation details but you can use them to write code.”

Step 4: GPT generates code using only the interface.

GPT sees signatures like get_animals_animals_events(cross_animal_query_list, bodypart_names, ...) with their docstrings, and writes code that calls these functions — without ever seeing the implementation behind each one. The real work happens in manager classes (animal_manager.py, object_manager.py, etc.) that GPT never touches.

The result:

GPT stopped inventing functions that didn’t exist. Token usage dropped because less context was needed. The system produced correct, executable code from plain English questions.

On a standardised animal behavior benchmark (MABe 2022 Behavior Challenge), this constrained system matched or outperformed dedicated machine learning models that had been specifically trained for the task. And with GPT-4 instead of GPT-3.5, the error rate on real user queries dropped from 18% to 10%.

What this means if you’re building (or buying) AI for your data

Here’s where this connects to what I think about every day:
building agentic data teams for SMEs.

Most of the AI hype you hear is about making models bigger, more capable, more autonomous. The promise is: just give the AI more freedom, and it’ll figure it out. But this paper demonstrates the opposite. The magic was in the constraints.

This maps directly to what I’ve learned from years of running engineering teams. The bottleneck in delivery is never talent. It’s orchestration. The handoffs, the QA loops, the “who checks if this works with our X number of existing features” problem. When you get the coordination right, a small team outperforms a big one. That’s true whether the team is human or artificial.

The takeaway

AmadeusGPT offers a crucial lesson for any business bringing AI into its data operations:

Design the interfaces your AI agents work through as carefully as you would design a job description for a new hire.

Define what each function does, what it accepts, and what it returns. Hide the rest. The narrower the surface area, the fewer ways the agent can be wrong.

And that principle extends beyond APIs. The researchers didn’t build one super-agent that did everything. They built a system of components with specific roles. A code generator agent, a rephraser agent, a self-correction agent, and an explainer agent. Each one had a narrow job and a clean interface to the others. Your agentic team should work the same way: one agent writes the query, another validates it, another explains the results.

They also planned for failure. The self-correction mechanism in AmadeusGPT didn’t prevent all errors, but it caught a significant number of them without human intervention. Good orchestration is not assuming that the first attempt will be perfect. It means building in checkpoints, retries, and graceful failure.

The Bottom line

The companies that will get the most value from AI in the next few years won’t be the ones using the most powerful models. They’ll be the ones who orchestrate AI agents most effectively with the right constraints, the right memory, and the right coordination among agents.

Paper: Ye, S., Lauer, J., Zhou, M., Mathis, A., & Mathis, M.W. (2023). AmadeusGPT: a natural language interface for interactive animal behavioral analysis. NeurIPS 2023.

Hello, I’m Ademar Tutor. I’m researching how to agentic data teams for SMEs. An AI system that delivers the data capabilities of a full data team, without the headcount. I’m currently completing my Master’s in AI at the University of Waikato. I share what I’m learning along the way! Follow me on LinkedIn if that’s a problem you’re thinking about too.

• Front-end: Next.js

• Back-end: FastAPI

• LLM integration: LangChain

I wanted a really easy way to deploy the app without handling too much of the config, similar to Heroku, Vercel, or Fly.io, but without the heavy lifting.

That’s when I stumbled on Azure Container Apps.

In this post, I’ll walk through what I learned deploying an AI evaluation system on Azure Container Apps and why this architecture is a surprisingly good fit if you’re building AI systems without a dedicated DevOps team.

A Quick Primer: AKS vs. Azure Container Apps

Azure Kubernetes Service (AKS) is Microsoft’s managed Kubernetes offering. Kubernetes itself is the industry-standard container orchestration platform. It handles deploying, scaling, and managing containerized applications across clusters of machines. “Managed” means Azure handles the control plane (the brain of the cluster), but you still manage worker nodes, networking, upgrades, and the endless YAML configuration that Kubernetes requires. AKS is powerful and flexible, which is why large engineering teams love it. But that power comes with complexity.

Azure Container Apps is a higher-level abstraction built on top of Kubernetes. The key difference: you never see or touch the Kubernetes layer. You give Azure your containers, tell it how you want them to scale and communicate, and it handles everything else: node management, networking, certificate rotation, cluster upgrades. You get the benefits of container orchestration (revisions, traffic splitting, autoscaling, managed ingress) without writing a single Kubernetes manifest.

Think of it this way: AKS gives you a fully-equipped commercial kitchen where you control every burner and oven. Container Apps gives you a meal delivery service; you provide the recipes (containers), and someone else handles the cooking infrastructure.

For solo builders and small teams, that tradeoff matters enormously.

Architecture Overview

Three containers, one environment, zero Kubernetes manifests.

Let me run you through how I set this up.

Step-by-Step: First-Time Setup

Throughout this tutorial, replace these placeholders with your own values:

1. Create a resource group

az group create \
  --name <your-app>-rg \
  --location southeastasia

2. Create a Container Apps environment

az containerapp env create \
  --name <your-app>-env \
  --resource-group <your-app>-rg \
  --location southeastasia

3. Build Docker images for linux/amd64

If you're on Apple Silicon, this step matters. Azure runs amd64, so you need to build for that architecture explicitly, otherwise feel free to skip this step:

docker build \
  --platform linux/amd64 \
  -t <your-app>-backend:latest \
  ./backend

docker build \
  --platform linux/amd64 \
  -t <your-app>-frontend:latest \
  ./frontend

4. Create Azure Container Registry

az acr create \
  --resource-group <your-app>-rg \
  --name <youracr> \
  --sku Basic \
  --admin-enabled true

Note: ACR names must be globally unique and contain only alphanumeric characters (no hyphens).

5. Login, tag, and push

az acr login --name <youracr>

docker tag <your-app>-backend:latest <youracr>.azurecr.io/<your-app>-backend:latest
docker tag <your-app>-frontend:latest <youracr>.azurecr.io/<your-app>-frontend:latest

docker push <youracr>.azurecr.io/<your-app>-backend:latest
docker push <youracr>.azurecr.io/<your-app>-frontend:latest

When it finishes, Azure prints your public URLs:

• https://backend.<suffix>.southeastasia.azurecontainerapps.io/

• https://frontend.<suffix>.southeastasia.azurecontainerapps.io/

What You'll Run Daily

Once the infrastructure is setup, every subsequent deploy can be done on a single script deploy-demo.sh. Here’s a run-down of what the script does:

Step 1: Build images with the backend URL baked in

docker build --platform linux/amd64 -t <your-app>-backend:latest ./backend

docker build --platform linux/amd64 \
  --build-arg NEXT_PUBLIC_API_BASE=https://backend.<env-suffix>.azurecontainerapps.io \
  -t <your-app>-frontend:latest ./frontend

That NEXT_PUBLIC_API_BASE argument matters. Next.js compiles environment variables prefixed with NEXT_PUBLIC_ into the browser bundle at build time. If you don't set it here, your frontend won't know where to find your backend.

Step 2: Push to the registry

docker tag <your-app>-backend:latest <youracr>.azurecr.io/<your-app>-backend:latest
docker tag <your-app>-frontend:latest <youracr>.azurecr.io/<your-app>-frontend:latest

docker push <youracr>.azurecr.io/<your-app>-backend:latest
docker push <youracr>.azurecr.io/<your-app>-frontend:latest

Step 3: Update the container apps

az containerapp update \
  --name backend \
  --resource-group <your-app>-rg \
  --image <youracr>.azurecr.io/<your-app>-backend:latest

az containerapp update \
  --name frontend \
  --resource-group <your-app>-rg \
  --image <youracr>.azurecr.io/<your-app>-frontend:latest

Each update command creates a new revision. The new revision starts receiving traffic automatically. If something breaks, you can route traffic back to the previous revision with one command.

Step 4: Patch environment variables

FRONTEND_FQDN=$(az containerapp show --name frontend --resource-group <your-app>-rg --query "properties.configuration.ingress.fqdn" -o tsv)

az containerapp update \
  --name backend \
  --resource-group <your-app>-rg \
  --set-env-vars \
    FRONTEND_URL=https://${FRONTEND_FQDN} \
    DATABASE_URL=postgresql+psycopg2://<db-user>:<db-password>@db:5432/<db-name> \
    ENV=production

Note that DATABASE_URL points to db, the internal service name. Your Postgres instance isn't exposed to the internet, only your backend can reach it.

The complete loop

./deploy-demo.sh
  → builds amd64 images
  → pushes to ACR
  → updates container apps (creates new revisions)
  → patches environment variables
  → done

Note:

I’ve experience a few times wherein sometimes the script does not work. This happens terminal has been open for days. Run these two commands and retry:

az login
az acr login --name <youracr>

Conclusion

Azure Container Apps isn’t “AKS lite.” It’s a different tradeoff entirely: you give up control over the orchestration layer, and in exchange, you get to focus on your actual product. And for my use case, Azure Container Apps fits the bill.

If you’re building something similar and want to compare notes, I’d like to hear about it. What’s your current deploy setup, and what’s painful about it?

• Engineers building LLM-as-a-Judge or LLM evaluation pipelines

• MLOps/Platform teams who need audit trails for AI outputs

• Tech leads evaluating how to manage prompt configuration at scale

What you’ll get:

• A concrete database pattern (Postgres + SQLAlchemy) for versioned prompt templates

• API design for tenant-scoped prompt activation

• An audit trail that links every assessment to its prompt version

I previously thought that “LLM apps are just prompts + glue.” Basically, they are just “LLM Wrappers.”

After building a few LLM apps, I disagree. The prompt isn’t copywriting. It’s a production configuration.

If your prompt changes, your system behavior changes. If your system behavior changes, your evaluation metrics shift. And if your evaluation shifts without an audit trail, you end up “improving” the product by accident because you changed the yardstick, not the thing being measured.

What happens when there is no prompt versioning?

Teams burn entire sprints chasing phantom regressions. Scores drop 15% after a “minor tweak.” Was the tweak the model, the data, or the prompt? Nobody knows, because nobody recorded which prompt produced which assessment.

Research on LLM-as-a-Judge systems shows that even small prompt variations can significantly affect scores, rankings, and bias characteristics, introducing what researchers call “evaluation instability driven by prompt mis-specification rather than model capability.”

A 2024 EMNLP study found that LLM performance is highly sensitive to the prompts utilized, and this variability poses challenges for accurate assessment.

Meanwhile, scoring bias and prompt sensitivity studies show that even state-of-the-art judges can have large per-instance score variance under simple prompt perturbations like rubric order or score anchoring.

If you’re using LLM-as-a-Judge in production, you’re one undocumented prompt edit away from:

• Unexplainable metric shifts that erode stakeholder trust

• Compliance gaps when auditors ask for evaluation lineage

• Wasted debugging cycles because you can’t isolate variables

• False confidence in “improvements” that were just measurement drift

That’s why modern teams treat prompts as versioned contracts and store them like contracts: in a database and recorded on every evaluation artifact. This fits the broader shift toward product-level evaluation (not “model vibes”) that’s showing up in 2025 eval playbooks.

Is this post for you?

Ask yourself:

• Have you ever shipped a prompt change and couldn’t explain why scores shifted?

• Do you store prompts in code but have no record of which version produced which output?

• Are different customers (or tenants) asking for different evaluation criteria and you’re copy-pasting prompts to handle it?

• Could you answer, right now, “which prompt version generated this assessment 3 months ago?”

If any of these hit home, keep reading. This post breaks down a concrete pattern from the app I’ve built llm-as-a-judge-sales-coach (multi-tenant “LLM as an evaluator” app for sales coaching.):

Prompt templates as evaluation contracts: versioned rubrics stored in Postgres, with an “active template per org” and an assessment audit trail that records prompt version.

Why is “prompt templates as contracts” trending?

LLM evaluation is prompt-sensitive. A small rubric tweak can swing agreement metrics or produce systematically different “judgments.” Surveys and platform docs now call out prompt templates as a key driver of bias/variance in LLM-as-judge evaluation, which is why teams increasingly treat evaluation prompts as managed artifacts.

The goal is:

1. Version the prompt like a contract (v0, v1, …)

2. Activate exactly one contract per tenant

3. Record which contract produced each assessment

4. Evaluate changes offline before promoting them

That’s what this implementation does.

The pattern: prompt templates as a first-class database model

In the app, a prompt template is a row in prompt_templates:

• organization_id (tenant boundary)

• name

• version (e.g., v0, v1, custom_v2)

• system_prompt

• user_template (must contain {transcript})

• is_active (only one active per org)

The model is explicit about the “one active per org” invariant:

class PromptTemplate(Base):

    """

    Only one template per organization can be active at a time.

    """

    version = Column(String(20), nullable=False, default=”v0”)

    is_active = Column(Boolean, default=False, server_default=”false”, nullable=False)

Full Source: backend/app/models/prompt_template.py

Contract enforcement: “active template per org”

This is where the pattern truly shines.

When you create or update a template with is_active=True, the CRUD layer deactivates all other templates for that org:

def _deactivate_org_templates(db, organization_id):

    db.query(PromptTemplate).filter(

        PromptTemplate.organization_id == organization_id,

        PromptTemplate.is_active == True,

    ).update({”is_active”: False})

def create(..., is_active=False):

    if is_active:

        _deactivate_org_templates(db, organization_id)

    ...

Full Source: backend/app/crud/prompt_template.py

That gives you a clean tenant-safe guarantee:

• each org can iterate on prompts independently

• the runtime scorer always has a single “source of truth”

• rollbacks become easy: just re-activate the prior version

The API exposes this in a very “config management” way:

• GET /prompt-templates/active → fetch the active contract

• POST /prompt-templates/{template_id}/activate → promote a version

@router.get(”/active”)

def get_active_template(...):

    template = template_crud.get_active_for_org(db, org_id)

    ...

@router.post(”/{template_id}/activate”)

def activate_template(...):

    updated = template_crud.update(db, template, is_active=True)

Full source: backend/app/routers/prompt_templates.py

The default contract: hardcoded v0, persisted at org creation

You still want a sane default so the system works on day one.

This repo keeps the initial prompt as code constants:

• SYSTEM (role + strict rules)

• USER_TEMPLATE (JSON schema + {transcript} insertion)

def create_default_for_org(db, organization_id):

    from app.prompts.prompt_templates import SYSTEM, USER_TEMPLATE

    return create(..., version=”v0”, system_prompt=SYSTEM, user_template=USER_TEMPLATE, is_active=True)

Full source: backend/app/crud/prompt_template.py

And the user template makes the “contract” concrete by embedding a strict JSON schema and requiring the transcript placeholder:

CONVERSATION TRANSCRIPT:

{transcript}

Provide your assessment as valid JSON matching the schema above.

Full source: backend/app/prompts/prompt_templates.py

Auditability: template version recorded with each assessment

This is the “evaluation contract” payoff.

The scoring pipeline returns (assessment_data, model_name, prompt_version):

template = get_active_template(...)

system, user = build_prompt(..., system_prompt=template.system_prompt, user_template=template.user_template)

prompt_version = template.version

...

return assessment_data, model_name, prompt_version

Full source: backend/app/services/scorer.py

Then the /assess router persists that prompt version on the Assessment record:

data, model_name, prompt_ver = score_transcript(...)

assessment = Assessment(
  transcript_id=transcript.id,
  scores=data[”scores”],
  coaching=data[”coaching”],
  model_name=model_name,
  prompt_version=prompt_ver,
)

Full source: backend/app/routers/assess.py

And the Assessment model explicitly indexes it for querying later:

prompt_version = Column(String, nullable=False, index=True)

Full source: backend/app/models/assessment.py

So months later you can answer:

• “Which prompt version produced this coaching output?”

• “Did v3 increase scores artificially vs v2?”

• “When we changed the rubric, what shifted?”

This is what it means to treat prompts like contracts: you can’t audit what you didn’t record.

How this enables “safe prompt iteration” without drift chaos

Once your prompt is a contract, you can run a clean workflow:

1. Draft v3 — You tweak the rubric to catch a new failure mode. No stress, it’s not live yet.

2. Preview it — Hit the preview endpoint. See exactly what output the new prompt produces. No guessing.

3. Evaluate offline — Run it against your gold set. Compare v3 vs v2 side-by-side. Did scores improve, or did you just move the goalposts?

4. Promote it — One API call. The new contract is live. The old one is still in the database if you need to roll back.

5. Observe with confidence — Every assessment is stamped. When your PM asks “what changed?” You know. When a customer disputes a score, you can show them exactly which rubric produced it.

This is the difference between “I think we improved the model” and “I can prove v3 outperforms v2 on our gold set, and here’s the audit trail.”

What does this actually save you?

For a team running 1,000+ evaluations/month, this pattern will pay for itself the first time the team needs to answer “why did tenant X’s scores change?”. A question that will take half a day and now takes one SQL query.

The takeaway

If you’re using LLM-as-a-Judge, your prompt isn’t “just a prompt.”

It’s the scoring contract. It defines what “good” means.

So treat it like production configuration:

• store it in the DB

• version it

• activate it per tenant

• stamp every assessment with its version (and eventually its ID)

• evaluate changes before you ship them

That’s how you keep evaluation from drifting and keep your improvements real.

The bottleneck at that time was clocks. The best clocks of the era were pendulum clocks. These clocks were great on land, terrible on a moving ship.

When the ship is being tossed around by the waves, the pendulum’s motion gets corrupted. And when your clock is wrong, you don’t just get the time wrong, you can end up navigating to the wrong place.

In 1735, everything changed when John Harrison built the first practical marine chronometer: a clock that kept accurate time at sea. Mechanically, the key shift was moving from pendulum regulation to balance + spring regulation (the same core concept behind mechanical watches).

The story seems familiar in the context of AI today.

A lot of LLM systems behave like pendulum clocks: impressive in calm conditions, unreliable when the environment shifts. Small changes in phrasing, missing context, messy inputs, or edge-case scenarios can swing outputs from spot-on to wildly off.

If we want AI apps that people can trust, we need the equivalent of Harrison’s chronometer: a mechanism that works at sea, not just in the lab.

The chronometer wasn’t a nicer map; it was a repeatable standard. In AI, evals are that standard: a way to measure reliability across rough conditions, not just demo conditions.

Evaluations turn model behavior into something we can measure, compare, and improve. Instead of trusting a prompt because it “sounds good,” we run it against a fixed gold set, quantify performance, and track whether changes actually make the system more consistent.

In other words: evals make reliability an engineering loop.

Below, I’ll show how I built that loop for an LLM-as-a-judge that produces ordinal coaching scores for the sales coach app I’ve built (https://github.com/iamademar/llm-as-a-judge-sales-coach), and why I evaluate it with Pearson r + Quadratic Weighted Kappa + ±1 accuracy.

The production scenario

The concrete scenario here is a sales coaching app I built recently.

It takes a sales call transcript and produces two outputs:

1. Coaching feedback for the conversation (what went well, what to improve, suggested next moves)

2. Ordinal 1–5 scores for each SPIN dimension (Situation, Problem, Implication, Need-Payoff)

Under the hood, that assessment is produced by a prompt-based LLM judge: we pass the transcript plus a structured rubric/template, and the model returns scores + rationale. Here’s the demo of how it works in production:

This kind of workflow shows up everywhere beyond sales coaching too, any time you’re using an LLM to grade, rank, extract, or decide: routing tickets, scoring leads, auditing calls, classifying documents, prioritizing follow-ups.

And in all of those cases, the real product isn’t “LLM output.”

The product is consistent decisions.

The reliability problem

Prompt-based judging is powerful, but it’s also easy to believe your own demo.

A prompt can look great on a few handpicked examples and still drift badly when:

• the transcript style changes,

• the call gets messy,

• the scenario is ambiguous,

• the customer isn’t cooperative,

• or the rep switches strategy mid-call.

This is where many LLM features die in production: not because the model is “bad,” but because reliability becomes unpredictable. When the output affects something important like coaching, performance reviews, or decision-making…

Unpredictable = Unshippable

So the real question isn’t “does the prompt work?”

It’s:

Does this prompt produce consistent, calibrated scores that match how a human coach would rate the same conversation?

And if the market shifts (new playbook, new ICP, new sales motion), can we recalibrate without guessing?

Because the implication of getting this wrong isn’t just a “wrong score.” It’s downstream damage:

• managers stop trusting the tool and revert to manual reviews,

• reps feel unfairly graded,

• enablement focuses on the wrong behaviors,

• and a promising AI feature is rolled back

The calibration loop

The fix is to treat the judge like an instrument that needs calibration.

We build a small gold set: a curated set of transcripts that clearly fit the SPIN framework, each labeled with “gold standard” SPIN scores (and ideally short notes explaining why).

Then we run the evaluator prompt over that same dataset and compare:

• model scores vs. gold scores (per SPIN dimension)

• aggregate performance across the dataset

• failure cases (where it disagrees and why)

If the metrics show the model is close enough and the failure cases are understandable, we treat the prompt as calibrated.

From there, every prompt change is tested the same way. That gives you two production-grade properties:

1. Confidence to ship: you can say “this version is better” with evidence

2. Regression detection: you catch degradations before users do

At this point, reliability becomes a repeatable loop.

Now we can talk about how to measure it correctly because the judge outputs 1–5 coaching scores, and those labels are ordinal: they’re ordered, and how far off you are matters.

Why “accuracy” is misleading for 1–5 coaching scores

Exact-match accuracy treats these as equally wrong:

• Gold = 5, Pred = 4 ✅ (close enough for coaching)

• Gold = 5, Pred = 1 ❌ (way off)

Both are simply “wrong” under accuracy.

But in rubric scoring, distance matters. That’s exactly the use case for weighted kappa (where disagreements are penalized based on severity), and why QWK shows up in ordinal scoring benchmarks like automated essay scoring.

So instead of one brittle metric, I intentionally use three complementary “views” of quality…

Why I use three metrics (not one)

Prompt-based scoring isn’t something you “get right” once. It’s something you calibrate, then keep calibrated.

And because these scores are ordinal (1–5), you want metrics that answer three different production questions:

1. Are we moving in the right direction when the gold label moves? (trend alignment)

2. Do we agree like a second rater would? (ordinal agreement)

3. Are we close enough that the coaching experience still feels right? (product tolerance)

That’s why I use:

• Pearson r → checks trend alignment

• Quadratic Weighted Kappa (QWK) → checks ordinal agreement and punishes big misses

• ±1 accuracy → checks “close enough” from a coaching / UX standpoint

Together, they give you a reliability signal that’s much harder to game than plain accuracy.

Metric 1: Pearson r (trend alignment)

In backend/app/services/evaluation_metrics.py, pearson_r(...) implements Pearson correlation with two practical “production-friendly” choices:

• If there’s only one data point, return 0.0

• If either series has zero variance (constant predictions or constant gold labels), return 0.0

# Handle single element case
if n == 1:
    return 0.0

# Handle zero variance case (constant array)
if var_true == 0.0 or var_pred == 0.0:
    return 0.0

Source code: https://github.com/iamademar/llm-as-a-judge-sales-coach/blob/main/backend/app/services/evaluation_metrics.py

Why this is useful in production

This prevents eval runs from exploding on edge cases and makes failure modes obvious.

If your judge collapses into “everything is a 3,” Pearson r doesn’t politely pretend you’re doing fine — it drops toward 0.

What Pearson r catches

• “When gold goes up, predictions go up too.”

• Broad trend / rank alignment across transcripts

What Pearson r can miss

• Calibration drift: you can be consistently off by +1 or −1 and still get decent correlation

Translation: Pearson r tells you if the judge has the right shape — not whether it’s well calibrated.

Metric 2: Quadratic Weighted Kappa (ordinal agreement)

In the same file, quadratic_weighted_kappa(...) treats the LLM judge like a second rater.

Two details matter here.

1) Quadratic penalty by distance

The weight matrix is explicitly quadratic:

weight_mat[i][j] = ((i - j) ** 2) / ((n_labels - 1) ** 2)

So:

• off by 1 → small penalty

• off by 2+ → much larger penalty

That matches how humans interpret rubric scores: a “4 vs 5” disagreement is a rounding error; “1 vs 5” is a broken judge.

2) Sensible handling of degenerate cases

If there’s no variation in labels (n_labels == 1), the function returns 1.0:

if n_labels == 1:
    return 1.0

This avoids the annoying situation where your eval run fails just because a small slice of data happens to be uniform.

What QWK catches

• “We agree like raters would.”

• Big misses are heavily penalized — exactly what you want for coaching credibility

What QWK helps prevent (this is the production killer)

• A judge that’s “directionally okay” but still frequently off by 2–4 points

• The kind of disagreement that makes reps say: “This is random.”

Translation: QWK tells you whether the judge is trustworthy as a rater, not just correlated.

Metric 3: ±1 accuracy (coaching tolerance)

The repo’s plus_minus_one_accuracy(...) is a pragmatic product metric:

n_correct = sum(
    1 for yt, yp in zip(y_true, y_pred)
    if abs(yt - yp) <= 1
)
return n_correct / len(y_true)

This answers the question your users actually feel:

How often is the score close enough that the coaching still makes sense?

Why it matters

In coaching workflows, “4 vs 5” usually yields similar guidance. Even if the exact number differs, the actionable feedback is often basically the same.

So ±1 accuracy is the metric that maps closest to perceived reliability.

What ±1 accuracy catches

• “Close enough” performance that preserves a good UX

• Whether prompt changes are making the judge more usable

What it helps prevent

• Quiet drift that technically looks “fine” on correlation but feels worse to users

• The slow slide into “this tool is kind of flaky”

Translation: ±1 is your product guardrail metric.

How to interpret the three together

Here’s the simplest way I’ve found to read these in practice:

• Pearson r high, QWK low → good direction, poor agreement (calibration/labeling mismatch)

• QWK high, ±1 low → rare but catastrophic misses are happening (bad failure modes)

• ±1 high, QWK mediocre → UX feels okay, but big misses exist (risk of trust collapse)

• All three improving together → your judge is getting reliably better, not just “better on paper”

That’s the real point of mixing metrics: you’re not optimizing a number, you’re making a judge that stays reliable when transcripts get messy and the world changes.

Bringing it back to the chronometer

Harrison didn’t make the ocean calmer. He built an instrument that stayed accurate anyway and navigators trusted it because it held up under real conditions.

Offline evaluation is our chronometer: the mechanism that keeps an LLM judge reliable.

And when the sales team shifts direction in response to the market, you don’t throw the system away. You update the gold set to reflect the new playbook and real examples, re-run the evals, and keep the judge aligned.

You don’t control the sea.
You recalibrate the instrument so it still tells the truth when things get rough.

We’re finally past the phase where “LLMs can sort of grade things”!

Today’s models are increasingly capable of acting as automated evaluators, and there’s now a growing body of research studying when (and how) LLM judgments align with human evaluation:

• Zheng et al. (2023) — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

• Liu et al. (2023) — G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment

• Kim et al. (2024) — Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models

• Chen et al. (2024) — Humans or LLMs as the Judge? A Study on Judgement Bias

• Huang et al. (2025) — An Empirical Study of LLM-as-a-Judge for LLM Evaluation

What better way to test this out than in sales coaching? Sales coaching is already a “judge” workflow.

Here’s a real-world scenario:
A sales coach listens to a call recording, then fills out a scorecard. They grade the rep across a playbook, let’s say SPIN (Situation, Problem, Implication, Need-Payoff) or your internal rubric, and add notes like:

• “They asked too many surface-level questions.”

• “They didn’t push on impact.”

• “Great discovery, but weak closing.”

The problem is that judgment is expensive, time-consuming, and possibly inconsistent. Two coaches can score the same call differently. Depending on mood, the same coach might score it differently a week later. And when a sales rep challenges the score and asks a question like “Why did I get a 2 on Implication?”, the explanation is usually buried in messy notes.

That’s why sales coaching is a perfect proving ground for LLM-as-a-Judge: it forces you to treat evaluation as a system, not a one-off prompt.

If the model is going to “judge” a call, it needs to do what good coaches do:

1. apply a consistent rubric,

2. produce structured scores, and

3. justify those scores with evidence from the call (transcript).

If an AI coaching system that can’t explain why a score was given or reproduce that score the next week, it really isn’t an effective AI system.

In this post, I’ll walk through a production-grade LLM-as-a-Judge architecture for sales coaching. I’ll show how transcripts are evaluated, validated, stored, and continuously calibrated, using my open-source reference implementation: https://github.com/iamademar/llm-as-a-judge-sales-coach/

Demo app deployed in Azure.

The Design: A Four-Stage Pipeline

The assessment pipeline follows this sequence:

Transcript → Judge → Validation → Persistence

Stage 1: Transcript Ingestion

Every assessment starts with a raw conversation transcript. The frontend captures or uploads conversation data, which flows through our FastAPI backend:

# 1) Persist transcript first
transcript = Transcript(
    representative_id=req.metadata.get("representative_id"),
    buyer_id=req.metadata.get("buyer_id"),
    call_metadata=req.metadata,
    transcript=req.transcript,
)
db.add(transcript)
db.flush()  # Get ID before scoring

Why persist before scoring? This ensures that if the LLM call fails, the data is not lost. The transcript is the source of truth, and assessments are derived artifacts.

Stage 2: The Judge

The scoring service orchestrates the LLM evaluation. Here’s the complete flow from the scorer.py:

def score_transcript(
    transcript: str,
    organization_id: Optional[uuid.UUID] = None,
    db: Optional[Session] = None,
) -> tuple[dict, str, str]:
    """
    Score a sales transcript using SPIN framework via LLM.

    This is the main scoring pipeline that:
    1. Fetches organization's active prompt template from database
    2. Builds calibrated prompt (system + user) using the template
    3. Calls LLM to generate assessment
    4. Parses and validates JSON response
    5. Validates score ranges and required keys
    6. Returns assessment data with metadata

    Args:
        transcript: Sales conversation transcript with speaker tags (Rep:/Buyer:)
        organization_id: Organization UUID for fetching prompt template and LLM credentials
        db: Database session for template and credential lookup

    Returns:
        Tuple of (assessment_data, model_name, prompt_version) where:
        - assessment_data: dict with "scores" and "coaching" keys
        - model_name: LLM model identifier used
        - prompt_version: Prompt template version identifier

    Raises:
        ValueError: If transcript is empty, no active template found, or JSON parsing fails
        AssertionError: If scores are out of range [1, 5] or required keys missing

    Examples:
        >>> os.environ["MOCK_LLM"] = "true"
        >>> # Note: Requires organization with active template in test DB
        >>> data, model, version = score_transcript("Rep: Hi\\nBuyer: Hello", org_id, db)
        >>> "scores" in data and "coaching" in data
        True
    """
    # Step 1: Fetch organization's active prompt template
    if not organization_id or not db:
        raise ValueError("organization_id and db are required for scoring")

    template = template_crud.get_active_for_org(db, organization_id)
    if not template:
        raise ValueError(
            "No active prompt template found for organization. "
            "Please ensure a default template was created during setup."
        )

    # Step 2: Build calibrated prompt using org's template
    system, user = build_prompt(
        transcript,
        system_prompt=template.system_prompt,
        user_template=template.user_template,
    )
    prompt_version = template.version

    # Step 3: Call LLM (mock or real provider)
    raw_json = call_json(
        system,
        user,
        model=MODEL_NAME,
        temperature=0.0,
        response_format_json=True,
        organization_id=organization_id,
        db=db,
    )

    # Step 4: Parse JSON with guardrails
    data = parse_json_strict(raw_json)

    # Step 5: Validate required keys
    assert "scores" in data, "Missing required key 'scores' in LLM response"
    assert "coaching" in data, "Missing required key 'coaching' in LLM response"

    # Step 6: Validate score bounds (all must be 1-5)
    scores = data["scores"]
    required_score_keys = [
        "situation", "problem", "implication", "need_payoff",
        "flow", "tone", "engagement"
    ]

    for key in required_score_keys:
        assert key in scores, f"Missing required score key '{key}'"
        score_value = scores[key]
        assert isinstance(score_value, int), \
            f"Score '{key}' must be integer, got {type(score_value).__name__}"
        assert 1 <= score_value <= 5, \
            f"Score '{key}' must be in range [1, 5], got {score_value}"

    # Step 7: Validate coaching structure
    coaching = data["coaching"]
    required_coaching_keys = ["summary", "wins", "gaps", "next_actions"]

    for key in required_coaching_keys:
        assert key in coaching, f"Missing required coaching key '{key}'"

    # Return assessment with metadata for tracking
    return data, MODEL_NAME, prompt_version

Key design decision: The metadata alongside the assessment (model_name, prompt_version) is returned. This enables reproducibility and A/B testing of different prompt strategies.

Stage 3: Validation with Guardrails

LLMs rarely return clean JSON on first try. The guardrails handle the common failure modes:

def parse_json_strict(raw: str) -> dict:
    """
    Parse JSON with fallback strategies for LLM-generated content.

    Attempts multiple parsing strategies:
    1. Direct json.loads() on raw input
    2. Strip code fences (``` or ```json) and whitespace, then retry
    3. Raise ValueError with debugging context if all strategies fail

    Args:
        raw: Raw string that may contain JSON (possibly wrapped in code fences)

    Returns:
        Parsed JSON as a dictionary

    Raises:
        ValueError: If JSON cannot be parsed after all strategies, includes
                   first ~120 chars of input for debugging

    Examples:
        >>> parse_json_strict('{"key": "value"}')
        {'key': 'value'}

        >>> parse_json_strict('```json\\n{"key": "value"}\\n```')
        {'key': 'value'}

        >>> parse_json_strict('```\\n{"key": "value"}\\n```')
        {'key': 'value'}
    """
    # Strategy 1: Try parsing as-is
    try:
        return json.loads(raw)
    except (json.JSONDecodeError, TypeError):
        pass

    # Strategy 2: Strip code fences and whitespace
    # Remove leading/trailing whitespace and newlines
    cleaned = raw.strip()

    # Pattern to match code fences with optional language specifier
    # Matches: ```json ... ``` or ``` ... ```
    fence_pattern = r'^```(?:json)?\s*\n?(.*?)\n?```$'
    match = re.match(fence_pattern, cleaned, re.DOTALL)

    if match:
        # Extract content between fences
        cleaned = match.group(1).strip()
    else:
        # Even without full fence pattern, strip any leading/trailing backticks
        cleaned = cleaned.strip('`').strip()

    # Try parsing cleaned version
    try:
        return json.loads(cleaned)
    except (json.JSONDecodeError, TypeError) as e:
        # Strategy 3: Raise with context
        preview = raw[:120] if len(raw) > 120 else raw
        raise ValueError(
            f"Failed to parse JSON after cleanup strategies. "
            f"Error: {str(e)}. "
            f"Input preview (first 120 chars): {preview!r}"
        )

This parser handles:

• Markdown wrapping: \``json {…} ````

• Extra whitespace: Leading/trailing newlines and spaces

• Debugging failures: Provides first 120 chars of raw output when all strategies fail

Stage 4: Persistence

Once validated, the assessment is persisted with full metadata:

class Assessment(Base):
    """
    SPIN framework assessment model.

    Stores LLM-generated scores and coaching feedback for a transcript.
    Tracks model name and prompt version for reproducibility and evaluation.
    """
    __tablename__ = "assessments"

    id = Column(Integer, primary_key=True, index=True, autoincrement=True)
    transcript_id = Column(
        Integer,
        ForeignKey("transcripts.id", ondelete="CASCADE"),
        nullable=False,
        index=True,
        comment="Reference to parent transcript"
    )
    scores = Column(
        JSON,
        nullable=False,
        comment="SPIN scores: {situation, problem, implication, need_payoff, flow, tone, engagement}"
    )
    coaching = Column(
        JSON,
        nullable=False,
        comment="Coaching feedback: {summary, wins, gaps, next_actions}"
    )
    model_name = Column(
        String,
        nullable=False,
        index=True,
        comment="LLM model identifier (e.g., 'gpt-4o-mini', 'claude-3-sonnet')"
    )
    prompt_version = Column(
        String,
        nullable=False,
        index=True,
        comment="Prompt template version (e.g., 'spin_v1', 'spin_v2')"
    )
    latency_ms = Column(
        Integer,
        nullable=True,
        comment="LLM call latency in milliseconds"
    )
    created_at = Column(
        DateTime(timezone=True),
        server_default=func.now(),
        nullable=False,
        comment="Timestamp when assessment was created"
    )

    # Relationship to transcript
    transcript_ref = relationship("Transcript", back_populates="assessments")

    def __repr__(self):
        return (
            f"<Assessment(id={self.id}, transcript_id={self.transcript_id}, "
            f"model={self.model_name!r}, prompt_version={self.prompt_version!r})>"
        )

Why store model_name and prompt_version? This enables:

• Compare different LLM providers (GPT-4 vs Claude vs Gemini)

• Track prompt evolution and measure quality improvements

• Debug regressions when scores drift

• Run historical analyses on assessment quality

Note: This can be seen on the demo where each assessment is labell with the LLM used.

Reducing Drift with Rubrics and Schemas

What keeps the system reliable is the explicit agreements between the LLM and our application about output format and scoring criteria. We’ve built both in prompt and through Pydantic validators.

The Rubric: Calibrated Scoring Criteria

Every LLM evaluation uses a detailed rubric that defines each score on a 1-5 scale. Here’s an excerpt from the default prompt template used on the app:

SCORING RUBRIC (1-5 scale):

**situation** (1-5): Quality of situation questions
- 1: No situation questions; jumps to pitch
- 2: Minimal context gathering; superficial questions
- 3: Adequate situation questions covering basic context
- 4: Good situation questions establishing clear current state
- 5: Excellent situation questions; thorough understanding of buyer's environment

**problem** (1-5): Quality of problem questions
- 1: No problem identification; ignores pain points
- 2: Weak problem exploration; misses key issues
- 3: Identifies some problems but lacks depth
- 4: Good problem identification with clear pain points
- 5: Exceptional problem discovery; uncovers hidden issues

**implication** (1-5): Quality of implication questions
- 1: No exploration of consequences; stays surface-level
- 2: Minimal urgency building; weak consequence exploration
- 3: Some implication questions but lacks impact
- 4: Good implication development building urgency
- 5: Outstanding implication questions creating compelling urgency

**need_payoff** (1-5): Quality of need-payoff questions
- 1: No connection between solution and buyer value
- 2: Weak value proposition; generic benefits
- 3: Adequate need-payoff with some value connection
- 4: Strong need-payoff linking solution to specific pains
- 5: Exceptional need-payoff; buyer articulates own value

**flow** (1-5): Adherence to SPIN sequence (S→P→I→N)
- 1: Random questioning; no discernible structure
- 2: Poor flow; jumps between stages inconsistently
- 3: Follows SPIN loosely; some stage mixing
- 4: Good SPIN sequence with clear progression
- 5: Excellent SPIN flow; natural and purposeful transitions

**tone** (1-5): Professional, empathetic, confident, adaptive communication
- 1: Pitchy, monologue-style; ignores buyer cues
- 2: Inconsistent tone; occasional empathy gaps
- 3: Mixed empathy and clarity; adequate professionalism
- 4: Strong tone; professional, warm, and responsive
- 5: Exceptional tone; adaptive, empathetic, confident, concise

**engagement** (1-5): Active listening, reflection, buyer talk-time
- 1: Dominates conversation; no active listening
- 2: Limited listening; minimal buyer participation
- 3: Adequate engagement; balanced talk-time
- 4: Good engagement; actively listens and reflects
- 5: Outstanding engagement; buyer-led insights and high talk-time

Note: This can be prompt can be tweaked so when the sales team learns better ways to approach sales conversations, the “judgement” can be recalibrated.

This rubric serves multiple purposes:

1. Anchors the LLM’s judgment to specific observable behaviors

2. Reduces prompt sensitivity by explicitly defining what each score means

3. Enables inter-rater reliability when comparing LLM vs human evaluations

4. Documents the evaluation criteria for stakeholders and audits

The Schema: Enforcing Structure

Alongside the rubric, we embed a JSON schema directly in the prompt:

JSON SCHEMA (your response must match this exactly):
{{
  "type": "object",
  "properties": {{
    "scores": {{
      "type": "object",
      "properties": {{
        "situation": {{"type": "integer", "minimum": 1, "maximum": 5}},
        "problem": {{"type": "integer", "minimum": 1, "maximum": 5}},
        "implication": {{"type": "integer", "minimum": 1, "maximum": 5}},
        "need_payoff": {{"type": "integer", "minimum": 1, "maximum": 5}},
        "flow": {{"type": "integer", "minimum": 1, "maximum": 5}},
        "tone": {{"type": "integer", "minimum": 1, "maximum": 5}},
        "engagement": {{"type": "integer", "minimum": 1, "maximum": 5}}
      }},
      "required": ["situation", "problem", "implication", "need_payoff", "flow", "tone", "engagement"]
    }},
    "coaching": {{
      "type": "object",
      "properties": {{
        "summary": {{"type": "string"}},
        "wins": {{"type": "array", "items": {{"type": "string"}}}},
        "gaps": {{"type": "array", "items": {{"type": "string"}}}},
        "next_actions": {{"type": "array", "items": {{"type": "string"}}}}
      }},
      "required": ["summary", "wins", "gaps", "next_actions"]
    }}
  }},
  "required": ["scores", "coaching"]
}}

We also reinforce this with Pydantic models on the application side:

class AssessmentScores(BaseModel):
    """
    SPIN scoring model with additional conversation quality metrics.

    All scores must be integers in the range [1, 5] where:
    - 1 = Poor
    - 2 = Below Average
    - 3 = Average
    - 4 = Good
    - 5 = Excellent
    """
    situation: int = Field(..., ge=1, le=5, description="Quality of situation questions")
    problem: int = Field(..., ge=1, le=5, description="Quality of problem questions")
    implication: int = Field(..., ge=1, le=5, description="Quality of implication questions")
    need_payoff: int = Field(..., ge=1, le=5, description="Quality of need-payoff questions")
    flow: int = Field(..., ge=1, le=5, description="Overall conversation flow")
    tone: int = Field(..., ge=1, le=5, description="Tone and professionalism")
    engagement: int = Field(..., ge=1, le=5, description="Customer engagement level")

    @field_validator(
        "situation", "problem", "implication", "need_payoff",
        "flow", "tone", "engagement",
        mode="before"
    )
    @classmethod
    def validate_score_range(cls, v, info):
        """Ensure all scores are integers in valid range [1, 5]"""
        if not isinstance(v, int):
            raise ValueError(f"{info.field_name} must be an integer, got {type(v).__name__}")
        if v < 1 or v > 5:
            raise ValueError(f"{info.field_name} must be between 1 and 5, got {v}")
        return v

Why this dual approach?

• The prompt-embedded schema guides the LLM’s output generation

• The Pydantic validators catch any deviations before persistence

• Together, they form a contract that dramatically reduces drift

Failure Modes: What We Learned the Hard Way

Building this system revealed three major categories of failure. I’ve listed them below and the solution.

1. Inconsistency: Score Drift Over Time

Problem: Early versions showed score inflation—the LLM would gradually shift toward higher scores without calibration.

Solution:

• Explicit rubric anchoring (see above)

• Temperature set to 0.0 for deterministic outputs

• Tracking prompt_version to detect when drift correlates with prompt changes

# Deterministic LLM calls
raw_json = call_json(
    system,
    user,
    model=MODEL_NAME,
    temperature=0.0,  # No randomness
    response_format_json=True,
    organization_id=organization_id,
    db=db,
)

2. Verbosity: When LLMs Over-Explain

Problem: LLMs would add commentary like “Here’s my assessment:” or wrap JSON in markdown, breaking parsers.

Solution: Multi-strategy parsing with guardrails (see Stage 3 above) and explicit instructions:

# Immutable system prompt - instructs LLM behavior
SYSTEM = """You are a senior sales coach specializing in the SPIN (Situation, Problem, Implication, Need-Payoff) selling methodology.

Your task is to evaluate sales conversations and provide scoring and coaching feedback.

CRITICAL INSTRUCTIONS:
- Return STRICT JSON that exactly matches the provided JSON Schema
- Do NOT include any extra keys beyond those specified in the schema
- Do NOT wrap your response in markdown code blocks
- Ensure all scores are integers between 1 and 5 (inclusive)
- Base your assessment on evidence from the conversation transcript"""

The system prompt is immutable and enforces strict JSON mode.

3. Prompt Sensitivity: Small Changes, Big Impact

Problem: Tweaking prompt wording would cause score distributions to shift unexpectedly.

Solution:

• Store prompts as versioned database records (not hardcoded strings)

• Track prompt_version on every assessment

• Run evaluation datasets before promoting new prompt versions

class PromptTemplate(Base):
    """
    Prompt template for SPIN assessments.

    Each organization has at least one template (v0 default created automatically).
    Only one template per organization can be active at a time.
    """

    __tablename__ = "prompt_templates"

    id = Column(
        UUID(),
        primary_key=True,
        default=uuid.uuid4,
        index=True,
        comment="Unique template identifier",
    )
    organization_id = Column(
        UUID(),
        ForeignKey("organizations.id", ondelete="CASCADE"),
        nullable=False,
        index=True,
        comment="Organization this template belongs to",
    )
    name = Column(
        String(100),
        nullable=False,
        comment="Human-readable template name",
    )
    version = Column(
        String(20),
        nullable=False,
        default="v0",
        comment="Version identifier (e.g., 'v0', 'v1', 'custom_v2')",
    )
    system_prompt = Column(
        Text,
        nullable=False,
        comment="System prompt defining LLM behavior",
    )
    user_template = Column(
        Text,
        nullable=False,
        comment="User prompt template (must contain {transcript} placeholder)",
    )
    is_active = Column(
        Boolean,
        default=False,
        server_default="false",
        nullable=False,
        comment="Only one template per org can be active",
    )
    created_at = Column(
        DateTime(timezone=True),
        server_default=func.now(),
        nullable=False,
        comment="Template creation timestamp",
    )
    updated_at = Column(
        DateTime(timezone=True),
        server_default=func.now(),
        onupdate=func.now(),
        nullable=False,
        comment="Template last update timestamp",
    )

    # Relationships
    organization = relationship("Organization", back_populates="prompt_templates")
    evaluation_runs = relationship(
        "EvaluationRun",
        back_populates="prompt_template",
        cascade="all, delete-orphan",
        order_by="desc(EvaluationRun.created_at)",
    )

    @property
    def latest_evaluation(self):
        """Get the most recent evaluation run for this template."""
        return self.evaluation_runs[0] if self.evaluation_runs else None
    
    @property
    def best_qwk_score(self):
        """Get the best QWK score across all evaluations."""
        if not self.evaluation_runs:
            return None
        valid_runs = [run.macro_qwk for run in self.evaluation_runs if run.macro_qwk is not None]
        return max(valid_runs) if valid_runs else None

    def __repr__(self):
        return (
            f"<PromptTemplate(id={self.id}, "
            f"org={self.organization_id}, "
            f"name={self.name!r}, "
            f"version={self.version}, "
            f"is_active={self.is_active})>"
        )

This design enables A/B testing of prompt variants, provides the ability to roll back if a new prompt degrades quality, and allows for historical analysis of how prompts impact assessment quality.

System Architecture

If your looking at the code and need a map of how to read it, here’s the complete sequence diagram showing how a transcript flows through the system:

┌──────────┐
│ Frontend │
│ (Next.js)│
└────┬─────┘
     │ POST /assess
     │ { transcript, metadata }
     ▼
┌─────────────────────┐
│ FastAPI Router      │
│ (assess.py)         │
└────┬────────────────┘
     │ 1) Persist transcript
     ▼
┌─────────────────────┐
│ PostgreSQL          │
│ transcripts table   │
└────┬────────────────┘
     │ transcript_id
     ▼
┌─────────────────────┐
│ Scoring Service     │
│ (scorer.py)         │
└────┬────────────────┘
     │ 2) Fetch active prompt template
     ▼
┌─────────────────────┐
│ PostgreSQL          │
│ prompt_templates    │
└────┬────────────────┘
     │ system_prompt, user_template
     ▼
┌─────────────────────┐
│ Prompt Builder      │
│ (build_prompt)      │
└────┬────────────────┘
     │ 3) Build prompts with rubric + schema
     ▼
┌─────────────────────┐
│ LLM Client          │
│ (llm_client.py)     │
└────┬────────────────┘
     │ 4) Call OpenAI/Anthropic/Google
     ▼
┌─────────────────────┐
│ External LLM API    │
│ (GPT-4, Claude, etc)│
└────┬────────────────┘
     │ raw JSON response
     ▼
┌─────────────────────┐
│ JSON Guardrails     │
│ (parse_json_strict) │
└────┬────────────────┘
     │ 5) Parse & strip code fences
     ▼
┌─────────────────────┐
│ Schema Validator    │
│ (scorer.py)         │
└────┬────────────────┘
     │ 6) Validate scores [1-5], keys, types
     ▼
┌─────────────────────┐
│ PostgreSQL          │
│ assessments table   │
└────┬────────────────┘
     │ assessment_id, scores, coaching
     ▼
┌─────────────────────┐
│ FastAPI Response    │
│ (AssessResponse)    │
└────┬────────────────┘
     │ 7) Return to frontend
     ▼
┌──────────┐
│ Frontend │
│ Dashboard│
└──────────┘

Key Takeaways

If you’re building LLM-as-a-judge systems, these takeaways might be helpful:

1. Embed schemas in prompts — Don’t rely on post-hoc parsing alone

2. Version your prompts — Treat them like code, store in DB with metadata

3. Build guardrails at every layer — Prompt, parsing, application validation. Something like this:

   # Layer 1: Prompt schema (in system prompt)
   
   # Layer 2: Parsing guardrails
   data = parse_json_strict(raw_json)
   
   # Layer 3: Application validation
   assert “scores” in data, “Missing required key ‘scores’”
   for key in required_score_keys:
       assert key in scores, f”Missing score key ‘{key}’”
       assert 1 <= scores[key] <= 5, f”Score out of range: {scores[key]}”

   Note: For full details, see code on github.

4. Track metadata religiously — model_name, prompt_version, latency_ms enable debugging. Really helpful later on!

5. Evaluate before deploying — Use golden datasets with human labels for QA. In this scenario, these are conversations from sales reps that closed and considered a really good sales call.

6. Set temperature to 0.0 — Determinism reduces drift for structured outputs

LLMs are powerful judges, but they need strong contracts and validation to become reliable production systems.

Code References

All code examples in this post are from the open-source codebase:

• Scoring service: backend/app/services/scorer.py

• JSON guardrails: backend/app/utils/json_guardrails.py

• Assessment model: backend/app/models/assessment.py

• Assessment router: backend/app/routers/assess.py

• Prompt templates: backend/app/prompts/prompt_templates.py

• LLM client: backend/app/services/llm_client.py

• Prompt template model: backend/app/models/prompt_template.py

• Assessment schemas: backend/app/schemas/assessment.py

The full source is available in the repository:
https://github.com/iamademar/llm-as-a-judge-sales-coach

Questions or comments? I’d love to hear about your experiences building LLM judge systems. What failure modes have you encountered? What guardrails work best for your use case?

As the models improved, the friction didn’t disappear. It just changed shape. The challenge became managing tokens, deciding which files to reference, and constantly re-establishing state whenever tokens are almost out. The “intelligence” was there, but it added the responsibility of managing tokens. You had to manually select context, plan, validate, and correct errors every time.

When tools like Cursor first appeared, it felt like ChatGPT embedded inside an editor. It saved a few keystrokes and reduced context switching, but it didn’t immediately feel transformative. It was useful but still an incremental improvement.

That perception changed after using it across several real projects this year. Over time, Cursor stopped feeling like a chat tool and became something else entirely. The gains weren’t just coming from better prompts or smarter models. They stemmed from how the application itself had learned to work with LLMs.

It took a few real projects to see it clearly, but Cursor started removing friction in places I’d always assumed were just part of using LLMs.

Cursor does Automatic “context engineering” for you.

This is the biggest shift for me, and probably the easiest to miss if you haven’t felt the pain before, is context engineering.

With ChatGPT, you are the context engineer. You decide which files matter, copy snippets, summarise large modules, and hope you didn’t forget something important. When things go wrong, the model usually isn’t “wrong”. It’s responding to an incomplete or poorly structured context.

Cursor takes that responsibility.

Instead of asking you to supply context manually, it autonomously discovers, selects, and structures relevant code paths before the model ever reasons about your question.

In the screenshots below, I asked a deceptively simple question:

“I want to change the evaluation from online to offline. What files do I need to change?”

What happened next is the important part.

Cursor didn’t ask for files or offer generic advice. It inspected the repository itself, traced the relevant evaluation paths, inferred what “offline” meant from the code, and returned a structured answer spanning backend, frontend, configuration, and documentation.

Cursor uses the LLM throughout the process, including during code exploration. However, the exploration itself is constrained and scaffolded by deterministic tooling such as repository indexing, symbol graphs, and file search. The LLM helps reason about the code, but it isn’t responsible for discovering or assembling the context on its own.

Cursor orchestrates multiple LLM calls as DAGs

Another subtle shift is how Cursor uses large language models. It doesn’t rely on a single prompt that tries to do everything at once. Instead, it orchestrates multiple LLM calls, each with a narrowly defined role, and connects them into an execution graph that grows in complexity only when needed (like DAG).

When I asked:

“I want to change the evaluation from online to offline. What files do I need to change?”

You can see this happening in the screenshots.

Cursor explicitly reports multiple tool invocations during the process, progressively building up understanding before producing the explanation. That behavior is difficult to reproduce with a single-pass prompt, because the system needs to decide what to look at next based on what it has already learned.

Cursor provides application-specific UX for the human-in-the-loop.

Most LLM workflows still feel like “ask a question → get an answer.” Cursor turns it into a tighter cycle by giving you interactions that match how developers actually work. Instead of pushing everything through a chat window, it gives you features like inline edit, context-aware suggestions, and mode-driven flows that map nicely to an RPI workflow: Research → Planning → Implementation.

And this isn’t just useful on greenfield projects. It’s arguably more valuable when you’re stepping into an unfamiliar codebase. Imagine you’re new to a project and trying to understand how authentication works end-to-end. Instead of manually grepping for middleware, auth routes, token handling, and frontend state, you can start with a prompt like:

“Show me how authentication works in this app. Start from the front-end until the back-end. Give a workflow diagram.”

What I like about the result is that it doesn’t stay abstract. Cursor turns your question into a walkthrough: it identifies the real entry points (e.g., the login page and the /api/auth/login route), traces how requests are forwarded to the backend, and explains how tokens are stored and reused (access token vs refresh token, cookie handling, and the fetch helper that attaches auth).

The output becomes a navigable map you can verify immediately by jumping into the referenced files. So you’re not just reading an answer. You can actively validate your mental model of the system by adding logs or tweaking the code.

4. Cursor has introduced agents... and this is where things get existential

In mid-2025, agent-based workflows began to appear in mainstream dev tools. When OpenAI Codex was reintroduced, it showcased a new interaction model: you assign a task, a fleet of agents executes it, and you’re presented with a pull request to review. Cursor quickly followed with its own take on agent mode.

Up to this point, LLM tools, including Cursor were primarily collaborative. You stayed in the loop at every step: ask a question, review the answer, apply the change. Agents shift that boundary. Instead of collaborating within the task, you delegate the task and move into a reviewer role.

This looks like the “AGI endgame.” If a system can plan work, navigate a codebase, modify files, and submit a PR.

For me, it raises an uncomfortable question:

If AGI exists, why would we ask it to build software for accounting professionals instead of just doing the accounting itself?

This is where the idea of WaaS (Work as a Service) starts to feel more relevant than SaaS. The value shifts away from tools that enable humans to work toward systems that perform the work directly. Software stops being the product; execution does.

If that trend continues, it would shake up the outsourcing industry (BPO, software development, and managed services) as we know it.

I digress.

Agent mode today is still not reliable enough to fully cross that boundary. On real projects, agents can misunderstand intent, make locally “correct” changes that violate system-level assumptions, or drift once tasks span too many implicit constraints. The cost of review becomes really high because if the agent jumps into a rabbit hole, you know, you have to pick and choose the code. Unfortunately, it spent tokens already. And more tokens == more $$$.

This is true today. But will it be in the coming year?

Will agent workflows be genuinely reliable next year? Possibly. Will they redefine what “writing software” even means? Maybe. Either way, it’s one of the most interesting frontiers Cursor has opened up.

We will soon find out what happens in 2026.

See you then!

What sets your franchise apart is its smart weather-based sales prediction system. Your unique value proposition? Franchisees can check whether today is an optimal day (if you sell on this day, you're set for the month 😂) to set up shop outdoors. If the forecast isn’t favorable, they can take the day off. No wasted effort!

Now it’s time for the next step:
Build an app that predicts whether today is a good day to sell coffee outside.

Example is simple enough, right? 😂 After all, you’re not building the next ChatGPT, just trying to figure out if the weather says “sell” or “do not sell”

To get started, you look at historical data. After squinting at it long enough (and maybe one too many cups of coffee), you spot a pattern: when it’s sunny and the temperature is mild, it’s time to sell!

Here’s what your dataset looks like for the past five days:

data = [ 
  {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "No"},
  {"Outlook": "Sunny", "Temperature": "Mild", "Sell": "Yes"},
  {"Outlook": "Rainy", "Temperature": "Cool", "Sell": "Yes"},
  {"Outlook": "Overcast", "Temperature": "Hot", "Sell": "Yes"},
  {"Outlook": "Rainy", "Temperature": "Mild", "Sell": "No"}
]

If you’ve been developing apps for a long time like me, you’ll probably think of something like this immediately:

def is_good_day_to_sell(outlook, temperature):
    if outlook == "Sunny":
        if temperature == "Hot":
            return "No"
        else:
            return "Yes"
    elif outlook == "Rainy":
        if temperature == "Cool":
            return "Yes"
        else:
            return "No"
    elif outlook == "Overcast":
        return "Yes"


# What you are trying to predict:
# {"Outlook": "Sunny", "Temperature": "Hot", "Sell": ???}
prediction = is_good_day_to_sell("Sunny", "Hot")
print("Prediction:", prediction)

This function is_good_day_to_sell works! As long as the world never changes. It’s hardcoded based on the exact patterns you’ve seen so far. So if the input is familiar, it gives the correct answer.

But the moment a new combination shows up like this:

{"Outlook": "Sunny", "Temperature": "Cool", "Sell": "???" }

Should it return “Yes” or “No”?

Do you need to add more logical conditions? More “if”, “else”, or “loops”?

What you need is a function that can be trained to see new patterns and then predict. How do you do this?

To truly make your app smarter, you need to allow it to learn from data and adapt to new situations.

You are now leaving the realm of rule-based deterministic programming to data-driven statistical learning (machine learning).

Yes, it’s time for machine learning.

You’ve done your research, and you’ve settled on creating a simple Decision Tree for the app.

P.S. This is a simplified example, not exactly ready to run your coffee empire… but we’ll get there ☕

What’s a decision tree?

A decision tree is like a flowchart that helps you make a decision step-by-step. At each step (called a “node”), the computer asks a yes/no or greater/less than question about a feature (or attribute) of the data.

At the end of the path (a “leaf”), the tree gives you a prediction, like a category or a value.

When you look at the definition and image of a decision tree, you might wonder: Couldn’t this just be done with a few logical conditions?

But this brings us back to a key idea, we need algorithms that can learn from data and adapt to new situations. A decision tree isn’t static; it must grow and be pruned dynamically based on the data.

So, as I understand it, we are building a function that generates a function to predict for each new dataset.

So, the real question becomes: How do we build such a tree from data?

To do that, there are two key mathematical concepts you need to understand: Entropy and Information Gain.

No worries, we’ll walk through them step by step. We won’t dive into the formal math proofs, but we will focus on how to translate the equations into code.

Mathematical Concepts

1. Entropy

What is Entropy?
It’s the measure of how much impurity or uncertainty in a dataset.

• If all examples in a dataset belong to the same class, the entropy is 0 (pure).

• If the examples are equally mixed (e.g., 50% “Yes”, 50% “No”), the entropy is maximum (e.g., in most examples I found, they are 1 or 0).

Yep, the concept is too abstract.

So, let’s apply it to your dataset for some context:

data = [ 
  {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "No"},
  {"Outlook": "Sunny", "Temperature": "Mild", "Sell": "Yes"},
  {"Outlook": "Rainy", "Temperature": "Cool", "Sell": "Yes"},
  {"Outlook": "Overcast", "Temperature": "Hot", "Sell": "Yes"},
  {"Outlook": "Rainy", "Temperature": "Mild", "Sell": "No"}
]

Let’s calculate entropy for the target attribute (What we want to predict): “Sell”

• Total examples: 5

• “Yes”: 3

• “No”: 2

So:

Substituting the values for the Entropy formula we get:

✅ Entropy = 0.971 bits (Entropy for target attribute “Sell”)

You’ve learned the math, now you need to map it to code:

from collections import Counter
import math

# --- Entropy Calculation Function ---
def entropy(data, target):
    # Count the occurrences of each class label in the target attribute
    # For example, in the "Sell" column, we count how many times 
    # "Yes" and "No" appear
    counts = Counter(row[target] for row in data)

    # Total number of examples in the dataset
    total = len(data)

    # Apply the entropy formula:
    # E(S) = -∑ (p_i * log2(p_i)) for each class i
    # Where p_i is the proportion of class i in the dataset
    return -sum((count / total) * math.log2(count / total) for count in counts.values())

# ------------------------------------
# 🧠 Let’s Apply It to Your Dataset
dataset = [
    {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "No"},
    {"Outlook": "Sunny", "Temperature": "Mild", "Sell": "Yes"},
    {"Outlook": "Rainy", "Temperature": "Cool", "Sell": "Yes"},
    {"Outlook": "Overcast", "Temperature": "Hot", "Sell": "Yes"},
    {"Outlook": "Rainy", "Temperature": "Mild", "Sell": "No"}
]

# We want to calculate the entropy of the target attribute: "Sell"
# Breakdown:
# Total examples = 5
# "Yes" = 3 → p_Yes = 3/5 = 0.6
# "No"  = 2 → p_No  = 2/5 = 0.4

# Entropy formula:
# E(S) = -[0.6 * log2(0.6) + 0.4 * log2(0.4)]
#      ≈ -[0.6 * (-0.737) + 0.4 * (-1.322)]
#      ≈ 0.971 bits

# ✅ Now let's calculate it using our function:
result = entropy(dataset, "Sell")
print("Entropy of 'Sell':", round(result, 3))  # Expected output: 0.971

2. Information Gain

Information Gain (IG) measures how much uncertainty (entropy) is reduced after splitting the data based on an attribute.

Again, let’s make this definition more understandable with examples on your data:

dataset = [
    {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "No"},
    {"Outlook": "Sunny", "Temperature": "Mild", "Sell": "Yes"},
    {"Outlook": "Rainy", "Temperature": "Cool", "Sell": "Yes"},
    {"Outlook": "Overcast", "Temperature": "Hot", "Sell": "Yes"},
    {"Outlook": "Rainy", "Temperature": "Mild", "Sell": "No"}
]

Let’s compute the information gain of the attribute: “Outlook”

Step 1: Group by “Outlook”

+----------+---------+-------------+
| Outlook  | Subset  | Sell values |
+----------+---------+-------------+
| Sunny    | Row 1,2 | No, Yes     |
| Rainy    | Row 3,5 | Yes, No     |
| Overcast | Row 4   | Yes         |
+----------+---------+-------------+

Step 2: Compute the entropy of each subset

Outlook = Sunny (2 items)

• “No”: 1

• “Yes”: 1

Outlook = Rainy (2 items)

• “Yes”: 1

• “No”: 1

Outlook = Overcast (1 item)

• “Yes”: 1

✅ Information Gain for “Outlook” = 0.171

You’ve learned the math, now you need to map it to code. Note that this uses the entropy function we’ve discussed above:

def information_gain(data, feature, target):
    # Step 1️⃣: Compute the total entropy of the entire dataset using
    # only the target class
    # This corresponds to: E(S) = 0.971 in your example
    total_entropy = entropy(data, target)

    # Step 2️⃣: Get all unique values of the attribute 
    # we're evaluating (e.g., Outlook = Sunny, Rainy, Overcast)
    values = set(row[feature] for row in data)

    # Step 3️⃣: Initialize the weighted sum of entropies after the split
    weighted_entropy = 0.0

    # Step 4️⃣: For each value (e.g., "Sunny"), compute:
    for value in values:
        # a. Subset the dataset where feature == value 
        # (e.g., Outlook == Sunny)
        subset = [row for row in data if row[feature] == value]

        # b. Calculate the proportion of the subset relative 
        # to the total data
        weight = len(subset) / len(data)

        # c. Compute entropy of the subset (based only on 
        # the target variable "Sell")
        subset_entropy = entropy(subset, target)

        # d. Add this weighted entropy to the total
        weighted_entropy += weight * subset_entropy

    # Step 5️⃣: Subtract weighted entropy from total entropy to 
    # get Information Gain
    return total_entropy - weighted_entropy

    # 🧠 Let's Apply Information Gain to "Outlook"
    # Total entropy of "Sell" was calculated before: ≈ 0.971

    # We'll now compute the weighted entropy after splitting by "Outlook":
    # Outlook has 3 unique values: Sunny, Rainy, Overcast

    # 🌤 Sunny subset:
    #   [{"Sunny", "Hot", "No"}, {"Sunny", "Mild", "Yes"}]
    #   - 2 samples: 1 Yes, 1 No → entropy = 1.0
    #   - weight = 2/5 = 0.4
    #   - weighted entropy = 0.4 × 1.0 = 0.4

    # 🌧 Rainy subset:
    #   [{"Rainy", "Cool", "Yes"}, {"Rainy", "Mild", "No"}]
    #   - 2 samples: 1 Yes, 1 No → entropy = 1.0
    #   - weight = 2/5 = 0.4
    #   - weighted entropy = 0.4 × 1.0 = 0.4

    # ☁️ Overcast subset:
    #   [{"Overcast", "Hot", "Yes"}]
    #   - 1 sample: 1 Yes → entropy = 0
    #   - weight = 1/5 = 0.2
    #   - weighted entropy = 0.2 × 0 = 0.0

    # 🔄 Total Weighted Entropy = 0.4 + 0.4 + 0.0 = 0.8

    # ✅ Information Gain = Total Entropy - Weighted Entropy
    #                    = 0.971 - 0.8
    #                    = 0.171

    # Let's run the function:
    gain_outlook = information_gain(dataset, "Outlook", "Sell")
    print("Information Gain for 'Outlook':", round(gain_outlook, 3))  
    # Expected: 0.171

In decision trees, it is used to select the best attribute to split on, the one that provides the highest IG. This will become clearer later once I use the information gain function.

Building the Tree

We’ve built the functions for Entropy and Information Gain; the next step is to create a function that builds the tree.

Here’s the step on how we build the tree:

1. Calculate Information Gain for “Outlook” and “Temperature”.

2. Select the feature with higher Information Gain (likely “Outlook”)

3. Split the data based on Outlook values (Sunny, Rainy, Overcast)

4. Recursively build subtrees for each branch.

Here’s the code for that:

def build_tree(data, features, target):
    
    # Extract all target labels from the current data subset
    # Example: if target="Sell", labels = ["No", "Yes", "Yes", "Yes", "No"]
    labels = [row[target] for row in data]
    
    # BASE CASE 1: Pure Node (All labels are the same)
    # If all instances have the same target label, create a leaf node
    # Example: labels = ["Yes", "Yes", "Yes"] → return "Yes"
    if labels.count(labels[0]) == len(labels):
        return labels[0]
    
    # BASE CASE 2: No Features Left (Feature exhaustion)
    # If we've used all features but still have mixed labels,
    # return the most common label (majority vote)
    # Example: labels = ["No", "Yes", "No"] → return "No" (majority)
    if not features:
        return Counter(labels).most_common(1)[0][0]

    # RECURSIVE CASE: Feature Selection and Splitting
    
    # Step 1: Calculate information gain for each remaining feature
    # Dictionary comprehension creates: {"Outlook": 0.25, "Temperature": 0.15}
    gains = {feature: information_gain(data, feature, target) for feature in features}
    
    # Step 2: Select the feature with the highest information gain
    # Uses max() with key=gains.get to find the key with maximum value
    best_feature = max(gains, key=gains.get)

    # Step 3: Create the tree node structure
    # Creates a dictionary where the key is the splitting feature
    # Example: {"Outlook": {}} - empty dict will hold the branches
    tree = {best_feature: {}}
    
    # Step 4: Get all unique values for the chosen feature
    # Example: if best_feature="Outlook", values = {"Sunny", "Rainy", "Overcast"}
    values = set(row[best_feature] for row in data)
    
    # Step 5: For each unique value, create a branch by recursive splitting
    for value in values:
        # Filter data to only include rows with this feature value
        # Example: subset for "Sunny" = all rows where Outlook="Sunny"
        subset = [row for row in data if row[best_feature] == value]
        
        # Recursively build a subtree for this subset using remaining features
        # Remove the current feature from available features to prevent reuse
        # Example: if we used "Outlook", remaining features = ["Temperature"]
        subtree = build_tree(subset, [f for f in features if f != best_feature], target)
        
        # Attach the returned subtree to the current tree node
        # Example: tree["Outlook"]["Sunny"] = {subtree for sunny conditions}
        tree[best_feature][value] = subtree

    # Return the completed tree structure
    # Final structure: 
    # Outlook?
    #   ├── Sunny:
    #   │   └── Temperature?
    #   │       ├── Hot → Yes
    #   │       └── Mild → No
    #   ├── Rainy:
    #   │   └── Temperature?
    #   │       ├── Hot → Yes
    #   │       └── Cool → No
    #   └── Overcast:
    #       └── Temperature?
    #           ├── Hot → No
    #           └── Cool → Yes
    return tree

Updating the function

Based on what we’ve learned, now it’s time to update the is_good_day_to_sell function:

def is_good_day_to_sell(tree, instance):
    """
    Prediction function that traverses a decision tree to make 
    predictions for new instances.Implements the inference/prediction 
    phase of the decision tree algorithm using recursion.
    
    Parameters:
    - tree: The trained decision tree structure (nested dictionaries) 
            created by build_tree
    - instance: A dictionary representing a new data point to classify 
                (e.g., {"Outlook": "Sunny", "Temperature": "Mild"})
    
    Returns:
    - The classification label (e.g., "Yes" or "No")
    
    Example traversal for instance {"Outlook": "Sunny", "Temperature": "Mild"}:
    Step 1: tree is dict, feature="Outlook", value="Sunny" 
    Step 2: subtree = tree["Outlook"]["Sunny"] (Temperature subtree)
    Step 3: Recursive call with Temperature subtree
    Step 4: tree is dict, feature="Temperature", value="Mild"
    Step 5: subtree = "Yes" (leaf node)
    Step 6: tree is not dict, return "Yes"
    """
    
    # BASE CASE - Leaf Node: If tree is not a dictionary, 
    # we've reached a leaf node containing the final 
    # prediction/classification label
    if not isinstance(tree, dict):
        return tree
    
    # Get the splitting feature from the current tree node
    # Since each tree node is a dictionary with one key (the splitting feature),
    # this extracts that feature name. Example: {"Outlook": {...}} → feature = "Outlook"
    feature = next(iter(tree))
    
    # Look up the value of the current feature in the instance being classified
    # Example: If feature="Outlook" and instance={"Outlook": "Sunny", ...}, then value="Sunny"
    value = instance.get(feature)
    
    # Navigate to the subtree corresponding to the instance's feature value
    # Example: tree["Outlook"].get("Sunny") returns the subtree for sunny conditions
    subtree = tree[feature].get(value)
    
    # RECURSIVE CASE: Continue traversing down the tree with the selected subtree
    # This follows the decision path through the tree based on the instance's feature values
    # until reaching a leaf node containing the final prediction
    return is_good_day_to_sell(subtree, instance)

Evaluation

This wouldn't feel like an AI blog if we didn’t compare our new algorithm to the previous one. How does our new algorithm compared to our original one?

Here’s how the evaluation was done:

from build_tree import build_tree
from is_good_day_to_sell import is_good_day_to_sell
from static_tree_predict import static_tree_predict

# --- Dataset ---
dataset = [
    {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "No"},
    {"Outlook": "Sunny", "Temperature": "Mild", "Sell": "Yes"},
    {"Outlook": "Rainy", "Temperature": "Cool", "Sell": "Yes"},
    {"Outlook": "Overcast", "Temperature": "Hot", "Sell": "Yes"},
    {"Outlook": "Rainy", "Temperature": "Mild", "Sell": "No"}
]

features = ["Outlook", "Temperature"]
target = "Sell"

# Complex dataset
complex_dataset = [
    {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "Yes"},     # contradiction to static
    {"Outlook": "Sunny", "Temperature": "Cool", "Sell": "No"},     # contradiction
    {"Outlook": "Sunny", "Temperature": "Mild", "Sell": "Yes"},
    {"Outlook": "Rainy", "Temperature": "Hot", "Sell": "Yes"},     # unseen combo
    {"Outlook": "Rainy", "Temperature": "Cool", "Sell": "No"},     # contradiction
    {"Outlook": "Rainy", "Temperature": "Mild", "Sell": "Yes"},    # flipped
    {"Outlook": "Overcast", "Temperature": "Hot", "Sell": "Yes"},
    {"Outlook": "Overcast", "Temperature": "Cool", "Sell": "No"},  # exception
    {"Outlook": "Overcast", "Temperature": "Mild", "Sell": "Yes"},
    {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "Yes"},     # repeat for reinforcement
]

# Much more challenging dataset that exposes static tree limitations
challenging_dataset = [
    # Sunny days - Static tree says Hot=No, others=Yes
    # Let's make the pattern more complex: depends on both features together
    {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "Yes"},      # Static: No, should be Yes
    {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "Yes"},      # Reinforce pattern
    {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "Yes"},      # Static gets this wrong
    {"Outlook": "Sunny", "Temperature": "Mild", "Sell": "No"},      # Static: Yes, should be No
    {"Outlook": "Sunny", "Temperature": "Mild", "Sell": "No"},      # Reinforce
    {"Outlook": "Sunny", "Temperature": "Cool", "Sell": "Yes"},     # Static: Yes, correct by luck
    {"Outlook": "Sunny", "Temperature": "Cool", "Sell": "Yes"},
    
    # Rainy days - Static tree says Cool=Yes, others=No  
    # Let's completely flip this pattern
    {"Outlook": "Rainy", "Temperature": "Cool", "Sell": "No"},      # Static: Yes, should be No
    {"Outlook": "Rainy", "Temperature": "Cool", "Sell": "No"},      # Reinforce
    {"Outlook": "Rainy", "Temperature": "Cool", "Sell": "No"},      # Static gets this wrong
    {"Outlook": "Rainy", "Temperature": "Mild", "Sell": "Yes"},     # Static: No, should be Yes
    {"Outlook": "Rainy", "Temperature": "Mild", "Sell": "Yes"},     # Reinforce
    {"Outlook": "Rainy", "Temperature": "Hot", "Sell": "Yes"},      # Static: No, should be Yes
    {"Outlook": "Rainy", "Temperature": "Hot", "Sell": "Yes"},      # Reinforce
    
    # Overcast days - Static tree always says Yes
    # Let's make it depend on temperature in a different way
    {"Outlook": "Overcast", "Temperature": "Hot", "Sell": "No"},    # Static: Yes, should be No
    {"Outlook": "Overcast", "Temperature": "Hot", "Sell": "No"},    # Reinforce
    {"Outlook": "Overcast", "Temperature": "Hot", "Sell": "No"},    # Static gets this wrong
    {"Outlook": "Overcast", "Temperature": "Mild", "Sell": "Yes"},  # Static: Yes, correct by luck
    {"Outlook": "Overcast", "Temperature": "Mild", "Sell": "Yes"},
    {"Outlook": "Overcast", "Temperature": "Cool", "Sell": "Yes"},  # Static: Yes, correct by luck
    {"Outlook": "Overcast", "Temperature": "Cool", "Sell": "Yes"},
    
    # Additional edge cases to really challenge the static tree
    {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "Yes"},      # More evidence against static
    {"Outlook": "Rainy", "Temperature": "Cool", "Sell": "No"},      # More evidence against static
    {"Outlook": "Overcast", "Temperature": "Hot", "Sell": "No"},    # More evidence against static
]

# Extreme challenge dataset - completely opposite to static tree assumptions
extreme_dataset = [
    # Make the static tree fail catastrophically
    # Sunny + Hot should be Yes (static says No)
    {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "Yes"},
    {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "Yes"},
    {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "Yes"},
    {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "Yes"},
    {"Outlook": "Sunny", "Temperature": "Hot", "Sell": "Yes"},
    
    # Sunny + others should be No (static says Yes)
    {"Outlook": "Sunny", "Temperature": "Mild", "Sell": "No"},
    {"Outlook": "Sunny", "Temperature": "Mild", "Sell": "No"},
    {"Outlook": "Sunny", "Temperature": "Cool", "Sell": "No"},
    {"Outlook": "Sunny", "Temperature": "Cool", "Sell": "No"},
    
    # Rainy + Cool should be No (static says Yes)
    {"Outlook": "Rainy", "Temperature": "Cool", "Sell": "No"},
    {"Outlook": "Rainy", "Temperature": "Cool", "Sell": "No"},
    {"Outlook": "Rainy", "Temperature": "Cool", "Sell": "No"},
    {"Outlook": "Rainy", "Temperature": "Cool", "Sell": "No"},
    
    # Rainy + others should be Yes (static says No)
    {"Outlook": "Rainy", "Temperature": "Mild", "Sell": "Yes"},
    {"Outlook": "Rainy", "Temperature": "Mild", "Sell": "Yes"},
    {"Outlook": "Rainy", "Temperature": "Hot", "Sell": "Yes"},
    {"Outlook": "Rainy", "Temperature": "Hot", "Sell": "Yes"},
    
    # Overcast should sometimes be No (static always says Yes)
    {"Outlook": "Overcast", "Temperature": "Hot", "Sell": "No"},
    {"Outlook": "Overcast", "Temperature": "Hot", "Sell": "No"},
    {"Outlook": "Overcast", "Temperature": "Hot", "Sell": "No"},
    {"Outlook": "Overcast", "Temperature": "Mild", "Sell": "Yes"},
    {"Outlook": "Overcast", "Temperature": "Cool", "Sell": "Yes"},
]

def compare_predictions(tree, dataset, title=""):
    """Compare predictions between static tree and ML tree."""
    print(f"\n📊 {title}")
    print(f"{'Outlook':<10} {'Temp':<10} {'Actual':<7} {'Static':<8} {'ML Tree':<8} {'Match?'}")
    for row in dataset:
        actual = row["Sell"]
        static_pred = static_tree_predict(row)
        ml_pred = is_good_day_to_sell(tree, row)
        match = "✅" if static_pred == ml_pred else "❌"
        print(f"{row['Outlook']:<10} {row['Temperature']:<10} {actual:<7} {static_pred:<8} {ml_pred:<8} {match}")

def compute_accuracy(predict_fn, data, target):
    """Compute accuracy of a prediction function on a dataset."""
    correct = 0
    for row in data:
        if predict_fn(row) == row[target]:
            correct += 1
    return correct / len(data)

def detailed_comparison(tree, dataset, title=""):
    """Provide detailed analysis of where static vs ML tree differ"""
    print(f"\n🔍 DETAILED ANALYSIS: {title}")
    print("="*80)
    
    static_correct = 0
    ml_correct = 0
    both_correct = 0
    both_wrong = 0
    static_better = 0
    ml_better = 0
    
    disagreements = []
    
    for row in dataset:
        actual = row["Sell"]
        static_pred = static_tree_predict(row)
        ml_pred = is_good_day_to_sell(tree, row)
        
        static_right = static_pred == actual
        ml_right = ml_pred == actual
        
        if static_right:
            static_correct += 1
        if ml_right:
            ml_correct += 1
            
        if static_right and ml_right:
            both_correct += 1
        elif not static_right and not ml_right:
            both_wrong += 1
        elif static_right and not ml_right:
            static_better += 1
        elif not static_right and ml_right:
            ml_better += 1
            
        if static_pred != ml_pred:
            disagreements.append({
                'case': f"{row['Outlook']}-{row['Temperature']}",
                'actual': actual,
                'static': static_pred,
                'ml': ml_pred,
                'static_right': static_right,
                'ml_right': ml_right
            })
    
    total = len(dataset)
    print(f"📈 ACCURACY COMPARISON:")
    print(f"   Static Tree:  {static_correct:2d}/{total} = {static_correct/total:.1%}")
    print(f"   ML Tree:      {ml_correct:2d}/{total} = {ml_correct/total:.1%}")
    print(f"   Improvement:  {ml_correct-static_correct:+2d} predictions = {(ml_correct-static_correct)/total:+.1%}")
    
    print(f"\n🎯 AGREEMENT ANALYSIS:")
    print(f"   Both Correct: {both_correct:2d}/{total} = {both_correct/total:.1%}")
    print(f"   Both Wrong:   {both_wrong:2d}/{total} = {both_wrong/total:.1%}")
    print(f"   Static Better:{static_better:2d}/{total} = {static_better/total:.1%}")
    print(f"   ML Better:    {ml_better:2d}/{total} = {ml_better/total:.1%}")
    
    if disagreements:
        print(f"\n⚔️  DISAGREEMENTS ({len(disagreements)} cases):")
        print(f"{'Case':<15} {'Actual':<7} {'Static':<7} {'ML':<7} {'Winner'}")
        print("-" * 50)
        for d in disagreements:
            if d['static_right'] and not d['ml_right']:
                winner = "Static ✅"
            elif not d['static_right'] and d['ml_right']:
                winner = "ML ✅"
            elif not d['static_right'] and not d['ml_right']:
                winner = "Both ❌"
            else:
                winner = "Both ✅"
            print(f"{d['case']:<15} {d['actual']:<7} {d['static']:<7} {d['ml']:<7} {winner}")
    
    return {
        'static_accuracy': static_correct/total,
        'ml_accuracy': ml_correct/total,
        'improvement': (ml_correct-static_correct)/total,
        'disagreements': len(disagreements)
    }

if __name__ == "__main__":
    # Train the tree on original dataset
    tree = build_tree(dataset, features, target)
    print("🌳 Learned Tree Structure:")
    print(tree)

    # Simple demo
    print("\n" + "="*50)
    print("🚀 SIMPLE DEMO")
    print("="*50)
    test_instance = {"Outlook": "Sunny", "Temperature": "Mild"}
    print("Prediction for", test_instance, "->", is_good_day_to_sell(tree, test_instance))
    
    print("\n" + "="*100)
    print("🧪 TESTING DIFFERENT DATASETS TO EXPOSE STATIC TREE LIMITATIONS")
    print("="*100)

    # Test 1: Original simple data
    print("\n" + "🔹"*50)
    print("TEST 1: Original Training Data (should work well for both)")
    detailed_comparison(tree, dataset, "Original Training Data")

    # Test 2: Basic complex data  
    print("\n" + "🔹"*50)
    print("TEST 2: Basic Complex Data (some contradictions)")
    detailed_comparison(tree, complex_dataset, "Basic Complex Data")

    # Test 3: More challenging data
    print("\n" + "🔹"*50)
    print("TEST 3: Challenging Dataset (systematic patterns opposite to static tree)")
    challenging_tree = build_tree(challenging_dataset, features, target)
    print(f"🌳 New tree learned from challenging data: {challenging_tree}")
    detailed_comparison(challenging_tree, challenging_dataset, "Challenging Dataset")

    # Test 4: Extreme challenge
    print("\n" + "🔹"*50)
    print("TEST 4: Extreme Challenge (completely opposite patterns)")
    extreme_tree = build_tree(extreme_dataset, features, target)
    print(f"🌳 New tree learned from extreme data: {extreme_tree}")
    detailed_comparison(extreme_tree, extreme_dataset, "Extreme Challenge Dataset")

This is the result:

This isn’t a very sophisticated evaluation, but it will hopefully give you an idea of why the Decision Tree learner is better than the static one. By the way, I’m calling the first version of the “prediction” function as static_tree_predict to distinguish it from the learning one, which is called the is_good_day_to_sell.

The reasoning on how the datasets is “harder” is commented on the code.

✅ Test 1: Original Training Data

+------------+-------------+---------------------------------+
| Metric     | Static Tree | ML Tree                         |
+------------+-------------+---------------------------------+
| Accuracy   | 100%        | 100%                            |
| Improvement| 0%          | —                               |
| Agreement  | Perfect     | Both agreed on every prediction |
+------------+-------------+---------------------------------+

🧪 Test 2: Complex Dataset

+---------------+-------------+-------------------------+
| Metric        | Static Tree | ML Tree                 |
+---------------+-------------+-------------------------+
| Accuracy      | 30%         | 40%                     |
| Improvement   | +10%        | +1 correct prediction   |
| Disagreements | 1 case      | ML better in that case  |
+---------------+-------------+-------------------------+

🔥 Test 3: Challenging Dataset

+---------------+-------------+---------------------------+
| Metric        | Static Tree | ML Tree                   |
+---------------+-------------+---------------------------+
| Accuracy      | 30%         | 40%                       |
| Improvement   | +10%        | +1 correct prediction     |
| Disagreements | 1 case      | ML better in that case    |
+---------------+-------------+---------------------------+

🚨 Test 4: Extreme Challenge Dataset

+---------------+-------------+---------------------------+
| Metric        | Static Tree | ML Tree                   |
+---------------+-------------+---------------------------+
| Accuracy      | 9.1%        | 100%                      |
| Improvement   | +90.9%      | +20 correct predictions   |
| Disagreements | 20 cases    | All won by ML Tree        |
+---------------+-------------+---------------------------+

Observations

The Decision Tree (ML tree), learns from data and adapts well to new patterns, showing significant improvement when the data distribution changes. However, it may fail when trained on flipped or misleading data and then tested on the original, revealing a risk of overfitting.

In contrast, the static tree (original code) relies on hardcoded logic and only performs well under fixed patterns. It fails catastrophically when exposed to unseen or contradictory inputs and cannot update itself without manual reprogramming, making it brittle in dynamic environments.

Conclusion

You’ve built the app ! Now it’s time to launch your startup and chase that unicorn status! 🦄 Go make it happen!

☕ Just a Quick Reality Check

Okay, this whole post is a beginner-friendly crash course in Decision Trees. It’s meant to help you see how machines can learn from patterns, not replace scikit-learn, your favorite ML library, or actual data science best practices.

The code here is more of a proof of concept than something you’d ship to production. No error handling, no fancy pruning, no cross-validation. Just vibes, recursion, and decision-making magic. ✨

Thanks for sticking with me this far! I know it got a bit lengthy. I couldn’t quite trim it down as much as I wanted. As a little reward, here’s a photo of my Dobermann pup, Claude, named after Claude Shannon, the father of information theory (yes, the same one behind Entropy and Information Gain).

🕰 A Lesson from History

The emergence of electricity in the late 19th and early 20th centuries forced many lamp-making companies into bankruptcy. But Edward Miller & Co., a prominent oil-lamp manufacturer based in Meriden, Connecticut, faced this pivotal moment differently. As electricity became more accessible, oil-lamp seems became more and more irrelevant.

Edward Miller & Co. didn’t resist the change. Instead they adapted. The company retrained its artisans, engineers, and factory workers to master electrical bulb design and production. The transition from handcrafted oil lamps to modern electric fixtures.

This wasn’t just a technological shift in their company.

It was an organizational transformation!

⚡ The Foundation Model Shift

The release of foundation models like GPT, Claude, and open-source versions started a pivotal shift in the AI landscape.

Initially, you needed elite research teams and millions to setup an infrastructure for AI. Now you can do the same thing via a simple API call.

AI became cheaper to implement!

Everyone wanted in. Many companies tried integrated AI into their products.

But beneath the surface, a surprising number still struggle to truly adapt. Ironically, even teams with established MLOps teams. How could these be?

If your ML Ops team is one of those, don’t worry. You’re not alone. Even companies with top-tier data science talent are facing growing pains. But the ones who act now have a window of opportunity to leap ahead.

🤖 Why Traditional ML Ops Teams Struggle to Adapt?

Historically, ML Ops revolves around supervised learning.

What is supervised learning?

You collect labeled data, train a model, deploy it, monitor it, and retrain.

Examples of supervised learning are:

• Fraud Detection: They train models on years of labeled transaction data to identify patterns indicative of fraudulent activity. Is this fraud or not?

• Product Recommendation: Teams use historical user behavior and purchase data to train models that suggest relevant products. Is this a relevant recommendation or not?

• Email Spam Detection: ML systems are trained on user-labeled emails to classify and filter out spam messages. Is this spam or not?

But foundation models flip this script. They’re trained on huge and generalized data. They don’t need your labeled dataset to get started.

The challenge now with accessible foundation model is: how to prompt, fine-tune, or steer the model to deliver value. This demands a new mindset, one where your team needs to understand how to use models initially without direct training and with minimal data

🏓 Ending the AI Ping-Pong

Given how easy it is to start building AI feature, ML teams and product teams fall into this trap:

• CEO: Let’s integrate AI!

• ML team ask product managers: “What do you want AI to do?”

• Product team ask ML engineers: “What can AI do?”

This loop goes nowhere. Without a shared workflow, nobody moves forward. This gap is exactly what AI Engineering aims to bridge.

Start by solving something real. Then ask, “Can AI make this faster, easier, smarter, or more scalable?”

🧠 Rethinking ML Pipelines with AI Engineering + Design Thinking

The way we integrate AI into products is shifting fundamentally. No longer do we start with a dataset and model. Instead, we begin with users: their pains, needs, and behavior.

🤖 AI Engineering (Design Thinking–Driven)

Inspired by Chip Huyen’s AI Engineering book, I’ve adapted the workflow to start with customers, not the dataset. From my learnings of building and working with startups (many of which didn’t quite take off 😅), the rise of accessible foundation models should give us a chance to shift our focus from “What can AI do?” to “How can AI make our customers’ lives easier?”

❌ Traditional ML Engineering:

Data ➡ Model ➡ Product ➡ Customer

1. Data: A team collects years of historical transaction records, each labeled as “fraud” or “not fraud.”

2. Model: They train a supervised learning model to recognize patterns that indicate potential fraud.

3. Product: The model is deployed into the company’s payment processing system to flag or block suspicious transactions in real-time.

4. Customer: Customers benefit indirectly through enhanced security and fewer fraudulent charges on their accounts.

It’s a proven system, but it assumes you already have structured data and a clear pattern to learn.

✅ AI Engineering (Design Thinking–Driven):

Customer ➡ Product ➡ Data ➡ Model

1. Customer: A product team notices that users often struggle to write effective product descriptions. This is a real, high-friction user problem.

2. Product: They respond by designing a “smart suggestion” feature directly within the editor to support users as they write.

3. Data: As users begin interacting with this feature, the team collects anonymized drafts and completions to understand usage patterns and pain points.

4. Model: With this data, they evaluate what foundation models can assist with (e.g., tone, clarity, SEO) and fine-tune or chain prompts to deliver helpful, personalized writing suggestions. If this is not enough, you can then later decide to augment the model with retrieval (RAG), integrate external tools (like grammar checkers or SEO analyzers), or design workflows with human-in-the-loop for higher quality control.

The process starts by solving a real user problem instead of building a model and looking for where to apply it. After identifying the problem, strategically bring in AI where it creates value.

You can see companies like Notion, Canva, and Shopify are already embracing this approach. Embedding AI seamlessly into their products that it feels like a natural extension of the user experience. From Notion’s AI that enhances productivity without disrupting workflow, to Canva’s Magic Studio that empowers non-designers to create effortlessly, to Shopify’s Sidekick that supports merchants like a real teammate. These platforms prove that AI can be invisible in interface, intuitive in use, and deeply human in intent.

🎯 Takeaway

We’re early in this transformation. But like Edward Miller & Co., the companies who adapt now will be the ones who lead.

• Don’t wait for ML teams to be told what to build.

• Don’t wait for product teams to "get AI."

• Start by solving something real. Then ask, “Can AI make this faster, easier, smarter, or more scalable?”

AI Engineering isn’t a rebrand. It’s a reorganization.

Here’s a photo of me trying to reorg myself to learn what is possible with AI through the Master of Artificial Intelligence program at the University of Waikato, home of Weka.

References

Historic New England. (n.d.). Electric lamp factory workers [Photograph]. https://images.historicnewengland.org/rs/138533/scr/default.jpg

Huyen, C. (2024). The AI engineering book. Claypot AI. https://huyenchip.com/ai-engineering/book/

Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., ... & Liang, P. (2021). On the opportunities and risks of foundation models (Stanford CRFM Report). https://crfm.stanford.edu/report.html

Zhang, A., Lipton, Z. C., Li, M., & Smola, A. J. (2021). Dive into deep learning (Version 1.0.0). https://d2l.ai

Anthropic. (2024). How Notion uses Claude to help people work faster and smarter. https://www.anthropic.com/customers/notion ↩

Canva. (2023). Canva launches Magic Studio, a suite of AI tools for everyday creators. https://www.canva.com/newsroom/news/canva-ai-launches ↩

Upskillist. (2024). Hiring AI over humans? Decoding Shopify’s bold new strategy. https://www.upskillist.com/blog/hiring-ai-over-humans-decoding-shopifys-bold-new-strategy ↩

I dove into integrating PayPal subscriptions into a SaaS app using Rails on the backend and Next.js on the frontend. It took quite a bit of time, so I decided to document the process, tough decisions, tackling some tricky technical challenges, and building a smooth subscription workflow that really works.

Background 🌐

The challenge? I was building a subscription payment system for a SaaS startup that was part of the 2024 Cohort of AUT Venture’s IDEA2IMPACT program. Initially, I was all set to use Stripe because of its awesome developer support and ease of use. But when I found out Stripe wasn’t available in our target country, I switched gears and went with PayPal instead.

Why PayPal & The Initial Hurdles 🤔

Why PayPal?

✅ A solid alternative when Stripe isn’t an option

🔄 Supports recurring billing

💳 A well-established payment provider

The Challenges I Faced:

• Unlike Stripe, PayPal’s subscription API meant I had to make a bunch of manual API calls.

• There’s no slick built-in UI for managing subscriptions.

• And on top of that, PayPal webhooks were a bit hit or miss in production.

Rolling Up My Sleeves with API Calls 🔧

Since PayPal doesn’t offer an out-of-the-box UI for subscriptions, I had to set everything up manually using cURL requests. Here’s a quick rundown of the process to setup subscriptions in your terminal:

Step 1: Generate an Access Token 🔑

encoded_credentials=$(echo -n "CLIENT_ID:CLIENT_SECRET" | base64 -w 0)
curl -v -X POST "https://api-m.sandbox.paypal.com/v1/oauth2/token" \
-H "Content-Type: application/x-www-form-urlencoded" \
-H "Authorization: Basic $encoded_credentials" \
-d "grant_type=client_credentials"

Step 2: Create a Product 🛍️

curl -v -X POST "https://api-m.sandbox.paypal.com/v1/catalogs/products" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ACCESS_TOKEN" \
-d '{
  "name": "My SaaS Product",
  "description": "SaaS subscription product",
  "type": "SERVICE",
  "category": "SOFTWARE",
  "home_url": "https://my-saas.com/"
}'

Step 3: Create a Billing Plan 💸

curl -v -X POST "https://api-m.sandbox.paypal.com/v1/billing/plans" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ACCESS_TOKEN" \
-d '{
  "product_id": "PROD-12345",
  "name": "Starter Plan",
  "description": "Basic subscription plan",
  "status": "ACTIVE",
  "billing_cycles": [
    {
      "frequency": { "interval_unit": "MONTH", "interval_count": 1 },
      "tenure_type": "REGULAR",
      "sequence": 1,
      "total_cycles": 0,
      "pricing_scheme": { "fixed_price": { "value": "49.99", "currency_code": "USD" } }
    }
  ],
  "payment_preferences": { "auto_bill_outstanding": true, "payment_failure_threshold": 3 }
}'

Bringing It All Together in Rails & Next.js 🔄

Here’s how I built the subscription flow:

• Frontend (Next.js): I rendered a PayPal button using the PayPal Button SDK. When a user clicked it and subscribed, PayPal returned a subscriptionID, which was then sent to my Rails backend.

• Backend (Rails): I saved the subscription details and set up scheduled jobs to periodically sync with PayPal for any updates.

Next.js Code for the PayPal Button 💻

const PaypalSubscriptionButton = ({ planId }) => {
  useEffect(() => {
    window.paypal.Buttons({
      createSubscription: (data, actions) => actions.subscription.create({ plan_id: planId }),
      onApprove: (data) => handleSubscriptionSuccess(data.subscriptionID),
    }).render("#paypal-button");
  }, []);

  const handleSubscriptionSuccess = async (subscriptionID) => {
    await fetch("/api/company_subscription", {
      method: "POST",
      body: JSON.stringify({ 
        subscription: { paypal_subscription_id: subscriptionID, plan_id: planId } 
      }),
    });
  };

  return <div id="paypal-button"></div>;
};

Rails API Endpoint to Save the Subscription 📝

class Api::CompanySubscriptionsController < ApplicationController
  def create
    subscription_params = params.require(:subscription).permit(:paypal_subscription_id, :plan_id)
    subscription = Subscription.create!(subscription_params)
    render json: { subscription: subscription }, status: :created
  end
end

Keeping Track of Subscription Status ⏱️

Because PayPal webhooks were a bit unreliable, I set up a scheduled job that periodically fetches subscription updates:

class SyncPaypalSubscriptionDetailsJob < ApplicationJob
  def perform(subscription_id)
    subscription = Subscription.find_by(id: subscription_id)
    return unless subscription
    return if subscription.paypal_subscription_id.blank?

    paypal_service = PaypalSubscriptionService.new
    subscription_details = paypal_service.get_subscription_details(subscription.paypal_subscription_id)

    subscription.update(
      status: subscription_details["status"].downcase,
      next_billing_date: Time.zone.parse(subscription_details.dig("billing_info", "next_billing_time"))
    )
  end
end

Final Takeaway 🎯

This project was a deep dive into integrating PayPal subscriptions in a SaaS environment. While PayPal’s API required more hands-on work compared to Stripe, I managed to build a robust system by using scheduled polling to handle subscription updates.

Starting with a lean MVP and iterating as needed turned out to be a winning strategy.

References:

• Github repo: https://github.com/iamademar/nextjs-rails-paypal-demo

By analyzing past marketing data, we can build a machine learning model to:
✅ Cut marketing costs by focusing on high-potential leads.
✅ Boost efficiency by reducing unnecessary calls.
✅ Enhance customer experience by avoiding spam.
✅ Increase revenue through better conversions.

In this article, we’ll explore the dataset, train a predictive AI model, and see how banks can apply real-time insights to optimize campaigns. 🚀

🎯 Business Impact: Why is this Useful?

This model allows the bank to:

1. Reduce Marketing Costs → The bank can avoid calling uninterested customers.

2. Improve Efficiency → Focus on high-potential leads, increasing campaign success rates.

3. Enhance Customer Experience → Avoid spamming uninterested customers.

4. Increase Revenue → More term deposit subscriptions lead to better bank profitability.

📌 Background: Why Train the Model?

The Portuguese banking institution has been running direct phone-based marketing campaigns to encourage customers to subscribe to term deposits. However, these campaigns involve multiple calls per customer, leading to high operational costs and wasted resources on uninterested clients.

To optimize these efforts, the bank wants to train a machine learning model that can predict which customers are most likely to subscribe to a term deposit. This prediction will allow the bank to:

• Focus marketing efforts on highly interested customers.

• Reduce unnecessary phone calls and operational costs.

• Improve conversion rates and overall campaign efficiency.

The dataset contains detailed information about customers and their interactions with previous marketing campaigns, including demographics, past interactions, and loan history.

Dataset: https://archive.ics.uci.edu/dataset/222/bank+marketing

📊 Understanding the Dataset

The dataset consists of 41,188 customer records with 20 features, collected between May 2008 and November 2010. The target variable (y) is binary:

• "yes" → The customer subscribed to a term deposit.

• "no" → The customer did not subscribe.

🔹 Key Features in the Dataset

1. Customer Demographics

   • age: Age of the customer.

   • job: Type of job (e.g., "admin", "blue-collar", "student").

   • marital: Marital status (e.g., "married", "single").

   • education: Level of education (e.g., "university degree", "high school").

2. Financial Information

   • balance: Customer’s average yearly balance (in euros).

   • default: Whether the customer has credit in default.

   • housing: Whether the customer has a housing loan.

   • loan: Whether the customer has a personal loan.

3. Marketing Campaign Interactions

   • contact: Type of contact used (cellular vs. telephone).

   • day_of_week & month: When the customer was last contacted.

   • duration: Last call duration (⚠️ should be discarded for realistic predictions as it’s known after the call).

   • campaign: Number of contacts in the current campaign.

   • pdays: Days since the customer was last contacted (-1 means no prior contact).

   • previous: Number of previous contacts.

   • poutcome: Outcome of the previous campaign (e.g., "failure", "success").

🛠️ Code

Step 1: Load the Dataset

import pandas as pd

# Load the dataset
df = pd.read_csv("bank-additional-full.csv", sep=";")  # Using semicolon as separator

• The bank has recorded customer responses from past campaigns.

• The dataset is loaded for preprocessing and analysis.

Step 2: Data Preprocessing

Before training a model, we need to clean and transform the data.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, LabelEncoder, OneHotEncoder
from sklearn.compose import ColumnTransformer

# Identify categorical and numerical features
categorical_features = ["job", "marital", "education", "contact", "month", "poutcome"]
numerical_features = ["age", "balance", "campaign", "pdays", "previous"]

# Drop 'duration' since it's not available before a call is made
df = df.drop(columns=["duration"])

# Encode categorical and numerical data
preprocessor = ColumnTransformer(
    transformers=[
        ("num", StandardScaler(), numerical_features),  # Scale numerical values
        ("cat", OneHotEncoder(), categorical_features)  # Convert categorical values
    ]
)

# Encode target variable
le = LabelEncoder()
df["y"] = le.fit_transform(df["y"])  # Convert 'yes'/'no' to 1/0

# Split into train and test sets
X = df.drop(columns=["y"])
y = df["y"]
X_train, X_test, y_train, y_test = train_test_split(preprocessor.fit_transform(X), y, test_size=0.2, random_state=42)

Why do this?

• StandardScaler ensures that numeric values are normalized.

• OneHotEncoder converts categorical values into machine-readable format.

• Drop ‘duration’ because it is only available after the call.

• Train-test split ensures we test our model on unseen customers.

Step 3: Train a Deep Learning Model

The bank wants to use a neural network to capture complex relationships between customer data and subscription likelihood.

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout

# Define a neural network model
model = Sequential([
    Dense(64, input_dim=X_train.shape[1], activation="relu"),
    Dropout(0.5),  # Prevents overfitting
    Dense(32, activation="relu"),
    Dropout(0.5),
    Dense(1, activation="sigmoid")  # Outputs probability (0 to 1)
])

# Compile the model
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])

# Train the model
history = model.fit(X_train, y_train, epochs=50, batch_size=32, validation_split=0.2, verbose=1)

The deep learning model used in this script is a Multi-Layer Perceptron (MLP), a type of Artificial Neural Network (ANN). It consists of fully connected (dense) layers and is designed for binary classification.

📌 Why Use This Deep Learning Model for This Dataset?

A Multi-Layer Perceptron (MLP) is used because:

1. Handles Both Categorical & Numerical Data

• The dataset includes both categorical (e.g., job, marital status) and numerical (e.g., balance, age) features.

• The preprocessing pipeline encodes categorical variables and normalizes numerical values, making it well-suited for an MLP.

2. Captures Complex Patterns

• Bank marketing data has non-linear relationships between features.

• MLP can detect interactions between variables (e.g., how age + job type + previous campaign success influences customer behavior).

3. Better Than Logistic Regression for Large Datasets

• Logistic Regression works well for simple binary classification but struggles with high-dimensional data.

• MLP can learn complex decision boundaries from features.

4. Dropout Layers Help Prevent Overfitting

• Overfitting is a common issue in marketing datasets.

• Dropout (0.5) ensures the model generalizes well to unseen data.

Step 3: Train a Deep Learning Model

The bank wants to use a neural network to capture complex relationships between customer data and subscription likelihood.

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout

# Define a neural network model
model = Sequential([
    Dense(64, input_dim=X_train.shape[1], activation="relu"),
    Dropout(0.5),  # Prevents overfitting
    Dense(32, activation="relu"),
    Dropout(0.5),
    Dense(1, activation="sigmoid")  # Outputs probability (0 to 1)
])

# Compile the model
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])

# Train the model
history = model.fit(X_train, y_train, epochs=50, batch_size=32, validation_split=0.2, verbose=1)

Step 4: Making Predictions on New Customers

The trained model can now be used to predict if new customers are likely to subscribe.

# Predict on test data
y_pred = model.predict(X_test)
y_pred = (y_pred > 0.5).astype(int).flatten()  # Convert probability to binary class

• If y_pred = 1, the customer is likely to subscribe.

• If y_pred = 0, the customer is unlikely to subscribe.

Step 5: Evaluating the Model’s Performance

How well does the model perform in predicting customer subscriptions?

from sklearn.metrics import precision_score, recall_score, f1_score

# Evaluate model performance
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
f1 = f1_score(y_test, y_pred)

print(f"Precision: {precision:.3f}")
print(f"Recall: {recall:.3f}")
print(f"F1 Score: {f1:.3f}")

✅ Precision → How many of the predicted "yes" were actually correct?

✅ Recall → How many actual "yes" were correctly identified?

✅ F1 Score → A balance between Precision and Recall.

📌 How can we integrated the model we developed?

We can create a real-time prediction for Bank Marketing Campaigns. Here's how I imagine it would look like.

🏆 Scenario

A surveyor is using a mobile application during a field campaign to gather customer information. 📱

• 📝 Data Collection

  • The surveyor inputs the following details into the app for a new customer, John:
    ✅ Age: 35
    ✅ Marital Status: Married
    ✅ Job: Blue-collar
    ✅ Housing Loan: No
    ✅ Personal Loan: No
    ✅ Contact Method: Cellular
    ✅ Outcome of Previous Campaign: Success

• ⚡ Real-Time Prediction

  • 📤 Upon submitting John's information, the app sends a request to the Django-based API hosting the predictive model.

  • 💡 The API processes the data and returns a prediction:

  • 📊 Probability of Subscription: 78%

• 🎯 Actionable Insight

  • 🔔 The app displays a notification:

  • "High likelihood of subscription. Prioritize follow-up with this customer."

  • 💬 The surveyor, armed with this insight, can tailor the conversation to increase the chances of conversion.

🚀 With this AI-driven approach, marketing efforts become more efficient, improving success rates while reducing wasted resources!

P.S.

• You can run this code on Google Colab.

• Github repo here.

So I tried prompting ChatGPT, and the initial version was something like this:

After a few tries of those basic prompts and not getting the result. I realized it wasn’t as straightforward as I initially thought.

Here’s how I navigated the process and ultimately succeeded:

Exploring Prompt Engineering with ChatGPT

I began using basic prompts, but they didn’t yield the desired results. This led me to experiment with Instruction-Following Prompting.

What is Instruction-Following Prompting?

In the context of large language models (LLMs), this technique involves giving clear, detailed instructions to guide the AI in producing the desired output while adhering to specific constraints.

In the context of AI and large language models (LLMs), this technique involves giving clear, detailed instructions to guide the AI in producing the desired output while adhering to specific constraints.

Here’s the approach I used:

1. Inputs: Provided two Excel files containing the class schedules.
2. Instructions: Included target courses, ensured minimum time between classes, etc.
3. Constraints: Specified time slots to avoid.
4. Desired Output: An Excel file with separate sheets for each schedule option.

Here’s how I applied the technique:

But for some reason, the prompt returned timeouts:

Determined to find a solution, I wrote my own Python scripts to handle the scheduling process. I started by diagramming the schedule creation process. 

Python Code:

However, as I ran the code, it took longer than expected.

The reason behind this is the find_schedules. The function is recursive and can potentially explore all combinations of courses. The time complexity can be approximated as O(K^T). If the number of target courses (k) increases, the number of potential schedules grows rapidly, leading to longer execution times.

P.S. If you are unfamiliar with the Big O, read it here.

Now, it could be the reason why ChatGPT was timing out. It was possibly running a code with  O(K^T) time complexity.

O(K^T) Workaround

I made the code work by tweaking the code and limiting the number of results.

Here’s the code:

AI theories

After some research, I found a better way to solve the issue. How? By using AI theories, specifically Constraint Satisfaction Problems (CSP).

What is CSP?

Imagine you are planning a small dinner party with three friends: Alice, Bob, and Carol. You need to seat them around a table, but there are a few rules:

1. Alice doesn’t want to sit next to Bob.
2. Bob wants to sit next to Carol.
3. Carol doesn’t want to sit at the head of the table.

In this example:

• The variables are the seats at the table.

• The possible values are Alice, Bob, and Carol.

• The constraints are the seating preferences of your friends.

A Constraint Satisfaction Problem (CSP) would help you find a seating arrangement that satisfies all these rules. For instance, you might end up with Alice, Carol, and Bob sitting in that order around the table, ensuring that all preferences are respected.

CSP requires three components:

1. Variables: In the context of a class scheduling problem, the variables are the classes that need to be scheduled. Each variable (class) is associated with specific attributes, such as course name, instructor, and number of students.

2. Domains: Each variable's domain represents the possible values that can be assigned to it. The domain includes time slots, days of the week, and available rooms for class scheduling. For example, the domain of a class variable could consist of all possible time slots and room combinations.

3. Constraints: Constraints are the rules that must be followed when assigning variable values. In the class scheduling problem, typical constraints include:

• Time Constraints: Ensuring that classes do not overlap for students or instructors.

• Room Constraints: Assigning classes to rooms that accommodate the expected number of students and have the necessary equipment.

• Instructor Constraints: Ensuring an instructor is not scheduled to teach two classes simultaneously and respecting their availability.

• Student Preferences: Accommodating preferences or requirements for specific time slots.

• Departmental Requirements: Meeting specific scheduling needs, such as required courses being available at particular times.

The goal of CSP is to find an assignment of values to the variables (class scheduling) that satisfies all the constraints (requirements for each class).

Note: Need more information on how the CSP algorithm works? You can view this Stanford lecture series.

How do you implement CSP? 

There’s actually a library built by Google to help you resolve the CSP problem. Google OR-Tools is specifically used to solve constraint satisfaction problems (CSPs) and other combinatorial optimization problems.

Implementing the solution

Question: Can I create a custom CSP (Constraint Satisfaction Problem) solution, or can I just use ChatGPT to handle it?

If latency isn’t a significant concern, the critical factor here is cost.

ChatGPT’s pricing is based on the number of tokens processed, both input and output. As of writing this, the GPT-4 model costs around $0.03 per 1,000 tokens for the standard version.

After running some cost estimates with ChatGPT, handling a single scheduling problem comes to about $0.075.

For a scheduling service like Calendly, which charges $10 per month, using ChatGPT to handle scheduling tasks would allow only about 133 requests per user each month ($10 / $0.075). This is in addition to their regular server costs. The company will lose money if a user exceeds or approaches 133 requests.

In conclusion, leveraging ChatGPT can simplify development and offer powerful AI capabilities, but the cost per request could become a limiting factor. Building a custom CSP solution might involve more upfront work but could provide a more cost-effective approach in the long run.

Hope you learned something! Happy coding!

P.S. The complete code here: https://github.com/iamademar/class_schedule_problem

Add this to your Gemfile:

Run bundle install and add an initializer on config/initializers/cors.rb

The next step is to add devise, devise-jwt and jsonapi-serializer gem:

Setup Devise

Do the typical devise installation  and create a devise User model then do the following modifications.

Before running the devise migration, you can add the following lines:

We will be using this for the revocation strategy. Next is to add the strategy to the user model and configure it to use the correct revocation strategy:

Setup devise-jwt

Set up a secret we will be using for JWT. On the terminal do this:

Save the key on an ENV variable. In my case, I'm saving the variable as DEVISE_JWT_SECRET_KEY.

Update config/initializers/devise.rb:

Setup jsonapi_serializer

Generate the serializer that we will be using later when we update the devise controllers so we return a json:

Update Devise controllers

Update the Registration controller to respond to JSON requests

Update the Session controller to respond to JSON requests

If you try out the API you will encounter this error:

ActionDispatch::Request::Session::DisabledSessionError (Your application has sessions disabled. To write to the session you must first configure a session store):

This is because the Rails application session store is not configured properly. 

You can configure it by add a `config/initiliazers/session_store.rb` file. Then add this line:

On the `config/application.rb`, add the following lines:

Testing with Postman

User Registration

User Login

Accessing weather endpoint with logged-in user

User logout and accessing weather endpoint

Hello there!

Do you have a startup idea or an exciting project you’re passionate about? I’d love to bring your vision to life!

I’m a software developer with 13 years of experience in building apps for startups, I specialize in Rails + Hotwire/React.

Whether you’re looking to innovate, grow your business, or bring a creative idea to the forefront, I’m here to provide tailored solutions that meet your unique needs.

Let’s collaborate to make something amazing!

Sincerely,
Ademar Tutor
hey@ademartutor.com

Let's work together!

Screenshot sourced from Behance.

Step 1: Rails Application Setup

Create new Rails API app
Run rails new weather_api_app --api to create a new Rails API application. The --api flag tells Rails to set up the application specifically for an API.

I'm assuming you already have Rails in your system.

Add Dependencies

Install the following gems to your Gemfile:

💎 httparty - Responsible for making HTTP requests to the OpenWeatherMap API.
💎 opentelemetry-sdk, opentelemetry-exporter-otlp, opentelemetry-instrumentation-all - OpenTelemetry gems for observability.  You can learn about how to set them up here.

Step 2: Weather Data Retrieval

Generate Controller

rails g controller Weather show

Update the WeatherController code

The routes should look like this:

You can now fetch data from the API using  a CURL request (or use Postman):

curl "http://localhost:3000/weather/show?city=Colombo"

That's the simplest way of requesting data from OpenWeatherMap API.

🚀  Refactoring the code to follow the Single Responsibility Principle

What we will do here is keep our controllers slim and focused only on handling web requests. Moving the API request logic into a service.

Here's how you do it:

1. Create a PORO for Fetching Weather Data

2. Update the WeatherController

Modify your WeatherController to use this new WeatherFetcher service:

With the OpenTelemetry and Jaeger setup, you can observe your application like the video below.

Step 3: Instrumentation with OpenTelemetry

What is instrumentation? Instrumentation is the act of adding observability code to the app yourself. The simple analogy is like adding `logger.info` but sending the data to OpenTelemetry and Jaeger instead of the console.

Global Tracer

The first thing we need to do is to acquire a Tracer from the `config/initializers/opentelemetry.rb`:

This "Tracer" will be globally available to any part of the app.

Adding a Span

We want to add a new span to break down the execution of a process into smaller, measurable units. In this case, we want to isolate the OpenWeatherMap request.

On the `app/services/weather_fetcher.rb`, let's add code to create a new span on the fetch method:

Adding an Event

On the private method `fetch`, let's add an event. To signify during the start and end of the OpenWeatherMap API request.

What's an event? 

A span event is a human-readable message on a span that represents “something happening” during it’s lifetime.

On the `app/services/weather_fetcher.rb`, let's update the fetch method:

The new weather_fetcher.rb with instrumentation should look like this:

You should be able to see the changes on the new span and event on Jaeger UI.

Hello there!

Do you have a startup idea or an exciting project you’re passionate about? I’d love to bring your vision to life!

I’m a software developer with 13 years of experience in building apps for startups, I specialize in Rails + Hotwire/React.

Whether you’re looking to innovate, grow your business, or bring a creative idea to the forefront, I’m here to provide tailored solutions that meet your unique needs.

Let’s collaborate to make something amazing!

Sincerely,
Ademar Tutor
hey@ademartutor.com

Let's work together!

However, in a microservices architecture, a single transaction or request might pass through many services hosted on different machines or across different data centers. This dispersion makes it hard to "observe."

Does it apply to a Rails monolith?

Yes, it does! 

Let's say you are working on an e-commerce application. You are fixing a production bug for the checkout workflow on this application.

The workflow includes the following actions:

• Charging credit cards via a 3rd party payment gateway (Stripe, Braintree and etc.)

• Sending a customer an email notification (Sendgrid, Twilio)

• Send updates to your order management system responsible for handling your inventory.

The bug only arises during peak hours. 

This would only mean that one or two of the processes took too long to respond and caused the entire workflow to fail. 

You do not monitor your Rails application's internal components, but you must also monitor the interactions with these external services.

This is where OpenTelemetry comes in.

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework that helps you gather metrics, traces, and logs.

Why do I need that? I already have New Relic, Skylight, and AppSignal.

OpenTelemetry is vendor-neutral. It doesn't lock you into a specific observability platform. You can switch to another vendor. Plus, OpenTelemetry is incubated by CNCF and is supported by industry leaders. This means it's here to stay!

Would OpenTelemetry replace New Relic, Skylight, and AppSignal?

OpenTelemetry specifies how data should be structured, collected, and sent over the network, but it doesn't provide a backend system to store, visualize, or analyze this data. 

You can send this data to New Relic or AppSignal to store, visualize, or analyze. 

But you can set up open-source solutions to view your data. This is where Jaeger comes in.

Jaeger maps the flow of requests and data as they traverse a distributed system. These requests may make calls to multiple services, which may introduce their own delays or errors.

https://www.jaegertracing.io/

How do you set up OpenTelemetry + Jaeger with a Ruby on Rails application?

These are the step-by-step process for integrating OpenTelemetry and Jaeger with your Ruby on Rails app.

1) Install the necessary gems

gem "opentelemetry-sdk"
gem "opentelemetry-exporter-otlp"
gem "opentelemetry-instrumentation-all"

2) Run Rails app and set the tracer exporter to console.

env OTEL_TRACES_EXPORTER=console rails server

Now, see if you can see the logs on the server. The logs should show something like this:

3) You can then set up a Jaeger backend to pass data. You can do that by running this image on your terminal:

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 14250:14250 \
  -p 9411:9411 \
  jaegertracing/all-in-one:latest

You will see the Jaeger on your docker desktop. You need to make sure the container is running.

Once it's running you should be able to access the Jaeger UI on your browser via http://localhost:16686/search

By default, traces are sent to an OTLP endpoint listening on localhost:4318. Your rails app should be set up automatically. However, if you need to update the endpoint, you can set it up manually by adding an ENV variable.

env OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" rails server

You should be able to see the request on Jaeger UI:

Hello there!

Do you have a startup idea or an exciting project you’re passionate about? I’d love to bring your vision to life!

I’m a software developer with 13 years of experience in building apps for startups, I specialize in Rails + Hotwire/React.

Whether you’re looking to innovate, grow your business, or bring a creative idea to the forefront, I’m here to provide tailored solutions that meet your unique needs.

Let’s collaborate to make something amazing!

Sincerely,
Ademar Tutor
hey@ademartutor.com

Let's work together!

I’m working on a super secret app to revolutionize service field businesses 😜. Yesterday, the codebase had the opportunity to be viewed by eyes from the outside world for the first time and got feedback.

One of the things that stood out was the code related to converting Requests to Quotes. This workflow:

This particular code:

This involved the callback after_create!

I know, I know. Callbacks are evil!

Well, my lame excuse at the time of development is I could not find the correct language for the conversion process when developing the ubiquitous language.

Based on the stakeholders, whenever they need the quote to be produced, they just say to the employee: “Palihug ko pa.quote ani dong/dai. “

This can be loosely translated as: “Hey [name], can you please create a quote for this customer’s request?”

At that time of development, I could not find the right words for it. Based on what I was hearing from the stakeholders, they just said to create a quote.

So, I opted for the easiest solution available, callbacks!

Why are callbacks bad?

It is probably not that concerning in this particular case since the workflow is pretty simple.

However, I’ve been burned by this before! When I was working on an e-commerce application, we used a after_save callback that sent app notifications to listing owners when their listings were updated. When we ran a script that was intended to update silently thousands of listings on a Sunday evening. Long story short, it was not a very silent Sunday 😅.

So, how do we move away from callbacks?

Modeling/developing/writing a better ubiquitous language for the domain you are working on helps you write better code.

In my case, I could not find the right technical words while conversing with the stakeholders.

However, now that I started developing the user interface, it seems it’s pretty straightforward:

So, what does the change look like in the code?

Step 1: Create a PORO to handle the business logic

Things to note on the code above:

• I created a PORO (Plain Old Ruby Object) for the Request to Quote Conversion business logic.

• Later, if we want to send email or mobile notifications to admins or employees, we should update this class.

• If anything fails, this class should handle the failure gracefully, especially if it involves payments, etc.

• Now that it’s a class, creating unit tests for this specific business logic would be easier, ensuring that this process is more robust when change requests come in later.

Step 2: Update controllers

You can then use the PORO on the controller like this:

Optional Steps:

You can actually do it a step further and follow DHH’s approach, creating a method for the request model:

Then, you can easily create a call this method on the controller like so:

Now, the code is more readable and safer in the future!

Question: Are callbacks evil?

There are two camps for this, people who are for and against callbacks.

So which camp do I align?

My take is:
As developers, we are constantly understanding the domains that we are working on and developing a unique ubiquitous language for them.

Initially, we may not have the right words for it, thus we need to use convenient “generic” tools such as callbacks. But as soon as you learn more about the domain, and build a better ubiquitous language, the design of your code should follow also.

Hello there!

Do you have a startup idea or an exciting project you’re passionate about? I’d love to bring your vision to life!

I’m a software developer with 13 years of experience in building apps for startups, I specialize in Rails + Hotwire/React.

Whether you’re looking to innovate, grow your business, or bring a creative idea to the forefront, I’m here to provide tailored solutions that meet your unique needs.

Let’s collaborate to make something amazing!

Sincerely,
Ademar Tutor
hey@ademartutor.com

Let's work together!

Objective

Our objective in this blog is to create a chatbot that is an expert in Amazon’s Help & Customer Service. The chatbot should be able to answer queries from Amazon’s Customer Service Page. Hopefully, this app will help Amazon save a couple of million dollars in Customer Support spending 😅.

On this blog, our limitation is that we will only build the back end of the chatbot. We’ll do UI and other improvements later.

Tech Stack

• PostgreSQL - We’ll use PostgreSQL with a vector search extension. Why use PostgreSQL? PostgreSQL is mature and well-established, which means it has a wide range of features, strong community support, and extensive documentation.

• pgvector - What is pgvector? pgvector is a PostgreSQL extension for vector similarity search.

• langchainrb_rails - Think of it as searchkick for elasticsearch, but this gem is specifically for AI.

• langchainrb - Think of it as a framework for integrating AI with Ruby applications.

• neighbor - This gem is responsible with vector search in PostgreSQL.

• OpenAI - Large Language Model that we will use.

Workflow

This is the workflow of how we’ll get the embedding for the Policy model.

This is the workflow of how we’ll ask a question.

Implementation

Step 1

Let’s create a new rails app.

Step 2

Let’s create a Policy model. This is where we store Amazon’s help & customer service.

Step 3

We then update seeds.rb to save the policy pages of Amazon on the Policy model. The seeds.rb should look like this:

Step 4

The next step is to install langchainrb_rails

Step 5

After install langchainrb_rails , the next step is to run the rails generator to add vectorsearch to the Policy model.

Step 6

The next step is to generate embeddings of the Policy model. In the console, do this:

Step 7

After embedding, the next step is to set up a connection to OpenAI's GPT-4 model using the Langchain library, configure a vector search mechanism, and then use this setup to process and respond to a question.

Step 8

Now, it's time to test the chatbot in the terminal.

Hello there!

Do you have a startup idea or an exciting project you’re passionate about? I’d love to bring your vision to life!

I’m a software developer with 13 years of experience in building apps for startups, I specialize in Rails + Hotwire/React.

Whether you’re looking to innovate, grow your business, or bring a creative idea to the forefront, I’m here to provide tailored solutions that meet your unique needs.

Let’s collaborate to make something amazing!

Sincerely,
Ademar Tutor
hey@ademartutor.com

Let's work together!

Previously, we've built a basic terminal chatbot with Ruby. Now, we are going to focus on PrompTemplates.

What are PromptTemplates? 

PromptTemplates in Langchain.rb are game-changers. They allow you to create flexible, dynamic prompts with placeholders, which can be filled in with user input or other data at runtime.

This adaptability is critical in prompt engineering, where the goal is to generate prompts that elicit the most accurate and relevant responses from an AI model.

What is Prompt Engineering?

Prompt engineering is the art of crafting prompts that guide AI models to provide the desired output. The way you phrase a prompt can significantly impact the model's response.

PromptTemplates offer a way to experiment with different phrasings and structures without hard-coding every possible prompt.

Building an example chatbot

Imagine building a chatbot that explains to preschools what specific profession and skills they need to succeed in that profession.

Step 1

Require the necessary libraries: OpenAI and Langchain.

Step 2

Set the logger level of Langchain to info, which means it will log informational messages. It is not a required step, but it's always good to know what's happening in the background with Langchain.rb

Step 3

Initialize a new Langchain::LLM::OpenAI object. This object is used to interact with OpenAI's API.

You need to provide your API key and other options.

llm_options is an empty hash here, but it can be used to pass additional options.

default_options sets the default language model to GPT-4 for chat completions.

Step 4

Create a new conversation object. This object will manage the conversation flow using the OpenAI model.

Step 5

Create a new PromptTemplate object. This template has a placeholder `{profession}` that can be filled in. `input_variables` defines the placeholders expected in the template.

Step 6

Prompt the user for the profession.

Step 7

Format the prompt template with the user's input and send the message to the OpenAI model. The response from the model is then printed out.

Full Code

Hello there!

Do you have a startup idea or an exciting project you’re passionate about? I’d love to bring your vision to life!

I’m a software developer with 13 years of experience in building apps for startups, I specialize in Rails + Hotwire/React.

Whether you’re looking to innovate, grow your business, or bring a creative idea to the forefront, I’m here to provide tailored solutions that meet your unique needs.

Let’s collaborate to make something amazing!

Sincerely,
Ademar Tutor
hey@ademartutor.com

Let's work together!

In this tutorial, we'll delve into creating your first AI-powered apps using Ruby, Langchain.rb library and OpenAI's GPTPT-4 API. 

Objective

We'll create a terminal-based chatbot that allows you to OpenAI'S GPT-4. This chatbot will be able to process user inputs and generate relevant, context-aware responses.

1. First, we'll need to create a ruby file. Let's call this chat.rb

2. What we want to do next is import the necessary libraries

• OpenAI: The openai library is a Ruby client for interacting with OpenAI's API.

• Langchain: Langchain is a Ruby library designed for building language applications. 

• reline: Reline is a Ruby library used for readline-compatible input in command-line applications.

3. Create an instance of `Langchain::LLM::OpenAI`. This class is a wrapper around OpenAI's API, specifically tailored for language models (LLMs). 

Side Note: The great thing with Langchain.rb is it supports other models, too, offering flexibility to switch between different AI models based on requirements.

4. Chat Interaction

The method `prompt_for_message`, handles multiline user inputs and implements commands for ending the input or exiting the program. The commands are: "done", "end", "eof", and "exit"

5. Interactive Loop

The code continuously fetches user messages and uses `chat.message` to obtain and display responses from the AI, creating a dynamic conversation flow.

Full code of chatbot.rb

Hello there!

Do you have a startup idea or an exciting project you’re passionate about? I’d love to bring your vision to life!

I’m a software developer with 13 years of experience in building apps for startups, I specialize in Rails + Hotwire/React.

Whether you’re looking to innovate, grow your business, or bring a creative idea to the forefront, I’m here to provide tailored solutions that meet your unique needs.

Let’s collaborate to make something amazing!

Sincerely,
Ademar Tutor
hey@ademartutor.com

Let's work together!

In college, Big-O Notations was taught in one of our courses. I really could not connect the dots back then. Big-O notations felt too theoretical at that time.

So, as a professional developer, I relied upon performance tools like rack-mini-profiler, Benchmark, bullet, etc.

When 3rd party monitoring apps like AppSignal and New Relic started becoming good at monitoring performance, I began creating rules when writing code to avoid performance issues later when my code hits production.

These are some of my "performance" rules:

1. Be Mindful with Loops

Rule: Reduce database hits inside loops.

# Bad
User.all.each do |user|
  user.update_attribute(:status, "active")
end

# Good
User.update_all(status: "active")

2. Optimize Database Queries

Rule: Avoid N+1 queries by eager loading associated records.

# Bad
User.all.each do |user|
  puts user.profile.name
end

# Good
User.includes(:profile).each do |user|
  puts user.profile.name
end

Note: Although, if you use the Russian Doll Caching strategy, this might not apply.

3. Background Jobs for Time-Consuming Tasks

Rule: If all else fails, move long-running tasks to background jobs. Pray that the user does not need the data ASAP 🙏

# Using Sidekiq or similar
LongTaskWorker.perform_async(user_id)

I have realized you can get by with your app's performance by learning those simple rules.

However, having a solid foundation in your bag that has been used since 1894 might be a good idea.

What is Big-O notation?

In plain words, Big-O notation describes the performance of your code using algebraic terms. Performance in this context refers to either the speed of execution or the amount of RAM the code consumes.

In computer science, these concepts are more formally known as 'time complexity' for speed and 'space complexity' for RAM usage. Big-O notation is used for both, but we're going to focus on speed here to simplify our discussion.

Big-O in real-life

In a web application, handling a process that involves sorting or searching through 100 data entries will naturally be slower compared to dealing with just 10 entries.

The key question is: how much slower is it? Does the time taken increase 10x, a 100x, or perhaps even a 1000x?

This variance in processing speed might not be evident when dealing with smaller data sets.

However, the situation changes if your application's performance degrades significantly with the addition of each new entry in the database.

Such a decline in speed becomes a major concern, particularly as the volume of data grows larger in the app.

Understanding Big O notation is crucial in this context. It helps developers predict how their code will scale as data volumes increase, allowing them to choose the most efficient algorithms and data structures.

By anticipating performance issues before they occur, developers can optimize their applications for speed and efficiency, ensuring a smoother user experience even as data grows. This foresight is invaluable in maintaining a high-performing web application over time.

Here's a super accurate emoji-based representation of Big-O notations to give you an idea of how fast they are as your data scales:

• 🚀 O(1) - Speed does not depend on the scale of the dataset

• 🚄 O(log n) - If you have ten times more data, it only takes twice as long to process.

• 🚗 O(n) - If the amount of data increases tenfold, the processing time also increases tenfold.

• 🚲 O(n log n) - If you increase the data by ten times, the processing time will roughly increase by twenty times.

• 🏃‍♂️ O(n^2) - If the amount of data is multiplied by ten, the processing time increases by a factor of one hundred. 

• 🚶‍♂️ O(2^n)- Doubling the amount of data leads to an exponential increase in processing time, making it significantly longer.

Big-O Notations In-depth

🚀 O(1)

O(1) indicates that the execution time of an operation remains constant, regardless of the size of the data set.

Example of accessing an element in a hash; the time taken is independent of the hash's size:

# Access time remains 
# constant for these

hash_with_100_items[:a]
hash_with_1000_items[:a]
hash_with_10000_items[:a]

As a result, hashes generally perform better than arrays for large datasets. Here's another Ruby example:

# Retrieving an item 
# from arrays of varying 
# sizes

array_with_100_items[50]
array_with_1000_items[50]
array_with_10000_items[50]

In this case, fetching an element at a specific index (like the 50th position) takes the same amount of time, regardless of the array's size.

🚄 O(log n)

O(log n) describes an algorithm where the time it takes to complete a task increases logarithmically in relation to the size of the dataset.

This means that each time the dataset doubles in size, the number of steps required increases by a relatively small, fixed amount.

O(log n) operations are efficient and much faster than linear time operations for large datasets.

Here is a Ruby example illustrating O(log n) complexity:

Binary Search:

A binary search algorithm in a sorted array is a classic example of O(log n) complexity. At each step, it divides the dataset in half, significantly reducing the number of elements to search through.

Here’s an example of a Ruby code of binary search. We compare the time difference with benchmark when we double the dataset.

Code:

require 'benchmark'

def binary_search(array, key)
  low = 0
  high = array.length - 1

  # This loop runs as long 
  # as the 'low' index is 
  # less than or equal to  
  # the 'high' index
  while low <= high
    # Find the middle 
    # index of the 
    # current range
    mid = (low + high) / 2
    value = array[mid]

    # If the value is less 
    # than the key, narrow 
    # the search to the 
    # upper half of the 
    # array
    if value < key
      low = mid + 1
    # If the value is 
    # greater than the 
    # key, narrow the 
    # search to the 
    # lower half of 
    # the array
    elsif value > key
      high = mid - 1
    # If the key is found, 
    # return its index
    else
      return mid
    end
  end

  # If the key is not 
  # found in the 
  # array, return nil
  nil
end

# Example usage with an 
# array of 100 million 
# elements
array = (1..100000000).to_a
puts Benchmark.measure {
  binary_search(array, 30)
}

# Example usage with an 
# array of 200 million 
# elements
array = (1..200000000).to_a
puts Benchmark.measure {
  binary_search(array, 30)
}

# Example usage with an 
# array of 300 million 
# elements
array = (1..300000000).to_a
puts Benchmark.measure {
  binary_search(array, 30)
}

In this code, the binary search algorithm has a time complexity of O(log n) because it divides the search interval in half with each step.

In each iteration of the while loop, it either finds the element or halves the number of elements to check.

This halving process continues until the element is found or the interval is empty.

As a result, the time to complete the search increases logarithmically with the size of the array (n), making binary search efficient for large datasets.

Result:

🚗 O(n)

O(n) describes the time complexity of an algorithm where the time to complete a task grows linearly and in direct proportion to the size of the input data set. This means that if you double the number of elements in your data set, the algorithm will take twice as long to complete.

Here is a Ruby example illustrating O(n) complexity:

Summing all elements in an array:

In this example, the time to sum all elements increases linearly with the number of elements in the array.

Code:

require 'benchmark'

def sum_array(array)
  sum = 0
  # Iterating over each 
  # element in the array
  array.each do |element|
    # Adding each element 
    # to the sum
    sum += element  
  end
  # Returning the 
  # total sum
  sum  
end

# Summing an array of 
# 100 million elements
array = (1..100000000).to_a
puts Benchmark.measure {
  # The time taken here is 
  # proportional to the 
  # array's size
  sum_array(array)  
}

# Summing an array of 
# 200 million elements
array = (1..200000000).to_a
puts Benchmark.measure {
  # Time taken will be roughly 
  # twice that of the 100 
  # million array
  sum_array(array)  
}

# Summing an array of 
# 300 million elements
array = (1..300000000).to_a
puts Benchmark.measure {
  # Time taken will be roughly 
  # three times that of the 
  # 100 million array
  sum_array(array)  
}

In this code, the sum_array function has an O(n) complexity because it involves a single loop that iterates over each element of the array exactly once.

The time taken to execute this function scales linearly with the number of elements in the array (n).

Therefore, if you double the size of the array, the time to sum the array also doubles, which is characteristic of linear time complexity.

Result:

🚲 O(n log n)

O(n log n) describes the time complexity of an algorithm where the time to complete a task increases at a rate proportional to the number of elements in the data set (n) multiplied by the logarithm of the number of elements (log n).

Here is a Ruby example illustrating O(n log n) complexity:

Merge Sort:

Merge Sort is a classic example of an algorithm with O(n log n) complexity. It divides the array into halves, sorts each half, and then merges them back together.

Code:

require 'benchmark'

def merge_sort(array)
  # Base case: a single 
  # element is already 
  # sorted
  if array.length <= 1
    array
  else
    # Find the middle 
    # of the array
    mid = array.length / 2

    # Recursively sort 
    # the left half
    left = merge_sort(array[0...mid])
    # Recursively sort 
    # the right half
    right = merge_sort(array[mid..-1])

    # Merge the two 
    # sorted halves
    merge(left, right)
  end
end

def merge(left, right)
  sorted = []
  # Until one of the arrays 
  # is empty,pick the smaller 
  # element from the front 
  # of each array
  until left.empty? || right.empty?
    if left.first <= right.first
      sorted << left.shift
    else
      sorted << right.shift
    end
  end

  # Concatenate the remaining 
  # elements (one of the arrays 
  # may have elements left)
  sorted.concat(left).concat(right)
end


# Benchmarking with 
# 10 million elements
array = (1..10000000).to_a
puts Benchmark.measure {
  merge_sort(array)
}

# Benchmarking with 
# 20 million elements
array = (1..20000000).to_a
puts Benchmark.measure {
  merge_sort(array)
}

# Benchmarking with 
# 30 million elements
array = (1..30000000).to_a
puts Benchmark.measure {
  merge_sort(array)
}

The merge_sort function exhibits O(n log n) complexity because it involves a divide-and-conquer approach.

The array is repeatedly divided into halves (resulting in a logarithmic number of divisions, i.e., log n), and each half is sorted and then merged.

The merging process for each level is linear in the size of the input (i.e., n), leading to a total time complexity of O(n log n).

This makes merge sort much more efficient for large datasets compared to algorithms with quadratic time complexities (like bubble sort or insertion sort), especially as the size of the data increases.

Result:

🏃‍♂️ O(n^2)

O(n^2) describes the time complexity of an algorithm where the time to complete a task is proportional to the square of the number of elements in the input dataset. This means that if the number of elements doubles, the time to complete the task increases by four times. Algorithms with O(n^2) complexity are generally less efficient for large datasets compared to O(n log n) or O(n) algorithms.

Here is a Ruby example illustrating O(n^2) complexity.

Bubble Sort:

Bubble Sort is a simple sorting algorithm that repeatedly steps through the list, compares adjacent elements, and swaps them if they are in the wrong order. The pass through the list is repeated until the list is sorted.

Code:

require 'benchmark'

def bubble_sort(array)
  n = array.length
  # Loop until no more swaps are needed
  loop do
    swapped = false
    # Iterate through the array. The '-1' is because Ruby arrays are zero-indexed,
    # and we're comparing each element with the next.
    (n-1).times do |i|
      # Compare adjacent elements
      if array[i] > array[i+1]
        # Swap if they are in the wrong order
        array[i], array[i+1] = array[i+1], array[i]
        # Indicate a swap occurred
        swapped = true
      end
    end
    # Break the loop if no swaps occurred in the last pass
    break unless swapped
  end
  # Return the sorted array
  array
end

# Benchmarking with 10,000 elements
array = (1..10000).to_a.shuffle
puts Benchmark.measure {
  bubble_sort(array)
}

# Benchmarking with 20,000 elements
array = (1..20000).to_a.shuffle
puts Benchmark.measure {
  bubble_sort(array)
}

# Benchmarking with 30,000 elements
array = (1..30000).to_a.shuffle
puts Benchmark.measure {
  bubble_sort(array)
}

In this bubble_sort method, the O(n^2) complexity arises from the nested loop structure:

a) The loop construct potentially iterates n times (in the worst case where the array is in reverse order).

b) Inside this loop, there's a (n-1).times loop that also iterates up to n-1 times for each outer loop iteration.

This nesting of loops, where each loop can run 'n' times, leads to the time complexity of O(n^2).

Result:

🚶‍♂️ O(2^n)

O(2^n) describes the time complexity of an algorithm where the time to complete a task doubles with each additional element in the input data set. This kind of complexity is typical in algorithms that involve recursive solutions to problems where the solution involves creating two or more subproblems for each problem. Such algorithms are often seen as inefficient for large datasets due to their exponential time growth.

Here is a Ruby example illustrating O(2^n) complexity.

Fibonacci Sequence (Recursive Implementation):

A classic example of O(2^n) complexity is the recursive calculation of the Fibonacci sequence, where each number is the sum of the two preceding ones.

Code:

require 'benchmark'

def fibonacci(n)
  return n if n <= 1
  fibonacci(n - 1) + fibonacci(n - 2)
end

puts Benchmark.measure {
  fibonacci(10)
}

puts Benchmark.measure {
  fibonacci(20)
}

puts Benchmark.measure {
  fibonacci(30)
}

The number of operations grows exponentially with the input size, characteristic of O(2^n) time complexity. This exponential growth makes these algorithms impractical for large input sizes due to the rapid increase in computation time.

Result:

Summary

You should not be using these code examples in production apps. There are built-in Ruby sort and search. And if you are sorting records (in PostgreSQL), please use the database feature.

I hope you can appreciate Big-O notation and apply this to your daily work. 🚀

Hello there!

Do you have a startup idea or an exciting project you’re passionate about? I’d love to bring your vision to life!

I’m a software developer with 13 years of experience in building apps for startups, I specialize in Rails + Hotwire/React.

Whether you’re looking to innovate, grow your business, or bring a creative idea to the forefront, I’m here to provide tailored solutions that meet your unique needs.

Let’s collaborate to make something amazing!

Sincerely,
Ademar Tutor
hey@ademartutor.com

Let's work together!