Don’t Use AI to Run Repeatable Tasks. Use AI to Build Them.

One thing I am learning while building AI workflows:

If a task is repeatable and the steps are known, don’t ask an AI agent to perform the entire task using natural language every time. Use AI to generate the code once, then run that code repeatedly.

This saves money, runs faster, is easier to test, and produces more predictable results.

I am not saying we should avoid AI. I am saying we should use it only where intelligence is actually needed.

A Simple Slack Digest Example

Assume we want to create a daily digest from several Slack channels.

The workflow is straightforward:

  1. Fetch messages from the last 24 hours.
  2. Find threads with replies.
  3. Fetch all messages in each thread.
  4. Remove bot messages and duplicates.
  5. Summarize each useful thread.
  6. Post the digest to another channel.

We could give this entire instruction to a powerful AI model:

Go through these Slack channels, collect all messages and threads from the last 24 hours, summarize them, and post the digest here.

It may work well. But why do we need an expensive LLM to fetch channels, calculate timestamps, handle pagination, open threads, remove duplicates, and format the result?

These are normal programming tasks.

Instead, use AI to generate a Python service that handles the predictable steps through Slack APIs. Send only the cleaned thread content to an LLM for summarization. Even for that step, a smaller and cheaper model may be enough.

Use the powerful model to build the workflow, not to run every step of it every day.

Why It Matters

Cost

In a fully agentic workflow, the model repeatedly consumes:

  • Instructions and tool definitions
  • Channel and thread metadata
  • API responses
  • Raw messages
  • Intermediate tool-call results
  • Final summaries

Most of those tokens are used to manage the workflow, not to understand the content.

It may look cheap for one agent and one run. Multiply it across many channels, teams, agents, and daily executions, and the industry may be wasting millions or even billions of tokens on work that normal code could handle.

Code can fetch and clean the data first. The LLM should receive only what it needs to understand.

Speed

An agent may fetch a channel, inspect the response, decide what to do next, call another tool, and repeat.

Code can fetch multiple channels and threads concurrently without waiting for an LLM decision between every API call.

Software should do the software work.

Predictability

Natural-language instructions leave room for interpretation.

What is an “important” thread? Should bot messages be included? What happens if the job runs twice? What if one API call fails?

In code, we can define these rules clearly:

  • Use exact start and end timestamps.
  • Ignore known bot users.
  • Deduplicate using message IDs.
  • Retry failed requests.
  • Do not post the same digest twice.
  • Apply clear limits for channels, messages, and threads.

The summaries may still vary slightly, but the workflow around them becomes predictable.

Unit Tests Instead of Evals Everywhere

If the entire workflow is controlled by natural language, we need AI evals to check whether the agent followed every instruction.

That becomes costly quickly. We start writing evals for pagination, timestamps, filtering, duplicate handling, retries, permissions, and many other combinations.

If those steps are code, normal unit tests are enough:

  • Did we calculate the correct 24-hour window?
  • Did we fetch all thread replies?
  • Did we handle pagination?
  • Did we filter bot messages?
  • Did we avoid duplicate posts?

Unit tests are fast, cheap, and deterministic.

We still need evals for the actual AI work, such as summary quality, relevance, and factual accuracy. But we don’t need an LLM eval to test whether a timestamp filter works.

Use unit tests for code. Use evals for the intelligence layer.

Where Should We Use AI?

Use code when the steps and rules are known.

Use an LLM when the task requires language understanding, summarization, classification, or judgment.

For rare and difficult cases, route the request to a stronger model. The common path can run using code and a smaller model.

This boundary is important. Otherwise, we may end up using the most expensive model as a scheduler, API client, loop, and formatter.

Monthly AI Budgets Will Force Better Design

I think this is the right time for companies across the industry to get serious about monthly AI budgets.

For example, if a team must build and operate its AI workflows within $2,000 per month, it will naturally ask:

  • Can this step be code?
  • Why are we sending this data to the model?
  • Can we filter or reduce it first?
  • Can we cache the result?
  • Can a smaller model handle this?
  • Can we use the expensive model only for difficult cases?
  • Can one shared service replace several similar agents?

A budget is not only a cost-control mechanism. It forces teams to make better engineering decisions.

Many useful innovations come from constraints. If there is no limit, the easiest solution is to send everything to a powerful model and let it figure things out. With a clear budget, teams start designing for efficiency.

The Principle

My principle is simple:

Don’t use AI to run repeatable automation. Use AI to build the automation, and keep AI in the workflow only where intelligence is required.

The first approach is easier for a demo.

The second approach is what we can afford, test, debug, and scale.