Automating the Work Nobody Wants to Do

Every night, a job on a site I run checks the current price of roughly a thousand products. It takes about fourteen minutes. Nobody was ever going to do that by hand, which is exactly the point. The work was not hard. It was boring and endless, and boring and endless is the profile of a task that quietly never gets done.

TL;DR: The tasks worth automating are rarely the interesting ones. What decides whether an automated job is still useful a year later is not the code - it is three design choices: store a summary instead of everything, record a result even when there was nothing to find, and count what you could not answer separately from what failed. Then write down every trade-off you knowingly made, or someone will reverse it without knowing they did.

What Counts as Worth Automating?

A good target repeats on a schedule, does not change much between runs, and would make a human slow, inconsistent, and resentful.

That last part matters more than people expect. Tedious manual work does not get done badly - it gets skipped, then rushed before a deadline, then skipped again. The automation is not replacing a careful process. It is replacing a gap. The work worth automating is the work nobody misses.

Store a Summary, Not Everything

The first decision was what to keep. Each product had roughly fifty listings behind it every night. Keeping all of them would have been around fifteen million records a year, growing forever, to answer a question nobody had asked. Instead the job stores three numbers per product per night: the cheapest, the typical, and the highest. That answers the real question - what did this cost, and is it moving - in a fraction of the space.

One subtlety goes well beyond pricing. The cheapest listing is almost always somebody advertising a bad-faith number they never intend to honour, so tracking the minimum means tracking the least trustworthy participant in the market. Any time you reduce a pile of data to one figure, ask which figure describes reality and which describes an outlier.

Record a Result Even When There Was Nothing to Find

The second decision was to write a record on nights when the job found nothing at all.

That sounds like clutter. It is the most valuable line in the design. Without it, "we checked and there was nothing there" and "the job never ran" look identical - both are an absence. Months later, staring at a gap, you cannot tell whether the world was quiet or your automation was dead.

This is the one people skip and regret. An automated process should leave evidence that it ran, especially when it had nothing to say, because silence looks exactly like success until the day it does not.

Count What You Could Not Answer, Separately From What Failed

Of the thousand products checked on the first full run, 437 came back with no matching listings at all. Nothing broke. Almost all of them were older hardware nobody sells any more, and the honest answer to "what does this cost today" was "nothing is for sale."

Had those been logged as failures, the run would have reported a 44 percent failure rate and I would have spent an evening chasing a bug that did not exist. Worse, a real failure sitting in that pile would have been invisible. So the job counts three things separately: answered, could not answer, and broke.

Most reporting collapses "no result" into "error" because it is easier. It costs you the ability to notice the thing that matters.

The Failure That Produced No Error Anywhere

Here is the one that stung. The routing layer had an address the automation needed configured with a trailing slash, so a closely related address it also called did not match the rule. Instead of failing, that request fell through to the website and came back with a perfectly valid page and a success code.

No error was logged. Nothing alerted. Every dashboard stayed green. The feature did nothing, successfully, and would have kept doing nothing indefinitely.

What caught it was a test that derives the list of addresses from the code itself, then asserts the routing configuration mentions every one. It did not need to know the right answer - only that the two halves agreed.

The lesson is worth more than the bug: a successful response is not the same as a correct one. Any check that only asks "did we get an answer" will pass forever on the wrong answer.

Write Down the Trade-Off, or Someone Will Reverse It

The data source's terms cap how long the information may be displayed. Honouring that exactly would have left the feature blank most of the day, which makes it worthless to whoever is looking at it.

So I made a call: widen the window knowingly, write down in plain language that it departs from the terms, and record the compliant alternative right beside it, in three separate places, so anyone who finds one finds the reasoning too.

An undocumented trade-off looks like a bug to the next person who reads it. They will helpfully fix it, quietly undo a deliberate decision, and nobody will know until something breaks.

What to Ask About Your Own Operation

What does someone here do every week that never changes? Not the judgement calls - the mechanical repetition. That is your list.

Could we tell the difference between "nothing happened" and "it stopped running"? If the answer is no, you do not have automation yet. You have a script and some optimism.

Cutting repetitive work comes from the same instinct as choosing tools that do not bill you per seat: spend money and attention on the parts of the business that need a human.

If you have a weekly task that never changes and never quite gets done, let's talk.