The Dependency Nobody Owned. Dead Nine Years And Still Callable.

Nobody notices a report that arrives. That is the whole trouble with it.

The call comes on the fourth working day of the month, from someone in finance who waited three days before being a bother about it. The monthly reconciliation pack did not land, and it has landed on the second working day of every month for as long as she has had the job.

Four hours later you have it. A scheduled task on a utility server, calling a Python script from a home directory belonging to an account disabled fourteen months ago when its owner left. It queried two databases, built a workbook, emailed it, and kept doing that for a year after everyone forgot it existed. It stopped because a credential it carried inline expired.

I have been handed some version of that server four times now. The script was never the defect — the script worked. The defect was that it had no owner, and that is a technical condition with predictable failure modes, not a filing problem.

The comfortable theory

We have a CMDB. Discovery runs nightly. If it is on the network it is in the inventory, so there is nothing here we do not know about.

All of that was true above. The host was inventoried, patched and monitored, and CPU, memory, disk and uptime stayed green straight through the outage, because by every measure the inventory collected, nothing was wrong.

An inventory of hosts tells you what you own. It does not tell you what depends on it, and only one of those questions wakes you up.

The sentence GitLab wrote down

On 31 January 2017 a GitLab engineer working late on replication problems removed a PostgreSQL data directory on the primary database instead of the secondary. "Of around 300 GB only about 4.5 GB is left." That part is a bad night, and bad nights happen.

What came next is the part worth re-reading. GitLab had five backup and replication mechanisms and, in their own words, "none are working reliably or set up in the first place." The regular backups ran, but "the files were only a few bytes in size." The S3 bucket was empty. And the SQL dumps — the one most teams would name first if asked which backup they trusted — were failing on a version mismatch: "the backup procedure was using pg_dump 9.2, while our database is running PostgreSQL 9.6."

That last job was not failing silently. It reported its failure, every night, by email. From the postmortem: "Unfortunately DMARC was not enabled for the cronjob emails, resulting in them being rejected by the receiver. This means we were never aware of the backups failing, until it was too late."

Then GitLab wrote the sentence I have quoted in meetings ever since: "Because there was no ownership, as a result nobody was responsible for testing this procedure."

The job was not silent. It was shouting into a channel with no listener, which is a different failure that looks identical from your desk.

They recovered from a staging copy roughly six hours old, losing an estimated "at least 5000 projects, 5000 comments, and roughly 700 users." On the follow-up list: "Assign an owner for data durability."

Dead nine years and still callable

The second shape is worse, because the thing is not even running.

The SEC's order against Knight Capital Americas describes order-routing code called Power Peg. "In 2003, Knight ceased using the Power Peg functionality." It was not removed: "Despite the lack of use, the Power Peg functionality remained present and callable." In 2005 Knight moved the function tracking cumulative shares to an earlier point in the code sequence and "did not retest the Power Peg code after moving the cumulative quantity function." Nine-year-old dead code had quietly lost its stop condition, and nobody knew, because nobody is assigned to code nobody uses.

On 1 August 2012 a technician deployed new code to seven of the eight servers running that system; "Knight did not have a second technician review this deployment." The eighth server, still holding the old code, turned 212 incoming parent orders into "4 million executions in 154 stocks for more than 397 million shares in approximately 45 minutes." Knight "realized a $460 million loss."

Hold it at arm's length — a settled order, consented to without admitting or denying the findings. The sequence still says something plain. Unused is not the same as absent, and an unowned component does not get retested when something next to it changes, because nobody's job is to ask.

The one column a scanner cannot fill

Both standards you are probably already measured against ask for an owner.

Control CM-8 in NIST SP 800-53 wants an inventory carrying what is needed "to achieve system component accountability," and its discussion is specific: that information "includes the system name, software owners, software version numbers." The enhancement CM-8(4) goes further and asks for a means of identifying the individuals responsible for administering each component — but it is an enhancement, selected at the high baseline. The owner's name is in the catalog as an option.

CIS Control 1 is blunter, on the rationale that "enterprises cannot defend what they do not know they have." Safeguard 1.1 asks the inventory to record network address, hardware address, machine name, asset owner, department, and whether the asset is approved on the network.

Look at the asymmetry in that list. Address, hardware address, machine name — a discovery tool fills those nightly for free, and CIS names the kinds to use for it (1.3 active discovery, 1.4 DHCP logging, 1.5 passive discovery). Owner and department are the only two no scanner can determine, so those are the two that stay blank. The inventory grows more accurate every night and no more useful, because accuracy was never the missing part.

Why it ends up this way

Inventories were built to answer audit questions — how many do we have, are they licensed, are they patched. "What breaks if this stops" was never on the form, and the people who buy the tool are not the people who get the call.

Ownership also has to be created by naming someone, and naming someone attaches a phone number to a 2 a.m. failure. Nobody refuses that out loud; it just never reaches the top of a list. Meanwhile the script keeps working, which removes the only pressure that would force the conversation.

What I would actually do

Inventory outputs, not hosts. List what the business consumes: every recurring report, file drop, data feed, dashboard, invoice batch, nightly extract. For each, write down what produces it and who notices within a day if it stops. Working backwards from outputs finds orphans; working forwards from the server list never has.

Make the leaver's checklist a discovery tool. Offboarding already hunts for a departing person's accounts. Extend it to their scheduled tasks and crontabs, anything running as their credentials, their name in a connection string, their tokens, mailbox rules feeding a process — while they still answer the phone. Then do the same for every non-human account that authenticated this month.

Alert on silence. GitLab's job told the truth every night to a receiver that threw it away. Anything that matters should emit a heartbeat when it succeeds, with the alert wired to that heartbeat's absence. "It told me nothing was wrong" and "it told me nothing" must not look the same on your screen.

Review what you inherited as though you are about to carry it. That is the question at the heart of a production readiness review at Google: "If I were on-call for this service right now, what are the problems I'd want to fix?" The same write-up offers a warning worth stealing — "the absence of recent postmortems is a red flag."

Delete things on purpose. Power Peg sat dead for nine years because leaving it cost nothing that showed up anywhere. Retiring dormant code and jobs is reliability work, not housekeeping. If you cannot delete it, make it fail loudly rather than quietly.

I have never worked anywhere with a complete picture of this, including the places where the picture was my job. What changed is the order I look in. The servers on the list are not what worry me. It is the thing that has run correctly and unattended for four years, because the day it stops is the day you meet whoever was relying on it.