EN
ItalianoEnglish

Real IT doesn't happen
in presentationsit happensin the field

Four stories from everyday work: a Friday evening that could have ended badly, a fault that was not a fault, a move with everything still running and an assistant that won everyone over by keeping quiet.

19:40

The phone rings at 7:40 pm on a Friday. On the other end is the head of accounts at a distribution company: around sixty people, three warehouses, an ERP that never sleeps. She says something that anyone in this line of work learns to recognise instantly: I can't open a file, and it has a strange name.

By 7:55 pm there are dozens of files with strange names. By 8:15 pm there are hundreds, and a document appears in a shared folder explaining, in polite, menacing English, how much it would cost to get them back.

Don't switch anything off

The first decision is the one that feels least natural: do not switch off the servers. Switching off destroys the evidence needed to understand what happened, and can interrupt encryption halfway through, leaving the data in a worse state.

Instead, you isolate the machines from the network, unplug whatever needs unplugging and record the situation as it stands. Then you open the plan.

The plan was real

It had been written months earlier together with the client, setting out what to do and in what order, with the phone numbers of the right people. And the backups existed. Not the kind everyone says they have, but real ones: copies isolated from the network, with restore tests run every month and formally recorded.

The last test had been nineteen days earlier. We knew how long it would take to get the ERP back up, because we had already done it as an exercise.

A weekend's work

Saturday was spent working out how they had got in: a mailbox compromised weeks earlier, access that had stayed under the radar. Sunday went on restoring, cleaning up, changing every credential and sealing the breach.

By nine on Monday morning the company was up and running. It lost a few hours of data, an amount that was known and accepted in advance: that too was a choice written into the plan, not a surprise discovered after the fact.

The difference

Paying the ransom was never discussed, because there was no need. And that is the point to take away: the difference between an incident and a disaster is not decided on the night.

It is decided in the quiet months, when testing a restore feels like wasted time and segmenting a network feels like pedantry.

The takeaway

A backup is not what you save: it is what you can restore. And calm during an incident is not a personality trait. It is a procedure written in advance.

One supplier at a time

Take an engineering company of around forty people. For years it did what almost every company does: add a supplier whenever a new need came up. Email was handled by one firm, the ERP by another, the network by a trusted technician who came when he could.

Then the phone system, the printers, the website, the CCTV. Seven suppliers in all. Each one good at its own piece, none responsible for the whole.

The Monday the email stopped

One Monday morning, email stops working. Not for everyone: only for some people, which is worse, because it makes the problem hard to describe and easy to bat back and forth.

The email supplier says the problem is the network. The network technician says it is the domain. Whoever manages the domain does not reply until the next day. Meanwhile the company is not sending orders or receiving confirmations, and the people answering the phone do not know what to tell customers.

Three days

The fault itself was minor: a missed renewal and a configuration changed by someone who had not documented it. It took three days to fix, two and a half of them spent working out whose job it was.

The real cost was not technical. It was the time, how it looked to customers, and the discovery that nobody knew who was in charge of what.

The map nobody had

When we arrived, the request was: give us a quote for email. The honest answer was different: the problem is not email. We proposed an assessment first, in other words a map.

The inventory brought to light what nobody could see any more: credentials that existed only in individual people's memories, equipment that had been out of support for years, overlapping contracts that paid twice for the same service.

A single person in charge

The real work was not replacing the suppliers. Some stayed, because they were doing a good job. It was making someone responsible for the whole: a single set of documentation, credentials kept safe and tracked, a point of contact who always answers and also coordinates the others.

The first measurable effect came with the next fault, a few months later: one afternoon's work, a single point of contact, no passing the buck.

The takeaway

The most expensive supplier is not the one with the highest rates: it is the setup that nobody answers for. Before asking for yet another quote, ask yourself who has the map.

A move with everything still running

On paper it was a supply job: a virtualisation cluster that had reached the end of support and needed replacing. In reality it was a move with everything still running.

The infrastructure ran around seventy virtual machines, including an ERP used across three shifts, six days a week. Storage was at 92%, performance had been dropping for months, and nobody could any longer say for certain what could be switched off.

Measure first

For the first month we installed nothing. We measured: three months of real metrics on CPU, memory, storage latency and hourly load profiles. Sizing based on what people say, where every server is by definition essential, produces clusters that are oversized where it is not needed and undersized where it hurts.

The measurements turned up the usual surprises: snapshot chains forgotten for months, machines still running that nobody needed any more, and CPU ready values that explained slowdowns users had stopped reporting out of resignation.

Waves

The new cluster was built alongside the old one, on the same network, with the new storage already in production. The migrations started with the least critical machines.

Not out of caution for its own sake: the aim was to test the method on what could afford a mistake, and reach the important machines with a well-practised procedure. Before each wave, backups were checked and the rollback criteria were written down.

The plan for when it goes wrong

For the ERP, the agreed window was four hours on a Saturday night. The runbook gave a timing for every step and, above all, set rollback criteria in advance: if the database is not in sync by a set time, you roll back, no discussion.

It is the part most often missing from projects. Not the plan for when everything goes well: the plan for when it goes wrong, decided while everyone is still thinking clearly and nobody is in a hurry.

Forty minutes

The switchover took forty minutes of the four hours allowed. On Monday morning not a single ticket came in.

The following week, someone in the accounts department asked, quite innocently, whether the migration had been postponed. It is the best compliment this work can receive.

The takeaway

A successful migration is a boring event, and boredom is not luck: it is by design. If the rollback plan is decided on the night of the switchover, it is already too late.

Thirty years of documents

The group had thirty years of documents: procedures, contracts, technical manuals, minutes, quotes. And mixed in, as always, personnel files, confidential agreements, a legal dispute.

It was the kind of request we now get every week: we want an internal assistant that answers people's questions using our documents.

The silent risk

Discussion of these systems almost always centres on a single risk, error: the confidently invented answer, the so-called hallucination. But in business settings there is a quieter, more costly risk: the right answer given to the wrong person.

An assistant that has read everything and answers everyone is not a productivity project. It is an incident waiting to happen.

Permissions before the model

So the project started there, not with the model. Document retrieval was built to respect the original permissions: the assistant only searches places the person asking could already access on their own, with their own credentials.

Every answer cites its sources, with a link to the document it comes from. And when the context is not enough for a well-founded answer, the assistant says so: "I don't know, with the information available" is an answer the project was designed to give, not a system failure.

The test bench

Before writing a single line of integration code, we wrote the evaluation set: a few hundred questions with their expected answers, including trick questions, built specifically to tempt the system into making things up or straying into areas it should not.

Every change goes through that test bench before it reaches users. Everything the assistant receives and answers is logged, because a system that talks to the company must be auditable like any other business system.

The decisive demo

The demonstration that decided the project was not a brilliant answer. In a meeting, an executive asked the assistant for a department's pay data. The assistant replied that those documents were not accessible with his profile, and told him who to ask.

There was a moment of silence, then the sentence that made the whole job worthwhile: so we really can use it.

The takeaway

People soon tire of a system that answers everything; they come to regret one that answers too much. In a business, the reliability of an AI is also measured by what it refuses to say.

Every story starts with an initial consultation.

If you recognised your company in one of these situations, or want to avoid recognising it in the future, let's talk: no obligation and no presentations.

Tell us about your situation