The report said HIGH. The raw rows said MEDIUM.

When someone resigns, a client sometimes wants to know one thing before they go: did anything walk out the door with them?

I have a report for that. It pulls the last 90 days of the person’s Microsoft 365 audit log over Graph (SharePoint, OneDrive, Exchange), checks it against a list of red flags such as bulk downloads, forwarding rules, external shares and heavy sync activity, has Claude write the findings up, and renders a branded PDF of about 16 pages. Nothing goes to the client until I’ve read it.

This week there were two departures. One report came back MEDIUM. The other scored six red flags and came back HIGH.

It never went to the client as HIGH. It was re-issued as MEDIUM, with a note in the report saying why. Here’s what changed my mind.

Twenty minutes in the rows

The collector already writes every event it pulls to a spreadsheet: timestamp, file, folder, device, network address, size. The report summarises those rows. I went back and read the ones behind the flags.

The “bulk download session” on the last day:

• Lasted 130 seconds, not the hour the summary claimed.
• Touched two neighbouring folders in one project, on one site. The report said ten folders across five sites, because the detector counted the whole day’s activity, not the session’s.
• Came to 47 files and about 16 MB.
• Wasn’t new. It was the twelfth day in under three weeks they’d downloaded from that same set of documents.
• Pulled one PDF four times over the morning. That’s a file that won’t open, not someone collecting.
• Came from a company-managed laptop in an ordinary browser, on the same network they’d used on most of the 90 days.

And across the full 90 days: no shares, no re-uploads, no email attachments. Nothing moved anywhere after it was downloaded. A cluster of deletions on the last afternoon turned out to be personal paperwork, not project files.

The flag said 104 downloads in the final fortnight, and that starts an argument. “Two minutes, 16 MB, documents they’d been downloading for weeks” ends it.

A count is not a judgement

The risk rating was a count: how many flags fired. It couldn’t tell that a flag had an innocent explanation, and it couldn’t tell a strong flag from a weak one.

Some flags nearly settle it on their own. An export of a whole mailbox to a file, or a rule forwarding mail to an outside address, is hard to explain away. Others fire on everybody. The sync flag triggers above 100 sync operations, which almost any laptop user with OneDrive crosses without trying. It fired on both reports this week, meant nothing either time, and cost each of them a whole risk level.

Counting treats those as equal. So every rating this engine issued before this week came from a count, and I’m treating all of them as unreviewed.

The model wasn’t the problem

It would be easy to blame the AI here, and it would be wrong. The written analysis was sound on what it was given. What it was given was summaries.

The step where Claude writes the findings never saw a single timestamp, network address, device flag or folder path. All of that was sitting in the spreadsheet the collector had already written, and all of it was thrown away before the model got the data. Every mistake I found was in the summarising or in the template, not in the reasoning.

Six sentences that were never checked

Reading every page of both reports turned up something worse than a wrong rating: sentences the template printed no matter what the data said.

• “Recommendation: Proceed with standard offboarding procedures.” The last line of every report this engine had ever produced, at every risk level. On the HIGH report, it sat under a conclusion asking the client to preserve the laptop as evidence.
• “Zero forwarding rules found.” Hard-coded. True on both reports, by luck. Anyone who had set up a forwarding rule would have been told they hadn’t.
• “Total deletions: N, minimal, no evidence of evidence destruction.” The word “minimal” was printed whatever N was.
• “No cross-user OneDrive access detected,” printed on page 12 of reports whose pages 4 to 7 listed colleagues’ drives the person had opened.
• An after-hours column that read 0 on every day, because the prompt told the model to put 0 there, while the summary above it reported hundreds of after-hours operations.
• A line of my own prompt wording, printed in the client’s PDF.

And a bonus: the risk badge was sized to fit “OVERALL RISK: LOW”. MEDIUM came out as “VERALL RISK: MEDIU”, clipped at both ends. Nobody had noticed, because every earlier report had come back LOW.

Every one of these passed the tests. The tests checked that a PDF rendered with the right sections. None of them could check whether a sentence was true.

All six are fixed. Each one now depends on the number it describes, or states the number and leaves the verdict to the written analysis, or is gone.

Run the report before you remove the licence

One more thing, and it’s about offboarding order rather than code. This person’s licence was removed the same day the report ran. That starts a countdown of about 30 days on their OneDrive and mailbox, which is exactly the material the report was about to recommend keeping.

The order is now:

1. Last working day.
2. Block sign-in, and separately revoke their sign-in tokens.
3. Run the report.
4. Then decide what to do with the licence.

Step 2 is two things on purpose. The audit log showed authentication events on this account in the days after it was disabled. Graph can’t tell a person signing in from a device quietly refreshing a token it already had, so I can’t say which it was. Disabling an account isn’t the same as ending its access.

What changed, and what hasn’t yet

A reviewer can now override the rating, but only with a written reason, and the report prints both levels and the reason on page 2. A rating can move. It can’t move silently.

That’s the stopgap. The real fixes are still on the list:

• Give the model the raw rows for the windows that matter (the download session, the deletions, the sent mail), not just totals.
• Add a novelty test to the download flag: had this folder been touched before the window? That alone would have corrected this rating.
• Put device, network and bytes in the report. Managed or unmanaged device is the most useful single fact in the whole dataset, and it’s already collected.
• Weight the flags instead of counting them.
• Render every template at every risk level against data built to make each sentence false, and fail on any sentence that survives unchanged.

The report I nearly sent was well formatted, confident, and wrong about the only thing the client would really read: the conclusion. Twenty minutes with the raw rows was worth more than the whole pipeline.