ReferenceAsk
Remy Reference/Architecture/Architecture/Content Inspection
21Architecture

Content Inspection

Every file written to app storage scanned for malware and personal data, including files that reach storage without passing through a platform pod, with the verdict shown as a badge and the app never slowed.

Files written to an app's storage are scanned for malware with ClamAV and for personal data with Presidio, and the result is attached to the object as a badge in the workspace dashboard. Scanning is a reporting feature: files stay instantly readable to the app, and the scan is a background fact about the object. Coverage has limits, stated below: antivirus reads objects up to 20MB, personal-data scanning reads text it can extract within its own caps, and a few platform-owned objects are not scanned. This chapter covers the trigger, the scanners, and where the results land.


#The Trigger

Files reach storage in two ways: through the platform's brokered file routes, and directly from a browser using a presigned upload grant. In the second path, no platform pod ever sees the bytes, so the trigger cannot live in application code. It lives in S3.

Every app-file bucket in every region emits object-created notifications to that region's SNS topic. Each topic delivers, with raw message delivery, into one central file-scan queue. A worker processor drains the queue and adapts each raw S3 event into a routable message. For a created object, it records a pending scan row keyed on the object's physical identity (region, bucket, key) and runs the antivirus check right away.

The personal-data check waits until the object has sat for an hour. Its verdict only matters for files that stay: an app's temporary files are gone by then, and a burst of writes drains at the analyzer's pace instead of all landing at once. A self-host's store also posts removals, and for a removed object the worker deletes the row once the object is confirmed gone.

Two kinds of object get no row: platform-internal stores, which never appear in the dashboard, and the JSON sidecars the platform writes beside generated media, which are its own record of a file already scanned. Objects under an organization or user root outside any app are scanned the same way as an app's.

The worker fleet does not scale on this queue's depth. A scan's ceiling is the PII analyzer's pods, which the baseline workers already fill, so a burst waits in the queue. See Scale and Reliability.

An hourly sweep claims the rows that still have a check pending and have been untouched for an hour. These are either personal-data checks whose object has sat that long, or scans whose message ran past the queue's time limit. The sweep puts their keys back on the queue for the workers. A scan that finds its object gone deletes the row, and a daily sweep drops any flagged row whose object is gone.


#The Scanners

Scanning runs against a standalone content-inspection tier on the trusted cluster. It has three Deployments: ClamAV, Presidio's analyzer, and Presidio's anonymizer, which serves the PII redaction step rather than the scanner. Each has its own Service. All sit deliberately outside the platform's deploy rollout, so an app deploy never restarts a scanner or fails because of one.

Antivirus. ClamAV runs as a daemon with its signature database baked into the image, so a fresh pod is scannable immediately and the updater fetches only deltas after boot. The worker streams the object's bytes to the daemon over its stream protocol and reads the verdict. Objects over 20MB are recorded as skipped. A signature-freshness check reads the daemon's engine and database version. Two replicas behind a disruption budget keep one available through signature reloads and node drains.

PII. Microsoft Presidio's analyzer runs as a service with an English NLP model. The worker extracts text from the object using a deliberately lightweight, local, free extraction method:

  • a ranged read of the first 5MB of a text-like file;
  • a parser for PDF, Word and Excel files up to 15MB;
  • a skip for images, image-only PDFs and other types.

Only the first 100,000 characters extracted are analyzed. The text is analyzed against a curated set of entity classes: the identifiers and credentials that mean a file may expose someone's personal data. Noisy classes are excluded, and low-confidence matches are dropped. The result is aggregated to counts per entity type; the text itself is never stored.

Extraction here is separate from the ingest-grade extraction used for data sources, which routes binary formats to a document model. Public-bucket keys hash their owner and cannot be attributed to a workspace, so a paid model call per uploaded file would have no one to bill and is not made.


#Results

Each scan row carries an AV status and a PII status: pending, clean, flagged, error, or skipped, with the entity counts for PII. The workspace's file dashboard shows a badge per file and per store. An Uploads view lists what end users uploaded through the interface SDK and what the platform generated for the app, and the owner can inspect and delete a flagged file there. The dashboard's manage routes refuse to hand out URLs or previews for an object whose AV status is flagged. The serving data plane does not gate on scan rows; the app's own reads and its users' reads are never blocked by the scanner.

What This Gives a Workspace
Stored files scanned, including files that never passed through the platform, within the limits above.
A malware verdict and a personal-data summary per object, visible where the files are managed.
Scanning that never slows the app or its users.
Scanners the platform runs itself, on its own cluster, with nothing sent to a third party.