Deduplicating Media Uploads With File Fingerprints

Media libraries fill up in a predictable way. The organization's logo is uploaded for the header, again for the footer, again for an email template and again for a social card. A staff photo appears on three pages and gets uploaded each time. An import brings in hundreds of images, many of them the same picture at the same size.

Left alone, that leads to cluttered libraries, wasted storage and confusion about which copy is the "real" one. Our platform handles it with a technique that is simple to describe: file fingerprints.

What a file fingerprint is

A fingerprint, more formally a cryptographic hash, is a short string calculated from every byte of a file. Two files with exactly the same contents produce exactly the same fingerprint. Change a single pixel and the fingerprint changes completely.

We calculate a fingerprint for every file that enters an account's media library and store it alongside the file's other details. Comparing two fingerprints is fast, much faster than comparing two images byte by byte.

How we use it

When a new upload arrives, we calculate its fingerprint and look for an existing item with the same fingerprint. If we find one, and it belongs to the same account, is stored in the same place and is still present on disk, we reuse it. The upload field receives the existing file's location, and no second copy is stored.

If any of those conditions fails, we store the new file normally. That cautious approach avoids surprises: we never reuse a file that has gone missing, sits in a private location or belongs to someone else.

Never across accounts

The most important rule is the boundary. An identical image uploaded by two different organizations is stored twice, once in each account's library. Sharing a single copy between customers might save a little space, but it would tie their data together. If one organization deleted the file, would it disappear for the other? Could one learn that the other had the same file? Those questions have no good answers, so we avoid them entirely.

Fingerprints are compared only within the account that is uploading, which matches the broader way we keep each customer's files in their own space.

Imports benefit the most

Website imports are where deduplication pays off the most. Older sites often reference the same image from many pages, sometimes under slightly different addresses. By fingerprinting each downloaded file, our importers store one copy and point every page at it. The customer ends up with a clean library that reflects what their site actually uses.

What fingerprints cannot do

A fingerprint only matches identical files. A resized version, a recompressed copy or a cropped variation of the same photo produces a different fingerprint, so those remain separate items. Detecting visually similar images is a different, much harder problem, and we do not pretend that hashing solves it.

That is fine for our purposes. The goal is to stop the most common kind of clutter, exact repeats, with a method that is fast, predictable and impossible to get wrong in a way that mixes up customers.

Fingerprints have other uses

The same stored fingerprint helps in other places. It lets us confirm a file was copied correctly during backups and restores, detect when a file on disk has changed unexpectedly and verify that a downloaded asset matches what we expected. One small field, several quiet benefits.

A small feature in service of a bigger promise

Customers rarely notice deduplication. They just find that their media library makes sense and that the photo they uploaded last month is right where they expect it. That kind of invisible care runs through VoVo and the rest of our Technology & Impact work.

Frequently asked questions

What is a file hash?

A file hash is a short value calculated from a file's contents. Identical files produce identical hashes, so hashes are a fast way to spot exact duplicates.

Will deduplication merge my files with another organization's?

No. Duplicates are only recognized within the same account. Identical files in different accounts are stored separately.

Does deduplication find resized or edited copies?

No. It only recognizes exact copies. Resized, recompressed or cropped versions are treated as different files.