Skip to content
All projects

DupeClean Pro

A desktop app that finds duplicate photos, videos and documents and cleans them up with a full undo.

Role
Designer and developer
Year
2026

Source is in a private repository.

Illustration of a desktop app matching two identical photos, with space to reclaim and a dated folder tree

What it is

A cross-platform desktop app for Windows, macOS and Linux that finds duplicate pictures, videos and documents across folders and external drives, recommends the best copy to keep, and organises the rest into a dated folder tree.

The problem

Photo, video and document libraries collect copies under different names across many drives. Reading every byte of every file is slow on large libraries, and a clean-up tool that removes the wrong file loses the user's trust for good.

Architecture

Requests flow left to right. Hover a component for its role.

DupeClean Pro architecturefile queueon demandlive groupsPyQt6 desktop UI: dashboard, compare, clean-upPyQt6 desktop UIQThread workers: signals onlyQThread workersDirectory scanner: os.scandir, cancellableDirectory scannerDuplicate finder: size, 64 KB, full xxHashDuplicate finderSQLite cache: WAL, size and mtimeSQLite cacheComparators: pHash, ffprobe, MinHashComparatorsOrganizer: date folders, never overwritesOrganizerUndo journal and trash: JSON journal, send2trashUndo journal andtrash
Solid line: synchronous callDashed line: through a queueDashed box: external or infrastructure
  • PyQt6 desktop UI client, dashboard, compare, clean-up
  • QThread workers worker, signals only
  • Directory scanner service, os.scandir, cancellable
  • Duplicate finder service, size, 64 KB, full xxHash
  • SQLite cache data, WAL, size and mtime
  • Comparators service, pHash, ffprobe, MinHash
  • Organizer service, date folders, never overwrites
  • Undo journal and trash data, JSON journal, send2trash

Key decisions

  • Cheap checks first

    Files are grouped by size, then by an xxHash of their first 64 KB, and only the survivors are hashed in full in 8 MB chunks, so most files are never read completely and a 50 GB file never sits in memory.

  • Never lose a file

    The organizer never overwrites (clashes get _2 and _3 suffixes), never hard-deletes (files go to the system trash), and journals every step before the next, so a whole session can be undone.

  • An engine with no Qt in it

    Scanning, hashing and comparison live in a Qt-free core that runs on QThread workers and talks to the window only through signals. The interface stays responsive, and the same engine runs headless from a command-line tool.

  • Judge content, not names

    Similarity scores compare pixels, frames and text only. Filename closeness is reported separately, because real duplicates often have unrelated names.

Highlights

  • Duplicates stream into the results as they are confirmed, and Stop works while files are being hashed.
  • Rescans skip unchanged files using a SQLite cache keyed on size and modification time.
  • Pictures compared by perceptual hash, sharpness and EXIF; videos by frame fingerprints; documents by MinHash text similarity.
  • Keeps the display awake during long scans on macOS, Windows and Linux.
  • 174 automated tests at 92 per cent coverage, and PyInstaller build specs for all three platforms.

Stack

Python 3.11PyQt6SQLitexxHashOpenCVPillowPyMuPDFFFmpegimagehashdatasketchPyInstallerpytest-qtuv