diglloydTools IntegrityChecker Java Version: Finds Duplicates, Saved me 100GB!
diglloydTools
A cool new feature of diglloydTools IntegrityChecker 'icj' (Java version) is finding duplicate files; it saved me a whopping 100GB! Over the past few years, I had inadvertantly made copies when intending to move files due to a silent Finder bug: if files are marked as locked, the Finder makes copies—yikes!
By using the SHA-1 hashes, exact matches can be found unerringly and all but instantly (once the hashes are done via the 'update' command).
This example is after the 100GB duplications were found and deleted, so there is only 2.4 GiB left of duplicates.
Note: this listing is not displayed fully or properly due to bugs in web browsers.
# find all duplicates on volume Archive diglloydMP:DIGLLOYD lloyd$ icj dupes Archive # icj version 1.0 beta 5 @ 2016-08-13 17:38 # Copyright 2016 DIGLLOYD INC. All Rights Reserved # Use of this software requires a license. https://macperformanceguide.com/Software-License.html # Sat Aug 13 20:33:49 PDT 2016 [1]={/Volumes/Archive}
Finding folders...4240.6920.9043.12381.14350.16676.16676 12003 ms to find 16676 folders
Loading hash data concurrently for 16676 folders... 0.100.200.300.400.500.600.700.800.900.1000.1100.1200....16500.16600....waiting for I/O to finish...15652.16676 Loaded hash data for 16676 folders in 23822 ms
995 ms to find duplicates
1 duplicate of "/Volumes/Archive/WebSiteOriginals/Infrared-S3-D70-5D/2006-0829-IR-camera-compare-all-orig/S3/091/DSCF0472.RAF" waste = 24 MiB
/Volumes/Archive/WebSiteOriginals/Infrared-S3-D70-5D/2006-0829-IR-camera-compare/S3/091/DSCF0472.RAF
1 duplicate of "/Volumes/Archive/ArticlesAndReviews/D2XColor/materials/2005-0423 ManyTests/1/color-balance-correction/LCDPhotos/IMG_0044.JPG" waste = 1644 KiB
/Volumes/Archive/ArticlesAndReviews/D2XColor/LCDPhotos/IMG_0044.JPG
... lines omitted for brevity ...
4 duplicates of "/Volumes/Archive/ArticlesAndReviews/D2X-vs-1DsMII/publish/logos/diglloyd36.gif" waste = 2644 bytes
/Volumes/Archive/ArticlesAndReviews/ShiftLensStitching/publish/logos/diglloyd36.gif
/Volumes/Archive/ArticlesAndReviews/D200-vs-D2X/publish/logos/diglloyd36.gif
/Volumes/Archive/ArticlesAndReviews/TheSharpestImage/original/publish/logos/diglloyd36.gif
/Volumes/Archive/ArticlesAndReviews/CoastalOptics60f4/CoastalOptics_60f4/publish/logos/diglloyd36.gif 1 duplicate of "/Volumes/Archive/WebSiteOriginals/Infrared-S3-D70-5D/2006-0829-IR-camera-compare/D70/no-filter/_DSC5742.xmp" waste = 4380 bytes
/Volumes/Archive/WebSiteOriginals/Infrared-S3-D70-5D/2006-0829-IR-camera-compare-all-orig/D70/no-filter/_DSC5742.xmp
....
WASTED SPACE from duplicate files: 2388 MiB
