Finding Duplicates

Geeqie provides a utility to find images that have similar attributes or content.

To display a new Find Duplicates Window select File ▸ Find duplicates .

Adding files to be compared

Right-click either comparison pane and choose Append list to add the file list currently displayed in the main window, including a filtered folder list or collection. Choose Add folder… to select a folder directly; enable Include subfolders to add its contents recursively. Files are added to the pane whose menu you opened. These commands are unavailable while another file list is being added.

Add files to be compared using drag and drop. Drop files or folders onto the Find Duplicates window to add them to the list of files to compare. When one or more folders are dropped onto the window a menu will appear allowing you to choose the desired action:

Add contents
The contents of dropped folders will added to the window.
Add contents recursive
The contents of dropped folders and all sub folders will be added to the window.
Skip folders
Ignore folders contained in the drop list.

When files are added to the window, the comparison is restarted to include the new files.

Symbolic links to folders will be followed, but symbolic links to files will be ignored. Only actual files will be tested.

Comparison methods

The attribute to use for two images to match can be selected with the Compare by: drop down menu. Each method is explained below:

Name

The file name.

Name case-insensitive

The file name but ignoring case.

Size

The file size.

Date

The file date.

Dimensions

The image dimensions.

Checksum

The MD5 file checksum.

Path

The complete path to file.

Similarity (high)

Very similar image content. Preset to 95%.

Similarity

Similar image content. Preset to 90%.

Similarity (low)

Slightly similar image content. Preset to 85%.

Similarity (custom)

The percentage value used to consider two images a match is configured in the spin box at the bottom of the window. With the standard algorithm, setting a value below 85 is not likely to be useful. When pHash is selected in the Duplicates window, percentages measure matching hash bits instead of RGB similarity; see Similarity Algorithms for threshold details.

Name ≠ Content

Show images with the same name but different content.

Because pairs of files are being compared, when there are multiple hits care should be taken when analyzing the results.

Name case-insensitive ≠ Content

Show images with the same name, ignoring case, but different content.

Because pairs of files are being compared, when there are multiple hits care should be taken when analyzing the results.

Show all

Do not compare. Show all images.

Results list

Files that match with the selected comparison method will appear in the list. Matching files are grouped in alternating color.

The order of the result list can be changed by clicking on the column header. This will re-order the images within each set. When comparing by image content similarity, the matching sets will be sorted by order of rank starting with the files that are most similar.

A context menu is available for the result list by right clicking the mouse or pressing the Menu key when a row has the focus.

Groups in the results list may be selected by using the keyboard. Refer to Keyboard Shortcuts.

The selection can be changed using the keyboard and mouse the same as in a file pane of the main window.

The image Dimensions column of the result list will only contain dimension information when comparing by dimensions, or when the data is easily available from memory or has been read from the cache.

Selecting duplicates by file size

The results context menu's Select group 1 duplicates command selects the first displayed file in each match group. Select group 2 duplicates selects all remaining files in each match group, including groups containing more than two files. These commands replace the current selection. Here, group 1 and group 2 refer to row positions within each displayed match group, not the two input file sets used by Compare two file sets.

To retain the largest file in each match group, click the Size column header until the largest file appears first in each group, then choose Select group 2 duplicates. All other files in every match group will be selected. Review the selection before deleting or moving files.

To retain the smallest file instead, sort with the smallest file first and use the same selection command. Sorting changes clear the selection, so sort before selecting. Size refers to bytes, not image dimensions. When sizes are equal, one file remains unselected without any quality-based preference; a larger file is not necessarily a higher-quality image.

Exporting results for scripts

After comparison finishes, use the results context menu's Export submenu to choose Export to JSON or Run script with JSON…. JSON exports all result groups, including unselected members, and records which files were selected. CSV and tab-delimited exports continue to export selected rows.

To run a script, choose an executable script with an interpreter line, such as #!/usr/bin/env python3. Geeqie passes one argument: the path of a temporary JSON snapshot. The script runs asynchronously. Geeqie removes the snapshot when the script exits, and reports launch failures or unsuccessful exit status. The snapshot is captured when the menu command is invoked; later selection changes do not affect it.

The command geeqie --dupes-export-json writes JSON from the most recently opened Find Duplicates window to standard output without changing its selection:

geeqie --dupes-export-json > duplicates.json

Check schema (org.geeqie.duplicates), version (1), and status before processing. A status of ready means comparison is complete; incomplete and no_window return an empty groups array. An empty completed search also has status ready.

Completed exports include stable, untranslated criterion names in criteria, the numeric match_mask from DupeMatchType, two_sets, algorithm (standard or phash), threshold in percent, ignore_orientation, and ignore_same_folder. Algorithm and threshold are null for comparisons that do not use similarity.

Each group has an export-local numeric id, a representative_id, and a members array. Each member includes its file URI as id, its path, source set (1 or 2), selected flag, byte size, modification time mtime in seconds since the Unix epoch, and width and height when known. Unknown dimensions are null. A file may occur in multiple groups and retains the same URI identifier.

The similarity_to_representative value is the numeric percentage stored for that member's match to its group representative. It is null for the representative itself and for comparisons without similarity. It does not describe every possible pair of group members. The root score_semantics field is member_to_representative_percent. JSON strings escape quotes, backslashes and control characters; numbers do not depend on locale.

For example, an executable Python script can read the snapshot and print each group's paths:

#!/usr/bin/env python3
import json
import sys

with open(sys.argv[1], encoding="utf-8") as source:
    results = json.load(source)
if (results["schema"], results["version"], results["status"]) != ("org.geeqie.duplicates", 1, "ready"):
    raise SystemExit("No completed duplicate results")
for group in results["groups"]:
    print(group["id"], [member["path"] for member in group["members"]])

Status bar

Along the bottom of the Find Duplicates window is an area that displays the count of files contained in the window, and the number of files in the result list.

The status bar will also display the status of an active compare operation using the progress bar. A compare operation involves 2 or 3 stages, depending on the type of comparison. These are the stages in order:

  1. If necessary, extra data is read into memory for the comparison stage and the progress bar will indicate this stage with text such as “Reading dimensions...”, “Reading checksums...”, or “Reading similarity data...”.

  2. The images are compared using the selected method, the progress bar will indicate this stage with the text “Comparing...”.
  3. The results are sorted for display, the progress bar will indicate this stage with the text “Sorting...”.

    Stage 1 is only used for the Dimensions, Checksum, and Similarity compare methods.

    If the time to complete a stage will be significant, an estimated time to completion will also be displayed in the progress bar. The estimated time only refers to the current stage, other stages are not included in the estimate. The time estimate is displayed using the format MINUTES:SECONDS.

Thumbnails

Thumbnails can be displayed beside each image in the result list by enabling the Thumbnails check box.

Ignore Orientation

When selected and a similarity compare is being used, the images are checked against 4 rotations: 0°, 90°, 180°, 270°, plus flip and mirror.

Sort

The normal sort order is for groups (in the case of Similarity checks) with the highest number of near-100% matches to be at the top of the list.

If this box is checked, groups with the lowest number of matches are placed at the top of the list.

Ignore matches within the same folder

Enable this check box to compare files only when their immediate parent folders differ. This is useful when each folder contains a burst or sequence of similar images and you want to find matches across folders. Subfolders are treated as separate folders. The option applies to all comparison methods and to both single-set and two-file-set searches.

The option is off by default in each new Find Duplicates window. Changing it restarts the comparison. Each result group contains files that match its representative in another folder; two members of the same group can still belong to the same folder. A file can appear in more than one group when necessary to show cross-folder matches.

Compare two file sets

Sometimes it is useful to compare one group of files to another, different group of files. Enable this check box to compare two groups of files. When enabled, a second list will appear and files can be added to this list using the same methods for the main list.

When comparing two file sets the results list will display matches between the two lists. For each match group, the first file is always from the main group, and the remaining files are always from the second group.

Drag and Drop

Drag and drop can be initiated with the primary or middle mouse buttons. Dragging a file that is selected will include all selected files in the drag. Dragging a file that is not selected will first change the selection to the dragged file, and clear the previous selection.

Image Data Window

Ctrl+Shift+Right Mouse click : Use this to display a dialog containing the data stored for the clicked image file. This is usually only useful for debugging purposes.