Ttyjfyk

I heavily depend on rsync for remote data transfer and backup purposes, occasionally on different filesystem types. AFAIK rsync by default compares file size and modification timestamps to find out which files need to be transfered. This behavior can be changed to just file sizes using --size-only flag and can be replaced by file checksums using --checksum flag.

Sometimes It's not quite clear to me why rsync decides a big tree of files should be re-transfered, when I know destination has the same data as source.

Is there an rsync option or something similar which helps to easily find out the difference between two file trees, for example for each file it could print one of timestamps, owner/group, permissions, file size/content, etc.

First, if there is such an option in rsync, I am not aware of it and cannot find it in the man page. This is unfortunate, since it would provide what you need quickly, and presumably for any possible rsync use case.

Second, I think that if one were to try to do this with existing rsync options (basically --dry-run combined with a glut of possible combinations of checksum, date-is-newer, date-is-older, file-should-be-deleted, ad infinitum) would require several passes of rsync, one for each possible reason a file might get rsynced across. This seems unwieldy for your purposes.

If you can deal with a tool that compares two trees and reports the differences and interpolating its output to suit your needs, then mtree might work for your needs.

mtree is specifically designed to compare two directory trees and report the differences.

On the source machine, create an mtree spec (including sha256 hash) of your desired source tree:

# mtree -K sha256 -cp /my_path > my_path.mtree

Then rsync the tree across, and run mtree on the remote machine to see whether the remote directory tree /remote_path matches the local /my_path:

# rsync -a --delete /my_path/ remote:/remote_path/
# ssh remote mtree -p /remote_path < my_path.mtree && echo match
match

Now let's go mess with the remote tree a little. Long story short, I created a destination file farkle which doesn't exist on the source, removed a file file.txt, and changed the timestamp on a file data.txt.

# ssh remote mtree -p remote_path < my_path.mtree || echo fail
.: modification time (Wed Apr 17 10:49:32 2019, Wed Apr 17 11:55:10 2019)
extra: farkle
data.txt: 
 modification time (Wed Apr 17 10:45:02 2019, Wed Apr 17 11:56:27 2019)
foo: size (32, 29)
 modification time (Tue Apr 16 10:16:44 2019, Wed Apr 17 11:53:18 2019)
 sha256digest (0x6082aa7261362c4c71c82adf492bc724de53a5814e64b233c43c6775efeb1dd0, 0x2d2537ea27c27dfb2c1690c51c652b9ada32adc29a91b732c24939dcff371cd6)
./file.txt missing
fail

mtree is informing you that:

• 
  farkle exists on the destination, but not the source; it would be
  deleted if you specify --delete in your rsync command line.


• 
  data.txt has not changed in content (sha256 hash) or size, but has
  a newer timestamp; copying based on checksum alone might not re-transfer the file, nor would rsyncing only "newer" files, since the destination's file is now newer than the source.


• 
  file.txt is absent on the destination host, and would presumably be rsynced across on the next run, barring some exclusion criterion.

So this doesn't quite give you a direct rationale from rsync about why it is or isn't copying what you think it should, but it does give you a hard, verifiable report of what is actually on the ground with respect to the local and remote directory trees.

If you retain the my_path.mtree output file methodically, such as by including a datestamp in the filename, you can also run mtree on the local machine to compare the existing /my_path tree against a previous run (yesterday's? last week's?) to see whether something has changed locally that you might not have been expecting.

mtree is included in some Unix distributions, but if yours is lacking mtree, I have had success installing Archie Cobbs' version from GitHub.