[tahoe-dev] big data usability question

[Florian Hofmann]

> So, for Big Data, you think in terms of crawlers, spiders, programs that
> walk the data at a manageable rate, learn what's changed, update remote
> copies, test and repair shares, etc. rsync itself is not a great tool
> for this unless you know the directory structure ahead of time and only
> run rsync on a small piece of it at a time (something which can complete
> in less than an hour or so).

Not really related, but i wonder if and when rsync will support
filesystems that keep track of what has changed since a certain point
in history (thinking of btrfs and/or zfs here). This would make
crawling of rsync near instantaneous.