Magic-Folder: add status command-line tool

[dawuud]

We need a tool to display which files are currently syncing. User's will want to know when some local data is finished persisting to a remote grid. The status command could have many iterations. I am only interested in making a simple command-line tool that displays a list of files that are currently being synced... but perhaps this could in the future specify percentage synced per file... and have many other interfaces for different operating systems.

[dawuud]

i added the boiler plate for the status command... ​https://github.com/david415/tahoe-lafs/tree/2569.magic-status-cli.0 ​https://github.com/david415/tahoe-lafs/commit/aa40fd0c2f03b773de3380982521b9ac9a9505ec

perhaps the existing tahoe-lafs http api has enough power to create the status command? or perhaps we must create additional http "commands"?

[meejah]

On ​https://github.com/meejah/tahoe-lafs/tree/2569.magic-status-cli.1 I have been working on this further. So far:

• an IProgress API (modifieds IReadable also)
• modified all IReadable things to use progress API properly
• simple HTML status page
• status page can emit JSON too
• flesh out "tahoe magic-folder status" further to print information about local + remote DMDs and in-progress downloads

Note that the CLI adds a dependency on "humanize" currently, as it's awesome for printing out human-readable times, deltas, and file-sizes.

[meejah]

status

[meejah]

Based on discussion during N&B today, I will add at least a token-based authentication to the status API to prove that you're a local client with access to the "private" area of your tahoe install (e.g. to prevent a malicious web request revealing filenames of shared items etc).

Similar to ​https://github.com/warner/petmail/blob/master/docs/steal_this_software.md#secure-all-http-frontend-access-tokens-event-channels

[dawuud]

another divide by zero error fix here:

• ​https://github.com/david415/tahoe-lafs/tree/2569.magic-status-cli.1

• ​https://github.com/david415/tahoe-lafs/commit/f98370d35dce4b348d4186ca23124b1aecf0f3a2

also... should we have another feature:

• exit-on-sync!

this might be useful for writing bash scripts that need to perform an operation after the magic-folder is done syncing local or remote content. In this case the user would specify a time duration in which if no remote or local changes are observed then we exit... that is the status command exits... the tahoe gateway of course continues to run!

[meejah]

There's also already a whole IUploadStatus and IDownloadStatus status-collecting interface. I haven't figured out how it works yet though, nor how to tie in magic-folder ... but I presume that's "the" way, rather than the pass-in-progress indicator I added?

[dawuud]

Remove redundant pending list

​https://github.com/david415/tahoe-lafs/commit/2230db45f75208839eb3347a97a4c792f1a4e259

[dawuud]

my latest branch: ​https://github.com/david415/tahoe-lafs/tree/2569.magic-status-cli.2

fixes : ​https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2577

it appears that status files in progress only display downloading files but not uploads. i think "in progress" is too vague and it should have two sections; downloads and uploads.

[meejah]

Yes, I agree about the two sections.

Uploads don't work at all yet, really.

[daira]

This is not a blocker for merging Magic Folder to trunk.

[meejah]

Daira reviewed the diff to give a preliminary opinion on new progress interface etc; these are my notes from that (any errors are my fault):

• look in collections for limited-length queue abstraction
• 'deque' is the right abstraction for the down/up-load history
• likes QueueItem? refactoring
• just want PercentProgress? (and change interface to include set_progress_total())
• @implementer -> implements (and/or when did zope.interface introduce decorator?)
• there's some commented-out-code in downloader that should go to queue mixin

[daira]

deque has a maxlen parameter to ​its constructor:

If maxlen is not specified or is None, deques may grow to an arbitrary length. Otherwise, the deque is bounded to the specified maximum length. Once a bounded length deque is full, when new items are added, a corresponding number of items are discarded from the opposite end. Bounded length deques provide functionality similar to the tail filter in Unix. They are also useful for tracking transactions and other pools of data where only the most recent activity is of interest.

[daira]

According to ​the changelog, zope.interface added @implementer in version 3.6.0. Tahoe already requires at least that version, so there is no compatibility problem. I've opened #2636 to switch to the new syntax everywhere.

[dawuud]

what's the next step to resolve this ticket?

[daira]

Sorry, I lost track of this. Are the issues from comment:11 resolved? It needs rebasing onto 2438.magic-folder-stable.8, also.

[meejah]

see ​https://github.com/tahoe-lafs/tahoe-lafs/pull/230

[meejah]

2569.magic-status.3 can be reviewed

[daira]

This is now on 2438.magic-folder-stable.12.