wui: "since" and "announced" columns are confusing

[leif]

The list of storage servers in the gateway's web ui includes two columns containing timestamps, labeled "since" and "announced".

I expected that "announced" would be the most recent time that introducer received an announcement from the storage server, and "since" would be the time that the gateway's current connection to the storage server was opened (or, in the case of a disconnected server, the time that the connection was lost).

Instead, it appears that "announced" is actually the time that the gateway first received the announcement from the introducer (which is the time that the running instance of the gateway first successfully connected to the introducer, except in the less common case of storage servers that the introducer learned about after that time).

It appears that the "since" column is doing what I expected in the case of connected servers (showing when the connection was made) but in the case of disconnected servers it shows the current time (when the page was rendered).

The "announced" column is rarely useful with its current behavior, and perhaps should simply be removed.

The "since" column is useful for connected servers, as it shows how long they've been connected. For disconnected servers the current behavior is also somewhat useful as a stopgap until #1972 (wui: show time page was rendered) is implemented.

However, for disconnected servers there are two things that would be more useful to know, neither of which are currently displayed: when it became disconnected (or better, when it last responded to communication), and when the last connection attempt was made. (Or is that what the current time actually is? Is it constantly attempting to reconnect?)

Related: #816 (Add ping-all-servers button to welcome page)

[daira]

Merging in #311 (add "last-heard-from" timestamp to welcome page):

The welcome page currently lists all storage servers that we're connected to. It should ask Foolscap for the last-heard-from timestamp (which should be less than 10 minutes for real connections, since the keepalives insure that we'll get some amount of traffic at least that frequently) and display it in the same table.

Since that ticket was filed, foolscap has acquired an rref.getDataLastReceivedAt() method. It was ​added in foolscap 0.6.1, which Tahoe-LAFS already depends on.

[daira]

Merging in #1174 (say more explicitly what the 'since' date refers to for servers on the welcome page):

<kpreid> I really wish the welcome page grid info had last-seen dates for servers

<trel1023> and their storage status (on/off)

<trel1023> rather, whether they were accepting new shares

[...]

<warner> FYI, the "since" column was supposed to show that (if the server has never connected, it's supposed to show the announcement-receipt time. If it is currently connected, it's supposed to show when the connection was established. If the server was once connected and is no longer connected, it's supposed to show when the connection was lost).

<warner> I'm pretty sure it's broken for at least one of these three cases, though, I remember being surprised by the timestamp it showed.

<daira> That sounds rather unintuitive. How about changing it to explicitly say what the time means, e.g. one of:

"Announced at ..."

"Connected since ..."

"Disconnected since ..."

?

[daira]

Note that given the existence of rref.getDataLastReceivedAt(), which should already give us enough information to display something sensible, I think this ticket can be implemented independently of #816.

[leif]

I've started working on this.

​This branch removes the Since and Announced columns, as well as the Service column (which I think only ever contains "storage") and replaces them with "Last RX" (using rref.getDataLastReceivedAt) and "Status".

The Status column also says "Connected" or "Disconnected", which addresses #1961, but I don't really like seeing those words repeated so much. Maybe it would be better to have two separate tables for connected and disconnected servers?

The timestamps displayed in these new columns are relative to the page render time but there are absolute timestamps in a tooltip. Is there any objection to changing the TIME_FORMAT on the absolute timestamps to ISO 8601?

[leif]

I forgot to mention: the issue with the "Since" entry for disconnected servers displaying the time the page was rendered was actually only happening for servers that the gateway hadn't ever connected to.

The cause is that server.get_last_loss_time() returns None, which was being passed to time.localtime() which of course helpfully returns the present time when it gets no argument OR if its argument is None.

My branch says N/A in this case instead.

[daira]

This ticket has a patch, but no unit tests. It would be a candidate for 1.11 if it had tests.

[leif]

I will try to get tests for this done in time for 1.11.

[leif]

I wrote some tests and opened a pull request: ​https://github.com/tahoe-lafs/tahoe-lafs/pull/129

[zooko]

I reviewed the code and found only one minor issue (​https://github.com/leif/tahoe-lafs/commit/486dbfc7bd3c0bbba42a6df8e4564601120aec0e#commitcomment-8720650).

Good patch! And good design! Thank you. ☺

Here are some issues I have with the design:

• I'm reluctant to remove the "when did we first hear about this server" information, because I think it could sometimes be useful for diagnostics in a live system. Is it removed here because it clutters up the UI, or because it was confusingly represented, or because we didn't think of a reason that it would be useful? Could we change this patch to keep that information visible?
• We tend to use RFC-3339 absolute timestamps in other places, and I'm a bit uncomfortable with using deltas-until-now in some places and absolute timestamps in others. Do we have a principled reason for which representation to use? Or is it obvious to users which one to expect in each case? Personally, I think I would prefer absolute timestamps, although I haven't tried using the delta timestamps yet to see if they grow on me. Please see #1077, which is a ticket to make all of our time representations be consistently represented to the user and consistently implemented for the developer.
• FYI there is a bug in Foolscap such that the connected/disconnected status is sometimes incorrect. (See #816.) I guess this is a separate issue from this ticket, so I don't request that we deal with it right now, but I'm just letting you know that there is some closely related functionality (i.e. remove the "status" display entirely) that we might want to do soon. (Also, even if there weren't that bug in Foolscap, it is fundamentally impossible for connected/disconnected status to be always correct, because connections are an illusion.)

Okay, so the last of my three comments is merely a note about a related ticket, the first is a request for justification from someone else or else change this branch to put back that information, and the middle is a request for justification from someone else or else change this branch to make the timestamps look the same here as elsewhere in the code.

Thanks!

[daira]

I get a test failure:

[FAIL]
Traceback (most recent call last):
  File "/home/davidsarah/tahoe/git/trunk/src/allmydata/test/test_web.py", line 638, in _check
    self.failUnlessIn('<div class="status-indicator service-Connected"></div>\n<div class="status-description">Connected<br /><a class="timestamp" title="00:00:10 01-Jan-1970">1d0h0m50s</a></div></td>', res_u)
  File "/usr/lib/python2.7/dist-packages/twisted/trial/_synctest.py", line 416, in assertIn
    % (containee, container))
twisted.trial.unittest.FailTest: '<div class="status-indicator service-Connected"></div>\n<div class="status-description">Connected<br /><a class="timestamp" title="00:00:10 01-Jan-1970">1d0h0m50s</a></div></td>' not in u'<!DOCTYPE html\n  PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"\n  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n<html lang="en">\n  <head>\n    <meta charset="utf-8" />\n    <title>Tahoe-LAFS - Welcome</title>\n    <meta content="width=device-width, initial-scale=1.0" name="viewport" />\n    <meta content="Tahoe-LAFS is a free and open distributed storage system" name="description" />\n    <meta content="Tahoe-LAFS" name="author" />\n\n    <!-- Le styles -->\n    <link href="/css/bootstrap.css" rel="stylesheet" />\n    <link href="/css/new-tahoe.css" rel="stylesheet" />\n\n    <!-- Le fav and touch icons -->\n    <link href="/icon.png" rel="shortcut icon" />\n  </head>\n\n  <body>\n\n    <div class="navbar navbar-fixed-top">\n      <div class="navbar-inner">\n        <div class="container-fluid">\n          <a href="/" class="brand"><img src="/img/logo.png" alt="Tahoe-LAFS" /></a>\n          <table class="node-info pull-right">\n            <tr>\n              <th>Nickname:</th>\n              <td>fake_nickname \u263a</td>\n            </tr>\n            <tr>\n              <th>Node ID:</th>\n              <td title="TubID: tubid">v0-nodeid</td>\n            </tr>\n          </table>\n        </div>\n      </div>\n    </div>\n\n    <div class="container-fluid">\n      <div class="row-fluid">\n        <div class="span3">\n          <div class="well sidebar-nav nav">\n             <div class="nav-header">Open Tahoe-URI:</div>\n             <div class="nav-form">\n               <form action="uri" method="get" enctype="multipart/form-data">\n                 <input type="text" name="uri" />\n                 <p><input type="submit" class="btn" value="View File or Directory \xbb" /></p>\n               </form>\n            </div>\n            <hr />\n\n            <div class="nav-header">Download Tahoe-URI:</div>\n            <div class="nav-form">\n              <form action="uri" method="get" enctype="multipart/form-data">\n                <label for="download-uri">\n                  URI\n                  <input type="text" name="uri" />\n                </label>\n                <label for="download-filename">\n                  Filename\n                  <input type="text" name="filename" />\n                </label>\n                <input type="hidden" name="save" value="true" />\n                <p><input type="submit" class="btn" value="Download File \xbb" /></p>\n              </form>\n            </div>\n            <hr />\n\n            <div class="nav-header">Upload File</div>\n            <div class="nav-form">\n              <form action="uri" method="post" enctype="multipart/form-data">\n                <input type="file" class="freeform-input-file" name="file" />\n                <input type="hidden" name="t" value="upload" />\n\n                <label class="radio" for="upload-chk"> \n                  <input checked="checked" type="radio" id="upload-chk" value="chk" name="format" />\n                  Immutable\n                </label>\n\n                <label class="radio" for="upload-sdmf">\n                  <input type="radio" id="upload-sdmf" value="sdmf" name="format" />\n                  <acronym title="Small Distributed Mutable File">SDMF</acronym>\n                </label>\n\n                <label class="radio" for="upload-mdmf">\n                  <input type="radio" id="upload-mdmf" value="mdmf" name="format" />\n                  <acronym title="Medium Distributed Mutable File">MDMF</acronym> (experimental)\n                </label>\n\n                <p><input type="submit" class="btn" value="Upload File \xbb" /></p>\n              </form>\n            </div>\n            <hr />\n\n            <div class="nav-header">Create Directory</div>\n            <div class="nav-form">\n              <form action="uri" method="post" enctype="multipart/form-data">\n                <label class="radio" for="mkdir-sdmf"> \n                  <input checked="checked" type="radio" id="mkdir-sdmf" value="sdmf" name="format" />\n                  <acronym title="Small Distributed Mutable File">SDMF</acronym>\n                </label>\n\n                <label class="radio" for="mkdir-mdmf">\n                  <input type="radio" id="mkdir-mdmf" value="mdmf" name="format" />\n                  <acronym title="Medium Distributed Mutable File">MDMF</acronym> (experimental)\n                </label>\n\n                <input type="hidden" name="t" value="mkdir" />\n                <input type="hidden" name="redirect_to_result" value="true" />\n                <input type="submit" class="btn" value="Create a directory \xbb" />\n              </form>\n            </div>\n\n          </div><!--/.well -->\n          <div class="well sidebar-nav">\n            <div class="nav-header">\n              <ul class="nav nav-list">\n                <li class="nav-header">Tools</li>\n                <li><a href="status">Recent and Active Operations</a></li>\n                <li><a href="statistics">Operational Statistics</a></li>\n              </ul>\n            </div>\n            <hr />\n            <div class="nav-header">\n              <ul class="nav nav-list">\n                <li class="nav-header">Save incident report</li>\n                <li><div><form action="report_incident" method="post" enctype="multipart/form-data"><fieldset><input type="hidden" name="t" value="report-incident" />What went wrong?\xa0\xa0<input type="text" name="details" />\xa0\xa0<input type="submit" value="Save \xbb" /></fieldset></form></div></li>\n              </ul>\n            </div>\n          </div><!--/.well -->\n        </div><!--/span-->\n        <div class="span9">\n          <div style="margin-bottom: 16px">\n            <h1 style="font-size: 48px">Grid Status</h1>\n          </div>\n          <div class="grid-status">\n            <div class="row-fluid">\n              <div class="span6">\n                <div>\n                  <h3>\n                    <div class="status-indicator connected-no"></div>\n                    <div>Introducer not connected</div>\n                  </h3>\n                  <div class="furl">/[censored]</div>\n                </div>\n                <div>\n                  <h3>\n                    <div class="status-indicator connected-not-configured"></div>\n                    <div>Helper</div>\n                  </h3>\n                  <div class="furl">None</div>\n                </div>\n              </div><!--/span-->\n              <div class="span6">\n                <div class="span4 services">\n                  <h3>Services</h3>\n                  <div><ul><li><a href="storage">Storage Server</a>: not accepting new shares (read-only)</li><li>Not running helper</li></ul></div>\n                </div><!--/span-->\n              </div><!--/span-->\n            </div><!--/row-->\n          </div>\n          <div class="row-fluid">\n            <h2>\n              Connected to <span>1</span>\n              of <span>2</span> known storage servers\n            </h2>\n          </div><!--/row-->\n          <table class="table table-striped table-bordered peer-status"><tr>\n                <td><h3>Status</h3></td>\n                <td><h3>Nickname</h3></td>\n                <td><h3>Address</h3></td>\n                <td><h3>Last RX</h3></td>\n                <td><h3>Version</h3></td>\n                <td><h3>Available</h3></td>\n              </tr><tr>\n              <td class="service-connection-status">\n              <div class="status-indicator service-Disconnected"></div>\n<div class="status-description">Disconnected<br /><a class="timestamp" title="01:00:25 01-Jan-1970">1d0h0m35s</a></div></td>\n              <td class="nickname-and-peerid">\n                <div class="nickname">disconnected_nickname \u263b</div>\n                <div class="nodeid">n52gqzlsl5xg6zdfnfsa</div>\n              </td>\n              <td class="address">N/A</td>\n              <td class="service-last-received-data"><a class="timestamp" title="01:00:35 01-Jan-1970">1d0h0m25s</a></td>\n              <td class="service-version">allmydata-tahoe-fake</td>\n              <td class="service-available-space">123.5kB</td>\n            </tr><tr>\n              <td class="service-connection-status">\n              <div class="status-indicator service-Connected"></div>\n<div class="status-description">Connected<br /><a class="timestamp" title="01:00:10 01-Jan-1970">1d0h0m50s</a></div></td>\n              <td class="nickname-and-peerid">\n                <div class="nickname">other_nickname \u263b</div>\n                <div class="nodeid">n52gqzlsl5xg6zdfnfsa</div>\n              </td>\n              <td class="address"></td>\n              <td class="service-last-received-data"><a class="timestamp" title="01:00:30 01-Jan-1970">1d0h0m30s</a></td>\n              <td class="service-version">allmydata-tahoe-fake</td>\n              <td class="service-available-space">123.5kB</td>\n            </tr></table>\n        </div><!--/span-->\n      </div><!--/row-->\n\n      <hr />\n\n      <footer>\n        <p>\xa9 <a href="https://tahoe-lafs.org/">Tahoe-LAFS Software Foundation 2013</a></p>\n        <p class="minutia">Page rendered at <span>17:44:14 25-Nov-2014</span></p>\n        <p class="minutia">allmydata-tahoe: 1.10.0.post212 [1973-status-last-rx-rebased: 486dbfc7bd3c0bbba42a6df8e4564601120aec0e]\nfoolscap: 0.6.4\npycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958\nzfec: 1.4.5\nTwisted: 13.0.0\nNevow: 0.11.1\nzope.interface: unknown\npython: 2.7.5+\nplatform: Linux-LinuxMint_1-x86_64-64bit_ELF\npyOpenSSL: 0.14\nsimplejson: 2.6.2\npycrypto: 2.6\npyasn1: 0.1.7\nmock: 1.0.1\nsetuptools: 0.6c16dev5\nservice-identity: 14.0.0\ncharacteristic: 14.1.0\npyasn1-modules: 0.0.5\ncryptography: 0.3\ncffi: 0.8.2\nsix: 1.8.0\npycparser: 2.10\n</p>\n        <p class="minutia">Tahoe-LAFS code imported from: <span>&lt;module \'allmydata\' from \'/home/davidsarah/tahoe/git/trunk/src/allmydata/__init__.py\'&gt;</span></p>\n      </footer>\n\n    </div><!--/.fluid-container-->\n  </body>\n</html>'

allmydata.test.test_web.Web.test_welcome
-------------------------------------------------------------------------------

(In case it is relevant, my current local timezone is the same as UTC.)

[daira]

Zooko and I agreed that "when did we first hear about this server" is rarely meaningful and sometimes confusing. Usually, it would be roughly the time at which the introducer restarts, except when the server is *newly* added since the introducer last restarted (not when the server reboots). So we are now okay with removing this information.

[leif]

daira, are you sure your timezone is UTC? mine is UTC and the tests pass; from the test failure, I'd guess you're in UTC+1.

I will soon make the tests timezone-independent, and address the comments on github (fix the whitespace, and add spaces between the time units).

zooko, I hope you can come around to liking the relative timestamps, because I like them much better!

[daira]

Replying to leif:

daira, are you sure your timezone is UTC?

Positive.

[zooko]

Replying to leif:

zooko, I hope you can come around to liking the relative timestamps, because I like them much better!

I'm open to the idea because it might be a usability improvement, at least for some users, but I'm concerned about possibly introducing inconsistency by doing it in this patch.

How about if we make this a separate ticket? For #1983 ticket we continue to use the same format of timestamps that we currently use elsewhere in the UI, and then we separately discuss converting all of our timestamps to be consistently showing relative. That discussion should probably be on #1077.

That would help unblock this ticket (#1973).

[leif]

Here is the latest version of my work on this issue:

​https://github.com/tahoe-lafs/tahoe-lafs/compare/master...leif:1973.status-last-rx-rebased.8

This puts relative timestamps in the server column on the welcome page, and normalizes the timestamps everywhere (i think) else in the web UI.

I *think* this is ready to merge. It addresses all of the github comments in the previous versions linked from the commit message.

[daira]

LGTM modulo ​https://github.com/tahoe-lafs/tahoe-lafs/commit/559846c0beb4190809c1c901444b878a4edad8e0#commitcomment-15463357.

[leif]

merged in ​https://github.com/tahoe-lafs/tahoe-lafs/commit/a2d724aab7bf6dd2f7ffde37cb21c570ba773bf8

[warner]

Milestone renamed