remove foolscap 'reference'-token bug workaround in mutable publish

[warner]

To close this ticket, remove the workaround which was needed only to avoid a Foolscap bug which is long since fixed.

---

Foolscap 0.2.5 had a bug in which an inbound constraint (say aTupleConstraint) wouldn't accept a 'reference' token that pointed at a tuple. This would occur when the same tuple was sent multiple times in a single request.

The mutable-file publish code, when it is creating a brand-new mutable file, and when there are fewer than N servers (so multiple shares are sent to the same server), can trigger this case, because the same assert-that-there-are-no-shares test vector (a static tuple of numbers and strings) is used once per share. Python seems to compile the tuple once, so foolscap sees the same object appear in the outbound arguments multiple times, and uses the 'reference' token to preserve this (unintentional) relationship.

The workaround is to construct this tuple at runtime, basically changing:

 for x in something:
   testv = (0, 1, 'eq', '')
   stuff.append(testv)

into:

 for x in something:
   testv = tuple([0, 1, 'eq', ''])
   stuff.append(testv)

The constraint bug was fixed in foolscap-0.2.6, but since it's an inbound-side bug, what really matters is the version of foolscap on the other end of the wire, so to interoperate with older versions of tahoe (which may be using older versions of foolscap), we must avoid sending the 'reference' token at all. So this workaround (which was removed in f7b4c45d4651b25b because tahoe now requires a newer version of foolscap) needs to be reinstated.

In addition, I've seen another problem that relates to this code. A few days ago I started noticing failures in the buildbot 'speed test', which runs trunk code against a statically configured 'perfnet' (with a client node on prodcs2 and four storage nodes plus introducer on tahoebs1). Four nodes and ten shares means multiple shares per node, so the testv-and-readv-and-writev call gets multiple shares, which means a 'reference' token.

The storage server appears to still have problems with the 'reference' token, even though it's running foolscap-0.3.1 . Sometimes I see a Violation that claims getObject(reftoken) returned a non-tuple. Sometimes I see a "dangling reference" BananaError, which indicates that getObject() failed to find the reference. The BananaError causes the connection to be dropped, so any remaining writev calls fail with a DeadReferenceError.

I can provoke this behavior with a local trunk tree against the storage nodes on tahoebs1. I haven't been able to reproduce this with local storage nodes (running foolscap-0.3.1). If I reintroduce the tuple([0,1,'eq','']) workaround, the problem seems to go away.

If I revert the recent versioning changes (which tries to call 'get_version' on the storage server, and falls back to a default value if that fails), then the problem seems to go away. The tahoebs1 perfnet nodes are a few months old (they are running r2901), so they don't have get_version(). So I'm suspecting that the Violation that gets raised somehow messes things up, such that getObject() remembers some bit of bogus state, so later getObject() calls either get a bad value or no value at all.

I'm going to commit the tuple([0,1,'eq','']) workaround, since we need it anyways. But at some point we need to come back to this and examine it further, because I still don't understand the new problem.

The first step will be to add instrumentation to the storage servers, which means first bouncing one of the tahoebs1 nodes and see if it still has the problem. (running the same code locally didn't experience the problem; the only difference I can think of is that the tahoebs1 nodes have been running for two months, and might have all experienced something in that time to mess them up now, whereas my local nodes are younger).

[warner]

More notes on the second problem:

• bouncing the storage server does *not* make the problem go away
• adding instrumentation to TupleConstraint reveals that the "not a tuple" complaint is triggered by a list. Instead of getting a tuple, it got a list of [(0, sharedata)] (a list with one element, which is a tuple with two elements). That looks like a write vector.
  • in one test, that write vector was sent as object 79. A subsequent 'reference' token cited object 78. Maybe there's sometimes an off-by-one error?
  • further testing suggests that the setObject() counters are completely broken

[warner]

​Foolscap#104 created to track this one.

[warner]

Ok, I think I have Foolscap #104 figured out: shared references that follow a Violation on the wire will be broken. I won't be likely to fix it until next week, since GC is more important, but by avoiding 'reference' tokens (i.e. avoiding shared references), Tahoe should be ok for now.

[davidsarah]

Which versions of Tahoe used foolscap 0.2.5, 0.2.6 and 0.3.1?

[zooko]

Versions of Tahoe-LAFS don't require specific versions of foolscap, only "at least this version" versions of foolscap, so older versions of Tahoe-LAFS could have been paired with newer versions of foolscap. Here are the details:

• current darcs head of Tahoe-LAFS requires foolscap >= 0.4.1, since: r3874 (2009-05-23)
• before that it required foolscap >= 0.4.0, since r3870 (2009-05-22)
• before that it required foolscap >= 0.3.1, since r2968 (2008-09-20)
• before that it required foolscap >= 0.3.0, since r2825 (2008-08-05)
• before that it required foolscap >= 0.2.9, since r2729 (2008-07-30)
• before that it required foolscap >= 0.2.8, since r2666 (2008-06-09)
• before that it required foolscap >= 0.2.5, since r2381 (2008-04-03)
• before that it required foolscap >= 0.2.4, since r2062 (2008-02-05)

[zooko]

Oh, but I didn't answer which versions of Tahoe-LAFS were involved. I get that information by looking at The Parade of Release Notes:

• Tahoe-LAFS v1.6.1 r4244 (2010-02-24)
• Tahoe-LAFS v1.6.0 r4220 (2010-02-02)
• Tahoe-LAFS v1.5.0 r4037 (2009-08-02)
• Tahoe-LAFS v1.4.1 r3853 (2009-04-14)
• Tahoe-LAFS v1.3.0 r3620 (2009-02-14)
• Tahoe-LAFS v1.2.0 r2789 (2008-07-22)
• Tahoe-LAFS v1.1.0 r2693 (2008-06-11)
• Tahoe-LAFS v1.0.0 r2348 (2008-03-26)

[zooko]

I guess we can safely remove the work-around now.

[zooko]

Well, let's leave it in for one more major release... :-)

[davidsarah]

​http://foolscap.lothar.com/trac/ticket/104 was fixed in foolscap 0.4, which was first required by Tahoe-LAFS r3870, released as Tahoe-LAFS v1.5.0.

relnotes.txt​ says:

Servers from this release can serve clients of all versions back to v1.0 and clients from this release can use servers of all versions back to v1.0.

which I believe is already incorrect, although I can't remember why not. Should we be making this claim when we don't test it?