uploading a directory causes a brief dirnode exception

[warner]

I'm not sure why, but uploading a directory (the tahoe source tree), using 'curl', causes a (benign) exception in the vdrive server:

LOCAL:   File "/home/amduser/trunk/instdir/lib/allmydata/dirnode.py", line 81, in get
LOCAL:     raise KeyError("unable to find key %s" % idlib.b2a(key))
LOCAL: exceptions.KeyError: 'unable to find key uumzl6h4bokrkamr2z742hexgyg6oin2elowlkdrhfpd5hair3ya===='

I suspect that part of the upload process creates an empty directory and then tries to use it before the operation has completed, or something. I need to poke through the existing dirnodes and find which one references this one.

[warner]

the exception is marked as a LOCAL foolscap one, but the other three nodes don't have a corresponding REMOTE exception in their logs. It's possible that there's a fourth node out there that's connecting and trying to find its long-lost private dirnode, but the storage index keeps changing, and it seems to happen right at the beginning of a large directory upload.

[warner]

the same sort of exception occurred when I uploaded a file (with an HTTP PUT) that required a new directory to be created. Maybe we create a directory twice?

[warner]

oh, duh, this this normal. webish.PutHandler?._get_or_create_directories() will use the Deferred equivalent of:

try:
    child = parent.get(name)
except KeyError:
    child = parent.create_directory(name)

and the exceptions I'm seeing are because the (caught) KeyError? is sent via foolscap, which is currently configured to log all exceptions, even ones that are eventually caught and handled.

So this is spurious, and will go away when we get some better logging mechanism. As robk pointed out today, it would be nice if Foolscap had a way to say "I'm expecting to see exceptions like X and Y and Z, so don't freak out if you see them", and then we could use that information to decide what severity to use when logging the exceptions.