ValueError (need more than 8 values to unpack) during mutable repair

[vikarti]

Situation: Deep check was started from WUI on windows node(Windows Server 2008R2 if that's important). I manually deleted some random shares in storage on one of other nodes(running Mac OS X if that's matter) and after that - stopped that other node.(I was checking how it will react to HDD failing)

Suddenly windows node's MUI deep check went to:

<type 'exceptions.ValueError'>: need more than 8 values to unpack
C:\tahoe\tahoe-1.9.3\support\Lib\site-packages\twisted-10.1.0-py2.6-win-amd64.egg\twisted\internet\defer.py, line 441 in _runCallbacks
439                    self._runningCallbacks = True
440                    try:
441                        self.result = callback(self.result, *args, **kw)
442                    finally:
Locals
callback	<function _repair_finished at 0x0000000006EB2908>
self	<Deferred at 0x7f5a888L current result: None>
args	()
kw	{}
c:\tahoe\tahoe-1.9.3\src\allmydata\mutable\checker.py, line 269 in _repair_finished
267            r = CheckResults(from_string(self._node.get_uri()), self._storage_index)
268            self.cr_results.post_repair_results = r
269            self._fill_checker_results(repair_results.servermap, r)
270            self.cr_results.repair_results = repair_results # TODO?
Locals
repair_results	<allmydata.mutable.repairer.RepairResults instance at 0x000000000630F708>
r	<allmydata.check_results.CheckResults instance at 0x0000000004C7B8C8>
self	<allmydata.mutable.checker.MutableCheckAndRepairer instance at 0x00000000074A9A88>
Globals
CheckResults	<class allmydata.check_results.CheckResults at 0x00000000047A8258>
from_string	<function from_string at 0x0000000004673278>
c:\tahoe\tahoe-1.9.3\src\allmydata\mutable\checker.py, line 130 in _fill_checker_results
128        summary = []
129        vmap = smap.make_versionmap()
130        recoverable = smap.recoverable_versions()
131        unrecoverable = smap.unrecoverable_versions()
Locals
smap	<allmydata.mutable.servermap.ServerMap instance at 0x00000000074A33C8>
summary	[]
vmap	{(7, 'K\x03V%\x98\\\x990"\xaf\x17o;W\xe9\xfc\xdb\xf6\x13\x1dB\xf8\'\xc9K\xb89\xa8\xd0\xac_\xc4', 3, 10, 131073, 1104, '\x01\x00\x00\x00\x00\x00\x00\x00\x07K\x03V%\x98\\\x990"\xaf\x17o;W\xe9\xfc\xdb\xf6\x13\x1dB\xf8\'\xc9K\xb89\xa8\xd0\xac_\xc4\x03\n\x00\x00\x00\x00\x00\x02\x00\x01\x00\x00\x00\x00\x00\x00\x04P', (('EOF', 2583), ('verification_key', 1730), ('enc_privkey', 123), ('share_data', 2167), ('signature', 1474), ('block_hash_tree', 2551), ('share_hash_chain', 1338), ('verification_key_end', 2022))): set([(9, "'W\xb0P\xc3\xeaH\xfc\x18\xe5\xdb\xc2\x87\x8b\xab\x8efn\xfa\xaa", 1328894782.632), (6, '\x1f\x0b\xf9H\x83\xca\xeas\xea5\xdf\x8eo\xd2u\xfa\xb1\x87\xe1\xd9', 1328894782.632), (0, 'YE\xcb1\x8af0\x918J\xed\x1d@ed\xf6B\xd0?q', 1328894782.632), (1, '\xcb\xd9\xdc\xa2\x07m\xd8\x10Szq\x1b\xde\x8c\xf5\xf6\xa3\x97\x10\x93', 1328894782.632), (2, ']\xe9\x1d:\x1b\x05\x04\xba\xa4\xf3\xc7xuq\xa83\xaeT\x04\x90', 1328894782.632), (5, '\x9e\xf2\xc7\x10 \x90\x80W\xd0 \xce\xe2\xf2\xc2+Z\xaf\xb2\xb1+', 1328894782.632), (8, '\xfd\xc7gI\xc4m\xba!\x97\xf9\xe6\x82`\xf16\xb4\xfaY\xbf7', 1328894782.632), (7, '\xc1|\x03\xe0\x1c"\xb6\xd58\xccO\xc6\x90\xf6\xe8f\x1dk\xc3\x86', 1328894782.632), (4, '\xba=\x91Y\x1a\xf3&\xaeXm\xc8\x1b\x0eO4!\xef\xa9g.', 1328894782.632), (3, '\xb0%K\xba\x97\xea\xc3\xc0E {k\xf0\x9a\x1d\xfe\x01\xd1\xea\xe1', 1328894782.632)])}
c:\tahoe\tahoe-1.9.3\src\allmydata\mutable\servermap.py, line 258 in recoverable_versions
256        for (verinfo, shares) in versionmap.items():
257            (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
258             offsets_tuple) = verinfo
259            shnums = set([shnum for (shnum, peerid, timestamp) in shares])
Locals
verinfo	(7, 'K\x03V%\x98\\\x990"\xaf\x17o;W\xe9\xfc\xdb\xf6\x13\x1dB\xf8\'\xc9K\xb89\xa8\xd0\xac_\xc4', 3, 10, 131073, 1104, '\x01\x00\x00\x00\x00\x00\x00\x00\x07K\x03V%\x98\\\x990"\xaf\x17o;W\xe9\xfc\xdb\xf6\x13\x1dB\xf8\'\xc9K\xb89\xa8\xd0\xac_\xc4\x03\n\x00\x00\x00\x00\x00\x02\x00\x01\x00\x00\x00\x00\x00\x00\x04P', (('EOF', 2583), ('verification_key', 1730), ('enc_privkey', 123), ('share_data', 2167), ('signature', 1474), ('block_hash_tree', 2551), ('share_hash_chain', 1338), ('verification_key_end', 2022)))
versionmap	{(7, 'K\x03V%\x98\\\x990"\xaf\x17o;W\xe9\xfc\xdb\xf6\x13\x1dB\xf8\'\xc9K\xb89\xa8\xd0\xac_\xc4', 3, 10, 131073, 1104, '\x01\x00\x00\x00\x00\x00\x00\x00\x07K\x03V%\x98\\\x990"\xaf\x17o;W\xe9\xfc\xdb\xf6\x13\x1dB\xf8\'\xc9K\xb89\xa8\xd0\xac_\xc4\x03\n\x00\x00\x00\x00\x00\x02\x00\x01\x00\x00\x00\x00\x00\x00\x04P', (('EOF', 2583), ('verification_key', 1730), ('enc_privkey', 123), ('share_data', 2167), ('signature', 1474), ('block_hash_tree', 2551), ('share_hash_chain', 1338), ('verification_key_end', 2022))): set([(9, "'W\xb0P\xc3\xeaH\xfc\x18\xe5\xdb\xc2\x87\x8b\xab\x8efn\xfa\xaa", 1328894782.632), (6, '\x1f\x0b\xf9H\x83\xca\xeas\xea5\xdf\x8eo\xd2u\xfa\xb1\x87\xe1\xd9', 1328894782.632), (0, 'YE\xcb1\x8af0\x918J\xed\x1d@ed\xf6B\xd0?q', 1328894782.632), (1, '\xcb\xd9\xdc\xa2\x07m\xd8\x10Szq\x1b\xde\x8c\xf5\xf6\xa3\x97\x10\x93', 1328894782.632), (2, ']\xe9\x1d:\x1b\x05\x04\xba\xa4\xf3\xc7xuq\xa83\xaeT\x04\x90', 1328894782.632), (5, '\x9e\xf2\xc7\x10 \x90\x80W\xd0 \xce\xe2\xf2\xc2+Z\xaf\xb2\xb1+', 1328894782.632), (8, '\xfd\xc7gI\xc4m\xba!\x97\xf9\xe6\x82`\xf16\xb4\xfaY\xbf7', 1328894782.632), (7, '\xc1|\x03\xe0\x1c"\xb6\xd58\xccO\xc6\x90\xf6\xe8f\x1dk\xc3\x86', 1328894782.632), (4, '\xba=\x91Y\x1a\xf3&\xaeXm\xc8\x1b\x0eO4!\xef\xa9g.', 1328894782.632), (3, '\xb0%K\xba\x97\xea\xc3\xc0E {k\xf0\x9a\x1d\xfe\x01\xd1\xea\xe1', 1328894782.632)])}
shares	set([(9, "'W\xb0P\xc3\xeaH\xfc\x18\xe5\xdb\xc2\x87\x8b\xab\x8efn\xfa\xaa", 1328894782.632), (6, '\x1f\x0b\xf9H\x83\xca\xeas\xea5\xdf\x8eo\xd2u\xfa\xb1\x87\xe1\xd9', 1328894782.632), (0, 'YE\xcb1\x8af0\x918J\xed\x1d@ed\xf6B\xd0?q', 1328894782.632), (1, '\xcb\xd9\xdc\xa2\x07m\xd8\x10Szq\x1b\xde\x8c\xf5\xf6\xa3\x97\x10\x93', 1328894782.632), (2, ']\xe9\x1d:\x1b\x05\x04\xba\xa4\xf3\xc7xuq\xa83\xaeT\x04\x90', 1328894782.632), (5, '\x9e\xf2\xc7\x10 \x90\x80W\xd0 \xce\xe2\xf2\xc2+Z\xaf\xb2\xb1+', 1328894782.632), (8, '\xfd\xc7gI\xc4m\xba!\x97\xf9\xe6\x82`\xf16\xb4\xfaY\xbf7', 1328894782.632), (7, '\xc1|\x03\xe0\x1c"\xb6\xd58\xccO\xc6\x90\xf6\xe8f\x1dk\xc3\x86', 1328894782.632), (4, '\xba=\x91Y\x1a\xf3&\xaeXm\xc8\x1b\x0eO4!\xef\xa9g.', 1328894782.632), (3, '\xb0%K\xba\x97\xea\xc3\xc0E {k\xf0\x9a\x1d\xfe\x01\xd1\xea\xe1', 1328894782.632)])
<type 'exceptions.ValueError'>: need more than 8 values to unpack

instead of check results

no incident file was autocreated

after that deep check for same directory was initiated on Mac OS X node results mostly same

<type 'exceptions.ValueError'>: need more than 8 values to unpack
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py, line 542 in _runCallbacks
540                    current._runningCallbacks = True
541                    try:
542                        current.result = callback(current.result, *args, **kw)
543                    finally:
Locals
callback	<function _repair_finished at 0x111912e60>
args	()
current	<Deferred at 0x111768128 current result: None>
kw	{}
/Users/vikarti/tahoe-new/allmydata-tahoe-1.9.1/src/allmydata/mutable/checker.py, line 269 in _repair_finished
267            r = CheckResults(from_string(self._node.get_uri()), self._storage_index)
268            self.cr_results.post_repair_results = r
269            self._fill_checker_results(repair_results.servermap, r)
270            self.cr_results.repair_results = repair_results # TODO?
Locals
repair_results	<allmydata.mutable.repairer.RepairResults instance at 0x111633c68>
r	<allmydata.check_results.CheckResults instance at 0x1118e0ea8>
self	<allmydata.mutable.checker.MutableCheckAndRepairer instance at 0x111613320>
Globals
CheckResults	<class allmydata.check_results.CheckResults at 0x1105d4d50>
from_string	<function from_string at 0x110542140>
/Users/vikarti/tahoe-new/allmydata-tahoe-1.9.1/src/allmydata/mutable/checker.py, line 130 in _fill_checker_results
128        summary = []
129        vmap = smap.make_versionmap()
130        recoverable = smap.recoverable_versions()
131        unrecoverable = smap.unrecoverable_versions()
Locals
smap	<allmydata.mutable.servermap.ServerMap instance at 0x110f8e950>
summary	[]
vmap	{(5, '\xad|cW%\xbc\xbfn\xb9 \x84_2~\x96\xb1U\x8f\x08\xf4\xdc\xaf\xe4B\x12\xf6,%\xe5$\xee\x08', 3, 10, 131073, 348, '\x01\x00\x00\x00\x00\x00\x00\x00\x05\xad|cW%\xbc\xbfn\xb9 \x84_2~\x96\xb1U\x8f\x08\xf4\xdc\xaf\xe4B\x12\xf6,%\xe5$\xee\x08\x03\n\x00\x00\x00\x00\x00\x02\x00\x01\x00\x00\x00\x00\x00\x00\x01\\', (('EOF', 2331), ('verification_key', 1731), ('enc_privkey', 123), ('share_data', 2167), ('signature', 1475), ('block_hash_tree', 2299), ('share_hash_chain', 1339), ('verification_key_end', 2023))): set([(8, '\x1f\x0b\xf9H\x83\xca\xeas\xea5\xdf\x8eo\xd2u\xfa\xb1\x87\xe1\xd9', 1328895473.080371), (9, '\xb0%K\xba\x97\xea\xc3\xc0E {k\xf0\x9a\x1d\xfe\x01\xd1\xea\xe1', 1328895473.080371), (7, ']\xe9\x1d:\x1b\x05\x04\xba\xa4\xf3\xc7xuq\xa83\xaeT\x04\x90', 1328895473.080371), (5, '\xfd\xc7gI\xc4m\xba!\x97\xf9\xe6\x82`\xf16\xb4\xfaY\xbf7', 1328895473.080371), (6, '\xcb\xd9\xdc\xa2\x07m\xd8\x10Szq\x1b\xde\x8c\xf5\xf6\xa3\x97\x10\x93', 1328895473.080371), (3, 'YE\xcb1\x8af0\x918J\xed\x1d@ed\xf6B\xd0?q', 1328895473.080371), (4, "U\xcc\xfeLY\xfd\xda\x9e\xc1'0\x02\xbd\xfc\x19\x04\x17\n\xe2\x18", 1328895473.080371), (2, '\x9e\xf2\xc7\x10 \x90\x80W\xd0 \xce\xe2\xf2\xc2+Z\xaf\xb2\xb1+', 1328895473.080371), (1, '\xba=\x91Y\x1a\xf3&\xaeXm\xc8\x1b\x0eO4!\xef\xa9g.', 1328895473.080371), (0, "'W\xb0P\xc3\xeaH\xfc\x18\xe5\xdb\xc2\x87\x8b\xab\x8efn\xfa\xaa", 1328895473.080371)])}
/Users/vikarti/tahoe-new/allmydata-tahoe-1.9.1/src/allmydata/mutable/servermap.py, line 258 in recoverable_versions
256        for (verinfo, shares) in versionmap.items():
257            (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
258             offsets_tuple) = verinfo
259            shnums = set([shnum for (shnum, peerid, timestamp) in shares])
Locals
verinfo	(5, '\xad|cW%\xbc\xbfn\xb9 \x84_2~\x96\xb1U\x8f\x08\xf4\xdc\xaf\xe4B\x12\xf6,%\xe5$\xee\x08', 3, 10, 131073, 348, '\x01\x00\x00\x00\x00\x00\x00\x00\x05\xad|cW%\xbc\xbfn\xb9 \x84_2~\x96\xb1U\x8f\x08\xf4\xdc\xaf\xe4B\x12\xf6,%\xe5$\xee\x08\x03\n\x00\x00\x00\x00\x00\x02\x00\x01\x00\x00\x00\x00\x00\x00\x01\\', (('EOF', 2331), ('verification_key', 1731), ('enc_privkey', 123), ('share_data', 2167), ('signature', 1475), ('block_hash_tree', 2299), ('share_hash_chain', 1339), ('verification_key_end', 2023)))
versionmap	{(5, '\xad|cW%\xbc\xbfn\xb9 \x84_2~\x96\xb1U\x8f\x08\xf4\xdc\xaf\xe4B\x12\xf6,%\xe5$\xee\x08', 3, 10, 131073, 348, '\x01\x00\x00\x00\x00\x00\x00\x00\x05\xad|cW%\xbc\xbfn\xb9 \x84_2~\x96\xb1U\x8f\x08\xf4\xdc\xaf\xe4B\x12\xf6,%\xe5$\xee\x08\x03\n\x00\x00\x00\x00\x00\x02\x00\x01\x00\x00\x00\x00\x00\x00\x01\\', (('EOF', 2331), ('verification_key', 1731), ('enc_privkey', 123), ('share_data', 2167), ('signature', 1475), ('block_hash_tree', 2299), ('share_hash_chain', 1339), ('verification_key_end', 2023))): set([(8, '\x1f\x0b\xf9H\x83\xca\xeas\xea5\xdf\x8eo\xd2u\xfa\xb1\x87\xe1\xd9', 1328895473.080371), (9, '\xb0%K\xba\x97\xea\xc3\xc0E {k\xf0\x9a\x1d\xfe\x01\xd1\xea\xe1', 1328895473.080371), (7, ']\xe9\x1d:\x1b\x05\x04\xba\xa4\xf3\xc7xuq\xa83\xaeT\x04\x90', 1328895473.080371), (5, '\xfd\xc7gI\xc4m\xba!\x97\xf9\xe6\x82`\xf16\xb4\xfaY\xbf7', 1328895473.080371), (6, '\xcb\xd9\xdc\xa2\x07m\xd8\x10Szq\x1b\xde\x8c\xf5\xf6\xa3\x97\x10\x93', 1328895473.080371), (3, 'YE\xcb1\x8af0\x918J\xed\x1d@ed\xf6B\xd0?q', 1328895473.080371), (4, "U\xcc\xfeLY\xfd\xda\x9e\xc1'0\x02\xbd\xfc\x19\x04\x17\n\xe2\x18", 1328895473.080371), (2, '\x9e\xf2\xc7\x10 \x90\x80W\xd0 \xce\xe2\xf2\xc2+Z\xaf\xb2\xb1+', 1328895473.080371), (1, '\xba=\x91Y\x1a\xf3&\xaeXm\xc8\x1b\x0eO4!\xef\xa9g.', 1328895473.080371), (0, "'W\xb0P\xc3\xeaH\xfc\x18\xe5\xdb\xc2\x87\x8b\xab\x8efn\xfa\xaa", 1328895473.080371)])}
shares	set([(8, '\x1f\x0b\xf9H\x83\xca\xeas\xea5\xdf\x8eo\xd2u\xfa\xb1\x87\xe1\xd9', 1328895473.080371), (9, '\xb0%K\xba\x97\xea\xc3\xc0E {k\xf0\x9a\x1d\xfe\x01\xd1\xea\xe1', 1328895473.080371), (7, ']\xe9\x1d:\x1b\x05\x04\xba\xa4\xf3\xc7xuq\xa83\xaeT\x04\x90', 1328895473.080371), (5, '\xfd\xc7gI\xc4m\xba!\x97\xf9\xe6\x82`\xf16\xb4\xfaY\xbf7', 1328895473.080371), (6, '\xcb\xd9\xdc\xa2\x07m\xd8\x10Szq\x1b\xde\x8c\xf5\xf6\xa3\x97\x10\x93', 1328895473.080371), (3, 'YE\xcb1\x8af0\x918J\xed\x1d@ed\xf6B\xd0?q', 1328895473.080371), (4, "U\xcc\xfeLY\xfd\xda\x9e\xc1'0\x02\xbd\xfc\x19\x04\x17\n\xe2\x18", 1328895473.080371), (2, '\x9e\xf2\xc7\x10 \x90\x80W\xd0 \xce\xe2\xf2\xc2+Z\xaf\xb2\xb1+', 1328895473.080371), (1, '\xba=\x91Y\x1a\xf3&\xaeXm\xc8\x1b\x0eO4!\xef\xa9g.', 1328895473.080371), (0, "'W\xb0P\xc3\xeaH\xfc\x18\xe5\xdb\xc2\x87\x8b\xab\x8efn\xfa\xaa", 1328895473.080371)])
<type 'exceptions.ValueError'>: need more than 8 values to unpack

again no incident file was created

attempts to restart deep check results in same dump

[davidsarah]

Even though the results on Windows and Mac OS X are slightly different, it's unlikely that this is OS-specific; that's more likely to be nondeterminism due to timing, dict enumeration order, etc.

vikarti, is this with mutable.format set to mdmf?

[davidsarah]

Assuming that this is a duplicate of #1669 due to the similarity of symptoms and error message. (That ticket also has a "RuntimeError: Request.finish called on a request after its connection was lost" in the log, but I suspect that may be a red herring.)