Opened at 2015-04-10T23:59:39Z
Closed at 2015-04-28T17:16:55Z
#2400 closed defect (fixed)
OpenSSL.crypto.Error (unknown message digest algorithm) when starting a node, using OpenSSL 1.0.1k-fips
| Reported by: | daira | Owned by: | daira |
|---|---|---|---|
| Priority: | major | Milestone: | 1.10.1 |
| Component: | packaging | Version: | 1.10.0 |
| Keywords: | packaging openssl fips tahoe-start error test-needed | Cc: | |
| Launchpad Bug: |
Description
Gabe reported:
was able to build tahoe successfully, and can 'create-client' as well, however tahoe start fails due to pyOpenSSL error:
Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/twisted/application/app.py", line 645, in run
runApp(config)
File "/usr/lib64/python2.7/site-packages/twisted/scripts/twistd.py", line 23, in runApp
_SomeApplicationRunner(config).run()
File "/usr/lib64/python2.7/site-packages/twisted/application/app.py", line 379, in run
self.application = self.createOrGetApplication()
File "/usr/lib64/python2.7/site-packages/twisted/application/app.py", line 444, in createOrGetApplication
application = getApplication(self.config, passphrase)
--- <exception caught here> ---
File "/usr/lib64/python2.7/site-packages/twisted/application/app.py", line 455, in getApplication
application = service.loadApplication(filename, style, passphrase)
File "/usr/lib64/python2.7/site-packages/twisted/application/service.py", line 403, in loadApplication
application = sob.loadValueFromFile(filename, 'application', passphrase)
File "/usr/lib64/python2.7/site-packages/twisted/persisted/sob.py", line 210, in loadValueFromFile
exec fileObj in d, d
File "tahoe-client.tac", line 10, in <module>
c = client.Client()
File "/home/gabeos/tahoe/src/allmydata/client.py", line 130, in __init__
node.Node.__init__(self, basedir)
File "/home/gabeos/tahoe/src/allmydata/node.py", line 82, in __init__
self.create_tub()
File "/home/gabeos/tahoe/src/allmydata/node.py", line 174, in create_tub
self.tub = Tub(certFile=certfile)
File "/usr/lib/python2.7/site-packages/foolscap/pb.py", line 240, in __init__
self.setupEncryptionFile(certFile)
File "/usr/lib/python2.7/site-packages/foolscap/pb.py", line 252, in setupEncryptionFile
self.setupEncryption(certData)
File "/usr/lib/python2.7/site-packages/foolscap/pb.py", line 267, in setupEncryption
cert = self.createCertificate()
File "/usr/lib/python2.7/site-packages/foolscap/pb.py", line 476, in createCertificate
132)
File "/usr/lib64/python2.7/site-packages/twisted/internet/_sslverify.py", line 853, in signCertificateRequest
hlreq = CertificateRequest.load(requestData, requestFormat)
File "/usr/lib64/python2.7/site-packages/twisted/internet/_sslverify.py", line 571, in load
if not req.verify(req.get_pubkey()):
OpenSSL.crypto.Error: [('asn1 encoding routines', 'ASN1_item_verify', 'unknown message digest algorithm')]
Failed to load application: [('asn1 encoding routines', 'ASN1_item_verify', 'unknown message digest algorithm')]
There are reports of this issue popping up online, but seems that consensus is that it is due to openssl versions < [0.98], at which point SHA-256 algorithms were added. However, I'm on Fedora 21, running openssl version 1.0.1k-fips, and definitely have sha* available.
python -c 'import ssl; print ssl.OPENSSL_VERSION'
OpenSSL 1.0.1k-fips 8 Jan 2015
anyway, if you have any thoughts or suggestions, it'd be much appreciated. Thanks, Gabe
Attachments (1)
Change History (13)
comment:1 Changed at 2015-04-11T00:05:47Z by daira
comment:2 Changed at 2015-04-11T00:06:45Z by daira
- Keywords error added
comment:3 Changed at 2015-04-11T00:34:26Z by daira
It appears that foolscap, before the fix to http://foolscap.lothar.com/trac/ticket/141, generates certificates that use RSA with MD5 as the signature algorithm. Although RSA-MD5 is listed in the algorithms in comment:1, it may be that OpenSSL in FIPS mode is (not entirely unreasonably) refusing to use it to sign certificates.
comment:4 Changed at 2015-04-13T19:41:00Z by daira
This was indeed the cause. There's another change needed to foolscap needed, as described at http://foolscap.lothar.com/trac/ticket/141#comment:7.
Here's what Gabe wrote about the debugging of this problem:
FWIW, looks like the error in twisted is coming from a method that doesn't directly take digestAlgorithm as a parameter--the "sha256" argument change in the patch goes into /usr/lib64/python2.7/site-packages/twisted/internet/_sslverify.py in the signCertificateRequest method just fine (verified via printing statements), but the error is coming from the load method, and signCertificateRequest doesn't pass digestAlgorithm on to load as a parameter. Don't really know if that's useful, since I'm not familiar with these packages at all, but it makes sense to me that the patch wouldn't affect this particular error, since the argument doesn't influence the call site of the error.
I wrote the requestData parameter to a file, however, and toyed around with it using the system openssl, to avoid any pythonic errors and came up with the following output (openssl.cnf attached):
~ openssl req -noout -text -sha256 -inform der -verify -verbose -modulus -in tahoeReqData Using configuration from /etc/pki/tls/openssl.cnf 140135384266608:error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm:a_verify.c:191: ~ openssl req -noout -text -md5 -inform der -verify -verbose -modulus -in tahoeReqData Using configuration from /etc/pki/tls/openssl.cnf 140343873476464:error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm:a_verify.c:191: ~ openssl req -noout -text -md5 -inform der -verify -verbose -in tahoeReqData Using configuration from /etc/pki/tls/openssl.cnf 140596394153840:error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm:a_verify.c:191: ~ openssl req -noout -text -sha256 -inform der -verify -verbose -in tahoeReqData Using configuration from /etc/pki/tls/openssl.cnf 140197933606768:error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm:a_verify.c:191: gabeos ~ openssl req -noout -text -inform der -verify -verbose -in tahoeReqData Using configuration from /etc/pki/tls/openssl.cnf 140231916164976:error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm:a_verify.c:191: ~ openssl req -noout -text -inform der -verbose -in tahoeReqData Using configuration from /etc/pki/tls/openssl.cnf Certificate Request: Data: Version: 0 (0x0) Subject: CN=newpb_thingy Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:99:ff:6c:b8:ad:68:bc:42:c7:e9:9d:c7:5b:b3: c2:50:84:b3:ad:0e:cc:fa:01:e6:27:8a:87:24:1a: 20:e2:31:54:86:e0:8a:18:46:dd:5b:7d:92:28:5c: 05:14:c8:39:cc:15:33:72:65:f0:c2:cf:27:62:68: a4:ef:0a:b5:63:f5:91:fe:32:06:69:ad:76:67:1e: bb:5c:a8:b0:63:87:e2:eb:73:d7:18:15:9b:f3:75: 0a:7a:c4:f8:6d:f5:4a:a8:a8:d7:c1:3f:1b:45:f6: d1:f7:4a:a5:5f:3a:91:e4:4b:4d:cb:ce:25:22:75: ce:24:18:31:df:e5:7e:7d:c4:28:a5:13:bd:de:fe: 7c:1d:ee:13:d6:ae:87:d0:9a:56:3d:f8:64:e1:46: 69:de:db:96:26:28:e2:ad:83:db:02:8c:50:39:71: e3:d9:4c:c3:1d:f1:ef:6c:d8:38:a1:46:c0:52:48: db:7c:75:7a:5e:04:17:08:76:d3:3d:a7:c0:2a:2b: 06:d6:60:fd:9b:18:74:b8:b1:3e:fb:52:68:3c:c3: 6b:68:e9:c4:20:a8:15:69:27:eb:32:3d:65:4b:c4: 1a:27:4c:6d:b8:cc:ce:4f:7c:32:9d:c7:5d:b9:ad: 03:7f:11:36:55:f7:2a:97:d6:23:5c:67:c7:15:cf: 74:57 Exponent: 65537 (0x10001) Attributes: a0:00 Signature Algorithm: md5WithRSAEncryption 48:a3:3c:fe:fa:0a:26:b7:d6:2c:de:48:a9:d3:8b:67:79:d1: dc:cc:e4:ab:8b:59:af:17:5a:d1:6c:40:43:27:34:2f:bb:f1: b8:50:81:9a:92:d1:6c:8a:ee:0c:fd:b1:06:c7:12:fe:ee:d0: 42:8a:84:70:ce:69:0c:a2:a7:41:4c:71:ee:26:df:e5:37:a7: a2:93:8f:b4:6c:74:f8:5d:b2:5f:a1:83:45:c3:f0:7b:31:a9: 7f:5c:9e:8c:eb:a5:d7:dd:ed:4b:39:3c:6f:8b:e3:5c:13:b5: e0:23:26:47:0a:e1:4b:00:fc:91:cd:6d:de:d3:2b:d7:b5:17: e7:7d:f1:a4:da:3f:af:78:22:dc:4f:26:92:f3:1c:53:a5:3f: c4:4c:ad:11:21:49:64:b8:9f:d4:ef:1d:0c:cb:14:17:63:b7: 84:81:2f:d8:d1:00:c6:44:b1:f9:24:a6:80:92:88:17:b3:58: 4c:30:29:80:96:54:e1:de:ee:88:44:cb:16:3d:04:6d:5b:04: 09:b9:52:88:12:c5:4d:5b:b4:87:f3:aa:a2:51:d7:fa:a5:29: 9d:63:fd:90:b1:f5:b7:28:48:cc:61:a0:64:da:c4:ee:68:f9: fc:f6:e6:24:c8:3a:33:ac:54:c4:4a:33:81:f5:d9:62:1f:9b: 49:5b:99:14
Seeing that it's still md5, I changed the foolscap/pb.py code to include digestAlgorithm="sha256" in the keypair.certificateRequest(..) method [...] which appears to have solved the problem.
I have the openssl output:
openssl req -noout -text -inform der -verbose -verify -in tahoeReqDataSHA256 Using configuration from /etc/pki/tls/openssl.cnf verify OK Certificate Request: Data: Version: 0 (0x0) Subject: CN=newpb_thingy Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:98:17:5d:bc:ef:7d:d3:b1:fd:f1:ea:02:5d:9d: 48:06:c4:4c:75:74:e5:2e:0d:09:c0:a1:58:d6:fe: d7:db:4d:2d:85:93:45:a2:3c:e7:55:08:b2:fb:9f: 7c:c9:d0:47:13:9f:60:33:78:42:5c:d9:5f:8e:a6: 0f:9b:90:38:ea:af:50:0b:51:16:33:79:58:16:e5: 28:b2:a1:6a:64:df:b1:f4:91:c3:0d:5c:25:49:6b: 44:6b:c1:88:4e:96:c9:81:fe:08:56:7d:0e:3c:40: 60:a4:51:6d:93:21:79:90:7d:ad:f9:de:fc:36:35: 51:82:bf:be:43:3e:0d:6e:26:c8:18:a8:44:44:3a: 72:7a:e6:0d:1c:93:e4:5c:45:5c:04:e5:7d:ef:2c: 0b:0c:76:4b:d3:85:24:c2:0d:d6:0c:51:2f:08:29: 5b:c0:98:5f:30:1d:a0:2e:ae:e9:e5:3d:b5:5d:79: 58:92:8f:0c:a8:10:61:1f:5a:62:81:85:fc:0c:c9: 09:9a:a3:84:13:52:74:37:ea:a1:87:93:70:86:0a: 52:02:c8:91:28:0e:05:13:18:81:3f:d2:d7:a1:7c: 54:20:17:fd:af:f3:59:82:29:73:0f:66:41:40:55: 79:f9:a3:78:17:34:33:61:b8:76:f4:ec:c6:14:f2: 10:25 Exponent: 65537 (0x10001) Attributes: a0:00 Signature Algorithm: sha256WithRSAEncryption 6e:2d:a2:51:3a:29:b6:3a:8a:56:43:1f:85:da:17:12:82:4f: b3:4f:0e:3d:fc:e1:83:f5:f0:0f:a1:42:af:33:43:78:66:d0: f3:b3:9f:14:7e:5a:bd:e6:c8:3a:a8:2b:54:e8:b8:f4:06:09: cf:a4:87:74:df:27:d3:18:61:f1:eb:7f:1a:48:35:92:70:09: 99:f7:85:4f:fb:0f:b1:6e:8e:2b:f1:f3:d5:9d:a2:8b:3f:bf: 5f:7f:82:36:93:26:94:f4:a4:ae:48:db:a0:b7:49:44:c3:a9: 6f:16:13:25:aa:34:4f:b9:26:15:59:96:2e:f7:ea:9f:eb:a9: 1a:e8:78:0e:2f:b5:69:65:20:3c:7a:30:e2:9b:09:f3:26:17: 36:2d:a8:2d:55:22:94:49:f7:84:da:e9:7a:54:a8:bb:7e:ce: 98:94:2a:e1:0a:14:45:db:fd:89:b1:ab:10:49:78:69:2b:36: 21:fa:8b:9b:1b:f2:55:ab:4c:65:07:92:ed:92:03:89:89:f6: 4e:da:2f:eb:6d:a5:7a:73:02:21:cd:4c:f2:41:62:47:0b:57: b8:43:6f:93:0a:9a:2c:c7:79:75:51:d7:68:41:62:52:7e:ad: 10:10:97:cd:b2:db:7c:22:90:82:c9:c8:f0:08:1d:4d:ff:03: d6:8e:ff:89
and
> ./allmydata-tahoe-1.10.0/bin/tahoe start STARTING '/home/gabeos/.tahoe' > ps aux | grep tahoe gabeos 14821 0.0 0.5 401200 65632 ? Sl 11:25 0:00 /usr/bin/python /home/gabeos/allmydata-tahoe-1.10.0/support/bin/tahoe start
So it was the correct site to be patching, the patch just didn't cover everything since twisted has md5 as the default digest algorithm param.
New foolscap/pb.py method should look like so:
def createCertificate(self):
# this is copied from test_sslverify.py
dn = crypto.DistinguishedName(commonName="newpb_thingy")
keypair = crypto.KeyPair.generate(size=2048)
# Following line is the change that wasn't in the patch.
# Otherwise req has signature algorithm md5WithRSAEncryption.
# Should be sha256WithRSAEncryption for OpenSSL 1.0.1k-fips
req = keypair.certificateRequest(dn, digestAlgorithm="sha256")
certData = keypair.signCertificateRequest(dn, req,
lambda dn: True,
1, # serial number
digestAlgorithm="sha256",
)
cert = keypair.newCertificate(certData)
#opts = cert.options()
# 'opts' can be given to reactor.listenSSL, or to transport.startTLS
return cert
Changed at 2015-04-13T19:42:05Z by daira
comment:5 Changed at 2015-04-13T19:46:52Z by daira
When this fix is released in foolscap 0.8, we should depend on that version, and write a unit test verifying that the foolscap cert generated for a Tahoe node has a 2048-bit key and sha256WithRSAEncryption as the algorithm.
comment:6 Changed at 2015-04-13T19:49:14Z by daira
- Component changed from code-network to packaging
- Keywords test-needed added; fedora removed
- Milestone changed from undecided to 1.10.1
comment:7 Changed at 2015-04-13T23:32:41Z by daira
Foolscap (master) fixed in http://foolscap.lothar.com/trac/changeset/2a76437af31c97bdbfdc65ffda4d0cb60a97684a/.
comment:8 Changed at 2015-04-17T03:04:10Z by warner
I released foolscap-0.8.0 with this fix a few days ago, please re-test with a dependency on that version.
comment:9 Changed at 2015-04-21T17:37:16Z by daira
- Status changed from new to assigned
I will ask Gabe to re-test this.
comment:10 Changed at 2015-04-22T08:36:53Z by daira
Gabe confirmed that it is fixed in foolscap 0.8.0. We still need to decide whether Tahoe-LAFS 1.10.1 will require foolscap 0.8.0. If it does, we can then include the test described in comment:5 .
comment:11 Changed at 2015-04-28T17:16:42Z by daira
Dependency fixed in [53ced4be8be6dbbba82e74e9acb851f90b552588/trunk]. We decided not to do the test.
comment:12 Changed at 2015-04-28T17:16:55Z by daira
- Resolution set to fixed
- Status changed from assigned to closed
It's quite unfortunate that the error message doesn't specify which message digest algorithm is unknown. I thought that the FIPS build might not include SHA-1, but Gabe posted (on the LeastAuthority Zendesk ticket) an algorithm list that does include that: