simplejson-2.0.1 causes unicode-vs-bytestring problems

[warner]

A new version of simplejson was released recently, and it appears that the behavior of simplejson.loads changed. Whereas loads used to always return unicode strings, now it will return a regular bytestring if the value will fit into one:

• simplejson-1.9.2:

  >>> simplejson.loads('"a"')
  u'a'
  

• simplejson-2.0.1:

  >>> simplejson.loads('"a"')
  'a'
  

So to be compatible with this newer version, Tahoe needs to carefully coerce the output of simplejson.loads and convert all the string-or-unicode objects into unicode objects.

Here is a list of the files that use simplejson.loads:

• dirnode.py
• web/directory.py
• scripts/tahoe_cp.py
• scripts/tahoe_mv.py
• scripts/tahoe_ls.py
• test/check_load.py
• test/test_web.py
• test/test_util.py
• test/test_system.py

[warner]

Fixed, in 935d1856967cb18e. It looks like simplejson-2.0.0 introduced some speedups, one of which is to optimize decoding of strings that happen to be all-ascii, resulting in this behavior change.