Turrns out JSON::XS had some pretty good ideas that I could borrow.
Some TODO's left and this needs benchmarking.