Description

liujimj

opened on Dec 1, 2018

Issue body actions

How difficult would it be to also expose the SAX interface so we can process files that are much larger than can fit in RAM?

I am currently using ijson which has a yajl backend, but it's about 5x slower than RapidJSON.

Activity

1. lelit commented on Dec 2, 2018

   

   lelit

   on Dec 2, 2018

   Contributor

   More actions

   I do not foresee particular difficulties: a quick look seems to tell that what is missing is a way to expose the handler object...

   Cannot say when I will be able to spend time on this, though, sorry.
2. liujimj commented on Dec 4, 2018

   

   liujimj

   on Dec 4, 2018

   Author

   More actions

   @lelit Thanks, I would work on it myself, but it's been a long time since I've touched C++.

   ijson seems pretty popular (500+ stars) precisely because incremental parsing is so useful. I would switch over to RapidJSON's SAX implementation if I had a choice.
3. lelit commented on Dec 4, 2018

   

   lelit

   on Dec 4, 2018 via email

   Contributor

   More actions

   I will have a look at the `ijson` interface, thanks. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. [email protected] | -- Fortunato Depero, 1929.
4. lelit commented on Dec 4, 2018

   

   lelit

   on Dec 4, 2018

   Contributor

   More actions

   A quick look at ijson reveals that what you want is not the SAX interface, but rather an iterative interface to the reader, something more complicated because RJ does not expose out-of-the-box, AFAICT.
5. liujimj commented on Dec 4, 2018

   

   liujimj

   on Dec 4, 2018

   Author

   More actions

   Yes, I think you are right, although it looks like there is are IterativeParse methods on the Reader class which takes a Handler: http://rapidjson.org/md_doc_sax.html (under 'Token-by-Token Parsing')

   Perhaps the easiest way to expose this is to write a pass-through Handler that just re-emits the events, that is very close to the interface that ijson provides. Not sure how fast an FFI call is now, if naive implementation is too slow then one could batch events to emit back to Python.

Issue actions