Wednesday, 24 May 2017

JSON REST endpoint loading data from mongo -- pass BSON along in actual binary?

I've got a REST endpoint whose purpose is to load a whole bunch of polygon coordinates (in the tens of thousands) from documents stored in a MongoDB collection. Originally I'd just been treating this "normally" -- my node.js script loads the data from Mongo, and passes it along to the client as JSON.

But the resulting JSON file was humongous, because of course I'm sending these gigantic arrays of floating-point numbers which are all represented as strings.

So, I learned about the "raw" flag in Mongo, which lets the node driver pull down the actual BSON. Then I pass that through the JSON endpoint and decode the binary client-side. Made quite an improvement. However, even though it's now much more efficiently encoded, I'm still dealing with a text file; just instead of looking like this:

coords: [[24.251251251,53.12312161], [24.532362362,55.112412], ...]

it now looks like this:

coords: [12,242,12,6,33,112,12,52...]

granted, the latter is much shorter than the former. But it's still a text representation of binary data, which is inherently inefficient.

Is there a way to send along the actual binary data, either for a particular property in the JSON, or maybe sending the whole file that way?



via DanM

No comments:

Post a Comment