mardi 31 mars 2015

How do I get an octet-stream in Node.js using http (or request)?

I'm writing a Node.js script to parse Wikidata database dumps and insert them into my own database. I'm having troubles downloading the files, as it seems both Node.js's http module and the request module from npm are ending the response stream too early. I'm getting the following error:



events.js:85
throw er; // Unhandled 'error' event
^
SyntaxError: Unexpected end of input
at parse (native)
at emit (/var/app/current/node_modules/split/index.js:27:17)
at next (/var/app/current/node_modules/split/index.js:48:7)
at Stream.<anonymous> (/var/app/current/node_modules/split/index.js:53:5)
at Stream.stream.write (/var/app/current/node_modules/split/node_modules/through/index.js:26:11)
at Gunzip.ondata (_stream_readable.js:540:20)
at Gunzip.emit (events.js:107:17)
at readableAddChunk (_stream_readable.js:163:16)
at Gunzip.Readable.push (_stream_readable.js:126:10)
at Gunzip.Transform.push (_stream_transform.js:140:32)


I'm using the following code to download the database dump:



var streamer = request.get("http:/dumps.wikimedia.org/other/wikidata/20150330.json.gz").pipe(require('zlib').createGunzip()).pipe(split(JSON.parse));

streamer.on('data', function(obj){
...
});


The error is the same whether I use request module or the built-in http module.


Aucun commentaire:

Enregistrer un commentaire