Wednesday, 31 May 2017

Nodejs request.get respond "Web Page Blocked!"

I used get() method of npm request to get content of external url. But there are some sites responding "Web Page Blocked!" such as http://gourmet.goo.ne.jp/

enter image description here

It is still ok when I access this site from browser.
Here is a part of code:

var url = 'http://gourmet.goo.ne.jp/';
var headers = {};
headers['X-Requested-With'] = 'XMLHttpRequest';
headers['Referer Policy'] = 'no-referrer-when-downgrade';
headers['referer'] = url;
var mod_request = require('request');
var req = mod_request.get({ uri: url, encoding: 'binary', headers: headers }, function(err, res, body) {  
   var result = res.body; // res.body contain error page when url is http://gourmet.goo.ne.jp/
   // Process content here
});

Could you please give me any advice?



via GSP

No comments:

Post a Comment