Sunday 21 May 2017

NodeJS emulate browser for get/post requests

There's a lot of mixed results when I search around for emulating a browser. Long story short, I need my Node server to do get & post requests. Usually I'd just do this with the http package. However, there is some anti-scripting things in place on the other side. Namely javascripts that let the server know it's a real browser. So, I need these to be executed.

I actually solved this problem like 5 years ago, but my site was only using PHP then. The solution involved using a Qt webkit widget, and a fake X-server. Not elegant, but it was pretty easy to do. The only javascript engines I found available in Perl, PHP, or Python at the time were crazy slow.

As NodeJS is built on V8, I gotta think there's an easy way to do this. For the record, I'm hoping to get something a la the following.

// Omitting some callbacks
http.get('http://remote.site', function(res) {
  res.on('end', function() {
      // previously accumulated data is the page returned by
      // the request.  Any thing found in a <script> tag would have
      // been executed.
  });   
});



via kiss-o-matic

No comments:

Post a Comment