Sunday, 7 May 2017

web scraper service node cherio dynamic select

I am trying to create web scrapper service with node.js that does the following for a site say amazon.com/mens/section/item1

  • get the name of the item (I can do this with cherrio)
  • get the price for the item (this also depends on the color of the item)
  • select the color and create json array for each color and its price.

I tried to use nighmarejs for this to select the color. As this event is dynamic in nature w.r.t selecting drop-down list. The site uses dojo.js Few Question here

  • Is nighmarejs the correct way to scrape the site for dynamic content (static content is straightforward with cherrio)

  • I was able to get say an h1.class text with nighmarejs but having trouble with select element

  • What is the right library to use for web scraping dynamic content and create a service out of it so I can make a curl request t it with query url=amazon.com/mens/section/item1 and response is

  • {url:myUrl, name:itemName, prices:{blue:10, black:15}}



via Mani

No comments:

Post a Comment