Tuesday, 4 April 2017

Crawl NodeJs and AngularJs sites with scrapy python

I am trying crawl websites with scrapy python, most of sites done successfully but most of sites are giving tough time, because they are running on Nodejs and angularjs framework or other java frameworks, scrapy crawler is unable to get the details from the pages. please here i need your kind attention. looking forward to your earliest help.

here you can find the code that initially i am using for test base.

import scrapy
from selenium import webdriver
from scrapy.http import TextResponse

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ['https://en-ae.wadi.com/home_entertainment-televisions/?ref=navigation']

    def parse(self, response):
        self.log('i have just visited the ' + response.url)
        yield{
            'product_name'  : response.css('p.description.ng-binding > span::text').extract_first(),
        }

Thanks in advance.



via Muhammad Zia Ur Rahman

No comments:

Post a Comment