Tuesday, 30 May 2017

Iterate and extract paginated content with node js and selenium

Recently I started development of a that act as a crawler.There are paginated pages that I want to extract data from. At the movement I can iterate pages but no idea about how to extract page content, I'm using bellow function.

var getAllPages = function (category) {
    var driver = new webdriver.Builder().forBrowser('chrome').build();
    var map = webdriver.promise.map;
    driver.get('https://www.test.com/products/index.asp?qu=' + category);
    driver.wait(until.elementLocated(By.className('product-slide-all')), 30000, 'Could not locate the element');
    driver.findElement(By.className("pagination")).findElements(webdriver.By.tagName('a')).then(function (elements) {
        elements.forEach(function (element) {
            driver.executeScript("arguments[0].click()", element);
        })
    })
} 

is there any way that I can wait until AJAX call finishes and then extract page content while iterating.

Thanks



via Lakmal

No comments:

Post a Comment