Thursday, 25 May 2017

Google Cloud Datastore, how to query for more results

Straight and simple, I have the following function, using Google Cloud Datastore Node.js API:

fetchAll(query, result=[], queryCursor=null) {
  this.debug(`datastoreService.fetchAll, queryCursor=${queryCursor}`);
  if (queryCursor !== null) {
    query.start(queryCursor);
  }
  return this.datastore.runQuery(query)
  .then( (results) => {
    result=result.concat(results[0]);
    if (results[1].moreResults === _datastore.NO_MORE_RESULTS) {
      return result;
    } else {
      this.debug(`results[1] = `, results[1]);
      this.debug(`fetch next with queryCursor=${results[1].endCursor}`);
      return this.fetchAll(query, result, results[1].endCursor);
    }
  });
}

The Datastore API object is in the variable this.datastore;

The goal of this function is to fetch all results for a given query, notwithstanding any limits on the number of items returned per single runQuery call.

I have not yet found out about any definite hard limits imposed by the Datastore API on this, and the documentation seems somewhat opaque on this point, but I only noticed that I always get results[1] = { moreResults: 'MORE_RESULTS_AFTER_LIMIT' }, indicating that there are still more results to be fetched, and the results[1].endCursor remains stuck on constant value that is passed on again on each iteration.

So, given some simple query that I plug into this function, I just go on running the query iteratively, setting the query start cursor (by doing query.start(queryCursor);) to the endCursor obtained in the result of the previous query. And my hope is, obviously, to obtain the next bunch of results on each successive query in this iteration. But I always get the same value for results[1].endCursor. My question is: Why?

Conceptually, I cannot see a difference to this example given in the Google Documentation:

// By default, google-cloud-node will automatically paginate through all of
// the results that match a query. However, this sample implements manual
// pagination using limits and cursor tokens.
function runPageQuery (pageCursor) {
  let query = datastore.createQuery('Task')
    .limit(pageSize);

  if (pageCursor) {
    query = query.start(pageCursor);
  }

  return datastore.runQuery(query)
    .then((results) => {
      const entities = results[0];
      const info = results[1];

      if (info.moreResults !== Datastore.NO_MORE_RESULTS) {
        // If there are more results to retrieve, the end cursor is
        // automatically set on `info`. To get this value directly, access
        // the `endCursor` property.
        return runPageQuery(info.endCursor)
          .then((results) => {
            // Concatenate entities
            results[0] = entities.concat(results[0]);
            return results;
          });
      }

      return [entities, info];
    });
}

(except for the fact, that I don't specify a limit on the size of the query result by myself, which I have also tried, by setting it to 1000, which does not change anything.)

Why does my code run into this infinite loop, stuck on each step at the same "endCursor"? And how do I correct this?

Also, what is the hard limit on the number of results obtained per call of datastore.runQuery()? I have not found this information in the Google Datastore documentation thus far.

Thanks.



via trollkotze

No comments:

Post a Comment