Tuesday 23 May 2017

How to slow down PgSQL results rows stream in javascript?

I'm experiencing out of memory issue in nodejs v4.5.0 with RXJS(5.4.0) and PostgreSQL (driver "pg": "6.1.4").

I manually create an observable of PgSQL rows like this:

return Rx.Observable.create((subscriber) => {
    pool.connect().then((client: pg.Client) => {
        const stream:any = client.query(query.toParam());
        stream.on('row', (row) => {
            subscriber.next(row);
        });

        stream.on('end', () => {
            subscriber.complete();
            client.release();
        });
    });
});

Then I'm attaching some operators to the rx observable and doing some processing. Note that the rows returned from database are a bit heavy.

Investigation leads me to one conclusion: Rows from database are returned much faster then they can be processed. There must be memory reserved for that heavy data in order to wait for processing, and this cause out of memory issue:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

Abort trap: 6

I don't see any option on PostgreSQL driver to pause the stream. Any ideas how I can resolve this issue?



via Jakub Bejnarowicz

No comments:

Post a Comment