Node JS goole oauth2 using access_token after expirition date

Here the story:

I'm developing iOS application, connected to Node JS server. I'm asking the user to authenticate and give me permissions to read and edit his google calendar. After the authentication is done, I'm getting auth_token, refresh_token and the expiry_date. I'm storing them, because my server app, has to do some API calls in the future (after the expiry date). On the server side I'm building oauth2Client object:

var oauth2Client = new auth.OAuth2(clientId, clientSecret, redirectUrl);
oauth2Client.credentials = {access_token:tokenData.accessToken, 
refresh_token:tokenData.refreshToken,
expiry_date:tokenData.expirationDate,token_type:'Bearer'};

var calendar = google.calendar('v3');
calendar.events.list({
    auth: auth,
    calendarId: calendarId,
    timeMin: dateFrom.toISOString(),
    timeMax:dateTo.toISOString(),
    singleEvents: true,
    orderBy: 'startTime'
}, function(err, response) {

});

It all works perfectly fine untiil the token expires. My question is: Can I refresh the token from my server without asking the user? Or what is the best practice in that case?

I tried:

oauth2Client.refreshAccessToken(function(err, tokens) {        
});

but it gives me unauthorized_client error

via gpopov