malformed requests arriving to nginx server

This issue is a mystery to me, and I will be extremely happy to find the solution. I'll provide the details of my environment I think are relevant, but if anything else is required please comment and I'll add.

The infrastructure:

host - digital ocean droplet

os - Ubuntu 17.04

reverse proxy - nginx 1.12.0 (issue happened in nginx 1.10.0 too)

app server - node-js 7.10.0 (happened on 7.9.0 too)

clients - Android (happens in several version incl. 5.0, 6.0.1, 7.1.1, so most likely not related) using OkHttp 3.8.0

So nginx forwards traffic to the nodejs, and nodejs processes the requests from the Android clients.

The issue:

For some clients, once in a while, for a specific query, the json received by nodejs is malformed. specifically, several bytes are written at the beginning of the body's json,

Here is one example:

2017-05-24 15:42:14.899 (+0300) - error: Error summary:
name: SyntaxError
status: 400
request: POST /api/v1/user_details
body: 
 J        ount_id":"217627","user":{"email": "abc@def.com" ....<rest of json is ok>

notice the beginning of the body - seems like several malformed bytes overwriting the beginning of the original json. sometimes there are no wierd bytes, only the original json is cut e.g. ser":{"email": "abc@def.com"... . It gets weirder - it happens only once in a while (maybe one in 100 calls), for only some clients, in a specific endpoint (although I don't have many endpoints, and this one is used alot) and even in those clients - only once in many requests. It is not specific to a certain device (happened on LG, Samsung, OnePlus at least), and from the logs I get in clients, using an OkHttp interceptor to see what data is sent, it appears correct. I even tried sending the same exact request which failed in the logs of a client, on my device, and it succeeded. I tried to reconstruct this bug with no success - I've bombarded the server with requests, from multiple devices, to see if it was a matter of big numbers, but none have produced this issue. Further in the investigation, I've tried to see where the data gets malformed exactly - I've noticed that before all middlewares, nodejs receives the malformed body. I've added logging of bodies in nginx, and noticed nginx itself receives the malformed data.

This is one of the weirdest issues I ever ran into, and I really hope someone could tell me what exactly is going.

Nginx Configuration:

/etc/nginx/sites-enabled/default

# HTTP - redirect all requests to HTTPS:
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name my-service.com, app.my-service.com, www.my-service.com;
    # letsencrypt auto-renew
    location ~ /.well-known {
        root /var/www/html;
    }
    location / {
        return         301 https://$server_name$request_uri;
    }
    #return 301 https://$server_name$request_uri;
}

#log_format postdata '[$time_local] "$request" $status '
#                    '$request_body';

server {
    # SSL configuration

    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;
    include snippets/ssl-my-service.com.conf;
    include snippets/ssl-params.conf;

    # Pass requests for / to localhost:3000:
    location / {
            #  used to log traffic's post body data when uncommented:
            #  access_log  /var/log/nginx/postdata.log  postdata;
            #  echo_read_request_body;

            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-NginX-Proxy true;

            proxy_pass https://localhost:3000/;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;

    }
}

snippets/ssl-params.conf

ssl_certificate /etc/letsencrypt/live/my-service.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/my-service.com/privkey.pem;

snippets/ssl-my-service.com.conf

# from https://cipherli.st/
# and https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html

ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
#ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
#ssl_ciphers EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!CAMELLIA;
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!MD5:!DES:!eNULL:!EXPORT:!3DES:!PSK:!LOW:!EXP:!SRP:!RC4:!DSS;
#ssl_ecdh_curve secp384r1;
ssl_ecdh_curve prime256v1;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
# Disable preloading HSTS for now.  You can use the commented out header line that includes
# the "preload" directive if you understand the implications.
#add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;

ssl_dhparam /etc/ssl/certs/dhparam.pem;

via sahar