Friday 17 March 2017

Node.js how to ungzip GET response made through TCP socket

When i send a GET request to Hearthstone website, and once i've concatenated all the sent frames, i still get a compressed string

My request:

GET /hearthstone/en/ HTTP/1.1
Host: eu.battle.net
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Referer: https://www.google.fr/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8,fr;q=0.6
Cookie: eu-cookie-compliance-agreed=1; _ga=GA1.3.780909635.1489783325; _gat_bnetgtm=1

My logs:

[Socket] > Connected !
Connected to 185.60.115.40
[Socket - Data] > data !
[Socket - Data] Handling head..
[Socket - Data] > data !
[Socket - Data] > data !
[Socket - Data] > data !
[Socket - Data] > data !
[Socket - Data] > data !
[Socket - Data] > data !
[Socket - Data] > data !
[Socket - Data] > data !
[Socket - Data] > data !
[Socket - Data] > data !
[Socket - Data] > data !
[Socket - Data] > data !
[Socket] > End message received !
{ '4331': '',
  'HTTP/1.1 200 OK': '',
  Date: 'Fri, 17 Mar 2017 21:22:08 GMT',
  Server: 'Apache',
  'X-Frame-Options': 'SAMEORIGIN',
  'Retry-After': '600',
  'Content-Language': 'en-GB',
  Vary: 'Accept-Encoding',
  'Content-Encoding': 'gzip',
  'Keep-Alive': 'timeout=5, max=4000',
  Connection: 'Keep-Alive',
  'Transfer-Encoding': 'chunked',
  'Content-Type': 'text/html;charset=UTF-8' }
      ����&���ܱm۞���m��۶m۶m�c��=s��=��I���/iҦM?uvA\7�HvĘ    �� ��~���:=d9�J�n8��ܪ}$k�7�V��'���q�u��7�W8i^��k��v���P�H�k����Q�Q*��" k����n�H���`�Fb�b��4'U���7�8Q���a�ɧ�P�^���F�A�G[Rn��7դ�CҸ�\0T����0����(����3mw�PX��ꇍ�D!����ǂ�����dUoS��1�H9l9h�)�!�����_�z>����g�+p��Hz4 M��\  �N��3��)Z�N�=�ƀ���]-R�ʅn$��_؉��l��k��=k
U�g%m�?�� ]>��i��[ɽ��.��5�6Ԧ�����|�l�ޔ��I�K���%�B�K8��M89X)ݿ�YT�Fk���}�d+L�xD�C��3D��
�f)n6ىp.5�h�Px��{rPC��Z
�qc6�Vq ����!x��ۀoW���A%W�rV�6.����8�ҹm��T)��Em�9���-;m�iIM����D   -a���M�����eb�O�����Ȓut"|�6�z�A�ޤ�0�\������5��1ٛe����     R���H�&E�O�%k�����L��a$�����*�+����}�h�do�s�4�����nB����[����' vE��$�!6�,~t�w�4��GZ�OK���3AP����+��K��}�m�u]e��G���o���k��c���qK�?��+�z-��5�J�����*��&[~��e=ǂQ�2�;EB�o,�>vWJ��E�$��R���s��M��,�J���2E7�cӬ��v0��W9u�r{=�:X� ��l)�5��BG�C�\�u��������*�*���wQ�"$\DPc-T��@,��iH���$��4���^-���iebը����y2�J/+\*�oj�_��К���ˬ�;�����
[...]

My code:

class HTTPRequest
{
    constructor ( client, request )
    {
        this.client = client
        this.request = request


        this.send()
    }


    send ()
    {
        this.handleResponse()
        this.client.write( this.request )
    }

    handleResponse ()
    {
        this.handleHead()
    }


    handleHead ()
    {
        this.client.socket.once('data', data =>
        {
            if (this.isHTTP(data) ) this.handleHTTPHead(data)

        })

    }

    /* Check if http message */
    isHTTP ( buffer )
    {
        if ( buffer == undefined ) buffer = this.buffer

        return ( buffer.toString().substring(0, 4) == 'HTTP' )
    }

    handleHTTPHead ( buffer )
    {
        console.log('[Socket - Data]', 'Handling head..')

        /* Handle head */
        let resDataUnparsed = buffer.toString().split('\r\n')
        let resData = {}

        for ( let key in resDataUnparsed )
        {
            let splited = resDataUnparsed[key].split(':')

            resData[ splited.shift() ] = splited.join(':').trim()
        }

        let keys = Object.keys( resData )
        let resHtml = ''

        for ( let i = keys.indexOf('') + 1; i < keys.length; i++ )
        {
            resHtml += keys[i]
            resHtml += resData[ keys[i] ]
            delete resData[ keys[i] ]
        }

        // Delete GET query end "flag" (...\r\n \r\n)
        delete resData['']

        this.resData = resData
        this.buffer = []
        this.handleHTTPBody()

        return this.resData
    }

    handleHTTPBody ()
    {
        let handle = ( data ) =>
        {
            this.buffer.push(data)
        }

        this.client.socket.on('data', handle)

        this.client.socket.on('end', () => {

            this.buffer = Buffer.concat(this.buffer)

            console.log( this.resData )
            let html = require('zlib').gzipSync(this.buffer)

            console.log(html.toString())
        }, 2000);

    }

}

And, what i don't understand, is why things goes wrong, Am i make this properly ?

Thanks !



via Alexandre Daubricourt

No comments:

Post a Comment