File: chunked.py

package info (click to toggle)
python-scrapy 1.0.3-2
  • links: PTS, VCS
  • area: main
  • in suites: stretch
  • size: 4,468 kB
  • ctags: 4,468
  • sloc: python: 22,154; xml: 199; makefile: 76; sh: 3
file content (13 lines) | stat: -rw-r--r-- 505 bytes parent folder | download | duplicates (3) 
1
2
3
4
5
6
7
8
9
10
11
12
13
 
from scrapy.utils.http import decode_chunked_transfer


class ChunkedTransferMiddleware(object):
    """This middleware adds support for chunked transfer encoding, as
    documented in: http://en.wikipedia.org/wiki/Chunked_transfer_encoding
    """

    def process_response(self, request, response, spider):
        if response.headers.get('Transfer-Encoding') == 'chunked':
            body = decode_chunked_transfer(response.body)
            return response.replace(body=body)
        return response