ó ç'ÊUc@sIdZddlZddlmZddlmZddlmZmZm Z m Z ddl m Z m Z ddlmZddlmZdd lmZdd lmZdd lmZmZdd „Zd ejfd„ƒYZdejfd„ƒYZddlm Z m!Z!m"Z"m#Z#m$Z$m%Z%dejfd„ƒYZ&dS(sV from twisted.internet import defer Tests borrowed from the twisted.web.client tests. iÿÿÿÿN(turlparse(tunittest(tservertstaticterrortutil(treactortdefer(tStringTransport(tFilePath(tWrappingFactory(t webclient(tRequesttHeaderscOs5d„}ddlm}|||d|||ŽjS(s-Adapted version of twisted.web.client.getPagec_sG|jddƒ}tjt||Žd|ƒ}|jjd„ƒ|S(NttimeouticSs|jS(N(tbody(tr((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyts(tpoptclienttScrapyHTTPClientFactoryR tdeferredt addCallback(targstkwargsRtf((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyt_clientfactorysiÿÿÿÿ(t_makeGetterFactorytcontextFactory(ttwisted.web.clientRR(turlRRRRR((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pytgetPages  tParseUrlTestCasecBs)eZdZd„Zd„Zd„ZRS(s.Test URL parsing facility and defaults values.cCs7tjt|ƒƒ}|j|j|j|j|jfS(N(RRR tschemetnetlocthosttporttpath(tselfRR((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyt_parse#scCsãd}dd||ddffdd||ddffdd||ddffd d|d |d dffd d|d |d dffd d|d |d dffdd||ddffdd||ddffdd||ddffdd||ddffdd||ddffdd|d|ddffdd)fdd*fdd||d dffd!d||d dffd"d|d|ddffd#d+fd%d,ff}x0|D](\}}|jtj|ƒ||ƒq³WdS(-Ns 127.0.0.1s#http://127.0.0.1?c=v&c2=v2#fragmentthttpiPs /?c=v&c2=v2s$http://127.0.0.1/?c=v&c2=v2#fragments#http://127.0.0.1/foo?c=v&c2=v2#frags/foo?c=v&c2=v2s'http://127.0.0.1:100?c=v&c2=v2#fragments:100ids$http://127.0.0.1:100/?c=v&c2=v2#frags'http://127.0.0.1:100/foo?c=v&c2=v2#fragshttp://127.0.0.1t/shttp://127.0.0.1/shttp://127.0.0.1/foos/fooshttp://127.0.0.1?param=values /?param=valueshttp://127.0.0.1/?param=valueshttp://127.0.0.1:12345/foos:12345i90shttp://spam:12345/foos spam:12345tspamshttp://spam.test.org/foos spam.test.orgshttps://127.0.0.1/foothttpsi»shttps://127.0.0.1/?param=valueshttps://127.0.0.1:12345/shttp://scrapytest.org/foo sscrapytest.orgshttp://egg:7890 segg:7890teggiÒ(shttps spam:12345sspami90s/foo(shttps spam.test.orgs spam.test.orgiPs/foo(shttpsscrapytest.orgsscrapytest.orgiPs/foo(shttpsegg:7890seggiÒR)(t assertEqualsRR'(R&tlipttestsRttest((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyt testParse's,   cCs¯d}|jdƒ}t|ƒ|j|ƒ\}}}}}|jt|tƒƒ|jt|tƒƒ|jt|tƒƒ|jt|tƒƒ|jt|tƒƒdS(sä L{client._parse} should return C{str} for the scheme, host, and path elements of its return tuple, even when passed an URL which has previously been passed to L{urlparse} as a C{unicode} string. uhttp://example.com/pathtasciiN(tencodeRR't assertTruet isinstancetstrtint(R&tbadInputt goodInputR!R"R#R$R%((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyt test_externalUnicodeInterferenceEs (t__name__t __module__t__doc__R'R1R:(((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyR s  tScrapyHTTPPageGetterTestscBs#eZd„Zd„Zd„ZRS(c Csytjtdddddidd6dd 6d d 6d d 6dd6ƒƒ}|j|dƒtjtdƒƒ}|j|dƒtjtdddddddidd6ƒƒ}|j|dƒtjtddddƒƒ}|j|dƒtjtdddidd6ddgd6ƒƒ}|j|dƒtjtdddtidd6ddgd6ƒƒƒ}|j|dƒdS(NRshttp://foo/barRs some datatheaderss example.nettHosttfoobles User-Agents blah blahtCookiet12981sContent-LengthtvaluetUsefulsGET /bar HTTP/1.0 Content-Length: 9 Useful: value Connection: close User-Agent: fooble Host: example.net Cookie: blah blah some datas GET /bar HTTP/1.0 Host: foo tmethodtPOSTs name=values!application/x-www-form-urlencodeds Content-TypesƒPOST /bar HTTP/1.0 Host: foo Connection: close Content-Type: application/x-www-form-urlencoded Content-Length: 10 name=values4POST /bar HTTP/1.0 Host: foo Content-Length: 0 tsingles X-Meta-Singletvalue1tvalue2sX-Meta-MultivaluedsoGET /bar HTTP/1.0 Host: foo X-Meta-Multivalued: value1 X-Meta-Multivalued: value2 X-Meta-Single: single (RRR t_testR (R&tfactory((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyttest_earlyHeadersYsP             cCs]tƒ}tjƒ}||_|j|ƒ|jt|jƒjƒƒt|jƒƒƒ|S(N( RRtScrapyHTTPPageGetterRLtmakeConnectiont assertEqualtsetRDt splitlines(R&RLt testvaluet transporttprotocol((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyRK³s    cCsžtjtddƒƒ}tjƒ}||_tƒ|_|jdƒ|jdƒ|jdƒ|jdƒ|j|jtidgd6d gd 6ƒƒdS( NRshttp://foo/barsHTTP/1.0 200 OK s Hello: World s Foo: Bar s tWorldtHellotBartFoo( RRR RNRLR R?t dataReceivedRP(R&RLRU((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyttest_non_standard_line_endings½s          (R;R<RMRKR[(((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyR>Ws Z (tForeverTakingResourcet ErrorResourcetNoLengthResourcetHostHeaderResourcetPayloadResourcetBrokenDownloadResourcetWebClientTestCasecBs˜eZd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Z d„Z d „Z d „Z d „Z d „Zd „Zd„Zd„ZRS(cCstjd|ddƒS(Nit interfaces 127.0.0.1(Rt listenTCP(R&tsite((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyt_listenÑscCs'|jƒ}tj|ƒt|ƒjdƒjdƒtj|ƒ}|jdt j dƒƒ|jdt ƒƒ|jdt ƒƒ|jdt ƒƒ|jdtƒƒ|jd tƒƒ|jd tƒƒtj|d dƒ|_t|jƒ|_|j|jƒ|_|jjƒj|_dS( Ntfilet 0123456789tredirects/filetwaitRtnolengthR#tpayloadtbrokenR(tmktemptostmkdirR tchildt setContentRtFiletputChildRtRedirectR\R]R^R_R`RaRtSitetNoneReR twrapperRfR$tgetHosttportno(R&tnameR((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pytsetUpÔs  cCs |jjƒS(N(R$t stopListening(R&((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyttearDownåscCsd|j|fS(Nshttp://127.0.0.1:%d/%s(Rz(R&R%((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pytgetURLèscCs2dd}t|jdƒd|ƒj|j|ƒS(NRhi RlR(RRRR-(R&ts((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyt testPayloadës cCsctjt|jdƒƒj|jd|jƒt|jdƒdidd6ƒj|jdƒgƒS(NR#s 127.0.0.1:%dR?swww.example.comR@(Rt gatherResultsRRRR-Rz(R&((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyttestHostHeaderïs(cCs,t|jdƒƒ}|j|jdƒ|S(sš L{client.getPage} returns a L{Deferred} which is called back with the body of the response if the default method B{GET} is used. RgRh(RRRR-(R&td((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyt test_getPage÷scsL‡fd†}tj|dƒjˆjdƒ|dƒjˆjdƒgƒS(s¶ L{client.getPage} returns a L{Deferred} which is called back with the empty string if the method is C{HEAD} and there is a successful response code. cstˆjdƒd|ƒS(NRgRF(RR(RF(R&(s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyt_getPagestheadttHEAD(RR‚RRP(R&R†((R&s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyttest_getPageHeadscCs9t|jdƒddƒ}|j|jd|jƒ|S(sÌ When a non-zero timeout is passed to L{getPage} and the page is retrieved before the timeout period elapses, the L{Deferred} is called back with the contents of the page. R#Rids 127.0.0.1:%d(RRRR-Rz(R&R„((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyttest_timeoutNotTriggeringscsJˆjtˆjdƒddƒtjƒ}‡fd†}|j|ƒ|S(sÔ When a non-zero timeout is passed to L{getPage} and that many seconds elapse before the server responds to the request. the L{Deferred} is errbacked with a L{error.TimeoutError}. RjRgíµ ÷Æ°>cs0ˆjjjƒ}|r,|djjƒn|S(Ni(Rxt protocolstkeysRTtloseConnection(t passthrought connected(R&(s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pytcleanup"s(t assertFailureRRRt TimeoutErrortaddBoth(R&tfinishedR‘((R&s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyttest_timeoutTriggerings   cCst|jdƒƒj|jƒS(Nt notsuchfile(RRRt _cbNoSuchFile(R&((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyt testNotFound.scCs|jd|kƒdS(Ns404 - No Such Resource(tassert_(R&tpageData((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyR˜1scCsk|jdƒ}tj|ƒ\}}}}}tjt|ƒƒ}tj|||ƒ|jj|j |ƒS(NRg( RRR'RR Rt connectTCPRRt_cbFactoryInfo(R&RR!R"R#R$R%RL((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyttestFactoryInfo4s cCsZ|j|jdƒ|j|jjdƒƒ|j|jdƒ|j|jddƒdS(Nt200sHTTP/tOKscontent-lengtht10(R-tstatusRštversiont startswithtmessagetresponse_headers(R&t ignoredResultRL((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyR;scCst|jdƒƒj|jƒS(NRi(RRRt _cbRedirect(R&((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyt testRedirectAscCs|j|dƒdS(Ns¼ click here (R-(R&R›((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyR¨Ds (R;R<RfR|R~RRRƒR…RŠR‹R–R™R˜RžRR©R¨(((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyRbÐs             ('R=Rotsix.moves.urllib.parseRt twisted.trialRt twisted.webRRRRttwisted.internetRRttwisted.test.proto_helpersRttwisted.python.filepathR ttwisted.protocols.policiesR tscrapy.core.downloaderR Rt scrapy.httpR R RwRtTestCaseR R>ttwisted.web.test.test_webclientR\R]R^R_R`RaRb(((s8/home/travis/build/scrapy/scrapy/tests/test_webclient.pyts " 7u.