1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409
|
# Dexador
[](https://travis-ci.org/fukamachi/dexador)
[](https://coveralls.io/r/fukamachi/dexador)
Dexador is yet another HTTP client for Common Lisp with neat APIs and connection-pooling.
## Warning
This software is still BETA quality. The APIs will be likely to change.
## Differences from Drakma
* Fast, particularly when requesting to the same host (See [Benchmark](#benchmark))
* Neat APIs
* Signal a condition when HTTP request failed
* OpenSSL isn't required for Windows
See also [a presentation given at Lisp Meetup #31](http://www.slideshare.net/fukamachi/dexador-rises).
## Usage
```common-lisp
(dex:get "http://lisp.org/")
(dex:post "https://example.com/login"
:content '(("name" . "fukamachi") ("password" . "1ispa1ien")))
```
### Posting a form-data
You can specify a form-data at `:content` in an association list. The data will be sent in `application/x-www-form-urlencoded` format.
```common-lisp
(dex:post "http://example.com/entry/create"
:content '(("title" . "The Truth About Lisp")
("body" . "In which the truth about lisp is revealed, and some alternatives are enumerated.")))
```
### Auto-detects Multipart
If the association list contains a pathname, the data will be sent as `multipart/form-data`.
```common-lisp
(dex:post "http://example.com/entry/create"
:content '(("photo" . #P"images/2015030201.jpg")))
```
### Following redirects (GET or HEAD)
If the server reports that the requested page has moved to a different location (indicated with a Location header and a 3XX response code), Dexador will redo the request on the new place, the fourth return value shows.
```common-lisp
(dex:head "http://lisp.org")
;=> ""
; 200
; #<HASH-TABLE :TEST EQUAL :COUNT 7 {100D2A47A3}>
; #<QURI.URI.HTTP:URI-HTTP http://lisp.org/index.html>
; NIL
```
You can limit the count of redirection by specifying `:max-redirects` with an integer. The default value is `5`.
### Using cookies
Dexador adopts [cl-cookie](https://github.com/fukamachi/cl-cookie) for its cookie management. All functions takes a cookie-jar instance at `:cookie-jar`.
```common-lisp
(defvar *cookie-jar* (cl-cookie:make-cookie-jar))
(dex:head "https://mixi.jp" :cookie-jar *cookie-jar* :verbose t)
;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
; HEAD / HTTP/1.1
; User-Agent: Dexador/0.1 (SBCL 1.2.9); Darwin; 14.1.0
; Host: mixi.jp
; Accept: */*
;
; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
; HTTP/1.1 200 OK
; Date: Tue, 10 Mar 2015 10:16:29 GMT
; Server: Apache
; X-Dealer: 152151
; X-XRDS-Location: https://mixi.jp/xrds.pl
; Cache-Control: no-cache
; Pragma: no-cache
; Vary: User-Agent
; Content-Type: text/html; charset=EUC-JP
; Set-Cookie: _auid=9d47ca5a00ce4980c41511beb2626fd4; domain=.mixi.jp; path=/; expires=Thu, 09-Mar-2017 10:16:29 GMT
; Set-Cookie: _lcp=8ee4121c9866435007fff2c90dc31a4d; domain=.mixi.jp; expires=Wed, 11-Mar-2015 10:16:29 GMT
; X-Content-Type-Options: nosniff
;
; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
;; Again
(dex:head "https://mixi.jp" :cookie-jar *cookie-jar* :verbose t)
;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
; HEAD / HTTP/1.1
; User-Agent: Dexador/0.1 (SBCL 1.2.9); Darwin; 14.1.0
; Host: mixi.jp
; Accept: */*
; Cookie: _auid=b878756ed71a0ed5bcf527e324c78f8c; _lcp=8ee4121c9866435007fff2c90dc31a4d
;
; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
; HTTP/1.1 200 OK
; Date: Tue, 10 Mar 2015 10:16:59 GMT
; Server: Apache
; X-Dealer: 152146
; X-XRDS-Location: https://mixi.jp/xrds.pl
; Cache-Control: no-cache
; Pragma: no-cache
; Vary: User-Agent
; Content-Type: text/html; charset=EUC-JP
; Set-Cookie: _auid=b878756ed71a0ed5bcf527e324c78f8c; domain=.mixi.jp; path=/; expires=Thu, 09-Mar-2017 10:16:59 GMT
; Set-Cookie: _lcp=8ee4121c9866435007fff2c90dc31a4d; domain=.mixi.jp; expires=Wed, 11-Mar-2015 10:16:59 GMT
; X-Content-Type-Options: nosniff
;
; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
```
### Basic Authorization
```common-lisp
(dex:head "http://www.hatena.ne.jp/" :basic-auth '("nitro_idiot" . "password") :verbose t)
;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
; HEAD / HTTP/1.1
; User-Agent: Dexador/0.1 (SBCL 1.2.9); Darwin; 14.1.0
; Host: www.hatena.ne.jp
; Accept: */*
; Authorization: Basic bml0cm9faWRpb3Q6cGFzc3dvcmQ=
;
; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
```
### Faking a User-Agent header
You can overwrite the default User-Agent header by simply specifying "User-Agent" in `:headers`.
```common-lisp
(dex:head "http://www.sbcl.org/" :verbose t)
;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
; HEAD / HTTP/1.1
; User-Agent: Dexador/0.1 (SBCL 1.2.6); Darwin; 14.1.0
; Host: www.sbcl.org
; Accept: */*
;
; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
(dex:head "http://www.sbcl.org/"
:headers '(("User-Agent" . "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/8.0.3 Safari/600.3.18"))
:verbose t)
;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
; HEAD / HTTP/1.1
; User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/8.0.3 Safari/600.3.18
; Host: www.sbcl.org
; Accept: */*
;
; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
```
### Reusing a connection
Dexador reuses a connection by default. As it skips a TCP handshake, it would be much faster when you send requests to the same host continuously.
### Handling unexpected HTTP status code
Dexador signals a condition `http-request-failed` when the server returned 4xx or 5xx status code.
```common-lisp
;; Handles 400 bad request
(handler-case (dex:get "http://lisp.org")
(dex:http-request-bad-request ()
;; Runs when 400 bad request returned
)
(dex:http-request-failed (e)
;; For other 4xx or 5xx
(format *error-output* "The server returned ~D" (dex:response-status e))))
;; Ignore 404 Not Found and continue
(handler-bind ((dex:http-request-not-found #'dex:ignore-and-continue))
(dex:get "http://lisp.org"))
;; Retry
(handler-bind ((dex:http-request-failed #'dex:retry-request))
(dex:get "http://lisp.org"))
;; Retry 5 times
(let ((retry-request (dex:retry-request 5 :interval 3)))
(handler-bind ((dex:http-request-failed retry-request))
(dex:get "http://lisp.org")))
```
### Proxy
You can connect via proxy.
```common-lisp
(dex:get "http://lisp.org/" :proxy "http://proxy.yourcompany.com:8080/")
```
You can connect via SOCKS5 proxy.
```common-lisp
(dex:get "https://www.facebookcorewwwi.onion/" :proxy "socks5://127.0.0.1:9150")
```
You can set the default proxy by setting
```dex:*default-proxy*```
which defaults to the value of the environment variable HTTPS_PROXY or HTTP_PROXY
## Functions
All functions take similar arguments.
- `uri` (string or quri:uri)
- `method` (keyword)
- The HTTP request method: `:GET`, `:HEAD`, `:OPTIONS`, `:PUT`, `:POST`, or `:DELETE`. The default is `:GET`.
- `version` (number)
- The version of the HTTP protocol: typically `1.0` or `1.1`. The default is `1.1`.
- `content` (string, alist or pathname)
- The body of the request.
- `headers` (alist)
- The headers of the request. If the value of a pair is `NIL`, the header won't be sent. You can overwrite the default headers (Host, User-Agent and Accept) by this with the same header name.
- `basic-auth` (cons of username and password)
- Username and password for basic authorization. This is a cons having username at car and password at cdr. (e.g. `'("foo" . "bar")`)
- `cookie-jar` (cookie-jar of [cl-cookie](https://github.com/fukamachi/cl-cookie))
- A cookie jar object.
- `connect-timeout` (fixnum)
- The seconds to timeout until the HTTP connection established. The default is `10`, the value of `*default-connect-timeout*`.
- `read-timeout` (fixnum)
- The seconds to timeout until the whole HTTP body read. The default is `10`, the value of `*default-read-timeout*`.
- `keep-alive` (boolean)
- A flag if the connection keep connected even after the HTTP request. The default is `T`.
- `use-connection-pool` (boolean)
- When combined with `:keep-alive t`, will internally cache the socket connection to web servers to avoid having to open new ones. This is compatible with `:want-stream t` (when you close the returned stream or it is garbage collected the connection will be returned to the pool). If you pass in a stream with `:stream` then the connection pool is not used (unless there is a redirect to a new web server). This is not supported when using the WINHTTP backend. The default is `T`.
- `max-redirects` (fixnum)
- The limit of redirections. The default is `5`. If the redirection exceeds the limit, functions return the last response (not raise a condition).
- `ssl-key-file`, `ssl-cert-file`, `ssl-key-password`
- for HTTPS connection
- `stream`
- The stream to write an HTTP request. This is a way to reuse a connection and commonly used with `:keep-alive T`. This allows the caller to do connection pooling, etc. It is easier to just use `:use-connection-pool t`, which is the default, and let the dexador internals take care of this for you (only supported for usocket backends).
- `verbose` (boolean)
- This option is for debugging. When `T`, it dumps the HTTP request headers.
- `force-binary` (boolean)
- A flag for suppressing auto-decoding of the response body.
<!-- - `force-string` -->
- `want-stream` (boolean)
- A flag to get the response body as a stream.
- `proxy` (string)
- for use proxy. defaults to the value of `dex:*default-proxy*` which defaults to the value of environment variables HTTPS_PROXY or HTTP_PROXY. Not supported on windows currently
- `insecure` (boolean)
- To bypass SSL certificate verification (use at your own risk). The default is `NIL`, the value of `*not-verify-ssl*`.
<!-- - `ca-path` -->
### \[Function\] request
```common-lisp
(dex:request uri &key (method get) (version 1.1) content headers
basic-auth cookie-jar (connect-timeout *default-connect-timeout*)
(read-timeout *default-read-timeout*) (keep-alive t) (use-connection-pool t)
(max-redirects 5) ssl-key-file ssl-cert-file ssl-key-password stream
(verbose *verbose*) force-binary force-string want-stream proxy
(insecure *not-verify-ssl*) ca-path)
;=> body
; status
; response-headers
; uri
; stream
```
Send an HTTP request to `uri`.
The `body` is an octet vector or a string if the `Content-Type` is `text/*`. If you always want it to return an octet vector, specify `:force-binary` as `T`.
The `status` is an integer which represents HTTP status code.
The `response-headers` is a hash table which represents HTTP response headers. Note that all hash keys are downcased like "content-type". If there's duplicate HTTP headers, those values are concatenated with a comma.
The `uri` is a [QURI](https://github.com/fukamachi/quri) object which represents the last URI Dexador requested.
The `stream` is a usocket stream to communicate with the HTTP server if the connection is still alive and can be reused. This value may be `NIL` if `:keep-alive` is `NIL` or the server closed the connection with `Connection: close` header or you are using `:use-connection-pool t` which handles re-using the connections for you.
This function signals `http-request-failed` when the HTTP status code is 4xx or 5xx.
### \[Function\] get
```common-lisp
(dex:get uri &key version headers basic-auth cookie-jar keep-alive
use-connection-pool connect-timeout read-timeout max-redirects
force-binary force-string want-stream ssl-key-file
ssl-cert-file ssl-key-password stream verbose proxy insecure
ca-path)
```
### \[Function\] post
```common-lisp
(dex:post uri &key version content headers basic-auth cookie-jar
keep-alive use-connection-pool connect-timeout read-timeout
force-binary force-string want-stream ssl-key-file
ssl-cert-file ssl-key-password stream verbose proxy insecure
ca-path)
```
### \[Function\] head
```common-lisp
(dex:head uri &key version headers basic-auth cookie-jar connect-timeout
read-timeout max-redirects ssl-key-file ssl-cert-file
ssl-key-password stream verbose proxy insecure ca-path)
```
### \[Function\] put
```common-lisp
(dex:put uri &key version content headers basic-auth cookie-jar
keep-alive use-connection-pool connect-timeout read-timeout
force-binary force-string want-stream ssl-key-file
ssl-cert-file ssl-key-password stream verbose proxy insecure
ca-path)
```
### \[Function\] patch
```common-lisp
(dex:patch uri &key version content headers basic-auth cookie-jar
keep-alive use-connection-pool connect-timeout read-timeout
force-binary force-string want-stream ssl-key-file
ssl-cert-file ssl-key-password stream verbose proxy insecure
ca-path)
```
### \[Function\] delete
```common-lisp
(dex:delete uri &key version headers basic-auth cookie-jar keep-alive
use-connection-pool connect-timeout read-timeout
force-binary force-string want-stream ssl-key-file
ssl-cert-file ssl-key-password stream verbose proxy insecure
ca-path)
```
### \[Function\] fetch
Send a GET request to `URI` and write the response body to the `DESTINATION`.
```common-lisp
(dex:fetch uri destination &key (if-exists error) verbose proxy insecure)
```
## Benchmark

* Server
* Sakura VPS 1GB
* nginx 1.2.7, KeepAlive On
* Client
* MacBook Pro OS X Yosemite (CPU: 3GHz Intel Core i7, Memory: 8GB)
* SBCL 1.2.9
* Downloads an HTML file (181 bytes).
### Drakma
```
(time (dotimes (i 30) (drakma:http-request "http://files.8arrow.org/181B.html")))
Evaluation took:
1.012 seconds of real time
0.174742 seconds of total run time (0.148141 user, 0.026601 system)
17.29% CPU
1,683 forms interpreted
500 lambdas converted
3,027,928,949 processor cycles
29,416,656 bytes consed
```
### Dexador
```
(time (dotimes (i 30) (dex:get "http://files.8arrow.org/181B.html")))
Evaluation took:
0.499 seconds of real time
0.028057 seconds of total run time (0.019234 user, 0.008823 system)
5.61% CPU
56 forms interpreted
16 lambdas converted
1,494,851,690 processor cycles
1,472,992 bytes consed
```
## See Also
* [fast-http](https://github.com/fukamachi/fast-http)
* [cl-cookie](https://github.com/fukamachi/cl-cookie)
* [QURI](https://github.com/fukamachi/quri)
## Author
* Eitaro Fukamachi (e.arrows@gmail.com)
## Copyright
Copyright (c) 2015 Eitaro Fukamachi (e.arrows@gmail.com)
## License
Licensed under the MIT License.
|