1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
:class:`~tornado.queues.Queue` example - a concurrent web spider
================================================================
.. currentmodule:: tornado.queues
Tornado's `tornado.queues` module (and the very similar ``Queue`` classes in
`asyncio`) implements an asynchronous producer / consumer pattern for
coroutines, analogous to the pattern implemented for threads by the Python
standard library's `queue` module.
A coroutine that yields `Queue.get` pauses until there is an item in the queue.
If the queue has a maximum size set, a coroutine that yields `Queue.put` pauses
until there is room for another item.
A `~Queue` maintains a count of unfinished tasks, which begins at zero.
`~Queue.put` increments the count; `~Queue.task_done` decrements it.
In the web-spider example here, the queue begins containing only base_url. When
a worker fetches a page it parses the links and puts new ones in the queue,
then calls `~Queue.task_done` to decrement the counter once. Eventually, a
worker fetches a page whose URLs have all been seen before, and there is also
no work left in the queue. Thus that worker's call to `~Queue.task_done`
decrements the counter to zero. The main coroutine, which is waiting for
`~Queue.join`, is unpaused and finishes.
.. literalinclude:: ../../demos/webspider/webspider.py
|
