1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
|
Cluster
=======
.. py:currentmodule:: django_q
Django Q2 uses Python's multiprocessing module to manage a pool of workers that will handle your tasks.
Start your cluster using Django's ``manage.py`` command::
$ python manage.py qcluster
You should see the cluster starting ::
10:57:40 [Q] INFO Q Cluster freddie-uncle-twenty-ten starting.
10:57:40 [Q] INFO Process-ede257774c4444c980ab479f10947acc ready for work at 31784
10:57:40 [Q] INFO Process-ed580482da3f42968230baa2e4253e42 ready for work at 31785
10:57:40 [Q] INFO Process-8a370dc2bc1d49aa9864e517c9895f74 ready for work at 31786
10:57:40 [Q] INFO Process-74912f9844264d1397c6e54476b530c0 ready for work at 31787
10:57:40 [Q] INFO Process-b00edb26c6074a6189e5696c60aeb35b ready for work at 31788
10:57:40 [Q] INFO Process-b0862965db04479f9784a26639ee51e0 ready for work at 31789
10:57:40 [Q] INFO Process-7e8abbb8ca2d4d9bb20a937dd5e2872b ready for work at 31790
10:57:40 [Q] INFO Process-b0862965db04479f9784a26639ee51e0 ready for work at 31791
10:57:40 [Q] INFO Process-67fa9461ac034736a766cd813f617e62 monitoring at 31792
10:57:40 [Q] INFO Process-eac052c646b2459797cee98bdb84c85d guarding cluster at 31783
10:57:40 [Q] INFO Process-5d98deb19b1e4b2da2ef1e5bd6824f75 pushing tasks at 31793
10:57:40 [Q] INFO Q Cluster freddie-uncle-twenty-ten running.
Stopping the cluster with ctrl-c or either the ``SIGTERM`` and ``SIGKILL`` signals, will initiate the :ref:`stop_procedure`::
16:44:12 [Q] INFO Q Cluster freddie-uncle-twenty-ten stopping.
16:44:12 [Q] INFO Process-eac052c646b2459797cee98bdb84c85d stopping cluster processes
16:44:13 [Q] INFO Process-5d98deb19b1e4b2da2ef1e5bd6824f75 stopped pushing tasks
16:44:13 [Q] INFO Process-b0862965db04479f9784a26639ee51e0 stopped doing work
16:44:13 [Q] INFO Process-7e8abbb8ca2d4d9bb20a937dd5e2872b stopped doing work
16:44:13 [Q] INFO Process-b0862965db04479f9784a26639ee51e0 stopped doing work
16:44:13 [Q] INFO Process-b00edb26c6074a6189e5696c60aeb35b stopped doing work
16:44:13 [Q] INFO Process-74912f9844264d1397c6e54476b530c0 stopped doing work
16:44:13 [Q] INFO Process-8a370dc2bc1d49aa9864e517c9895f74 stopped doing work
16:44:13 [Q] INFO Process-ed580482da3f42968230baa2e4253e42 stopped doing work
16:44:13 [Q] INFO Process-ede257774c4444c980ab479f10947acc stopped doing work
16:44:14 [Q] INFO Process-67fa9461ac034736a766cd813f617e62 stopped monitoring results
16:44:15 [Q] INFO Q Cluster freddie-uncle-twenty-ten has stopped.
The number of workers, optional timeouts, recycles and cpu_affinity can be controlled via the :doc:`configure` settings.
Multiple Clusters
-----------------
You can have multiple clusters on multiple machines, working on the same queue as long as:
- They connect to the same :doc:`broker<brokers>`.
- They use the same cluster name. See :doc:`configure`
- They share the same ``SECRET_KEY`` for Django.
.. _multiple-queues
Multiple Queues
-----------------
You can have multiple queues in one Django site, and use multiple cluster to work on each queue.
Different queues are identified by different queue names which are also cluster names.
To run an alternate cluster, e.g. to work on the 'long' queue, start your cluster with command::
# On Linux
$ Q_CLUSTER_NAME=long python manage.py qcluster
# On Windows
$ python manage.py qcluster --name long
You can set different Q_CLUSTER options for alternative clusters, such as 'timeout', 'queue_limit'
and any other options which are valid in :doc:`configure`. See :ref:`alt-clusters`.
.. note::
To use multiple queue, use the keyword argument `cluster` in async_task() and schedule():
* if `cluster` is not set (the default), async_task() and schedule() will be handled by the default cluster;
* if `cluster` is set, only clusters with matching cluster name will run the task or do the schedule.
Using a Procfile
----------------
If you host on `Heroku <https://heroku.com>`__ or you are using `Honcho <https://github.com/nickstenning/honcho>`__ you can start the cluster from a :file:`Procfile` with an entry like this::
worker: python manage.py qcluster
Process managers
----------------
While you certainly can run a Django Q2 with a process manager like `Supervisor <http://supervisord.org/>`__ or `Circus <https://circus.readthedocs.org/en/latest/>`__ it is not strictly necessary.
The cluster has an internal sentinel that checks the health of all the processes and recycles or reincarnates according to your settings or in case of unexpected crashes.
Because of the multiprocessing daemonic nature of the cluster, it is impossible for a process manager to determine the clusters health and resource usage.
An example :file:`circus.ini` ::
[circus]
check_delay = 5
endpoint = tcp://127.0.0.1:5555
pubsub_endpoint = tcp://127.0.0.1:5556
stats_endpoint = tcp://127.0.0.1:5557
[watcher:django_q]
cmd = python manage.py qcluster
numprocesses = 1
copy_env = True
Note that we only start one process. It is not a good idea to run multiple instances of the cluster in the same environment since this does nothing to increase performance and in all likelihood will diminish it.
Control your cluster using the ``workers``, ``recycle`` and ``timeout`` settings in your :doc:`configure`
An example :file:`supervisor.conf` ::
[program:django-q]
command = python manage.py qcluster
stopasgroup = true
Supervisor's ``stopasgroup`` will ensure that the single process doesn't leave orphan process on stop or restart.
Reference
---------
.. py:class:: Cluster
.. py:method:: start
Spawns a cluster and then returns
.. py:method:: stop
Initiates :ref:`stop_procedure` and waits for it to finish.
.. py:method:: stat
returns a :class:`Stat` object with the current cluster status.
.. py:attribute:: pid
The cluster process id.
.. py:attribute:: host
The current hostname
.. py:attribute:: sentinel
returns the :class:`multiprocessing.Process` containing the :ref:`sentinel`.
.. py:attribute:: timeout
The clusters timeout setting in seconds
.. py:attribute:: start_event
A :class:`multiprocessing.Event` indicating if the :ref:`sentinel` has finished starting the cluster
.. py:attribute:: stop_event
A :class:`multiprocessing.Event` used to instruct the :ref:`sentinel` to initiate the :ref:`stop_procedure`
.. py:attribute:: is_starting
Bool. Indicating that the cluster is busy starting up
.. py:attribute:: is_running
Bool. Tells you if the cluster is up and running.
.. py:attribute:: is_stopping
Bool. Shows that the stop procedure has been started.
.. py:attribute:: has_stopped
Bool. Tells you if the cluster has finished the stop procedure
|
