1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
|
======= ================================
SEP 11
Title Process models for Scrapy
Created 2009-11-16
Status Partially implemented - see #168
======= ================================
==================================
SEP-011: Process models for Scrapy
==================================
There is an interest of supporting different process models for Scrapy, mainly
to help prevent memory leaks which affect running all spiders in the same
process.
By running each spider on a separate process (or pool of processes) we'll be
able to "recycle" process when they exceed a maximum amount of memory.
Supported process models
========================
The user could choose between different process models:
1. in process (only method supported so far)
2. pooled processes (a predefined pool of N processes, which could run more than one spider each)
3. separate processes (one process per spider)
Using different processes would increase reliability at the cost of performance.
Another ideas to consider
=========================
- configuring pipeline process models - so that we can have a process exclusive
for running pipelines
- support writing spidersr in different languages when we don't use an in
process model
|
