File: README.md

package info (click to toggle)
plowshare 2.1.6-1~bpo8+1
  • links: PTS, VCS
  • area: main
  • in suites: jessie-backports
  • size: 712 kB
  • sloc: sh: 5,030; makefile: 92; sed: 13
file content (588 lines) | stat: -rw-r--r-- 17,034 bytes parent folder | download | duplicates (2)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
# Plowshare

## Introduction

Plowshare is a set of command-line tools (written entirely in Bash shell script) designed for managing file-sharing websites (aka Hosters).

Plowshare is divided into 6 scripts:
- *plowdown*, for downloading URLs
- *plowup*, for uploading files
- *plowdel*, for deleting remote files
- *plowlist*, for listing remote shared folders
- *plowprobe*, for retrieving information of downloading URLs
- *plowmod*, easy management (installation or update) of Plowshare modules

Plowshare itself doesn't support any websites (named *module*). It's just the core engine.
Concerning modules, few are available separately and must be installed in user directory (see [below](#install)).

### Features

- Small footprint (few shell scripts). No java, no python. Run fast on embedded devices.
- Few dependencies and portable. [Bash](https://www.gnu.org/software/bash/) and [cURL](http://curl.haxx.se/) are enough for most hosters.
- Modules (hoster plugins) are simple to write using [Plowshare API](#implement-your-own-modules).
- Support for automatic online captcha solver services.
- Cache mechanism: hoster session or cookie reuse (to avoid relogin).

## Install

See `INSTALL` file for details.

## Usage examples

All scripts share the same verbose options:
- `-v0` (be quiet, alias: `-q`)
- `-v1` (errors only)
- `-v2` (infos message; default)
- `-v3` (show all messages)
- `-v4` (show all messages, HTML pages and cookies, use this for bug report)

Getting help:
- `--help`
- `--longhelp` (additionally prints modules command-line options)

Exhaustive documentation is available in manpages.

All examples below are using fake links.

### Plowdown

Download a file from Rapidshare:

```sh
$ plowdown http://www.rapidshare.com/files/86545320/Tux-Trainer.rar
```

Download a file from Rapidgator using an account (free or premium):

```sh
$ plowdown -a 'myuser:mypassword' http://rapidgator.net/file/49b1b874
```

**Note**: `:` is the separator character for login and password.
Enclosing string using single quotes ensure against shell expansion.

Download a list of links (one link per line):

```sh
$ cat file_with_links.txt
# This is a comment
http://depositfiles.com/files/abcdefghi
http://www.rapidshare.com/files/86545320/Tux-Trainer_25-01-2008.rar
$ plowdown file_with_links.txt
```

Download a list of links (one link per line) commenting out (with `#`) those successfully downloaded:

```sh
$ plowdown -m file_with_links.txt
```

**Note**: Files are consecutively downloaded in the order read from input text file.

Download a file from Oron with Death by Captcha service:

```sh
$ plowdown --deathbycaptcha='user:pass' http://oron.com/dw726z0ohky5
```

Download a file from Rapidshare with a proxy (cURL supports `http_proxy` and `https_proxy` environment variables, default port is `3128`):

```sh
$ export http_proxy=http://xxx.xxx.xxx.xxx:80
$ plowdown http://www.rapidshare.com/files/86545320/Tux-Trainer.rar
```

Download a file with limiting the download speed (in bytes per second):

```sh
$ plowdown --max-rate 900K http://www.rapidshare.com/files/86545320/Tux-Trainer.rar
```

**Note**: Accepted prefixes are: `k`, `K`, `Ki`, `M`, `m`, `Mi`.

Download a file from Rapidshare (like firefox: append `.part` suffix to filename while file is being downloaded):

```sh
$ plowdown --temp-rename http://www.rapidshare.com/files/86545320/Tux-Trainer.rar
```

Download a password-protected file from Mediafire:

```sh
$ plowdown -p 'somepassword' http://www.mediafire.com/?mt0egmhietj60iy
```

**Note**: If you don't specify password and link requests it, you'll be prompted (stdin) for one.

Avoid never-ending downloads: limit the number of tries (for captchas) and wait delays for each link:

```sh
$ plowdown --max-retries=4 --timeout=3600 my_big_list_file.txt
```

Retrieve final url (don't use *plowdown* for download):

```sh
$ plowdown -q --skip-final --printf %d http://oron.com/dw726z0ohky5 | xargs wget
```

**Note**: This will not work if final url (remote host) requires a cookie. For anonynous users,
generated link has limited access in time and you can usually download file only once.

### Plowup

Upload a single file anonymously to BayFiles:

```sh
$ plowup bayfiles /tmp/foo.bar
```

Upload a bunch of files anonymously to 2Shared (doesn't recurse subdirectories):

```sh
$ plowup 2shared /path/myphotos/*
```

**Note**: `*` is a [wildcard character](http://en.wikipedia.org/wiki/Glob_%28programming%29) expanded by Bash interpreter.

Upload a file to Rapidshare with an account (premium or free)

```sh
$ plowup -a 'myuser:mypassword' rapidshare /path/xxx
```

Upload a file to Mirrorcreator changing remote filename:

```sh
$ plowup mirrorcreator /path/myfile.txt:anothername.txt
```

**Note**: `:` is the separator character for local filename and remote filename.

Upload a file to MegaShares (anonymously) and set description:

```sh
$ plowup -d "Important document" megashares /path/myfile.tex
```

Upload a file to Oron anonymously with a proxy:

```sh
$ export http_proxy=http://xxx.xxx.xxx.xxx:80
$ export https_proxy=http://xxx.xxx.xxx.xxx:80
$ plowup oron /path/myfile.txt
```

Abort slow upload (if rate is below limit during 30 seconds):

```sh
$ plowup --min-rate 100k mediafire /path/bigfile.zip
```

Modify remote filenames (example: `foobar.rar` gives `foobar-PLOW.rar`):

```sh
$ plowup --name='%g-PLOW.%x' mirrorcreator *.rar
```

**Remark**: cURL is not capable of uploading files containing a comma `,` in their filename, but *plowup* will
temporarily create a symlink for you.

Use cache over sessions to avoid multiple logins:

```sh
$ plowup --cache=shared -a 'user:pasword' 1fichier file1.zip
$ plowup --cache=shared 1fichier file2.zip
```

On first command line, login stage will be performed and session (token or cookie) will be saved in
`~/.config/plowshare/storage/module-name.txt`.
On second command line, *plowup* will reuse the data stored to bypass login step. You don't have to specify credentials.

**Note**: Only few hosters currently support cache mechanism. Have a look to
[Plowshare legacy modules matrix](https://github.com/mcrapet/plowshare-modules-legacy) for more information.

Custom results, print upload time, link and filename in HTML format:

```sh
$ plowup 1fichier -v0 --printf '<li>%T: <a href="%u">%f</a>%n' 5MiB.bin 10MB.bin
<li>11:12:42: <a href="https://1fichier.com/?52jwehc851">5MiB.bin</a>
<li>11:12:46: <a href="https://1fichier.com/?bn1jdvtpqi">10MB.bin</a>
```

### Plowdel

Delete a file from MegaShares (*delete link* required):

```sh
$ plowdel http://d01.megashares.com/?dl=6EUeDtS
```

Delete files (deletes are successive, not parallel):

```sh
$ plowdel http://d01.megashares.com/?dl=6EUeDtS http://depositfiles.com/rmv/1643181821669253
```

Delete a file from Rapidshare (account is required):

```sh
$ plowdel -a myuser:mypassword http://rapidshare.com/files/293672730/foo.rar
```

### Plowlist

List links contained in a shared folder link and download them all:

```sh
$ plowlist http://www.mediafire.com/?qouncpzfe74s9 > links.txt
$ plowdown -m links.txt
```

List two shared folders (first link is processed, then the second one, this is not parallel):

```sh
$ plowlist http://www.mediafire.com/?qouncpzfe74s9 http://www.sendspace.com/folder/5njdw7
```

**Remark**: Some hosters are handling tree folders, you must specify `-R`/`--recursive` command-line switch to *plowlist* for enabing recursive lisiting.

List some Sendspace web folder. Render results for vBulletin *BB* syntax:

```sh
$ plowlist --printf '[url=%u]%f[/url]%n' http://www.sendspace.com/folder/5njdw7
```

List links contained in a dummy web page. Render results as HTML list:

```sh
$ plowlist --fallback --printf '<li><a href="%u">%u</a></li>%n' \
      http://en.wikipedia.org/wiki/SI_prefix
```

### Plowprobe

Gather public information (filename, file size, file hash, ...) about a link.
No captcha solving is requested.

Filter alive links in a text file:

```sh
$ plowprobe file_with_links.txt > file_with_active_links.txt
```

Custom results as shell format, print links information (filename and size):

```sh
$ plowprobe --printf '#%f (%s)%n%u%n'  http://myhoster.com/files/5njdw7
#foo-bar.rar (134217728)
http://myhoster.com/files/5njdw7
```

Custom results as [JSON](http://json.org/) format, print links information (filename and size):

```sh
$ plowprobe --printf '{"url":"%U","size":%s}%n' http://myhoster.com/files/5njdw7
{"url":"http:\/\/myhoster.com\/files\/5njdw7","size":134217728}
```

Custom results: print *primary* url (if supported by hosters and implemented by module):

```sh
$ plowprobe --printf='%v%n' http://a5ts8yt25l.1fichier.com/
https://1fichier.com/?a5ts8yt25l
```

Use `-` argument to read from stdin:

```sh
$ plowlist http://pastebin.com/1d82F5sd | plowprobe - > filtered_list.txt
```

## Configuration file

Plowshare looks for `~/.config/plowshare/plowshare.conf` or `/etc/plowshare.conf` files.
Options given at command line can be stored in the file.

Example:
```ini
###
### Plowshare configuration file
### Line syntax: token = value
###

[General]
interface = eth1
captchabhood=cbhuser:cbhpass

rapidshare/a = matt:4deadbeef
mediafire/a = "matt:4 dead beef "
freakshare/b=plowshare:xxxxx

[Plowdown]
timeout=3600
#antigate=49b1b8740e4b51cf51838975de9e1c31

[Plowup]
max-retries=2
mirrorcreator/auth-free = foo:bar
mirrorcreator/count = 5

[Plowlist]
verbose = 3

#[Plowprobe]
```

Notes:
- Blank lines are ignored, and whitespace before and after a token or value is ignored, although a value can contain whitespace within.
- Lines which begin with a `#` are considered comments and ignored.
- Double quoting value is optional.
- Valid configuration token names are long-option command-line arguments of Plowshare. Tokens are always lowercase. For modules options, tokens are prepended by module name and a slash character. For example: `rapidshare/auth` is equivalent to `rapidshare/a` (short-option are also possible here). Another example: `freakshare/b` is equivalent to `freakshare/auth-free`.
- Options in general section prevail over `PlowXXX` section. Options given on the command line prevail over configuration file options.

You can disable usage of Plowshare config file by providing `--no-plowsharerc` command-line switch. You can also specify a custom config file using `--plowsharerc` switch.

## Use your own captcha solver

It is possible providing *plowdown* or *plowup* with `--captchaprogram` command-line switch followed by a path to a script or executable.

### Script exit status

- `0`: solving success. Captcha Word(s) must be echo'ed (on stdout).
- `$ERR_NOMODULE`: external solver is not able to solve requested captcha. Let *plowdown* continue solving it normally (will consider `--captchamethod` if specified).
- `$ERR_FATAL`: external solver failed.
- `$ERR_CAPTCHA`: external solver failed. Note: this exit code is eligible with retry policy (`-r`/`--max-retries`).

### Examples

Understanding example:

```sh
#!/bin/bash
# $1: module name
# $2: path to image
# $3: captcha type. For example: "recaptcha", "solvemedia", "digit-4".

declare -r ERR_NOMODULE=2
declare -r ERR_CAPTCHA=7

# We only support uploadhero, otherwise tell Plowshare to solve on its own
if [ "$1" != 'uploadhero' ]; then
    exit $ERR_NOMODULE
fi

# You can print message to stderr
echo "Module name: $1" >&2
echo "Image: $2" >&2

# Use stdout to send decoding result
echo "5ed1"
exit 0
```

Captcha emailing example:

```sh
#!/bin/bash
#
# Sends an email with image as attachment.
# Requires heirloom-mailx and not bsd-mailx.
#
# Here is my ~/.mailrc:
#
# account gmail {
# set from="My Name <xyz@gmail.com>"
# set smtp-use-starttls
# ssl-verify=ignore
# set smtp=smtp://smtp.gmail.com:587
# set smtp-auth=login
# set smtp-auth-user=xyz@gmail.com
# set smtp-auth-password="xxx"
# }

declare -r ERR_FATAL=1
declare -r MAILTO='xyz@gmail.com'

# Image file expected
if [ ! -f "$2" ]; then
    exit $ERR_FATAL
fi

BODY="Hi!

Here is a captcha to solve; it comes from $1."

mailx -A gmail -s 'Plowshare sends you an image!' \
    -a "$2" "$MAILTO" >/dev/null <<< "$BODY" || {
        echo 'mailx fatal error, abort' >&2;
        exit $ERR_FATAL;
}

echo 'Please check your email account and enter captcha solution here:' >&2
IFS= read -r
echo "$REPLY"
exit 0
```

Captcha FTP example:

```sh
#!/bin/bash
#
# Uploads the image to an FTP server in the LAN. If the server is not available
# (i.e. my computer is not running) or no CAPTCHA solution is entered for
# 15 minutes (i.e. I am occupied), let Plowshare try to handle the CAPTCHA.

declare -r MODULE=$1
declare -r FILE=$2
declare -r HINT=$3
declare -r DEST='192.168.1.3'
declare -r ERR_NOMODULE=2

# Prepend the used module to the image file name
curl --connect-timeout 30 -T "$FILE" --silent "ftp://$DEST/${MODULE}__${FILE##*/}" || \
    exit $ERR_NOMODULE

echo "Captcha from module '$MODULE' with hint '$HINT'" >&2
read -r -t 900 -p 'Enter code: ' RESPONSE || exit $ERR_NOMODULE
echo "$RESPONSE"

exit 0
```

Database using image hash as key:

```sh
#!/bin/sh
#
# Back to February 2009, Megaupload was using 4-character rotation captchas.
# For example:
# $ sqlite3 captchas.db
# sqlite> CREATE TABLE mu (md5sum text unique not null, captcha text not null);
# sqlite> INSERT INTO mu VALUES('fd3b2381269d702eccc509b8849e5b0d', 'RHD8');
# sqlite> INSERT INTO mu VALUES('04761dbbe2a45ca6720755bc324dd19c', 'EFC8');
# sqlite> .exit

if [ "$1" = megaupload ]; then
  DB="$HOME/captchas.db"
  MD5=$(md5sum -b "$1" | cut -c-32)
  if VAL=$(sqlite3 "$DB" "SELECT captcha FROM mu WHERE md5sum=\"$MD5\""); then
    echo "$VAL"
    exit 0
  fi
fi
exit 2
```

## Plowdown advanced use

### Hooks

It is possible to execute your own script before and after call to module download function.
Related command-line switches are `--run-before` and `--run-after`.

Possible usage:
- (before) Check (with *plowprobe*) if a file has already been downloaded (same filename, same file size/hash)
- (before) Inject your own cookie
- (after) Unrar archives
- (after) Add `--skip-final` command-line switch and do your custom final link download


Example 1: Skip all links coming from HotFile hoster

```sh
$ cat drophf.sh
#!/bin/bash
# $1: module name
# $2: download URL
# $3: cookie (empty) file given to download module function
# You can print messages to stderr. stdout will be trashed
declare -r ERR_NOMODULE=2
if [ "$1" = 'hotfile' ]; then
    echo "===[Pre-processing script skipping $2]===" >&2
    exit $ERR_NOMODULE
fi
exit 0

$ plowdown --run-before ./drophf.sh -m list_of_links.txt
```

Example 2: Use `wget` for final download (with possible required cookie file for last download)

```sh
$ cat finalwget.sh
#!/bin/bash
# $1: module name
# $2: download URL
# $3: cookie file fulfilled by download module function
# $4: final download URL
# $5: final filename (no path: --output-directory is ignored)
# You can print messages to stderr. stdout will be trashed
echo "===[Post-processing script for $1]===" >&2
echo "Temporary cookie file: $3" >&2
wget --no-verbose --load-cookies $3 -O $5 $4

$ plowdown --skip-final --run-after ./finalwget.sh \
    http://www.mediafire.com/?k10t0egmhi23f
```

Example 3: Use multiple connections for final download (usually only for premium account users)

```sh
$ cat finalaria.sh
#!/bin/bash
aria2c -x2 $4

$ plowdown -a user:password --skip-final --run-after ./finalaria.sh \
    http://depositfiles.com/files/fv2u9xqya
```

## Miscellaneous

### Additional cURL settings

For all network operations, Plowshare is relying on cURL. You can tweak some advanced settings if necessary.

For example (enforce IPv6):
```sh
$ echo 'ipv6' >>~/.curlrc
```

Use Plowshare with a SOCKS proxy:
```sh
$ ssh -f -N -D localhost:3128 user@my.proxy.machine.org
$ echo 'socks5=localhost:3128' >>~/.curlrc
```

**Note**: As Plowshare is dealing with verbose, be sure (if present) to have these cURL's options commented:
```
#verbose
#silent
#show-error
```

### Known limitations

For historical reasons or design choices, there are several known limitations to Plowshare.

1. You cannot enter through command-line several credentials for different hosts.
   It's because the modules option `-a`, `--auth`, `-b` or `--auth-free` have the same switch name.
   But you can do it with the configuration file.
2. Same restriction for passwords (also a module option). Only one password can be defined with `-p`, `--link-password` switch name.

### Implement your own modules

Plowshare exports a set of API to help text and HTML processing.
It is designed to be as simple as possible to develop new modules.
A module must be written in shell with portability in mind; one module matches one website.

- [New module documentation](https://github.com/mcrapet/plowshare/wiki/Modules)
- [API list](https://github.com/mcrapet/plowshare/wiki/API)

A common approach is to read existing modules source code.

## License

Plowshare is made available publicly under the GNU GPLv3 License.
Full license text is available in COPYING file.