1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
|
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<title>GAP (Utils) - Chapter 9: Web Downloads</title>
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
<meta name="generator" content="GAPDoc2HTML" />
<link rel="stylesheet" type="text/css" href="manual.css" />
<script src="manual.js" type="text/javascript"></script>
<script type="text/javascript">overwriteStyle();</script>
</head>
<body class="chap9" onload="jscontent()">
<div class="chlinktop"><span class="chlink1">Goto Chapter: </span><a href="chap0.html">Top</a> <a href="chap1.html">1</a> <a href="chap2.html">2</a> <a href="chap3.html">3</a> <a href="chap4.html">4</a> <a href="chap5.html">5</a> <a href="chap6.html">6</a> <a href="chap7.html">7</a> <a href="chap8.html">8</a> <a href="chap9.html">9</a> <a href="chap10.html">10</a> <a href="chap11.html">11</a> <a href="chap12.html">12</a> <a href="chapBib.html">Bib</a> <a href="chapInd.html">Ind</a> </div>
<div class="chlinkprevnexttop"> <a href="chap0.html">[Top of Book]</a> <a href="chap0.html#contents">[Contents]</a> <a href="chap8.html">[Previous Chapter]</a> <a href="chap10.html">[Next Chapter]</a> </div>
<p id="mathjaxlink" class="pcenter"><a href="chap9_mj.html">[MathJax on]</a></p>
<p><a id="X815B0C4B7EBE6E1E" name="X815B0C4B7EBE6E1E"></a></p>
<div class="ChapSects"><a href="chap9.html#X815B0C4B7EBE6E1E">9 <span class="Heading">Web Downloads</span></a>
<div class="ContSect"><span class="tocline"><span class="nocss"> </span><a href="chap9.html#X8758CB7F79EFB6ED">9.1 <span class="Heading">Functions for downloading files from the web</span></a>
</span>
<div class="ContSSBlock">
<span class="ContSS"><br /><span class="nocss"> </span><a href="chap9.html#X7A7438AE8448635E">9.1-1 Download</a></span>
<span class="ContSS"><br /><span class="nocss"> </span><a href="chap9.html#X85182BA486E3C2AA">9.1-2 <span class="Heading">User preference <code class="code">DownloadVerifyCertificate</code></span></a>
</span>
<span class="ContSS"><br /><span class="nocss"> </span><a href="chap9.html#X79E10E5B83EF929F">9.1-3 <span class="Heading">User preference <code class="code">DownloadMaxTime</code></span></a>
</span>
</div></div>
</div>
<h3>9 <span class="Heading">Web Downloads</span></h3>
<p>The <code class="code">Download</code> operation has been written by Thomas Breuer, incorporating a number of suggestions from Max Horn, for version 0.77 of <strong class="pkg">Utils</strong>. It implements downloading a file from within <strong class="pkg">GAP</strong>. It can use the <strong class="pkg">IO</strong> or <strong class="pkg">curlInterface</strong> packages, or <em>wget</em> or <em>curl</em>, if installed, and it can be extended with other download methods quite easily. It is envisaged that, once other packages have started to use it, and any problems have been addressed, that the functions will be transferred to the main <strong class="pkg">GAP</strong> library.</p>
<p><a id="X8758CB7F79EFB6ED" name="X8758CB7F79EFB6ED"></a></p>
<h4>9.1 <span class="Heading">Functions for downloading files from the web</span></h4>
<p><a id="X7A7438AE8448635E" name="X7A7438AE8448635E"></a></p>
<h5>9.1-1 Download</h5>
<div class="func"><table class="func" width="100%"><tr><td class="tdleft"><code class="func">‣ Download</code>( <var class="Arg">url</var>[, <var class="Arg">opt</var>] )</td><td class="tdright">( function )</td></tr></table></div>
<p>This function downloads the file with the web address <var class="Arg">url</var>, which must be a string.</p>
<p>The result is a record which has at least the component <code class="code">success</code>, with value <code class="keyw">true</code> if the download was successful and <code class="keyw">false</code> otherwise. In the former case, the component <code class="code">result</code> is bound, whose value is a string that contains the contents of the downloaded file. In the latter case, the component <code class="code">error</code> is bound, whose value is a string that describes the problem.</p>
<p>The function calls the methods stored in the global list <code class="code">Download_Methods</code> until one of them is successful. Currently there are methods based on the <strong class="pkg">GAP</strong> functions <code class="func">DownloadURL</code> (<a href="../../../pkg/curlinterface/doc/chap1.html#X7827F7DE85113D7E"><span class="RefLink">curl: DownloadURL</span></a>) and <code class="func">SingleHTTPRequest</code> (<a href="../../../pkg/io/doc/chap7.html#X7DBCB0B4801E4D6D"><span class="RefLink">IO: SingleHTTPRequest</span></a>), and methods based on the external programs <code class="code">wget</code> and <code class="code">curl</code>.</p>
<p>An optional record <var class="Arg">opt</var> can be given. The following components are supported.</p>
<dl>
<dt><strong class="Mark"><code class="code">maxTime</code></strong></dt>
<dd><p>If this component is bound then its value must be a nonnegative integer <span class="SimpleMath">n</span>, meaning that the function gives up after <span class="SimpleMath">n</span> seconds.</p>
<p>A zero value of <span class="SimpleMath">n</span> means that no timeout is set, the method will never give up in this case.</p>
<p>The default for <span class="SimpleMath">n</span> is given by the value of the user preference <code class="code">DownloadMaxTime</code> (see <a href="chap9.html#X79E10E5B83EF929F"><span class="RefLink">9.1-3</span></a>).</p>
</dd>
<dt><strong class="Mark"><code class="code">target</code></strong></dt>
<dd><p>If this component is bound then its value must be a string that is a local filename, and the function writes the downloaded contents to this file; the returned record does not have a <code class="code">result</code> component in this case.</p>
</dd>
<dt><strong class="Mark"><code class="code">verifyCert</code></strong></dt>
<dd><p>If this component is bound and has the value <code class="keyw">false</code> then those download methods that are based on <code class="code">curl</code> or <code class="code">wget</code> will omit the check of the server's certificate.</p>
<p>The same effect is achieved for all <code class="func">Download</code> calls by setting the user preference <code class="code">DownloadVerifyCertificate</code> (see <a href="chap9.html#X85182BA486E3C2AA"><span class="RefLink">9.1-2</span></a>) to <code class="keyw">false</code> and omitting the <code class="code">verifyCert</code> component from <var class="Arg">opt</var>.</p>
</dd>
</dl>
<div class="example"><pre>
<span class="GAPprompt">gap></span> <span class="GAPinput">url:= "https://www.gap-system.org/index.html";;</span>
<span class="GAPprompt">gap></span> <span class="GAPinput">res1:= Download( url );;</span>
<span class="GAPprompt">gap></span> <span class="GAPinput">res1.success;</span>
true
<span class="GAPprompt">gap></span> <span class="GAPinput">IsBound( res1.result ) and IsString( res1.result );</span>
true
<span class="GAPprompt">gap></span> <span class="GAPinput">res2:= Download( Concatenation( url, "xxx" ) );;</span>
<span class="GAPprompt">gap></span> <span class="GAPinput">res2.success;</span>
false
<span class="GAPprompt">gap></span> <span class="GAPinput">IsBound( res2.error ) and IsString( res2.error );</span>
true
</pre></div>
<p><a id="X85182BA486E3C2AA" name="X85182BA486E3C2AA"></a></p>
<h5>9.1-2 <span class="Heading">User preference <code class="code">DownloadVerifyCertificate</code></span></h5>
<p>The value <code class="keyw">true</code> (the default) means that the server's certificate is checked in calls of <code class="func">Download</code> (<a href="chap9.html#X7A7438AE8448635E"><span class="RefLink">9.1-1</span></a>), such that nothing gets downloaded if the certificate is invalid.</p>
<p>If the value is <code class="keyw">false</code> then download methods are supposed to omit the check of the server's certificate (this may not be supported by all download methods).</p>
<p>One can set the value of the preference to be <code class="code">val</code> via <code class="func">SetUserPreference</code> (<a href="../../../doc/ref/chap3.html#X7B0AD104839B6C3C"><span class="RefLink">Reference: SetUserPreference</span></a>), by calling <code class="code">SetUserPreference( "utils", "DownloadVerifyCertificate", val )</code>, and access the current value via <code class="func">UserPreference</code> (<a href="../../../doc/ref/chap3.html#X7B0AD104839B6C3C"><span class="RefLink">Reference: UserPreference</span></a>), by calling <code class="code">UserPreference( "utils", "DownloadVerifyCertificate" )</code>.</p>
<p>We recommend leaving this preference at its default value <code class="keyw">true</code>. Sometimes it can be necessary to change it, e.g. to work around issues with old operating systems which may not be able to correctly verify new certificates. In general it is better to update such a system, but if that is not an option, then disabling certificate checks may be a good last resort.</p>
<p><a id="X79E10E5B83EF929F" name="X79E10E5B83EF929F"></a></p>
<h5>9.1-3 <span class="Heading">User preference <code class="code">DownloadMaxTime</code></span></h5>
<p>The value <code class="code">0</code> (the default) means that no timeout is set in calls of <code class="func">Download</code> (<a href="chap9.html#X7A7438AE8448635E"><span class="RefLink">9.1-1</span></a>). If the value is a positive integer <span class="SimpleMath">n</span> then those download methods that support a timeout will give up after <span class="SimpleMath">n</span> seconds.</p>
<p>One can set the value of the preference to be <code class="code">val</code> via <code class="func">SetUserPreference</code> (<a href="../../../doc/ref/chap3.html#X7B0AD104839B6C3C"><span class="RefLink">Reference: SetUserPreference</span></a>), by calling <code class="code">SetUserPreference( "utils", "DownloadMaxTime", val )</code>, and access the current value via <code class="func">UserPreference</code> (<a href="../../../doc/ref/chap3.html#X7B0AD104839B6C3C"><span class="RefLink">Reference: UserPreference</span></a>), by calling <code class="code">UserPreference( "utils", "DownloadMaxTime" )</code>.</p>
<div class="chlinkprevnextbot"> <a href="chap0.html">[Top of Book]</a> <a href="chap0.html#contents">[Contents]</a> <a href="chap8.html">[Previous Chapter]</a> <a href="chap10.html">[Next Chapter]</a> </div>
<div class="chlinkbot"><span class="chlink1">Goto Chapter: </span><a href="chap0.html">Top</a> <a href="chap1.html">1</a> <a href="chap2.html">2</a> <a href="chap3.html">3</a> <a href="chap4.html">4</a> <a href="chap5.html">5</a> <a href="chap6.html">6</a> <a href="chap7.html">7</a> <a href="chap8.html">8</a> <a href="chap9.html">9</a> <a href="chap10.html">10</a> <a href="chap11.html">11</a> <a href="chap12.html">12</a> <a href="chapBib.html">Bib</a> <a href="chapInd.html">Ind</a> </div>
<hr />
<p class="foot">generated by <a href="https://www.math.rwth-aachen.de/~Frank.Luebeck/GAPDoc">GAPDoc2HTML</a></p>
</body>
</html>
|
