File: README.rst

package info (click to toggle)
python-configargparse 0.11.0-1~bpo8%2B1
  • links: PTS, VCS
  • area: main
  • in suites: jessie-backports
  • size: 272 kB
  • sloc: python: 1,356; makefile: 4
file content (383 lines) | stat: -rw-r--r-- 13,947 bytes parent folder | download | duplicates (2)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
Overview
~~~~~~~~

Applications with more than a handful of user-settable options are best
configured through a combination of command line args, config files,
hard-coded defaults, and in some cases, environment variables.

Python's command line parsing modules such as argparse have very limited
support for config files and environment variables, so this module
extends argparse to add these features.

|Travis CI Status for bw2/ConfigArgParse|  -- from `Travis CI <https://travis-ci.org/bw2/ConfigArgParse/>`_

Features
~~~~~~~~

-  command-line, config file, env var, and default settings can now be
   defined, documented, and parsed in one go using a single API (if a
   value is specified in more than one way then: command line >
   environment variables > config file values > defaults)
-  config files can have .ini or .yaml style syntax (eg. key=value or
   key: value)
-  user can provide a config file via a normal-looking command line arg
   (eg. -c path/to/config.txt) rather than the argparse-style @config.txt
-  one or more default config file paths can be specified
   (eg. ['/etc/bla.conf', '~/.my_config'] )
-  all argparse functionality is fully supported, so this module can
   serve as a drop-in replacement (verified by argparse unittests).
-  env vars and config file keys & syntax are automatically documented
   in the -h help message
-  new method :code:`print_values()` can report keys & values and where
   they were set (eg. command line, env var, config file, or default).
-  lite-weight (no 3rd-party library dependencies except (optionally) PyYAML)
-  extensible (:code:`ConfigFileParser` can be subclassed to define a new
   config file format)
-  unittested by running the unittests that came with argparse but on
   configargparse, and using tox to test with python2.7+ and python3+

Example
~~~~~~~

*my_script.py*:

Script that defines 4 options and a positional arg and then parses and prints the values. Also,
it prints out the help message as well as the string produced by :code:`format_values()` to show
what they look like.

.. code:: py

   import configargparse

   p = configargparse.ArgParser(default_config_files=['/etc/settings.ini', '~/.my_settings'])
   p.add('-c', '--my-config', required=True, is_config_file=True, help='config file path')
   p.add('--genome', required=True, help='path to genome file')  # this option can be set in a config file because it starts with '--'
   p.add('-v', help='verbose', action='store_true')
   p.add('-d', '--dbsnp', help='known variants .vcf', env_var='DBSNP_PATH')  # this option can be set in a config file because it starts with '--'
   p.add('vcf', nargs='+', help='variant file(s)')

   options = p.parse_args()

   print(options)
   print("----------")
   print(p.format_help())
   print("----------")
   print(p.format_values())    # useful for logging where different settings came from


*config.txt:*

Since the script above set the config file as required=True, lets create a config file to give it:

.. code:: py

    # settings for my_script.py
    genome = HCMV     # cytomegalovirus genome
    dbsnp = /data/dbsnp/variants.vcf


*command line:*

Now run the script and pass it the config file:

.. code:: bash

    python my_script.py --genome hg19 --my-config config.txt  f1.vcf  f2.vcf

*output:*

Here is the result:

.. code:: bash

    Namespace(dbsnp='/data/dbsnp/variants.vcf', genome='hg19', my_config='config.txt', vcf=['f1.vcf', 'f2.vcf'], verbose=False)
    ----------
    usage: my_script.py [-h] --genome GENOME [-v] -c MY_CONFIG [-d DBSNP]
                        vcf [vcf ...]
    Args that start with '--' (eg. --genome) can also be set in a config file
    (/etc/settings.ini or /home/jeff/.my_settings or provided via -c) by using
    .ini or .yaml-style syntax (eg. genome=value). Command-line values override
    environment variables which override config file values which override
    defaults.

    positional arguments:
      vcf                   variant file
    optional arguments:
      -h, --help            show this help message and exit
      --genome GENOME       path to genome file
      -v                    verbose
      -c MY_CONFIG, --my-config MY_CONFIG
                            config file path
      -d DBSNP, --dbsnp DBSNP
                            known variants .vcf [env var: DBSNP_PATH]
    ----------
    Command Line Args:   --genome hg19 --my-config config.txt f1.vcf f2.vcf
    Config File (config.txt):
      dbsnp:             /data/dbsnp/variants.vcf

Special Values
~~~~~~~~~~~~~~

Under the hood, configargparse handles environment variables and config file
values by converting them to their corresponding command line arg. For
example, "key = value" will be processed as if "--key value" was specified
on the command line.

Also, the following special values (whether in a config file or an environment
variable) are handled in a special way to support booleans and lists:

-  :code:`key = true` is handled as if "--key" was specified on the command line.
   In your python code this key must be defined as a boolean flag
   (eg. action="store_true" or similar).

-  :code:`key = [value1, value2, ...]` is handled as if "--key value1 --key value2"
   etc. was specified on the command line. In your python code this key must
   be defined as a list (eg. action="append").

Config File Syntax
~~~~~~~~~~~~~~~~~~

Only command line args that have a long version (eg. one that starts with '--')
can be set in a config file. For example, "--color" can be set by
putting "color=green" in a config file. The config file syntax depends on the
constuctor arg: :code:`config_file_parser_class` which can be set to one of the
provided classes: :code:`DefaultConfigFileParser` or :code:`YAMLConfigFileParser`,
or to your own subclass of the :code:`ConfigFileParser` abstract class.

*DefaultConfigFileParser*  - the full range of valid syntax is:

.. code:: yaml

        # this is a comment
        ; this is also a comment (.ini style)
        ---            # lines that start with --- are ignored (yaml style)
        -------------------
        [section]      # .ini-style section names are treated as comments

        # how to specify a key-value pair (all of these are equivalent):
        name value     # key is case sensitive: "Name" isn't "name"
        name = value   # (.ini style)  (white space is ignored, so name = value same as name=value)
        name: value    # (yaml style)
        --name value   # (argparse style)

        # how to set a flag arg (eg. arg which has action="store_true")
        --name
        name
        name = True    # "True" and "true" are the same

        # how to specify a list arg (eg. arg which has action="append")
        fruit = [apple, orange, lemon]
        indexes = [1, 12, 35 , 40]


*YAMLConfigFileParser*  - allows a subset of YAML syntax (http://goo.gl/VgT2DU)

.. code:: yaml

        # a comment
        name1: value
        name2: true    # "True" and "true" are the same

        fruit: [apple, orange, lemon]
        indexes: [1, 12, 35, 40]


ArgParser Singletons
~~~~~~~~~~~~~~~~~~~~~~~~~

To make it easier to configure different modules in an application,
configargparse provides globally-available ArgumentParser instances
via configargparse.getArgumentParser('name') (similar to
logging.getLogger('name')).

Here is an example of an application with a utils module that also
defines and retrieves its own command-line args.

*main.py*

.. code:: py

    import configargparse
    import utils

    p = configargparse.getArgumentParser()
    p.add_argument("-x", help="Main module setting")
    p.add_argument("--m-setting", help="Main module setting")
    options = p.parse_known_args()   # using p.parse_args() here may raise errors.

*utils.py*

.. code:: py

    import configargparse
    p = configargparse.getArgumentParser()
    p.add_argument("--utils-setting", help="Config-file-settable option for utils")

    if __name__ == "__main__":
       options = p.parse_known_args()

Help Formatters
~~~~~~~~~~~~~~~

:code:`ArgumentDefaultsRawHelpFormatter` is a new HelpFormatter that both adds
default values AND disables line-wrapping. It can be passed to the constructor:
:code:`ArgParser(.., formatter_class=ArgumentDefaultsRawHelpFormatter)`


Aliases
~~~~~~~

The configargparse.ArgumentParser API inherits its class and method
names from argparse and also provides the following shorter names for
convenience:

-  p = configargparse.getArgParser()  # get global singleton instance
-  p = configargparse.getParser()
-  p = configargparse.ArgParser()  # create a new instance
-  p = configargparse.Parser()
-  p.add_arg(..)
-  p.add(..)
-  options = p.parse(..)

HelpFormatters:

- RawFormatter = RawDescriptionHelpFormatter
- DefaultsFormatter = ArgumentDefaultsHelpFormatter
- DefaultsRawFormatter = ArgumentDefaultsRawHelpFormatter


Design Notes
~~~~~~~~~~~~

Unit tests:

tests/test_configargparse.py contains custom unittests for features
specific to this module (such as config file and env-var support), as
well as a hook to load and run argparse unittests (see the built-in
test.test_argparse module) but on configargparse in place of argparse.
This ensures that configargparse will work as a drop in replacement for
argparse in all usecases.

Previously existing modules (PyPI search keywords: config argparse):

-  argparse (built-in module python v2.7+ )

   -  Good:

      -  fully featured command line parsing
      -  can read args from files using an easy to understand mechanism

   -  Bad:

      -  syntax for specifying config file path is unusual (eg.
         @file.txt)and not described in the user help message.
      -  default config file syntax doesn't support comments and is
         unintuitive (eg. --namevalue)
      -  no support for environment variables

-  ConfArgParse v1.0.15
   (https://pypi.python.org/pypi/ConfArgParse)

   -  Good:

      -  extends argparse with support for config files parsed by
         ConfigParser
      -  clear documentation in README

   -  Bad:

      -  config file values are processed using
         ArgumentParser.set_defaults(..) which means "required" and
         "choices" are not handled as expected. For example, if you
         specify a required value in a config file, you still have to
         specify it again on the command line.
      -  doesn't work with python 3 yet
      -  no unit tests, code not well documented

-  appsettings v0.5 (https://pypi.python.org/pypi/appsettings)

   -  Good:

      -  supports config file (yaml format) and env_var parsing
      -  supports config-file-only setting for specifying lists and
         dicts

   -  Bad:

      -  passes in config file and env settings via parse_args
         namespace param
      -  tests not finished and don't work with python3 (import
         StringIO)

-  argparse_config v0.5.1
   (https://pypi.python.org/pypi/argparse_config)

   -  Good:

      -  similar features to ConfArgParse v1.0.15

   -  Bad:

      -  doesn't work with python3 (error during pip install)

-  yconf v0.3.2 - (https://pypi.python.org/pypi/yconf) - features
   and interface not that great
-  hieropt v0.3 - (https://pypi.python.org/pypi/hieropt) - doesn't
   appear to be maintained, couldn't find documentation

-  configurati v0.2.3 - (https://pypi.python.org/pypi/configurati)

   -  Good:

      -  JSON, YAML, or Python configuration files
      -  handles rich data structures such as dictionaries
      -  can group configuration names into sections (like .ini files)

   -  Bad:

      -  doesn't work with python3
      -  2+ years since last release to PyPI
      -  apparently unmaintained


Design choices:

1. all options must be settable via command line. Having options that
   can only be set using config files or env. vars adds complexity to
   the API, and is not a useful enough feature since the developer can
   split up options into sections and call a section "config file keys",
   with command line args that are just "--" plus the config key.
2. config file and env. var settings should be processed by appending
   them to the command line (another benefit of #1). This is an
   easy-to-implement solution and implicitly takes care of checking that
   all "required" args are provied, etc., plus the behavior should be
   easy for users to understand.
3. configargparse shouldn't override argparse's
   convert_arg_line_to_args method so that all argparse unit tests
   can be run on configargparse.
4. in terms of what to allow for config file keys, the "dest" value of
   an option can't serve as a valid config key because many options can
   have the same dest. Instead, since multiple options can't use the
   same long arg (eg. "--long-arg-x"), let the config key be either
   "--long-arg-x" or "long-arg-x". This means the developer can allow
   only a subset of the command-line args to be specified via config
   file (eg. short args like -x would be excluded). Also, that way
   config keys are automatically documented whenever the command line
   args are documented in the help message.
5. don't force users to put config file settings in the right .ini
   [sections]. This doesn't have a clear benefit since all options are
   command-line settable, and so have a globally unique key anyway.
   Enforcing sections just makes things harder for the user and adds
   complexity to the implementation.
6. if necessary, config-file-only args can be added later by
   implementing a separate add method and using the namespace arg as in
   appsettings_v0.5

Relevant sites:

-  http://stackoverflow.com/questions/6133517/parse-config-file-environment-and-command-line-arguments-to-get-a-single-coll
-  http://tricksntweaks.blogspot.com/2013_05_01_archive.html
-  http://www.youtube.com/watch?v=vvCwqHgZJc8#t=35


.. |Travis CI Status for bw2/ConfigArgParse| image:: https://travis-ci.org/bw2/ConfigArgParse.svg?branch=master