Parse multiple subcommands in python simultaneously or other way to group parsed arguments
Question: I am converting Bash shell installer utility to Python 2.7 and need to
implement complex CLI so I am able to parse tens of parameters (potentially up
to ~150). These are names of Puppet class variables in addition to a dozen of
generic deployment options, which where available in shell version.

However after I have started to add more variables I faced are several
challenges: 1\. I need to group parameters into separate dictionaries so
deployment options are separated from Puppet variables. If they are thrown
into the same bucket, then I will have to write some logic to sort them,
potentially renaming parameters and then dictionary merges will not be
trivial. 2\. There might be variables with the same name but belonging to
different Puppet class, so I thought subcommands would allow me to filter what
goes where and avoiding name collisions.

At the momment I have implemented parameter parsing via simply adding multiple
parsers:

    
    
    parser = argparse.ArgumentParser(description='deployment parameters.')
    env_select = parser.add_argument_group(None, 'Environment selection')
    env_select.add_argument('-c', '--client_id',  help='Client name to use.')
    env_select.add_argument('-e', '--environment', help='Environment name to use.')
    setup_type = parser.add_argument_group(None, 'What kind of setup should be done:')
    setup_type.add_argument('-i', '--install', choices=ANSWERS, metavar='', action=StoreBool, help='Yy/Nn Do normal install and configuration')
    # MORE setup options
    ...
    args, unk = parser.parse_known_args()
    config['deploy_cfg'].update(args.__dict__)
    
    pup_class1_parser = argparse.ArgumentParser(description=None)
    pup_class1 = pup_class1_parser.add_argument_group(None, 'Puppet variables')
    pup_class1.add_argument('--ad_domain', help='AD/LDAP domain name.')
    pup_class1.add_argument('--ad_host', help='AD/LDAP server name.')
    # Rest of the parameters
    
    args, unk = pup_class1_parser.parse_known_args()
    config['pup_class1'] = dict({})
    config['pup_class1'].update(args.__dict__)
    # Same for class2, class3 and so on.
    

The problem with this approach that it does not solve issue 2. Also first
parser consumes "-h" option and rest of parameters are not shown in help.

I have tried to use [example selected as an
answer](http://stackoverflow.com/questions/10448200/how-to-parse-multiple-sub-
commands-using-python-argparse) but I was not able to use both commands at
once.

    
    
    ## This function takes the 'extra' attribute from global namespace and re-parses it to create separate namespaces for all other chained commands.
    def parse_extra (parser, namespace):
      namespaces = []
      extra = namespace.extra
      while extra:
        n = parser.parse_args(extra)
        extra = n.extra
        namespaces.append(n)
    
      return namespaces
    
    pp = pprint.PrettyPrinter(indent=4)
    
    argparser=argparse.ArgumentParser()
    subparsers = argparser.add_subparsers(help='sub-command help', dest='subparser_name')
    
    parser_a = subparsers.add_parser('command_a', help = "command_a help")
    ## Setup options for parser_a
    parser_a.add_argument('--opt_a1', help='option a1')
    parser_a.add_argument('--opt_a2', help='option a2')
    
    parser_b = subparsers.add_parser('command_b', help = "command_b help")
    ## Setup options for parser_a
    parser_b.add_argument('--opt_b1', help='option b1')
    parser_b.add_argument('--opt_b2', help='option b2')
    
    
    ## Add nargs="*" for zero or more other commands
    argparser.add_argument('extra', nargs = "*", help = 'Other commands')
    
    namespace = argparser.parse_args()
    pp.pprint(namespace)
    extra_namespaces = parse_extra( argparser, namespace )
    pp.pprint(extra_namespaces)
    

Results me in:

    
    
    $ python argtest.py command_b --opt_b1 b1 --opt_b2 b2 command_a --opt_a1 a1
    usage: argtest.py [-h] {command_a,command_b} ... [extra [extra ...]]
    argtest.py: error: unrecognized arguments: command_a --opt_a1 a1
    

The same result was when I tried to define parent with two child parsers.

**QUESTIONS**

  1. Can I somehow use parser.add_argument_group for argument parsing or is it just for the grouping in help print out? It would solve issue 1 without missing help side effect. Passing it as `parse_known_args(namespace=argument_group)` (if I correctly recall my experiments) gets all the variables (thats ok) but also gets all Python object stuff in resulting dict (that's bad for hieradata YAML)
  2. What I am missing in the second example to allow to use multiple subcommands? Or is that impossible with argparse?
  3. Any other suggestion to group command line variables? I have looked at Click, but did not find any advantages over standard argparse for my task. 

Note: I am sysadmin, not a programmer so be gently on me for the non object
style coding. :)

Thank you

**RESOLVED** Argument grouping solved via the answer suggested by
[hpaulj](http://stackoverflow.com/users/901925/hpaulj).

    
    
    import argparse
    import pprint
    parser = argparse.ArgumentParser()
    
    group_list = ['group1', 'group2']
    
    group1 = parser.add_argument_group('group1')
    group1.add_argument('--test11', help="test11")
    group1.add_argument('--test12', help="test12")
    
    group2 = parser.add_argument_group('group2')
    group2.add_argument('--test21', help="test21")
    group2.add_argument('--test22', help="test22")
    
    args = parser.parse_args()
    pp = pprint.PrettyPrinter(indent=4)
    
    d = dict({})
    
    for group in parser._action_groups:
        if group.title in group_list:
            d[group.title]={a.dest:getattr(args,a.dest,None) for a in group._group_actions}
    
    print "Parsed arguments"
    pp.pprint(d)
    

This gets me desired result for the issue No.1. until I will have multiple
parameters with the same name. Solution may look ugly, but at least it works
as expected.

    
    
    python argtest4.py --test22 aa  --test11 yy11 --test21 aaa21
    Parsed arguments
    {   'group1': {   'test11': 'yy11', 'test12': None},
        'group2': {   'test21': 'aaa21', 'test22': 'aa'}}
    


Answer: Your question is too complicated to understand and respond to in one try. But
I'll throw out some preliminary ideas.

Yes, `argument_groups` are just a way of grouping arguments in the help. They
have no effect on parsing.

Another recent SO asked about parsing groups of arguments:

[Is it possible to only parse one argument group's parameters with
argparse?](http://stackoverflow.com/questions/31519997/is-it-possible-to-only-
parse-one-argument-groups-parameters-with-argparse)

That poster initially wanted to use a group as a parser, but the `argparse`
class structure does not allow that. `argparse` is written in object style.
`parser=ArguementParser...` creates one class of object,
`parser.add_arguement...` creates another, `add_argument_group...` yet
another. You customize it by subclassing `ArgumentParser` or `HelpFormatter`
or `Action` classes, etc.

I mentioned a `parents` mechanism. You define one or more parent parsers, and
use those to populate your 'main' parser. They could be run indepdently (with
parse_known_args), while the 'main' is used to handle help.

We also discussed grouping the arguments after parsing. A `namespace` is a
simple object, in which each argument is an attribute. It can also be
converted to a dictionary. It is easy to pull groups of items from a
dictionary.

There have SO questions about using multiple subparsers. That's an awkward
proposition. Possible, but not easy. Subparsers are like issueing a command to
a system program. You generally issue one command per call. You don't nest
them or issue sequences. You let shell piping and scripts handle multiple
actions.

`IPython` uses `argparse` to parse its inputs. It traps help first, and issues
its own message. Most arguments come from config files, so it is possible to
set values with default configs, custom configs and in the commandline. It's
an example of naming a very large set of arguments.

Subparsers let you use the same argument name, but without being able to
invoke multiple subparsers in one call that doesn't help much. And even if you
could invoke several subparsers, they would still put the arguments in the
same namespace. Also `argparse` tries to handle flaged arguments in an order
independent manner. So a `--foo` at the end of the command line gets parsed
the same as though it were at the start.

There was SO question where we discussed using argument names ('dest') like
`'group1.argument1'`, and I've even discussed using nested namespaces. I could
look those up if it would help.

* * *

Another thought - load `sys.argv` and partition it before passing it to one or
more parsers. You could split it on some key word, or on prefixes etc.

