text: # -*- coding: utf-8 -*-
  '''
  IPDB module
  ===========
  
  Basically, IPDB is a transactional database, containing
  records, that represent network stack objects. Any change
  in the database is not reflected immediately in OS, but
  waits until `commit()` is called. One failed operation
  during `commit()` rolls back all the changes, has been made
  so far. Moreover, IPDB has commit hooks API, that allows
  you to roll back changes depending on your own function
  calls, e.g. when a host or a network becomes unreachable.
  
  IPDB, NDB, IPRoute
  ------------------
  
  These modules use completely different approaches.
  
  * IPRoute just forwards requests to the kernel, and doesn't
    wait for the system state. So it's up to developer to check,
    whether the requested object is really set up or not.
  * IPDB is an asynchronously updated database, that starts
    several additional threads by default. If your project's policy
    doesn't allow implicit threads, keep it in mind. But unlike IPRoute,
    the IPDB ensures the changes to be reflected in the system.
  * NDB is like IPDB, and may obsolete it in the future. The difference
    is that IPDB creates Python object for every RTNL object, while
    NDB stores everything in an SQL DB, and creates objects on demand.
  
  Sync on commit::
  
      with IPDB() as ipdb:
          with ipdb.interfaces['eth0'] as i:
              i.up()
              i.add_ip('192.168.0.2/24')
              i.add_ip('192.168.0.3/24')
          # ---> <--- here you can expect `eth0` is up
          #           and has these two addresses, so
          #           the following code can rely on that
  
  So IPDB is updated asynchronously, but the `commit()`
  operation is synchronous. At least, it is planned to be such.
  
  NB: *In the example above `commit()` is implied with the
  `__exit__()` of the `with` statement.*
  
  The choice between IPDB and IPRoute depends on your project's
  workflow. If you plan to retrieve the system info not too
  often (or even once), or you are sure there will be not too
  many network object, it is better to use IPRoute. If you
  plan to lookup the network info on the regular basis and
  there can be loads of network objects, it is better to use
  IPDB or NDB. Why?
  
  IPRoute just loads what you ask -- and loads all the
  information you ask to. While IPDB and NDB load all the info upon
  startup, and later the DB is just updated by asynchronous broadcast
  netlink messages. Assume you want to lookup ARP cache that
  contains hundreds or even thousands of objects. Using
  IPRoute, you have to load all the ARP cache every time you
  want to make a lookup. While IPDB will load all the cache
  once, and then maintain it up-to-date just inserting new
  records or removing them by one.
  
  So, IPRoute is much simpler when you need to make a call and
  then exit, while IPDB is cheaper in terms of CPU performance
  if you implement a long-running program like a daemon.
  
  Memory footprint issues
  -----------------------
  
  One of the major issues with IPDB is its memory footprint. It
  proved to be not suitable for environments with thousands of
  routes or neighbours. Being a design issue, it could not be
  fixed, so a new module was started, NDB, that aims to replace
  IPDB, but it isn't yet production ready. IPDB is still more
  feature rich and in some sense more stable.
  
  IPDB and other software
  -----------------------
  
  IPDB is designed to be a non-exclusive network settings database.
  There may be several IPDB instances on the same OS, as well as
  other network management software, such as NetworkManager etc.
  
  The IPDB transactions should not interfere with other software
  settings, unless they touch the same objects. E.g., if IPDB
  brings an interface up, while NM shuts it down, there will be
  a race condition.
  
  An example::
  
      # IPDB code                       #  NetworkManager at the same time:
      ipdb.interfaces['eth0'].up()      #
      ipdb.interfaces['eth0'].commit()  #  $ sudo nmcli con down eth0
      # ---> <---
      # The eth0 state here is undefined. Some of the commands
      # above will fail
  
  But as long as the software doesn't touch the same objects, there
  will be no conflicts. Another example::
  
      # IPDB code                         # At the same time, NetworkManager
      with ipdb.interfaces['eth0'] as i:  # adds addresses:
          i.add_ip('172.16.254.2/24')     #  * 10.0.0.2/24
          i.add_ip('172.16.254.3/24')     #  * 10.0.0.3/24
      # ---> <---
      # At this point the eth0 interface will have all four addresses.
      # If the IPDB transaction fails by some reason, only IPDB addresses
      # will be rolled back.
  
  There may be a need to prevent other software from changing the network
  settings. There is no locking at the kernel level, but IPDB can revert
  all the changes as soon as they appear on the interface::
  
      # IPDB code
      ipdb.interfaces['eth0'].freeze()
                                         # Here some other software tries to
                                         # add an address, or to remove the old
                                         # one
      # ---> <---
      # At this point the eth0 interface will have all the same settings as
      # at the `freeze()` call moment. Newly added addresses will be removed,
      # all the deleted addresses will be restored.
      #
      # Please notice, that an address removal may cause also a routes removal,
      # and that is the thing that IPDB can not neither prevent, nor revert.
  
      ipdb.interfaces['eth0'].unfreeze()
  
  Quickstart
  ----------
  
  Simple tutorial::
  
      from pyroute2 import IPDB
      # several IPDB instances are supported within on process
      ipdb = IPDB()
  
      # commit is called automatically upon the exit from `with`
      # statement
      with ipdb.interfaces.eth0 as i:
          i.address = '00:11:22:33:44:55'
          i.ifname = 'bala'
          i.txqlen = 2000
  
      # basic routing support
      ipdb.routes.add({'dst': 'default',
                       'gateway': '10.0.0.1'}).commit()
  
      # do not forget to shutdown IPDB
      ipdb.release()
  
  Please, notice `ip.release()` call in the end. Though it is
  not forced in an interactive python session for the better
  user experience, it is required in the scripts to sync the
  IPDB state before exit.
  
  IPDB supports functional-like syntax also::
  
      from pyroute2 import IPDB
      with IPDB() as ipdb:
          intf = (ipdb.interfaces['eth0']
                  .add_ip('10.0.0.2/24')
                  .add_ip('10.0.0.3/24')
                  .set_address('00:11:22:33:44:55')
                  .set_mtu(1460)
                  .set_name('external')
                  .commit())
          # ---> <--- here you have the interface reference with
          #           all the changes applied: renamed, added ipaddr,
          #           changed macaddr and mtu.
          ...  # some code
  
      # pls notice, that the interface reference will not work
      # outside of `with IPDB() ...`
  
  Transaction modes
  -----------------
  IPDB has several operating modes:
  
      - 'implicit' (default) -- the first change starts an implicit
          transaction, that have to be committed
      - 'explicit' -- you have to begin() a transaction prior to
          make any change
  
  The default is to use implicit transaction. This behaviour
  can be changed in the future, so use 'mode' argument when
  creating IPDB instances.
  
  The sample session with explicit transactions::
  
      In [1]: from pyroute2 import IPDB
      In [2]: ip = IPDB(mode='explicit')
      In [3]: ifdb = ip.interfaces
      In [4]: ifdb.tap0.begin()
          Out[3]: UUID('7a637a44-8935-4395-b5e7-0ce40d31d937')
      In [5]: ifdb.tap0.up()
      In [6]: ifdb.tap0.address = '00:11:22:33:44:55'
      In [7]: ifdb.tap0.add_ip('10.0.0.1', 24)
      In [8]: ifdb.tap0.add_ip('10.0.0.2', 24)
      In [9]: ifdb.tap0.review()
          Out[8]:
          {'+ipaddr': set([('10.0.0.2', 24), ('10.0.0.1', 24)]),
           '-ipaddr': set([]),
           'address': '00:11:22:33:44:55',
           'flags': 4099}
      In [10]: ifdb.tap0.commit()
  
  
  Note, that you can `review()` the `current_tx` transaction,
  and `commit()` or `drop()` it. Also, multiple transactions
  are supported, use uuid returned by `begin()` to identify
  them.
  
  Actually, the form like 'ip.tap0.address' is an eye-candy.
  The IPDB objects are dictionaries, so you can write the code
  above as that::
  
      ipdb.interfaces['tap0'].down()
      ipdb.interfaces['tap0']['address'] = '00:11:22:33:44:55'
      ...
  
  Context managers
  ----------------
  
  Transactional objects (interfaces, routes) can act as context
  managers in the same way as IPDB does itself::
  
      with ipdb.interfaces.tap0 as i:
          i.address = '00:11:22:33:44:55'
          i.ifname = 'vpn'
          i.add_ip('10.0.0.1', 24)
          i.add_ip('10.0.0.1', 24)
  
  On exit, the context manager will authomatically `commit()`
  the transaction.
  
  Read-only interface views
  -------------------------
  
  Using an interface as a context manager **will** start a
  transaction. Sometimes it is not what one needs. To avoid
  unnecessary transactions, and to avoid the risk to occasionally
  change interface attributes, one can use read-only views::
  
      with ipdb.interfaces[1].ro as iface:
          print(iface.ifname)
          print(iface.address)
  
  The `.ro` view neither starts transactions, nor allows to
  change anything, raising the `RuntimeError` exception.
  
  The same read-only views are available for routes and rules.
  
  Create interfaces
  -----------------
  
  IPDB can also create virtual interfaces::
  
      with ipdb.create(kind='bridge', ifname='control') as i:
          i.add_port(ip.interfaces.eth1)
          i.add_port(ip.interfaces.eth2)
          i.add_ip('10.0.0.1/24')
  
  
  The `IPDB.create()` call has the same syntax as
  `IPRoute.link('add', ...)`, except you shouldn't specify
  the `'add'` command. Refer to `IPRoute` docs for details.
  
  Please notice, that the interface object stays in the database
  even if there was an error during the interface creation. It is
  done so to make it possible to fix the interface object and try
  to run `commit()` again. Or you can drop the interface object
  with the `.remove().commit()` call.
  
  IP address management
  ---------------------
  
  IP addresses on interfaces may be managed using `add_ip()` and
  `del_ip()`::
  
      with ipdb.interfaces['eth0'] as eth:
          eth.add_ip('10.0.0.1/24')
          eth.add_ip('10.0.0.2/24')
          eth.add_ip('2001:4c8:1023:108::39/64')
          eth.del_ip('172.16.12.5/24')
  
  The address format may be either a string with `'address/mask'`
  notation, or a pair of `'address', mask`::
  
      with ipdb.interfaces['eth0'] as eth:
          eth.add_ip('10.0.0.1', 24)
          eth.del_ip('172.16.12.5', 24)
  
  The `ipaddr` attribute contains all the IP addresses of the
  interface, which are acessible in different ways. Getting an
  iterator from `ipaddr` gives you a sequence of tuples
  `('address', mask)`::
  
      >>> for addr in ipdb.interfaces['eth0'].ipaddr:
      ...    print(ipaddr)
      ...
      ('10.0.0.2', 24)
      ('10.0.0.1', 24)
  
  Getting one IP from `ipaddr` returns a dict object with full spec:
  
      >>> ipdb.interfaces['eth0'].ipaddr[0]:
          {'family': 2,
           'broadcast': None,
           'flags': 128,
           'address': '10.0.0.2',
           'prefixlen': 24,
           'local': '10.0.0.2'}
  
      >>> ipdb.intefaces['eth0'].ipaddr['10.0.0.2/24']:
          {'family': 2,
           'broadcast': None,
           'flags': 128,
           'address': '10.0.0.2',
           'prefixlen': 24,
           'local': '10.0.0.2'}
  
  The API is a bit weird, but it's because of historical reasons. In
  the future it may be changed.
  
  Another feature of the `ipaddr` attribute is views::
  
      >>> ipdb.interfaces['eth0'].ipaddr.ipv4:
          (('10.0.0.2', 24), ('10.0.0.1', 24))
      >>> ipdb.interfaces['eth0'].ipaddr.ipv6:
          (('2001:4c8:1023:108::39', 64),)
  
  The views, as well as the `ipaddr` attribute itself are not supposed
  to be changed by user, but only by the internal API.
  
  Bridge interfaces
  -----------------
  
  Modern kernels provide possibility to manage bridge
  interface properties such as STP, forward delay, ageing
  time etc. Names of these properties start with `br_`, like
  `br_ageing_time`, `br_forward_delay` e.g.::
  
      [x for x in dir(ipdb.interfaces.virbr0) if x.startswith('br_')]
  
  Bridge ports
  ------------
  
  IPDB supports specific bridge port parameters, such as proxyarp,
  unicast/multicast flood, cost etc.::
  
      with ipdb.interfaces['br-port0'] as p:
          p.brport_cost = 200
          p.brport_unicast_flood = 0
          p.brport_proxyarp = 0
  
  Ports management
  ----------------
  
  IPDB provides a uniform API to manage bridge, bond and vrf ports::
  
      with ipdb.interfaces['br-int'] as br:
          br.add_port('veth0')
          br.add_port(ipdb.interfaces.veth1)
          br.add_port(700)
          br.del_port('veth2')
  
  Both `add_port()` and `del_port()` accept three types of arguments:
  
      * `'veth0'` -- interface name as a string
      * `ipdb.interfaces.veth1` -- IPDB interface object
      * `700` -- interface index, an integer
  
  Routes management
  -----------------
  
  IPDB has a simple yet useful routing management interface.
  
  Create a route
  ~~~~~~~~~~~~~~
  
  To add a route, there is an easy to use syntax::
  
      # spec as a dictionary
      spec = {'dst': '172.16.1.0/24',
              'oif': 4,
              'gateway': '192.168.122.60',
              'metrics': {'mtu': 1400,
                          'advmss': 500}}
  
      # pass spec as is
      ipdb.routes.add(spec).commit()
  
      # pass spec as kwargs
      ipdb.routes.add(**spec).commit()
  
      # use keyword arguments explicitly
      ipdb.routes.add(dst='172.16.1.0/24', oif=4, ...).commit()
  
  Please notice, that the device can be specified with `oif`
  (output interface) or `iif` (input interface), the `device`
  keyword is not supported anymore.
  
  More examples::
  
      # specify table and priority
      (ipdb.routes
       .add(dst='172.16.1.0/24',
            gateway='192.168.0.1',
            table=100,
            priority=10)
       .commit())
  
  The `priority` field is what the `iproute2` utility calls
  `metric` -- see also below.
  
  Get a route
  ~~~~~~~~~~~
  
  To access and change the routes, one can use notations as
  follows::
  
      # default table (254)
      #
      # change the route gateway and mtu
      #
      with ipdb.routes['172.16.1.0/24'] as route:
          route.gateway = '192.168.122.60'
          route.metrics.mtu = 1500
  
      # access the default route
      print(ipdb.routes['default'])
  
      # change the default gateway
      with ipdb.routes['default'] as route:
          route.gateway = '10.0.0.1'
  
  By default, the path `ipdb.routes` reflects only the main
  routing table (254). But Linux supports much more routing
  tables, so does IPDB::
  
      In [1]: ipdb.routes.tables.keys()
      Out[1]: [0, 254, 255]
  
      In [2]: len(ipdb.routes.tables[255])
      Out[2]: 11  # => 11 automatic routes in the table local
  
  It is important to understand, that routing tables keys in
  IPDB are not only the destination prefix. The key consists
  of 'prefix/mask' string and the route priority (if any)::
  
      In [1]: ipdb.routes.tables[254].idx.keys()
      Out[1]:
      [RouteKey(dst='default', table=254, family=2, ...),
       RouteKey(dst='172.17.0.0/16', table=254, ...),
       RouteKey(dst='172.16.254.0/24', table=254, ...),
       RouteKey(dst='192.168.122.0/24', table=254, ...),
       RouteKey(dst='fe80::/64', table=254, family=10, ...)]
  
  But a routing table in IPDB allows several variants of the
  route spec. The simplest case is to retrieve a route by
  prefix, if there is only one match::
  
      # get route by prefix
      ipdb.routes['172.16.1.0/24']
  
      # get route by a special name
      ipdb.routes['default']
  
  If there are more than one route that matches the spec, only
  the first one will be retrieved. One should iterate all the
  records and filter by a key to retrieve all matches::
  
      # only one route will be retrieved
      ipdb.routes['fe80::/64']
  
      # get all routes by this prefix
      [ x for x in ipdb.routes if x['dst'] == 'fe80::/64' ]
  
  It is also possible to use dicts as specs::
  
      # get IPv4 default route
      ipdb.routes[{'dst': 'default', 'family': AF_INET}]
  
      # get IPv6 default route
      ipdb.routes[{'dst': 'default', 'family': AF_INET6}]
  
      # get route by priority
      ipdb.routes.table[100][{'dst': '10.0.0.0/24', 'priority': 10}]
  
  While this notation returns one route, there is a method to get
  all the routes matching the spec::
  
      # get all the routes from all the tables via some interface
      ipdb.routes.filter({'oif': idx})
  
      # get all IPv6 routes from some table
      ipdb.routes.table[tnum].filter({'family': AF_INET6})
  
  Route metrics
  ~~~~~~~~~~~~~
  
  A special object is dedicated to route metrics, one can
  access it via `route.metrics` or `route['metrics']`::
  
      # these two statements are equal:
      with ipdb.routes['172.16.1.0/24'] as route:
          route['metrics']['mtu'] = 1400
  
      with ipdb.routes['172.16.1.0/24'] as route:
          route.metrics.mtu = 1400
  
  Possible metrics are defined in `rtmsg.py:rtmsg.metrics`,
  e.g. `RTAX_HOPLIMIT` means `hoplimit` metric etc.
  
  Multipath routing
  ~~~~~~~~~~~~~~~~~
  
  Multipath nexthops are managed via `route.add_nh()` and
  `route.del_nh()` methods. They are available to review via
  `route.multipath`, but one should not directly
  add/remove/modify nexthops in `route.multipath`, as the
  changes will not be committed correctly.
  
  To create a multipath route::
  
      ipdb.routes.add({'dst': '172.16.232.0/24',
                       'multipath': [{'gateway': '172.16.231.2',
                                      'hops': 2},
                                     {'gateway': '172.16.231.3',
                                      'hops': 1},
                                     {'gateway': '172.16.231.4'}]}).commit()
  
  To change a multipath route::
  
      with ipdb.routes['172.16.232.0/24'] as r:
          r.add_nh({'gateway': '172.16.231.5'})
          r.del_nh({'gateway': '172.16.231.4'})
  
  Another possible way is to create a normal route and turn
  it into multipath by `add_nh()`::
  
      # create a non-MP route with one gateway:
      (ipdb
       .routes
       .add({'dst': '172.16.232.0/24',
             'gateway': '172.16.231.2'})
       .commit())
  
      # turn it to become a MP route:
      (ipdb
       .routes['172.16.232.0/24']
       .add_nh({'gateway': '172.16.231.3'})
       .commit())
  
      # here the route will contain two NH records, with
      # gateways 172.16.231.2 and 172.16.231.3
  
      # remove one NH and turn the route to be a normal one
      (ipdb
       .routes['172.16.232.0/24']
       .del_nh({'gateway': '172.16.231.2'})
       .commit())
  
      # thereafter the traffic to 172.16.232.0/24 will go only
      # via 172.16.231.3
  
  Differences from the iproute2 syntax
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  By historical reasons, `iproute2` uses names that differs
  from what the kernel uses. E.g., `iproute2` uses `weight`
  for multipath route hops instead of `hops`, where
  `weight == (hops + 1)`. Thus, a route created with
  `hops == 2` will be listed by `iproute2` as `weight 3`.
  
  Another significant difference is `metrics`. The `pyroute2`
  library uses the kernel naming scheme, where `metrics` means
  mtu, rtt, window etc. The `iproute2` utility uses `metric`
  (not `metrics`) as a name for the `priority` field.
  
  In examples::
  
      # -------------------------------------------------------
      # iproute2 command:
      $ ip route add default \\
          nexthop via 172.16.0.1 weight 2 \\
          nexthop via 172.16.0.2 weight 9
  
      # pyroute2 code:
      (ipdb
       .routes
       .add({'dst': 'default',
             'multipath': [{'gateway': '172.16.0.1', 'hops': 1},
                           {'gateway': '172.16.0.2', 'hops': 8}])
       .commit())
  
      # -------------------------------------------------------
      # iproute2 command:
      $ ip route add default via 172.16.0.2 metric 200
  
      # pyroute2 code:
      (ipdb
       .routes
       .add({'dst': 'default',
             'gateway': '172.16.0.2',
             'priority': 200})
       .commit())
  
      # -------------------------------------------------------
      # iproute2 command:
      $ ip route add default via 172.16.0.2 mtu 1460
  
      # pyroute2 code:
      (ipdb
       .routes
       .add({'dst': 'default',
             'gateway': '172.16.0.2',
             'metrics': {'mtu': 1460}})
       .commit())
  
  Multipath default routes
  ~~~~~~~~~~~~~~~~~~~~~~~~
  
  .. warning::
      As of the merge of kill_rtcache into the kernel, and it's
      release in ~3.6, weighted default routes no longer work
      in Linux.
  
  Please refer to
  https://github.com/svinota/pyroute2/issues/171#issuecomment-149297244
  for details.
  
  Rules management
  ----------------
  
  IPDB provides a basic IP rules management system.
  
  Create a rule
  ~~~~~~~~~~~~~
  
  Syntax is almost the same as for routes::
  
      # rule spec
      spec = {'src': '172.16.1.0/24',
              'table': 200,
              'priority': 15000}
  
      ipdb.rules.add(spec).commit()
  
  Get a rule
  ~~~~~~~~~~
  
  The way IPDB handles IP rules is almost the same as routes,
  but rule keys are more complicated -- the Linux kernel
  doesn't use keys for rules, but instead iterates all the
  records until the first one w/o any attribute mismatch.
  
  The fields that the kernel uses to compare rules, IPDB uses
  as the key fields (see `pyroute2/ipdb/rule.py:RuleKey`)
  
  There are also more ways to find a record, as with routes::
  
      # 1. iterate all the records
      for record in ipdb.rules:
          match(record)
  
      # 2. an integer as the key matches the first
      #    rule with that priority
      ipdb.rules[32565]
  
      # 3. a dict as the key returns the first match
      #    for all the specified attrs
      ipdb.rules[{'dst': '10.0.0.0/24', 'table': 200}]
  
  Priorities
  ~~~~~~~~~~
  
  Thus, the rule priority is **not** a key, neither in the
  kernel, nor in IPDB. One should **not** rely on priorities
  as on keys, there may be several rules with the same
  priority, and it often happens, e.g. on Android systems.
  
  Persistence
  ~~~~~~~~~~~
  
  There is no *change* operation for the rule records in the
  kernel, so only *add/del* work. When IPDB changes a record,
  it effectively deletes the old one and creates the new with
  new parameters, but the object, referring the record, stays
  the same. Also that means, that IPDB can not recognize the
  situation, when someone else does the same. So if there is
  another program changing records by *del/add* operations,
  even another IPDB instance, referring objects in the IPDB
  will be recreated.
  
  Performance issues
  ------------------
  
  In the case of bursts of Netlink broadcast messages, all
  the activity of the pyroute2-based code in the async mode
  becomes suppressed to leave more CPU resources to the
  packet reader thread. So please be ready to cope with
  delays in the case of Netlink broadcast storms. It means
  also, that IPDB state will be synchronized with OS also
  after some delay.
  
  The class API
  -------------
  '''
  import sys
  import atexit
  import logging
  import traceback
  import threading
  import weakref
  try:
      import queue
  except ImportError:
      import Queue as queue  # The module is called 'Queue' in Python2
  
  from functools import partial
  from pprint import pprint
  from pyroute2 import config
  from pyroute2.common import uuid32
  from pyroute2.common import basestring
  from pyroute2.iproute import IPRoute
  from pyroute2.netlink.rtnl import RTM_GETLINK, RTMGRP_DEFAULTS
  from pyroute2.netlink.rtnl.ifinfmsg import ifinfmsg
  from pyroute2.ipdb import rules
  from pyroute2.ipdb import routes
  from pyroute2.ipdb import interfaces
  from pyroute2.ipdb.routes import BaseRoute
  from pyroute2.ipdb.exceptions import ShutdownException
  from pyroute2.ipdb.transactional import SYNC_TIMEOUT
  from pyroute2.ipdb.linkedset import IPaddrSet
  from pyroute2.ipdb.linkedset import SortedIPaddrSet
  from pyroute2.ipdb.utils import test_reachable_icmp
  
  log = logging.getLogger(__name__)
  
  
  class Watchdog(object):
      def __init__(self, ipdb, action, kwarg):
          self.event = threading.Event()
          self.is_set = False
          self.ipdb = ipdb
  
          def cb(ipdb, msg, _action):
              if _action != action:
                  return
  
              for key in kwarg:
                  if (msg.get(key, None) != kwarg[key]) and \
                          (msg.get_attr(msg.name2nla(key)) != kwarg[key]):
                      return
  
              self.is_set = True
              self.event.set()
          self.cb = cb
          # register callback prior to other things
          self.uuid = self.ipdb.register_callback(self.cb)
  
      def wait(self, timeout=SYNC_TIMEOUT):
          ret = self.event.wait(timeout=timeout)
          self.cancel()
          return ret
  
      def cancel(self):
          self.ipdb.unregister_callback(self.uuid)
  
  
  class _evq_context(object):
      '''
      Context manager class for the event queue used by the event loop
      '''
      def __init__(self, ipdb, qsize, block, timeout):
          self._ipdb = ipdb
          self._qsize = qsize
          self._block = block
          self._timeout = timeout
  
      def __enter__(self):
          # Context manager protocol
          self._ipdb._evq_lock.acquire()
          self._ipdb._evq = queue.Queue(maxsize=self._qsize)
          self._ipdb._evq_drop = 0
          return self
  
      def __exit__(self, exc_type, exc_value, traceback):
          # Context manager protocol
          self._ipdb._evq = None
          self._ipdb._evq_drop = 0
          self._ipdb._evq_lock.release()
  
      def __iter__(self):
          # Iterator protocol
          if not self._ipdb._evq:
              raise RuntimeError('eventqueue must be used '
                                 'as a context manager')
          return self
  
      def next(self):
          # Iterator protocol -- Python 2.x compatibility
          return self.__next__()
  
      def __next__(self):
          # Iterator protocol -- Python 3.x
          msg = self._ipdb._evq.get(self._block, self._timeout)
          self._ipdb._evq.task_done()
          if isinstance(msg, Exception):
              raise msg
          return msg
  
  
  class IPDB(object):
      '''
      The class that maintains information about network setup
      of the host. Monitoring netlink events allows it to react
      immediately. It uses no polling.
      '''
  
      def __init__(self, nl=None, mode='implicit',
                   restart_on_error=None, nl_async=None,
                   sndbuf=1048576, rcvbuf=1048576,
                   nl_bind_groups=RTMGRP_DEFAULTS,
                   ignore_rtables=None, callbacks=None,
                   sort_addresses=False, plugins=None):
          plugins = plugins or ['interfaces', 'routes', 'rules']
          pmap = {'interfaces': interfaces,
                  'routes': routes,
                  'rules': rules}
          self.mode = mode
          self.txdrop = False
          self._stdout = sys.stdout
          self._ipaddr_set = SortedIPaddrSet if sort_addresses else IPaddrSet
          self._event_map = {}
          self._deferred = {}
          self._ensure = []
          self._loaded = set()
          self._mthread = None
          self._nl_own = nl is None
          self._nl_async = config.ipdb_nl_async if nl_async is None else True
          self.mnl = None
          self.nl = nl
          self._sndbuf = sndbuf
          self._rcvbuf = rcvbuf
          self.nl_bind_groups = nl_bind_groups
          self._plugins = [pmap[x] for x in plugins if x in pmap]
          if isinstance(ignore_rtables, int):
              self._ignore_rtables = [ignore_rtables, ]
          elif isinstance(ignore_rtables, (list, tuple, set)):
              self._ignore_rtables = ignore_rtables
          else:
              self._ignore_rtables = []
          self._stop = False
          # see also 'register_callback'
          self._post_callbacks = {}
          self._pre_callbacks = {}
  
          # local event queues
          # - callbacks event queue
          self._cbq = queue.Queue(maxsize=8192)
          self._cbq_drop = 0
          # - users event queue
          self._evq = None
          self._evq_lock = threading.Lock()
          self._evq_drop = 0
  
          # locks and events
          self.exclusive = threading.RLock()
          self._shutdown_lock = threading.Lock()
  
          # register callbacks
          #
          # examples::
          #   def cb1(ipdb, msg, event):
          #       print(event, msg)
          #   def cb2(...):
          #       ...
          #
          #   # default mode: post
          #   IPDB(callbacks=[cb1, cb2])
          #   # specify the mode explicitly
          #   IPDB(callbacks=[(cb1, 'pre'), (cb2, 'post')])
          #
          for cba in callbacks or []:
              if not isinstance(cba, (tuple, list, set)):
                  cba = (cba, )
              self.register_callback(*cba)
  
          # load information
          self.restart_on_error = restart_on_error if \
              restart_on_error is not None else nl is None
  
          # init the database
          self.initdb()
  
          # init the dir() cache
          self.__dir_cache__ = [i for i in self.__class__.__dict__.keys()
                                if i[0] != '_']
          self.__dir_cache__.extend(list(self._deferred.keys()))
  
          def cleanup(ref):
              ipdb_obj = ref()
              if ipdb_obj is not None:
                  ipdb_obj.release()
          atexit.register(cleanup, weakref.ref(self))
  
      def __dir__(self):
          return self.__dir_cache__
  
      def __enter__(self):
          return self
  
      def __exit__(self, exc_type, exc_value, traceback):
          self.release()
  
      def _flush_db(self):
  
          def flush(idx):
              for key in tuple(idx.keys()):
                  try:
                      del idx[key]
                  except KeyError:
                      pass
          idx_list = []
          if 'interfaces' in self._loaded:
              for (key, dev) in self.by_name.items():
                  try:
                      # FIXME
                      self.interfaces._detach(key,
                                              dev['index'],
                                              dev.nlmsg)
                  except KeyError:
                      pass
              idx_list.append(self.ipaddr)
              idx_list.append(self.neighbours)
          if 'routes' in self._loaded:
              idx_list.extend([self.routes.tables[x] for x
                               in self.routes.tables.keys()])
          if 'rules' in self._loaded:
              idx_list.append(self.rules)
          for idx in idx_list:
              flush(idx)
  
      def initdb(self):
  
          # flush all the DB objects
          with self.exclusive:
  
              # explicitly cleanup object references
              for event in tuple(self._event_map):
                  del self._event_map[event]
  
              self._flush_db()
  
              # if the command socket is not provided, create it
              if self._nl_own:
                  if self.nl is not None:
                      self.nl.close()
                  self.nl = IPRoute(sndbuf=self._sndbuf, rcvbuf=self._rcvbuf)
              # setup monitoring socket
              if self.mnl is not None:
                  self._flush_mnl()
                  self.mnl.close()
              self.mnl = self.nl.clone()
              try:
                  self.mnl.bind(groups=self.nl_bind_groups,
                                async_cache=self._nl_async)
              except:
                  self.mnl.close()
                  if self._nl_own is None:
                      self.nl.close()
                  raise
  
              # explicitly cleanup references
              for key in tuple(self._deferred):
                  del self._deferred[key]
  
              for module in self._plugins:
                  if (module.groups & self.nl_bind_groups) != module.groups:
                      continue
                  for plugin in module.spec:
                      self._deferred[plugin['name']] = module.spec
                      if plugin['name'] in self._loaded:
                          delattr(self, plugin['name'])
                          self._loaded.remove(plugin['name'])
  
              # start service threads
              for tspec in (('_mthread', '_serve_main', 'IPDB main event loop'),
                            ('_cthread', '_serve_cb', 'IPDB cb event loop')):
                  tg = getattr(self, tspec[0], None)
                  if not getattr(tg, 'is_alive', lambda: False)():
                      tx = threading.Thread(name=tspec[2],
                                            target=getattr(self, tspec[1]))
                      setattr(self, tspec[0], tx)
                      tx.setDaemon(True)
                      tx.start()
  
      def __getattribute__(self, name):
          deferred = super(IPDB, self).__getattribute__('_deferred')
          if name in deferred:
              register = []
              spec = deferred[name]
              for plugin in spec:
                  obj = plugin['class'](self, **plugin['kwarg'])
                  setattr(self, plugin['name'], obj)
                  register.append(obj)
                  self._loaded.add(plugin['name'])
                  del deferred[plugin['name']]
              for obj in register:
                  if hasattr(obj, '_register'):
                      obj._register()
                  if hasattr(obj, '_event_map'):
                      for event in obj._event_map:
                          if event not in self._event_map:
                              self._event_map[event] = []
                          self._event_map[event].append(obj._event_map[event])
          return super(IPDB, self).__getattribute__(name)
  
      def register_callback(self, callback, mode='post'):
          '''
          IPDB callbacks are routines executed on a RT netlink
          message arrival. There are two types of callbacks:
          "post" and "pre" callbacks.
  
          ...
  
          "Post" callbacks are executed after the message is
          processed by IPDB and all corresponding objects are
          created or deleted. Using ipdb reference in "post"
          callbacks you will access the most up-to-date state
          of the IP database.
  
          "Post" callbacks are executed asynchronously in
          separate threads. These threads can work as long
          as you want them to. Callback threads are joined
          occasionally, so for a short time there can exist
          stopped threads.
  
          ...
  
          "Pre" callbacks are synchronous routines, executed
          before the message gets processed by IPDB. It gives
          you the way to patch arriving messages, but also
          places a restriction: until the callback exits, the
          main event IPDB loop is blocked.
  
          Normally, only "post" callbacks are required. But in
          some specific cases "pre" also can be useful.
  
          ...
  
          The routine, `register_callback()`, takes two arguments:
              - callback function
              - mode (optional, default="post")
  
          The callback should be a routine, that accepts three
          arguments::
  
              cb(ipdb, msg, action)
  
          Arguments are:
  
              - **ipdb** is a reference to IPDB instance, that invokes
                  the callback.
              - **msg** is a message arrived
              - **action** is just a msg['event'] field
  
          E.g., to work on a new interface, you should catch
          action == 'RTM_NEWLINK' and with the interface index
          (arrived in msg['index']) get it from IPDB::
  
              index = msg['index']
              interface = ipdb.interfaces[index]
          '''
          lock = threading.Lock()
  
          def safe(*argv, **kwarg):
              with lock:
                  callback(*argv, **kwarg)
  
          safe.hook = callback
          safe.lock = lock
          safe.uuid = uuid32()
  
          if mode == 'post':
              self._post_callbacks[safe.uuid] = safe
          elif mode == 'pre':
              self._pre_callbacks[safe.uuid] = safe
          else:
              raise KeyError('Unknown callback mode')
          return safe.uuid
  
      def unregister_callback(self, cuid, mode='post'):
          if mode == 'post':
              cbchain = self._post_callbacks
          elif mode == 'pre':
              cbchain = self._pre_callbacks
          else:
              raise KeyError('Unknown callback mode')
          safe = cbchain[cuid]
          with safe.lock:
              ret = cbchain.pop(cuid)
          return ret
  
      def eventqueue(self, qsize=8192, block=True, timeout=None):
          '''
          Initializes event queue and returns event queue context manager.
          Once the context manager is initialized, events start to be collected,
          so it is possible to read initial state from the system witout losing
          last moment changes, and once that is done, start processing events.
  
          Example:
  
              ipdb = IPDB()
              with ipdb.eventqueue() as evq:
                  my_state = ipdb.<needed_attribute>...
                  for msg in evq:
                      update_state_by_msg(my_state, msg)
          '''
          return _evq_context(self, qsize, block, timeout)
  
      def eventloop(self, qsize=8192, block=True, timeout=None):
          """
          Event generator for simple cases when there is no need for initial
          state setup. Initialize event queue and yield events as they happen.
          """
          with self.eventqueue(qsize=qsize, block=block, timeout=timeout) as evq:
              for msg in evq:
                  yield msg
  
      def release(self):
          '''
          Shutdown IPDB instance and sync the state. Since
          IPDB is asyncronous, some operations continue in the
          background, e.g. callbacks. So, prior to exit the
          script, it is required to properly shutdown IPDB.
  
          The shutdown sequence is not forced in an interactive
          python session, since it is easier for users and there
          is enough time to sync the state. But for the scripts
          the `release()` call is required.
          '''
          with self._shutdown_lock:
              if self._stop:
                  log.warning("shutdown in progress")
                  return
              self._stop = True
              self._cbq.put(ShutdownException("shutdown"))
  
              if self._mthread is not None:
                  self._flush_mnl()
                  self._mthread.join()
  
              if self.mnl is not None:
                  self.mnl.close()
                  self.mnl = None
  
              if self._nl_own:
                  self.nl.close()
                  self.nl = None
  
              self._flush_db()
  
      def _flush_mnl(self):
          if self.mnl is not None:
              # terminate the main loop
              for t in range(3):
                  try:
                      msg = ifinfmsg()
                      msg['index'] = 1
                      msg.reset()
                      self.mnl.put(msg, RTM_GETLINK)
                  except Exception as e:
                      log.error("shutdown error: %s", e)
                      # Just give up.
                      # We can not handle this case
  
      def create(self, kind, ifname, reuse=False, **kwarg):
          return self.interfaces.add(kind, ifname, reuse, **kwarg)
  
      def ensure(self, cmd='add', reachable=None, condition=None):
          if cmd == 'reset':
              self._ensure = []
          elif cmd == 'run':
              for f in self._ensure:
                  f()
          elif cmd == 'add':
              if isinstance(reachable, basestring):
                  reachable = reachable.split(':')
                  if len(reachable) == 1:
                      f = partial(test_reachable_icmp, reachable[0])
                  else:
                      raise NotImplementedError()
                  self._ensure.append(f)
              else:
                  if sys.stdin.isatty():
                      pprint(self._ensure, stream=self._stdout)
          elif cmd == 'print':
              pprint(self._ensure, stream=self._stdout)
          elif cmd == 'get':
              return self._ensure
          else:
              raise NotImplementedError()
  
      def items(self):
          # TODO: add support for filters?
  
          # iterate interfaces
          for ifname in getattr(self, 'by_name', {}):
              yield (('interfaces', ifname), self.interfaces[ifname])
  
          # iterate routes
          for table in getattr(getattr(self, 'routes', None),
                               'tables', {}):
              for key, route in self.routes.tables[table].items():
                  yield (('routes', table, key), route)
  
      def dump(self):
          ret = {}
          for key, obj in self.items():
              ptr = ret
              for step in key[:-1]:
                  if step not in ptr:
                      ptr[step] = {}
                  ptr = ptr[step]
              ptr[key[-1]] = obj
          return ret
  
      def load(self, config, ptr=None):
          if ptr is None:
              ptr = self
  
          for key in config:
              obj = getattr(ptr, key, None)
              if obj is not None:
                  if hasattr(obj, 'load'):
                      obj.load(config[key])
                  else:
                      self.load(config[key], ptr=obj)
              elif hasattr(ptr, 'add'):
                  ptr.add(**config[key])
  
          return self
  
      def review(self):
          ret = {}
          for key, obj in self.items():
              ptr = ret
              try:
                  rev = obj.review()
              except TypeError:
                  continue
  
              for step in key[:-1]:
                  if step not in ptr:
                      ptr[step] = {}
                  ptr = ptr[step]
              ptr[key[-1]] = rev
  
          if not ret:
              raise TypeError('no transaction started')
          return ret
  
      def drop(self):
          ok = False
          for key, obj in self.items():
              try:
                  obj.drop()
              except TypeError:
                  continue
              ok = True
          if not ok:
              raise TypeError('no transaction started')
  
      def commit(self, transactions=None, phase=1):
          # what to commit: either from transactions argument, or from
          # started transactions on existing objects
          if transactions is None:
              # collect interface transactions
              txlist = [(x, x.current_tx) for x
                        in getattr(self, 'by_name', {}).values()
                        if x.local_tx.values()]
              # collect route transactions
              for table in getattr(getattr(self, 'routes', None),
                                   'tables', {}).keys():
                  txlist.extend([(x, x.current_tx) for x in
                                 self.routes.tables[table]
                                 if x.local_tx.values()])
              transactions = txlist
  
          snapshots = []
          removed = []
  
          tx_ipdb_prio = []
          tx_main = []
          tx_prio1 = []
          tx_prio2 = []
          tx_prio3 = []
          for (target, tx) in transactions:
              # 8<------------------------------
              # first -- explicit priorities
              if tx['ipdb_priority']:
                  tx_ipdb_prio.append((target, tx))
                  continue
              # 8<------------------------------
              # routes
              if isinstance(target, BaseRoute):
                  tx_prio3.append((target, tx))
                  continue
              # 8<------------------------------
              # intefaces
              kind = target.get('kind', None)
              if kind in ('vlan', 'vxlan', 'gre', 'tuntap', 'vti', 'vti6',
                          'vrf'):
                  tx_prio1.append((target, tx))
              elif kind in ('bridge', 'bond'):
                  tx_prio2.append((target, tx))
              else:
                  tx_main.append((target, tx))
              # 8<------------------------------
  
          # explicitly sorted transactions
          tx_ipdb_prio = sorted(tx_ipdb_prio,
                                key=lambda x: x[1]['ipdb_priority'],
                                reverse=True)
  
          # FIXME: this should be documented
          #
          # The final transactions order:
          # 1. any txs with ipdb_priority (sorted by that field)
          #
          # Then come default priorities (no ipdb_priority specified):
          # 2. all the rest
          # 3. vlan, vxlan, gre, tuntap, vti, vrf
          # 4. bridge, bond
          # 5. routes
          transactions = tx_ipdb_prio + tx_main + tx_prio1 + tx_prio2 + tx_prio3
  
          try:
              for (target, tx) in transactions:
                  if target['ipdb_scope'] == 'detached':
                      continue
                  if tx['ipdb_scope'] == 'remove':
                      tx['ipdb_scope'] = 'shadow'
                      removed.append((target, tx))
                  if phase == 1:
                      s = (target, target.pick(detached=True))
                      snapshots.append(s)
                  # apply the changes, but NO rollback -- only phase 1
                  target.commit(transaction=tx,
                                commit_phase=phase,
                                commit_mask=phase)
                  # if the commit above fails, the next code
                  # branch will run rollbacks
          except Exception:
              if phase == 1:
                  # run rollbacks for ALL the collected transactions,
                  # even successful ones
                  self.fallen = transactions
                  txs = filter(lambda x: not ('create' ==
                                              x[0]['ipdb_scope'] ==
                                              x[1]['ipdb_scope']), snapshots)
                  self.commit(transactions=txs, phase=2)
              raise
          else:
              if phase == 1:
                  for (target, tx) in removed:
                      target['ipdb_scope'] = 'detached'
                      target.detach()
          finally:
              if phase == 1:
                  for (target, tx) in transactions:
                      target.drop(tx.uid)
  
          return self
  
      def watchdog(self, wdops='RTM_NEWLINK', **kwarg):
          return Watchdog(self, wdops, kwarg)
  
      def _serve_cb(self):
          ###
          # Callbacks thread working on a dedicated event queue.
          ###
  
          while not self._stop:
              msg = self._cbq.get()
              self._cbq.task_done()
              if isinstance(msg, ShutdownException):
                  return
              elif isinstance(msg, Exception):
                  raise msg
              for cb in tuple(self._post_callbacks.values()):
                  try:
                      cb(self, msg, msg['event'])
                  except:
                      pass
  
      def _serve_main(self):
          ###
          # Main monitoring cycle. It gets messages from the
          # default iproute queue and updates objects in the
          # database.
          ###
  
          while not self._stop:
              try:
                  messages = self.mnl.get()
                  ##
                  # Check it again
                  #
                  # NOTE: one should not run callbacks or
                  # anything like that after setting the
                  # _stop flag, since IPDB is not valid
                  # anymore
                  if self._stop:
                      break
              except Exception as e:
                  with self.exclusive:
                      if self._evq:
                          self._evq.put(e)
                          return
                  if self.restart_on_error:
                      log.error('Restarting IPDB instance after '
                                'error:\n%s', traceback.format_exc())
                      try:
                          self.initdb()
                      except:
                          log.error('Error restarting DB:\n%s',
                                    traceback.format_exc())
                          return
                      continue
                  else:
                      log.error('Emergency shutdown, cleanup manually')
                      raise RuntimeError('Emergency shutdown')
  
              for msg in messages:
                  # Run pre-callbacks
                  # NOTE: pre-callbacks are synchronous
                  for (cuid, cb) in tuple(self._pre_callbacks.items()):
                      try:
                          cb(self, msg, msg['event'])
                      except:
                          pass
  
                  with self.exclusive:
                      event = msg.get('event', None)
                      if event in self._event_map:
                          for func in self._event_map[event]:
                              func(msg)
  
                      # Post-callbacks
                      try:
                          self._cbq.put_nowait(msg)
                          if self._cbq_drop:
                              log.warning('dropped %d events',
                                          self._cbq_drop)
                              self._cbq_drop = 0
                      except queue.Full:
                          self._cbq_drop += 1
                      except Exception:
                          log.error('Emergency shutdown, cleanup manually')
                          raise RuntimeError('Emergency shutdown')
  
                      #
                      # Why not to put these two pieces of the code
                      # it in a routine?
                      #
                      # TODO: run performance tests with routines
  
                      # Users event queue
                      if self._evq:
                          try:
                              self._evq.put_nowait(msg)
                              if self._evq_drop:
                                  log.warning("dropped %d events",
                                              self._evq_drop)
                                  self._evq_drop = 0
                          except queue.Full:
                              self._evq_drop += 1
                          except Exception as e:
                              log.error('Emergency shutdown, cleanup manually')
                              raise RuntimeError('Emergency shutdown')
  