netdiff
Netdiff is a simple abstraction layer for parsing network topology data of
open source dynamic routing protocols or any other networking software
which has knowledge about the topology of a network.
Its goal is to allow applications like openwisp-network-topology to collect,
visualize and monitor network topology data without having to deal with
the details of each networking software from which the data is retrieved.
Features:
Install stable version from pypi
Install from pypi:
Install development version
Install tarball:
pip install https://github.com/openwisp/netdiff/tarball/master
Alternatively you can install via pip using git:
pip install -e git+git://github.com/openwisp/netdiff#egg=netdiff
If you want to contribute, install your cloned fork:
git clone git@github.com:<your_fork>/netdiff.git
cd netdiff
python setup.py develop
Basic Usage Example
Calculate diff of an OLSR 0.6.x topology:
from netdiff import OlsrParser
from netdiff import diff
old = OlsrParser(file="./stored-olsr.json")
new = OlsrParser(url="http://127.0.0.1:9090")
diff(old, new)
In alternative, you may also use the subtraction operator:
from netdiff import OlsrParser
from netdiff import diff
old = OlsrParser(file="./stored-olsr.json")
new = OlsrParser(url="http://127.0.0.1:9090")
old - new
The output will be an ordered dictionary with three keys:
Each key will contain a dict compatible with the NetJSON NetworkGraph
format representing respectively:
- the nodes and links that have been added to the topology
- the nodes and links that have been removed from the topology
- the nodes and links that are present in both topologies but their
attributes have changed
If no changes are present, keys will contain None.
So if between old and new there are no changes, the result will
be:
{"added": None, "removed": None, "changed": None}
While if there are changes, the result will look like:
{
"added": {
"type": "NetworkGraph",
"protocol": "OLSR",
"version": "0.6.6",
"revision": "5031a799fcbe17f61d57e387bc3806de",
"metric": "ETX",
"nodes": [
{
"id": "10.150.0.7",
"label": "Node A",
"local_addresses": [],
"properties": {},
},
{
"id": "10.150.0.6",
"label": "Node B",
"local_addresses": ["10.56.2.1"],
"properties": {"hostname": "nodeb.lan"},
},
],
"links": [
{
"source": "10.150.0.3",
"target": "10.150.0.7",
"cost": 1.50390625,
"cost_text": "",
"properties": {},
},
{
"source": "10.150.0.3",
"target": "10.150.0.6",
"cost": 1.0,
"cost_text": "",
"properties": {},
},
],
},
"removed": {
"type": "NetworkGraph",
"protocol": "OLSR",
"version": "0.6.6",
"revision": "5031a799fcbe17f61d57e387bc3806de",
"metric": "ETX",
"nodes": [
{
"id": "10.150.0.8",
"label": "Node C",
"local_addresses": [],
"properties": {},
}
],
"links": [
{
"source": "10.150.0.7",
"target": "10.150.0.8",
"cost": 1.0,
"cost_text": "",
"properties": {},
}
],
},
"changed": {
"type": "NetworkGraph",
"protocol": "OLSR",
"version": "0.6.6",
"revision": "5031a799fcbe17f61d57e387bc3806de",
"metric": "ETX",
"nodes": [],
"links": [
{
"source": "10.150.0.3",
"target": "10.150.0.2",
"cost": 1.0,
"cost_text": "",
"properties": {},
}
],
},
}
Parsers
Parsers are classes that extend netdiff.base.BaseParser and implement
a parse method which is in charge of converting a python data
structure into networkx.Graph object and return the result.
Parsers also have a json method which returns valid NetJSON output.
The available parsers are:
Initialization arguments
Data can be supplied in 3 different ways, in the following order of
precedence:
data: dict or str representing the topology/graph
url: URL to fetch data from
file: file path to retrieve data from
Other available arguments:
Initialization examples
Local file example:
from netdiff import BatmanParser
BatmanParser(file="./my-stored-topology.json")
HTTP example:
from netdiff import NetJsonParser
url = "https://raw.githubusercontent.com/interop-dev/netjson/master/examples/network-graph.json"
NetJsonParser(url=url)
Telnet example with timeout:
from netdiff import OlsrParser
OlsrParser(url="telnet://127.0.1", timeout=5)
HTTPS example with self-signed SSL certificate using verify=False:
from netdiff import NetJsonParser
OlsrParser(url="https://myserver.mydomain.com/topology.json", verify=False)
NetJSON output
Netdiff parsers can return a valid NetJSON NetworkGraph object:
from netdiff import OlsrParser
olsr = OlsrParser(url="telnet://127.0.0.1:9090")
# will return a dict
olsr.json(dict=True)
# will return a JSON formatted string
print(olsr.json(indent=4))
Output:
{
"type": "NetworkGraph",
"protocol": "OLSR",
"version": "0.6.6",
"revision": "5031a799fcbe17f61d57e387bc3806de",
"metric": "ETX",
"nodes": [
{
"id": "10.150.0.3"
},
{
"id": "10.150.0.2"
},
{
"id": "10.150.0.4"
}
],
"links": [
{
"source": "10.150.0.3",
"target": "10.150.0.2",
"cost": 2.4
},
{
"source": "10.150.0.3",
"target": "10.150.0.4",
"cost": 1.0
}
]
}
Exceptions
All the exceptions are subclasses of
netdiff.exceptions.NetdiffException.
ConversionException
netdiff.exceptions.ConversionException
Raised when netdiff can't recognize the format passed to the parser.
Not necessarily an error, should be caught and managed in order to support
additional formats.
The data which was retrieved from network/storage can be accessed via the
"data" attribute, eg:
def to_python(self, data):
try:
return super().to_python(data)
except ConversionException as e:
return self._txtinfo_to_jsoninfo(e.data)
ParserError
netdiff.exceptions.ParserError
Raised when the format is recognized but the data is invalid.
NetJsonError
netdiff.exceptions.NetJsonError
Raised when the json method of netdiff.parsers.BaseParser does not
have enough data to be compliant with the NetJSON NetworkGraph
specification.
TopologyRetrievalError
netdiff.exceptions.TopologyRetrievalError
Raised when it is not possible to retrieve the topology data (eg: the URL
might be temporary unreachable).
Specialized features
OpenVPN
By default, the OpenVPN parser uses the common name to identify a client,
this was chosen because if the public IP address is used, the same client
will not be recognized if it connects with a different IP address (very
probable since many ISPs use dynamic public IP addresses).
This does not work when the vpn server configuration allows different
clients to use the same common name (which is generally not recommended
anyway).
If you need to support legacy systems which are configured with the
OpenVPN duplicate-cn feature enabled, you can pass
duplicate_cn=True during the initialization of OpenvpnParser. This
will change the behavior of the parser so that each client is identified
by their common name and IP address (and additionally the port used if
there are multiple clients with same common name and IP).
Known Issues
ConnectionError: BadStatusLine
If you get a similar error when performing a request to the jsoninfo
plugin of olsrd (version 0.6 to 0.9) chances are high that http
headers are disabled.
To fix it turn on http headers in your olsrd configuration file, eg:
LoadPlugin "olsrd_jsoninfo.so.0.0"
{
PlParam "httpheaders" "yes" # add this line
PlParam "Port" "9090"
PlParam "accept" "0.0.0.0"
}
Running tests
Install your forked repo:
git clone git://github.com/<your_fork>/netdiff
cd netdiff/
python setup.py develop
Install test requirements:
pip install -r requirements-test.txt
Run tests with:
./runtests.py
./run-qa-checks
Alternatively, you can use the nose2 command (which has a ton of
available options):
nose2
nose2 tests.test_olsr # run only olsr related tests
nose2 tests/test_olsr.py # variant form of the previous command
nose2 tests.test_olsr:TestOlsrParser # variant form of the previous command
nose2 tests.test_olsr:TestOlsrParser.test_parse # run specific test
See test coverage with:
coverage run --source=netdiff runtests.py && coverage report
Contributing
Please refer to the OpenWISP contributing guidelines.
Support
See OpenWISP Support Channels.
Changelog
See CHANGES.
License
See LICENSE.