Watchdog – A Comprehensive Security Scanning and a Vulnerability Management Tool
GitHub - flipkart-incubator/watchdog: Watchdog - A Comprehensive Security Scanning and a Vulnerability Management Tool.
Watchdog  

  






Tool Description

Watchog is an integration of open source security tools aimed to provide a holistic security view for a given domain/IP. The way Watchdog is built, it can be used by product security teams, red teams and also by bug bounty hunters to get a 360° view of any Internet property it scans.
Given a list of domains/IP's it has the capability to perform a network scan, feed the output to open source web app scanners like Google's skip-fish and wapiti, perform tech stack analysis and determine if the stack has any known CVE’s.
Watchdog is designed considering the use case necessary to know all open services and its corresponding technologies for the endpoints you own, exposed over the Internet. As a company grows, it’s foot-prints grow on the World Wide Web leaving it's product security team with herculean task of maintaining an inventory of all the services and technologies exposed. This becomes further crucial at the event of a zero-day outbreak for a particular protocol or a third party product which might affect public endpoints of the company
WatchDog has the ability to scan all endpoints and perform technology version analysis on the services it detects and map this information with it’s rich CVE database maintained and updated locally.
Scan Engine:


Nmap
Skipfish
Wapiti
BuiltWith
Phantalyzer
Wappalyzer

Databases and collections:

Watchdog installs a local copy of CVE database which is a collection of following DB's :

cves (Common Vulnerabilities and Exposure items) - source NVD NIST
cpe (Common Platform Enumeration items) - source NVD NIST
cwe (Common Weakness Enumeration items) - source NVD NIST
capec (Common Attack Pattern Enumeration and Classification) - source NVD NIST
ranking (ranking rules per group) - local cve-search
d2sec (Exploitation reference from D2 Elliot Web Exploitation Framework) - source d2sec.com
MITRE Reference Key/Maps - source MITRE reference Key/Maps
ms - (Microsoft Bulletin (Security Vulnerabilities and Bulletin)) - source Microsoft
exploitdb (Offensive Security - Exploit Database) - source offensive security
info (metadata of each collection like last-modified) - local cve-search
via4 VIA4CVE cross-references.

What happens when you run watchdog:

Test domain: www.scanthis.com
Watchdog will perform following task on this domain:
a. Scan the domain to find visible open ports.
{e.g. output}
* 80  [Apache httpd 2.4.7 ((Debian))]
* 443 [Apache httpd 2.4.7 ((Debian))]
* 22  [OpenSSH 5.8p1_hpn13v10 (FreeBSD 20110102; protocol 2.0)]
* 21  [ProFTPD 1.3.3e]
* 993 [Plesk Courier imapd]

b. Perform tech-stack fingerprinting and identify all front-end and service level technologies running.
* jquery [1.8.1]
* php [5.5.9]
* twitter bootstrap [2.3]
* font awesome [**]
* google analytics [**]
* piwik []

c. Map the tech-stack versions with known vulnerabilities found in the master CVE database.
* [e.g. jquery 1.8.1 version has multiple CVE's - CVE-2012-6708, CVE-2015-9251]
* [e.g. php 5.5.9 version has multiple CVE's - CVE-2016-4073, CVE-2015-8835]
* [e.g. apache 2.4.7 version has multiple CVE's - CVE-2017-7679, CVE-2014-0226]

d. If step 1 detects any http services running [80/443] it will go ahead and perform a web application security scanning with wapiti and Skipfish.
f. Once the scan is complete the data will get populated on Watchdog’s UI which can be found at http://localhost/index.php
Installing Watchdog:

Quick Installation Video Link
Prerequisites & Softwares

Install PyV8:
Incase of any issues while installing, follow the below steps (workaround for Ubuntu 16+)
$ export LC_ALL=C
$ cd /tmp
$ pip install -e git://github.com/brokenseal/PyV8-OS-X#egg=pyv8
$ git clone https://github.com/emmetio/pyv8-binaries.git
$ unzip pyv8-binaries/pyv8-linux64.zip (or unzip appropriate zip file based on kernel version)
$ mv *PyV8* src/pyv8/pyv8/.

Clone Watchdog repository
$ git clone https://github.com/flipkart-incubator/watchdog.git
$ cd watchdog

Update the subdomains.txt file with your target subdomains
scanme.nmap.org
testphp.vulnweb.com

Run the installation script below
$ sudo chmod +x install.sh
$ sudo ./install.sh
* During installation, script prompts for web root directory. 
  Default directory /var/www/html will be taken automatically if not provided explicitly with-in 10 secs

Scanning with Watchdog:

Watchdog can be run using following command:

$ sudo python run.py

root@projectWatchdog:/watchdog# python run.py
usage: run.py [-h] [-c] [-iA INVENTORY_APPEND] [-iR INVENTORY_REPLACE]
              [-u {install,map,update}] [-s]

optional arguments:
  -h, --help            show this help message and exit
  -c, --config          to configure db structure
  -iA INVENTORY_APPEND, --inventory-append INVENTORY_APPEND
                        to append target to IP Inventory
  -iR INVENTORY_REPLACE, --inventory-replace INVENTORY_REPLACE
                        to replace targets in IP Inventory
  -u {install,map,update}, --updateCVEs {install,map,update}
                        to configure or update CVE database
  -s, --start           to start scanning engine

Configuring CVE-DB
a. Install cve-db using below command (Required to run at least once)
$sudo python run.py -u install

b. Map cves with cpes using below command [Required to run at-least once. The first run generally takes around 30~45 mins for the entire db to get populated. Recommended time: minimum 30mins]
$sudo python run.py -u map

c. Update the DB by using below command (optional / can run this once a month)
$sudo python run.py -u update

Adding new domains to the scan database:
Update the scan database with subdomains.txt file with new domains/IP's and run below commands
$ sudo python run.py -iA subdomains.txt (for appending targets to existing inventory)

$ sudo python run.py -iR subdomains.txt (for replacing targets in existing inventory)

Start Scanning:


Frontend can be accessed from http://localhost/index.php (or replace localhost with your web server address)
Dashboard Screenshots





Future Enhancements


The next release of watchdog will also support code scanning. Given a organisation's github link it will perform vulnerability analysis mapping with CVE database.
More tools to get added to the external scanning engine.

Contribution:

Lead Developer

Mohan KK @MohanKallepalli

Project Lead

Prajal Kulkarni

Project Team

Shubham Bansal
Prabhav Adhikari
Rohit Agrawal

Credits


Flipkart Security Team
NMAP
Wapiti
Skipfish
Phantalyser
CVE Search

