Host Scanner Scripts Git

Project summary

This repository hosts several miscellaneous utility scripts for the Host Scanner application.

get.sh and convert.sh

The first script downloads all the data files that are required for the various scripts to run. The second one runs the conversions.

When invoking the go scripts directly or via the converter script, the --json argument overrides the default behaviour of the serializeEntries() function within the scripts to dump the global entries list as an indented JSON to the specified output instead of the proprietary binary format. Similarly, the --nogz argument instructs the converter script not to gzip the output file after conversion.

This is useful for either debugging purposes or easy reuse of the data within 3rd-party applications. When reusing, please beware of the licenses under which these datasets are being distributed, as some do not allow commercial usage or restrict the licensing of the combined work.

Format

The file format which the source data is converted to is a generic binary format, having the following header:

┌ uint16      Package type
├ uint16      Package version
└[uint32      Number of entries]

The number of entries field is optional, however all files use it currently.

Strings are stored with a leading length indicator, and no trailing NULL:

┌ uint16      String length
└─ char       Characters

Integer types are encoded using little endian encoding.

cpe2hs.go

Converts NIST's Official Common Platform Enumeration (CPE) Dictionary to the binary format in use by the application.

Entries other than applications (a) and operating systems (o) are filtered, since they are not observed by the main application at this time.

Format

┌ uint16      Package type [0x0100]
├ uint16      Package version [0x0100]
├ uint32      Number of entries
└┬ string     CPE name
 ├ uint8      Number of common tokens
 ├─ string    Token
 ├ uint32     Number of versions
 └┬ string    CPE version
  ├ string    Version token
  ├ uint8     Number of version-specific tokens
  └─ string   Token

cpealt2hs.go

Since NIST's CVE database may use multiple CPE names to refer to the same application, the Debian Security team compiled a list of CPE aliases for use in their Security Tracker.

For example, nginx appears as both cpe:/a:nginx:nginx and cpe:/a:igor_sysoev:nginx, however there are more extreme cases, such as X11, which has 12 CPE names all referring to the same software package.

The alias database is licensed under MIT License (Expat) by the Debian Security team.

Format

┌ uint16      Package type [0x0200]
├ uint16      Package version [0x0100]
├ uint32      Number of entries
└┬ uint16     Number of aliases in entry
 └─ string    CPE name

cve2hs.go

Converts NIST's National Vulnerability Database (NVD) to an SQLite3 database to be queried by the application.

Entries not linked via CPE to at least one application or operating system are filtered, since they are of no use during automatic vulnerability discovery.

In order to run this script, you will need to first install the go-sqlite3 package with:

go get github.com/mattn/go-sqlite3

Since this dependency is a cgo package, you will need gcc in your %PATH%. As Cygwin is not supported by Go, you specifically need the MinGW version on Windows.

Tables

vulns (id int, cve text, date int, descr text, severity float, access char(1))
affected (vuln_id int, cpe text)

The access field represents the access vector, and can be:

• l for local: physical access or local presence is required to exploit.
• a for adjacent: attacker has to reside on the same local network.
• n for network: vulnerability is remotely exploitable over the Internet.

zudp2hs.go

Converts ZMap's UDP payloads to the binary format in use by the application.

The payload list is licensed under Apache License v2.0 by the Regents of the University of Michigan.

Format

┌ uint16      Package type [0x0A00]
├ uint16      Package version [0x0100]
├ uint32      Number of entries
└┬ string     Payload data
 ├ uint16     Number of ports in entry
 └─ uint16    Port number

nudp2hs.go

Converts Nmap's UDP payloads to the binary format in use by the application.

The payload list is licensed under GNU General Public License v2.0 by Insecure.Com LLC.

Format

┌ uint16      Package type [0x0A00]
├ uint16      Package version [0x0100]
├ uint32      Number of entries
└┬ string     Payload data
 ├ uint16     Number of ports in entry
 └─ uint16    Port number

ncpe2hs.go

Converts Nmap's service probes to the binary format in use by the application.

The service probes list is licensed under GNU General Public License v2.0 by Insecure.Com LLC.

Format

┌ uint16      Package type [0x0F00]
├ uint16      Package version [0x0100]
├ uint32      Number of entries
└┬ string     Regular expression
 ├ string     CPE name
 ├ string     Product
 └ string     Version

bsvr2hs.go

Converts Burp Suite Software Version Check's match rules to the binary format in use by the application.

The match rules list is licensed under Creative Commons Attribution-NonCommercial-ShareAlike v3.0 Unported License by August Detlefsen.

Format

┌ uint16      Package type [0x0F00]
├ uint16      Package version [0x0100]
├ uint32      Number of entries
└┬ string     Regular expression
 ├ string     CPE name
 ├ string     Product
 └ string     Version