Nmap API

NSE scripts have access to several Nmap facilities for writing flexible and elegant scripts. The API provides target host details such as port states and version detection results. It also offers an interface to the Nsock library for efficient network I/O.

Information Passed to a Script

An effective Nmap scripting engine requires more than just a Lua interpreter. Users need easy access to the information Nmap has learned about the target hosts. This data is passed as arguments to the NSE script's action method. The arguments, host and port, are Lua tables which contain information on the target against which the script is executed. If a script matched a hostrule, it gets only the host table, and if it matched a portrule it gets both host and port. The following list describes each variable in these two tables.

host

This table is passed as a parameter to the rule and action functions. It contains information on the operating system run by the host (if the -O switch was supplied), the IP address and the host name of the scanned target.

host.os

An array of OS match tables. An OS match consists of a human-readable name and an array of OS classes. Each OS class consists of a vendor, OS family, OS generation, device type, and an array of CPE entries for the class. (See the section called “Decoding the Reference Fingerprint Format” for a description of OS match fields.) Fields may be nil if they are not defined. The host.os table has this overall structure:

host.os = {
  {
    name = <string>,
    classes = {
      {
        vendor = <string>,
        osfamily = <string>,
        osgen = <string>,
        type = <string>,
        cpe = {
          "cpe:/<...>",
          [More CPE]
        }
      },
      [More classes]
    },
  },
  [More OS matches]
}

For example, an OS match on this nmap-os-db entry:

Fingerprint Linux 2.6.32 - 3.2
Class Linux | Linux | 2.6.X | general purpose
CPE cpe:/o:linux:linux_kernel:2.6
Class Linux | Linux | 3.X | general purpose
CPE cpe:/o:linux:linux_kernel:3

will result in this host.os table:

host.os = {
  {
    name = "Linux 2.6.32 - 3.2",
    classes = {
      {
        vendor = "Linux",
        osfamily = "Linux",
        osgen = "2.6.X",
        type = "general purpose",
        cpe = { "cpe:/o:linux:linux_kernel:2.6" }
      },
      {
        vendor = "Linux",
        osfamily = "Linux",
        osgen = "3.X",
        type = "general purpose",
        cpe = { "cpe:/o:linux:linux_kernel:3" }
      }
    },
  }
}

Only entries corresponding to perfect OS matches are put in the host.os table. If Nmap was run without the -O option, then host.os is nil.

host.ip

Contains a string representation of the IP address of the target host. If the scan was run against a host name and its DNS lookup returned more than one IP addresses, then the same IP address is used as the one chosen for the scan.

host.name

Contains the reverse DNS entry of the scanned target host represented as a string. If the host has no reverse DNS entry, the value of the field is an empty string.

host.targetname

Contains the name of the host as specified on the command line. If the target given on the command line contains a netmask or is an IP address the value of the field is nil.

host.reason

Contains a string representation of the reason why the target host is in its current state. The reason is given by the type of the packet that determined the state. For example, an echo-reply from an alive host.

host.reason_ttl

Contains the TTL value of the response packet, that was used to determine the status of the target host, when it arrived. This response packet is the packet that is also used to set host.reason.

host.directly_connected

A Boolean value indicating whether or not the target host is directly connected to (i.e. on the same network segment as) the host running Nmap.

host.mac_addr

MAC address of the destination host (six-byte-long binary string) if available, otherwise nil. The MAC address is generally only available for hosts directly connected on a LAN and only if Nmap is doing a raw packet scan such as SYN scan.

host.mac_addr_next_hop

MAC address of the first hop in the route to the host, or nil if not available.

host.mac_addr_src

Our own MAC address, which was used to connect to the host (either our network card's, or (with --spoof-mac) the spoofed address).

host.interface

A string containing the interface name (dnet-style) through which packets to the host are sent.

host.interface_mtu

The MTU (maximum transmission unit) for host.interface, or 0 if not known.

host.bin_ip

The target host's IP address as a 4-byte (IPv4) or 16-byte (IPv6) string.

host.bin_ip_src

Our host's (running Nmap) source IP address as a 4-byte (IPv4) or 16-byte (IPv6) string.

host.times

This table contains Nmap's timing data for the host (see the section called “Round Trip Time Estimation”). Its keys are srtt (smoothed round trip time), rttvar (round trip time variance), and timeout (the probe timeout), all given in floating-point seconds.

host.traceroute

This is an array of traceroute hops, present when the --traceroute option was used. Each entry is a host table with fields name, ip and srtt (round trip time). The TTL for an entry is implicit given its position in the table. An empty table represents a timed-out hop.

host.os_fp

If OS detection was performed, this is a string containing the OS fingerprint for the host. The format is described in the section called “Understanding an Nmap Fingerprint”.

port

The port table is passed to an NSE service script (i.e. only those with a portrule rather than a hostrule) in the same fashion as the host table. It contains information about the port against which the script is running. While this table is not passed to host scripts, port states on the target can still be requested from Nmap using the nmap.get_port_state() and nmap.get_ports() calls.

port.number

Contains the port number of the target port.

port.protocol

Defines the protocol of the target port. Valid values are "tcp" and "udp".

port.service

Contains a string representation of the service running on port.number as detected by the Nmap service detection. If the port.version.service_dtype field is "table", Nmap has guessed the service based on the port number. Otherwise version detection was able to determine the listening service and this field is equal to port.version.name.

port.reason

Contains a string representation of the reason why the target port is in its current state (given by port.state). The reason is given by the type of the packet that determined the state. For example, a RST packet from a closed port or SYN-ACK from an open port.

port.reason_ttl

Contains the TTL value of the response packet, that was used to determine the status of the target port, when it arrived. This response packet is the packet that is also used to set port.reason.

port.version

This entry is a table which contains information retrieved by the Nmap version scanning engine. Some of the values (such as service name, service type confidence, and the RPC-related values) may be retrieved by Nmap even if a version scan was not performed. Values which were not determined default to nil. The meaning of each value is given in the following table:

Table 9.1. port.version values
NameDescription
nameContains the service name Nmap decided on for the port.
name_confidenceEvaluates how confident Nmap is about the accuracy of name, from 1 (least confident) to 10. If port.version.service_dtype is "table", this is 3.
product, version, extrainfo, hostname, ostype, devicetypeThese five variables are the same as those described under <versioninfo> in the section called “match Directive”.
service_tunnelContains the string "none" or "ssl" based on whether or not Nmap used SSL tunneling to detect the service.
service_fpThe service fingerprint, if any, is provided in this value. This is described in the section called “Community Contributions”.
service_dtypeContains the string "table" or "probed" based on whether or not Nmap deduced port.version.name from the nmap-services file or from a service probe match.
cpeList of CPE codes for the detected service. As described in the official CPE specification these strings all start with the cpe:/ prefix.

port.state

Contains information on the state of the port. Service scripts are only run against ports in the open or open|filtered states, so port.state generally contains one of those values. Other values might appear if the port table is a result of the get_port_state or get_ports functions. You can adjust the port state using the nmap.set_port_state() call. This is normally done when an open|filtered port is determined to be open.

Network I/O API

To allow for efficient and parallelizable network I/O, NSE provides an interface to Nsock, the Nmap socket library. The smart callback mechanism Nsock uses is fully transparent to NSE scripts. The main benefit of NSE's sockets is that they never block on I/O operations, allowing many scripts to be run in parallel. The I/O parallelism is fully transparent to authors of NSE scripts. In NSE you can either program as if you were using a single non-blocking socket or you can program as if your connection is blocking. Even blocking I/O calls return once a specified timeout has been exceeded. Two flavors of Network I/O are supported: connect-style and raw packet.

Connect-style network I/O

This part of the network API should be suitable for most classical network uses: Users create a socket, connect it to a remote address, send and receive data and finally close the socket. Everything up to the Transport layer (which is either TCP, UDP or SSL) is handled by the library.

An NSE socket is created by calling nmap.new_socket, which returns a socket object. The socket object supports the usual connect, send, receive, and close methods. Additionally the functions receive_bytes, receive_lines, and receive_buf allow greater control over data reception. Example 9.3 shows the use of connect-style network operations. The try function is used for error handling, as described in the section called “Exception Handling”.

Example 9.3. Connect-style I/O
require("nmap")

local socket = nmap.new_socket()
socket:set_timeout(1000)
try = nmap.new_try(function() socket:close() end)
try(socket:connect(host.ip, port.number))
try(socket:send("login"))
response = try(socket:receive())
socket:close()

Raw packet network I/O

For those cases where the connection-oriented approach is too high-level, NSE provides script developers with the option of raw packet network I/O.

Raw packet reception is handled through a Libpcap wrapper inside the Nsock library. The steps are to open a capture device, register listeners with the device, and then process packets as they are received.

The pcap_open method creates a handle for raw socket reads from an ordinary socket object. This method takes a callback function, which computes a packet hash from a packet (including its headers). This hash can return any binary string, which is later compared to the strings registered with the pcap_register function. The packet hash callback will normally extract some portion of the packet, such as its source address.

The pcap reader is instructed to listen for certain packets using the pcap_register function. The function takes a binary string which is compared against the hash value of every packet received. Those packets whose hashes match any registered strings will be returned by the pcap_receive method. Register the empty string to receive all packets.

A script receives all packets for which a listener has been registered by calling the pcap_receive method. The method blocks until a packet is received or a timeout occurs.

The more general the packet hash computing function is kept, the more scripts may receive the packet and proceed with their execution. To handle packet capture inside your script you first have to create a socket with nmap.new_socket and later close the socket with socket_object:close—just like with the connection-based network I/O.

While receiving packets is important, sending them is certainly a key feature as well. To accomplish this, NSE provides access to sending at the IP and Ethernet layers. Raw packet writes do not use the same socket object as raw packet reads, so the nmap.new_dnet function is called to create the required object for sending. After this, a raw socket or Ethernet interface handle can be opened for use.

Once the dnet object is created, the function ip_open can be called to initialize the object for IP sending. ip_send sends the actual raw packet, which must start with the IP header. The dnet object places no restrictions on which IP hosts may be sent to, so the same object may be used to send to many different hosts while it is open. To close the raw socket, call ip_close.

For sending at a lower level than IP, NSE provides functions for writing Ethernet frames. ethernet_open initializes the dnet object for sending by opening an Ethernet interface. The raw frame is sent with ethernet_send. To close the handle, call ethernet_close.

Sometimes the easiest ways to understand complex APIs is by example. The ipidseq script included with Nmap uses raw IP packets to test hosts for suitability for Nmap's Idle Scan (-sI). The sniffer-detect script also included with Nmap uses raw Ethernet frames in an attempt to detect promiscuous-mode machines on the network (those running sniffers).

Structured and Unstructured Output

NSE scripts should usually return a table representing their output, one that is nicely organized and has thoughtfully chosen keys. Such a table will be automatically formatted for screen output and will be stored as nested elements in XML output. Having XML output broken down logically into keys and values makes it easier for other tools to make use of script output. It is possible for a script to return only a string, but doing so is deprecated. In the past, scripts could only return a string, and their output was simply copied to the XML as a blob of text–this is now known as unstructured output.

Suppose a script called user-list returns a table as shown in this code sample. The following paragraphs show how it appears in normal and XML output.

local output = stdnse.output_table()
output.hostname = "slimer"
output.users = {}
output.users[#output.users + 1] = "root"
output.users[#output.users + 1] = "foo"
output.users[#output.users + 1] = "bar"
return output

A Lua table is converted to a string for normal output. The way this works is: each nested table gets a new level of indentation. Table entries with string keys are preceded by the key and a colon; entries with integer keys simply appear in order. Unlike normal Lua tables, which are unordered, a table that comes from stdnse.output_table will keep its keys in the order they were inserted. Example 9.4, “Automatic formatting of NSE structured output” shows how the example table appears in normal output.

Example 9.4. Automatic formatting of NSE structured output
PORT     STATE SERVICE
1123/tcp open  unknown
| user-list:
|   hostname: slimer
|   users:
|     root
|     foo
|_    bar

The XML representation of a Lua table is constructed as follows. Nested table become table elements. Entries of tables that are not themselves tables become elem elements. Entries (whether table or elem) with string keys get a key attribute (e.g. <elem key="username">foo</elem>); entries with integer keys have no key element and their key is implicit in the order in which they appear.

In addition to the above, whatever normal output the script produces (even if automatically generated) is copied to the output attribute of the script element. Newlines and other special characters will be encoded as XML character entities, for example &#xa;. Example 9.5, “NSE structured output in XML” shows how the example table appears in XML.

Example 9.5. NSE structured output in XML
<script id="t" output="&#xa;hostname: slimer&#xa;users: &#xa;  root&#xa;  foo&#xa;  bar">
  <elem key="hostname">slimer</elem>
  <table key="users">
    <elem>root</elem>
    <elem>foo</elem>
    <elem>bar</elem>
  </table>
</script>

Some scripts need more control their normal output. This is the case, for example, with scripts that need to display complex tables. For complete control over the output, these scripts may do either of these things:

return a string as second return value, or
set the __tostring metamethod on the returned table.

The resulting string will be used in normal output, and the table will be used in XML as usual. The formatted string may contain newline characters to appear as multiple lines.

If the above code example were modified in this way to return a formatted string,

local output = stdnse.output_table()
output.hostname = "slimer"
output.users = {}
output.users[#output.users + 1] = "root"
output.users[#output.users + 1] = "foo"
output.users[#output.users + 1] = "bar"
local output_str = string.format("hostname: %s\n", output.hostname)
output_str = output_str .. "\n" .. stringaux.strjoin(", ", output.users)
return output, output_str

then the normal output would appear as follows:

PORT     STATE SERVICE
1123/tcp open  unknown
| user-list:
|   hostname: slimer
|_  users: root, foo, bar

There are conventions regarding the formatting of certain kinds of data in structured output. Users of NSE output benefit by being able to assume that some kinds of data, for instance dates and times, are formatted the same way, even in different scripts.

Network addresses, for example IPv4, IPv6, and MAC, are represented as strings.

Long hexadecimal strings such as public key fingerprints should be written using lower-case alphabetical characters and without separators such as colons.

Dates and times are formatted according to RFC 3339. If the time zone offset is known, they should appear like these examples:

2012-09-07T23:37:42+00:00
2012-09-07T23:37:42+02:00

If the time zone offset is not known (representing some unspecified local time), leave off the offset part:

2012-09-07T23:37:42

The library function datetime.format_timestamp code exists to format times for structured output. It takes an optional time zone offset in seconds and automatically shifts the date to be correct within that offset.

datetime.format_timestamp(os.time(), 0) --> "2012-09-07T23:37:42+00:00"

Exception Handling

NSE provides an exception handling mechanism which is not present in the base Lua language. It is tailored specifically for network I/O operations, and follows a functional programming paradigm rather than an object-oriented one. The nmap.new_try API method is used to create an exception handler. This method returns a function which takes a variable number of arguments that are assumed to be the return values of another function. If an exception is detected in the return values (the first return value is false), then the script execution is aborted and no output is produced. Optionally, you can pass a function to new_try which will be called if an exception is caught. The function would generally perform any required cleanup operations.

Example 9.6 shows cleanup exception handling at work. A new function named catch is defined to simply close the newly created socket in case of an error. It is then used to protect connection and communication attempts on that socket. If no catch function is specified, execution of the script aborts without further ado—open sockets will remain open until the next run of Lua's garbage collector. If the verbosity level is at least one or if the scan is performed in debugging mode, a description of the uncaught error condition is printed on standard output. Note that it is currently not easily possible to group several statements in one try block.

Example 9.6. Exception handling example
local result, socket, try, catch

result = ""
socket = nmap.new_socket()
catch = function() 
socket:close() 
end
try = nmap.new_try(catch)

try(socket:connect(host.ip, port.number))
result = try(socket:receive_lines(1))
try(socket:send(result))

Writing a function which is treated properly by the try/catch mechanism is straightforward. The function should return multiple values. The first value should be a Boolean which is true upon successful completion of the function and false (or nil) otherwise. If the function completed successfully, the try construct consumes the indicator value and returns the remaining values. If the function failed then the second returned value must be a string describing the error condition. Note that if the value is not nil or false it is treated as true so you can return your value in the normal case and return nil, <error description> if an error occurs.

The Registry

Scripts can share information by storing values in a register, which is a special table that can be accessed by all scripts. There is a global registry with the name nmap.registry, shared by all scripts. Each host additionally has its own registry called host.registry, where host is the host table passed to a script. Information in the registries is not stored between Nmap executions.

The global registry persists throughout an entire scan session. Scripts can use it, for example, to store values that will later be displayed by a postrule script. The per-host registries, on the other hand, only exist while a host is being scanned. They can be used to send information from one script to another one that runs against the same host. When possible, use the per-host registry; this not only saves you from having to make key names unique across hosts, but also allows the memory used by the registry to be reclaimed when it is no longer needed.

Here are examples of using both registries:

The portrule of the ssh-hostkey script collects SSH key fingerprints and stores them in the global nmap.registry so they can be printed later by the postrule.
The ssl-cert script collects SSL certificates and stores them in the per-host registry so that the ssl-google-cert-catalog script can use them without having to make another connection to the server.

Because every script can write to the global registry table, it is important to make the keys you use unique, to avoid overwriting the keys of other scripts (or the same script running in parallel).

Scripts that use the results of another script must declare it using the dependencies variable to make sure that the earlier script runs first.