During its initialization stage, Nmap loads the Lua interpreter and its provided libraries. These libraries are fully documented in the Lua Reference Manual. Here is a summary of the libraries, listed alphabetically by their namespace name:

In addition to loading the libraries provided by Lua, the nmap namespace functions are loaded. The search paths are the same directories that Nmap searches for its data files, except that the nselib directory is appended to each. At this stage any provided script arguments are stored inside the registry.

The next phase of NSE initialization is loading the selected scripts, based on the defaults or arguments provided to the --script option. The version category scripts are loaded as well if version detection was enabled. NSE first tries to interpret each --script argument as a category. This is done with a Lua C function in nse_init.cc named entry based on data from the script.db script categorization database. If the category is found, those scripts are loaded. Otherwise Nmap tries to interpret --script arguments as files or directories. If no files or directories with a given name are found in Nmap's search path, an error is raised and the Script Engine aborts.

If a directory is specified, all of the .nse files inside it are loaded. Each loaded file is executed by Lua. If a portrule is present, it is saved in the porttests table with a portrule key and file closure value. Otherwise, if the script has a hostrule, it is saved in the hosttests table in the same manner.

After initialization is finished, the hostrules and portrules are evaluated for each host in the current target group. The rules of every chosen script is tested against every host and (in the case of service scripts) each open and open|filtered port on the hosts. The combination can grow quite large, so portrules should be kept as simple as possible. Save any heavy computation for the script's action.

Next, a Lua thread is created for each of the matching script-target combinations. Each thread is stored with pertinent information such as the runlevel, target, target port (if applicable), host and port tables (passed to the action), and the script type (service or host script). The mainloop function then processes each runlevel grouping of threads in order.

Nmap performs NSE script scanning in parallel by taking advantage of Nmap's Nsock parallel I/O library and the Lua coroutines language feature. Coroutines offer collaborative multi-threading so that scripts can suspend themselves at defined points and allow other coroutines to execute. Network I/O, particularly waiting for responses from remote hosts, often involves long wait times, so this is when scripts yield to others. Key functions of the Nsock wrapper cause scripts to yield (pause). When Nsock finishes processing such a request, it makes a callback which causes the script to be pushed from the waiting queue back into the running queue so it can resume operations when its turn comes up again.

The mainloop function moves threads between the waiting and running queues as needed. A thread which yields is moved from the running queue into the waiting list. Running threads execute until they either yield, complete, or fail with an error. Threads are made ready to run (placed in the running queue) by calling process_waiting2running. This process of scheduling running threads and moving threads between queues continues until no threads exist in either queue.