Concurrency / Distributed Builds

Feature Description

Multiple shell actions (such as compiles) can be run in parallel to increase performance. This can be done by running multiple jobs on a single machine or by distributing jobs to remote machines in the network.

Impact

Performance

• Reduce build times by a factor of 4 to 6.

Productivity

• Shorter build time allows more builds to be run.
• Includes real-time output serializer (nmakelog) so output from concurrent jobs is not intermingled.

Consequence of Not Having the Feature

• Longer, less frequent builds.

Examples

In this example the Makefile makes 26 targets. Instead of compiling code we use virtual targets (no file is generated) with an action to sleep for 4 seconds. This simulates a compilation action that takes 4 seconds for each target.

:ALL: a b c d e f g h i j k l m n o p q r s t u v w x y z

a b c d e f g h i j k l m n o p q r s t u v w x y z : .VIRTUAL
	: $(<)
	sleep 4

The first build will not use concurrency, each target is made in serial. The output is truncated for the sake of brevity.

$ time nmake
+ : a
+ sleep 4
+ : b
+ sleep 4
+ : c
+ sleep 4
...
+ : x
+ sleep 4
+ : y
+ sleep 4
+ : z
+ sleep 4

real    1m45.52s
user    0m0.11s
sys     0m0.40s

Next we'll use the -j10 option, which tells nmake to run up to 10 concurrent shell jobs. Here we see about an 88% reduction in build time.

$ time nmake -j10
+ : a
+ sleep 4
+ : b
+ sleep 4
+ : c
+ sleep 4
...
+ : x
+ sleep 4
+ : y
+ sleep 4
+ : z
+ sleep 4

real    0m12.64s
user    0m0.15s
sys     0m0.54s

The amount of time reduction depends on the speed of the machine. A fast machine can handle more parallel jobs for a greater reduction. A machine with multiple CPU's will will be able to take advantage of the extra CPU's. And large projects can utilize idle machines on the network to distribute the load.

Example using standard make tool

Standard make does not have concurrency features to support multiple, concurrent jobs. GNU make does have a similar feature using the -j flag which results in similar time savings as above under the same conditions. Other make tools require additional utilities to distribute jobs to remote machines.