| User's Guide |
|
Definitions |
Configuration |
|
Administration File Structure Advanced Usage |
|
Execution |
Builder Builder is a package of software for doing a project's builds. It is project and platform independent, requiring no local modifications. All project and platform specific information is stored in external configuration files. In addition, these configuration files allow multiple, diverse build scenarios to defined and executed.
See the builder feature page for more details.Scenario Build scenarios are those build operations that you've defined for your project. A scenario might be the nightly build for a particular Sablime generic, for example. These scenarios can contain build node clean out operations, getversions, node copies, MR promotions, or other operations. You use builder to configure as many scenarios as you like, so that when you wish to run a build, you just instruct builder which scenario to execute, and it does the rest.
Project Some of the configuration information about a build may be applicable to multiple scenarios of your project. Builder's configurations are divided in two levels, the upper level is called the project and contains the more general items that apply to multiple scenarios. Below the project can be multiple scenarios which describe the details of that particular build operation.
back to top
We recommend installing builder into the build administrator's home directory in a sub-directory named builder. However, since ALL builder code and files are referenced relative to the environment variable BUILDERDIR, it really doesn't matter where that is.
There will be a substantial number of files created and stored under BUILDERDIR, so the file system that contains BUILDERDIR should be one with plenty of disk space.
back to topWe recommend that the builder owner be the same person who typically runs builds (the build administrator). On many machines, that is the Sablime user ID; but that is not a requirement.
When you run the nmake that installs builder into place under BUILDERDIR, it creates several setuid programs under BUILDERDIR/bin. These are setuid to the builder owner (who was the one running the nmake). These setuid programs make it possible for other people to run builder and still be able to write files to the BUILDERDIR area.
back to topSee the separate Instructions page for complete download and installation instructions.
back to topA single builder installation can be used to support builds on multiple platforms. Lets say, for example, that you have a file server that is shared by Suns and by HPs, and you wish to build on both platforms. If you install builder into a shared disk, you can use that single installation to support both builds.
See the Installation Instructions for how to install builder for shared installations.
See the separate User Guide entry for how to make use of the shared installation at runtime.
See the separate Getting Started page for instructions on how to begin using builder.
back to topYou have to have the environment variable BUILDERDIR set in order for builder to run. Set BUILDERDIR to the path to where builder was installed, then type $BUILDERDIR/bin/builder. If this is the first time you've run builder on this machine, it will automatically put you into the configuration mode, and will begin to ask you questions about your build environment. If you are the current build administrator on the machine, you'll probably have most of the answers handy. You can type help at any prompt, and builder will give you a more verbose explanation of what information it is requesting.
back to topBUILDERDIR must be defined (or supplied with the "-b" command line option) for builder to run. Putting BUILDERDIR in your PATH only helps you locate builder. builder itself doesn't care.
back to topSince builder requires that BUILDERDIR be defined, you can define it in the cron entry, and then execute builder. For example:
10 20 * * 0-5 /bin/ksh -c "export BUILDERDIR=/prime/gsa/builder;/prime/gsa/builder/bin/builder -p myprod1.0 -s nightly"
You can also use the "-b" option to specify BUILDERDIR:
10 20 * * 0-5 /bin/ksh -c "/prime/gsa/builder/bin/builder -p myprod1.0 -s nightly -b/prime/gsa/builder"
back to topbuilder requires no command line options. If you only have one project defined, it will use that one. For each project there is a default scenario, so you don't really need to specify that either. All the builder options can be configured into the scenario. You can, though, configure builder to accept options from the command line, and if you have multiple projects and scenarios you'll probably want to at least use the -p <project name> and -s <scenario name> options. You can use builder -h to get a complete list of the available options in man page format.
back to topbuilder -h will display a full help screen (several actually). If you're already running builder, you can type "help" AT ANY PROMPT, and it will give you context specific assistance.
back to topYou can type exit or quit at any prompt within builder, and the program will quit. If you were updating an existing project or scenario, the configuration parameters will be restored to their original values.
back to top
Making a new project or scenario
The first time you run builder on a machine, it will automatically put you into configuration mode; creating a new project and scenario.
You can user builder -f and/or -F to force re-configuration of a project (the -f) or the scenario (the -F). When it asks which project or scenario you want to re-configure, give it the new name. OR you can just type builder -p <new project name> and/or -s <new scenario name>. builder will notice that the named project/scenario doesn't exist and ask if you want to create it.
back to topChanging a project or scenario
You can type builder -f or builder -f -p <project name> to change a project.
You can type builder -F to cause builder to begin re-configuring an existing scenario. Use builder -s <scenario> -F to select a specific scenario.Builder will then step through each of the configuration items and ask you if the current value is OK. If you say yes, it'll go on to the next one, otherwise it will ask you for a new value for that item.
There are two shortcuts for configuration:
1) If you have already gotten to the item you wanted to change, answer YES! to the next Is that OK? question and builder will assume a yes answer for all the questions from there on out.
2) If you just want to skip to the next node, answer NEXT! to an Is that OK? question, and builder will assume a yes answer until the start of the next viewpath node.
back to topFirst, place a file named mail_template into the scenario directory ($BUILDERDIR/data/builds/<project>/<scenario>). That file, as the name says, is the template for your email. Any text in that file gets sent directly to the email, except for any of the following keywords which get replaced by the corresponding data. The keywords must be at the beginning of a line.
| Keyword |
|
|
|
The builder banner and version line |
|
|
The date/time that builder started |
|
|
The date/time that the build tool (nmake) was called |
|
|
The date/time when builder finished |
|
|
The date/time when the build tool (nmake) finished |
|
|
Information about the MRs on the source node. See below for variations. |
|
the MR Abstracts get displayed too. If the string "BASEONLY" is present on the same line as "MR_DATA", then only the MRs that began in the lowest configured State get reported on. If the string "UMRDATA" is present, then information about the included UMRs is also displayed |
|
|
|
The raw (complete) getversion output from each node where getversions were performed |
|
|
The raw (complete) output from the node_update command on each node where getversions were performed. |
|
|
The output from the KSH setup script |
|
|
The output from the configured "gating" script. |
|
|
Processing information on node copies, MR promotions, etc. that were configured to happen before the build |
|
|
Processing information on node copies, MR promotions, etc. that were configured to happen after the build |
|
|
A general summary of the build results |
|
|
The complete build output. |
|
|
Same as above, folded to 80 character line widths |
|
|
The Errors "grep"d from the Build Output |
|
|
A Categorized summary of the Build Errors |
|
|
Warnings "grep"d from the Build Output |
| BUILD_ERRS_WITH_MARKERS | The Build errors from the build output along with the START/FINISH
lines. (See below for maker definitions) |
|
|
The output from an "env" command run right before the build tool launch |
|
|
The output from the configured Post-Build script. |
|
|
A listing of the builder project, scenario, and node parameters in effect for this builder operation |
|
|
The Start and Finish times of many of the builder internal operations. |
So, if you wanted your email to look like this:
builder started at Tue Jun 22 07:54:00 EDT 1999
Your mail_template file should look like this:
To use the BUILD_ERRS_WITH_MARKERS keyword, you specify the Marker Strings after the keyword. In the above example template, the string BUILD_ERRS_WITH_MARKERS:START:END specifies that the Marker Strings are "START" and "END". The build output is grep'd for these strings as well as for the actual error and warning strings.
If you do not have a mail_template file in the $BUILDERDIR/<project>/<scenario> directory, the email format will revert to the default.
back to topBuild warnings in email output
The builder distribution places a file named warning_template in your $BUILDERDIR/extras directory. Take that file, place it into $BUILDERDIR/data/<project>/<scenario> and rename it warning_list. Then edit the file so it contains the strings that you want to be found in the build output and reported as warnings. In the example above, the warning "Warning: your license will expire some day" shows up in the build error report because the string "license will expire" was present in the warning_list file.
You can place the keyword BUILD_WARNINGS in your mail_template to cause only the warning output to be displayed. If you use the BUILD_ERRS_WITH_MARKERS keyword, the warnings will automatically be displayed as well. You must create a warning_list file, though, for either of them to occur.
back to top
The directories in BUILDERDIR are bin - containing the builder executables; lib - containing other builder functions; shipment - where you downloaded the distribution; and data - containing all the builder configuration and run-time files.
The bin directory also contains some builder utilities that aren't part of builder itself. In addition, there is an extras directory containing some sample files and special purpose files. See the Builder Extras and Utilities Page.The configuration files are stored in the builds sub-directory under the data directory: the project configuration items in the directory named for the project, and the scenario configuration files in sub-directories underneath the project.
The nodes sub-directory under the data directory contains the lock and log files for each build node. The log files store information about what activities happen on each build node, independent of which scenario performed the operation.
back to topEach time you run builder, it creates a whole set of output files designed to allow you to determine how things went. These are stored under the BUILDERDIR/data/builds/<project>/<scenario> directory in a directory named Dnnnnnnnnnn, where nnnnnnnnnn is the YYYYMMDDHHMM when the build was launched.
The file names should be fairly well explain their contents.Of course, most of the information that's important will have been sent to you by email, but occasionally it is necessary to look at the raw output.
back to topThe file "OperationsLog" contains detailed information about when each of the builder operations begins and ends, and can be helpful in debugging a builder operation. If you want builder to display more detailed information as it runs, set the environment variable DEBUG=true. You can get even more output by using the "-x" command line option. These debug settings were designed to help in the debug of the builder tool itself, though, so their applicability to user-level problem resolution is hit-and-miss.
back to topIf you have started a build operation and you need to terminate it, you should use the appropriate commands for your machine (kill -9 <pid>) to kill the nmake process(es). You can also kill the builder process itself using the same method. Since builder probably launched nmake and nmake may have launched any number of other processes, there is no effective way for builder itself to kill a build.
Having done that, you may need to remove nmake ".ml" files from the build node. Builder may have left behind some of its own lock files too, so you need to remove them using the "unlock_nodes" utility provided with the builder distribution. (unlock_nodes -p <project> -s <scenario>).
back to topBuilder keeps some lock files around so that it will know when another builder instance is using the build nodes. Builder usually cleans up after itself, but if the machine goes down, or somebody kills builder, then it may not remove the lock files. If you find yourself needing to remove the builder lock files, run the utility "unlock_nodes" (unlock_nodes -p <project> -s <scenario>).
back to top(Need to update this to show Builder2 statistics)
The builder distribution places an executable script named "statistics" into your $BUILDERDIR/bin.You can run statistics -h to get help information, but here's some examples:% statistics -p ccms9.0-s night -v -f # ######## builder - Software from Columbus STC Software Realization Services ######## reporting on all builds of ######## project ccms9.0 - scenario night # Start Time No. of MRs No. of Errs Finish Time # ---------- ---------- ----------- ----------- 9702271347 0 1027 9702280005 9702280010 0 0 9702280010 9703010010 0 0 9703010011 9703020010 0 0 9703020011 9703030010 0 0 9703030011 # ---------- ---------- ----------- ---------- 5 0 1027As you can see, each builder run for the specified scenario is reported, along with the number of MRs that were getversioned and the number of errors that were found. The start and finish times are also reported, and the elapsed time can be determined by simple math from those two.
The total number of builds, number of MRs and number of errors are also reported. The comment lines (beginning with "#") appear only if you specify "-v".
The statistics utility only will report on builder operations since the previous time that statistics was run. If you want to report on all builder operations, use the -f option. The report is displayed to the screen, and it is also stored in your $BUILDERDIR/<project>/<scenario> directory and named report.yymmddhhMM. If you want to keep regular statistics, you might want to put statistics -p <project> -s <scenario> -v into your "cron" table to run on the first of each month (for example).
back to top
Copy Gate and Prom Gate UtilitiesThese utilities are in $BUILDERDIR/bin, and permit the user to modify the "gating" behavior of builder with regard to node copies and MR promotions. In the normal case, builder will do these copies or promotions only if the number of errors is less than the defined error limit. With copy_gate and prom_gate, the user can specify that builder should unconditionally copy/promote or unconditionally NOT copy/promote. Furthermore, this can be specified to apply only for the next execution and then to revert to other behavior.
Each script displays a selection menu with these entries (table refers to "copy", "prom" is the same):
| Selection (Flag) | Explanation |
| copy_if_ok_repeat | Same as builder default behavior |
| force_nocopy_repeat | Unconditionally NOT copy |
| force_copy_repeat | Unconditionally copy |
| copy_if_ok_revert_to_force_copy | Conditionally copy this time, and then set the flag to force_copy_repeat for next time |
| copy_if_ok_revert_to_force_nocopy | Conditionally copy this time, and then set the flag to force_nocopy_repeat for next time |
| force_nocopy_revert_to_copy_ifok | Unconditionally NOT copy this time, and then set the flag to copy_if_ok_repeat |
| force_nocopy_revert_to_force_copy | Unconditionally NOT copy this time, and then set the flag to force_copy_repeat |
| force_copy_revert_to_copy_ifok | Unconditionally copy this time, and then set the flag to copy_if_ok_repeat |
| force_copy_revert_to_force_nocopy | Unconditionally copy this time, and then set the flag to force_nocopy_repeat |
Chaining builder executions
On some projects, the builds can occasionally take so long that one day's build is not finished by the time the next should be starting. Still, the project would like nightly builds. One solution to this is to chain the builder executions, so that builder re-schedules itself when it completes. For example, instead of having builder launched from cron, use the "PreFinish" hook (see Hooks) to have builder re-schedule itself using the "at" command:
echo "/bin/ksh $BUILDERDIR/bin/builder -pmyproj -smyscenario" | at 3am
If the above were put into the "PreFinish" hook, then builder would launch the next builder. If the current build happened to have run past 3 AM, then the next build would not start until the next day. Otherwise they would start once per day. Unlike cron, the operation launched by the "at" command will maintain the caller's environment, so "BUILDERDIR" will already be established.
Of course, the above script could be enhanced to be a little smarter about the re-scheduling - skipping certain week days for example.
This same concept can also be used for calling entirely different scenarios if, for example, your project has builds that depend on other builds to complete before they should launch.
Multiple Pass Builds
Occasionally, a project needs to have multiple nmake operations executed as part of the same build. For example, their Makefiles may be structured so that a complete "nmake" pass must be completed before a second "nmake install" action is useful. Or maybe they build in multiple separate sub-directories (and don't use nmake itself for the directory selection and navigation).
As another example, the nightly build of the "builder" project itself uses a two pass build: the first pass builds the project from the source and installs it into the execution bin. A second pass makes a cpio archive of the contents of the bin. The first pass uses the nmake arguments "-f Makefile.mk local install", while the second uses " package install".
To cause builder to use multiple pass builds, you simply supply builder with multiple sets of arguments to nmake, or supply it with multiple sets of sub-directories. If builder notices that the "build tool options" or the "subdirectories to build" contains the string "###" where the second "#" is an integer 2-9, builder will use the first portion of the string for the first pass, and each succeeding portion for each succeeding pass. For example, if the configured build tool options were "-f Makefile.mk local install #2# package install", builder would see this as a multiple pass build, and use "-f Makefile.mk local install" as the options on the first pass, and "package install" on the second.
This can be extended up through pass 9.
Hooks to User-supplied Code
Basic builder configurations already make use of external code such as the "ksh setup script" and the "gating script", etc. In addition to these, builder recognizes that there may be other times when user-supplied code needs to be executed. Just before or just after the getversion, for example.
Builder has a facility for calling external user-supplied programs at any of several critical points. See the Builder Hooks page for thorough information.
Distributed Builder
Builder comes with utilities to facilitate its use in a geographically distributed development environment. Note that this is not "distributed building" for the purposes of load balancing. That capability is built into "nmake" and is outside of builder's domain.
The Distributed Builder facility is currently in use within Sablime development itself. The documentation is still in the early stages, and should be improved shortly. The Distributed Builder facility makes use of several utilities within $BUILDERDIR/extras, and makes use of the hooks_template stored there.
If you used the shared installation feature to install builder, then you will have separate platform (or architecture) bin directories within $BUILDERDIR/bin. Each of these bins contains those few compiled executables that builder needs. The great majority of the builder software remains in the regular bin (since it is platform independent KSH).
If the variable ARCH is defined in the runtime environment, builder will put $BUILDERDIR/$ARCH/bin into the $PATH before the $BUILDERDIR/bin.
If an executable named "set_arch" is present in $BUILDERDIR/bin, builder will execute it and use the output as $ARCH. This takes precedence over any $ARCH value that just happens to be defined.