Alcatel-Lucent nmake: Newsletter No. 9

Highlights

Alcatel-Lucent nmake lu3.4 Released

Version lu3.4 was released June 29, 2001 for all Tier 1 and Tier 2 platforms. For a full list of fixes and enhancements see the release notes. The more significant new features and enhancements include the following:

Introduction of new dependency-based Java build support
Probe hints
Improved support for Sun WorkShopTM 6 C++ compiler
Support for C++ implicit includes
cc-% support for adding flags to CCFLAGS
Scoped CCFLAGS for :: operator
New what_version_nmake command

Check lu3.4 Permissions

For the lu3.4 release we changed from using cpio packages to tar packages because tar is a more widely used format. Unfortunately, since the release we have seen inconsistencies in how tar preserves the setuid bits when extracting the archive. We have seen tar behave differently when writing to different filesystems even on the same machine, sometimes preserving the setuid permissions and sometimes not. The 'p' flag for tar can be used to preserve the file modes, but using this also shows inconsistencies as it can sometimes also restores the owner (apparently depending on how the machine is configured to handle chown), which is not desired. For the time being we have added a step to the installation instructions on the download page to set the proper permissions where necessary. Since this step was not part of previous releases it may easily be over looked, and more than likely people have installed lu3.4 before the step was added to the page. If you have an opinion as to whether we should switch back to cpio, keep a step in the install instructions to set the permissions, or something else, please let us know.

If you have installed lu3.4 take a minute and double check the file permissions on two of the binaries. The files lib/ssd and lib/probe/probe should have the setuid bits set for the owner and lib/ssd should also have it set for the group. The 'ls -l' listing should show the following permissions for these two files:

    -rwsr-xr-x   lib/probe/probe
    -rwsr-sr-x   lib/ssd

If they are not set correctly the owner of these files can set them with the following commands:

    chmod 4755 lib/probe/probe
    chmod 6755 lib/ssd

Technical Notes

lu3.4 Java Support

In issue 7 we discussed the problems associated with building java code. We begin to address these problems with release lu3.4, which includes support for dependency based java builds. The initial java support is accomplished through the use of an external java scanner, called JavaDeps, to determine dependencies on java source code and classes. Dependencies can be tracked across java packages, which is recommended if multiple packages are being developed, for proper and complete maintenance of java code.

Dependency-based java build support is provided through the new :JAVA: operator. If your project is already using a custom defined :JAVA: operator then you will need to remove your custom rule to use the lu3.4 rule since the custom rule will take precedence over the default rule. Here is how to get started using the new :JAVA: operator.

Install the JavaDeps package

JavaDeps is not packaged with the nmake distribution and must be installed independently of the nmake package. Download and install JavaDeps according to the instructions on the JavaDeps download page.

Need a makefile

Unlike building C and C++ code the java makefile does not live in the same directory as the source code. The makefile must be placed at the root of the java package directory. For example, if your java source is in directory src/java/com/lucent/stc/pkg1/ and the full package name is com.lucent.stc.pkg1 then the makefile would be src/java/Makefile. This single makefile can build several java packages under the com/ directory which allows nmake to properly maintain dependencies across packages.

Using `:JAVA:`

Your new makefile will use the new :JAVA: operator, which requires one or more files or directories on the right-hand side. Files must be specified relative from the makefile directory (ie. com/lucent/src/pkg1/file.java) and may include shell pathname wildcards. In most cases it will be easier to specify directories instead of files. Specifying a directory is the same as specifying all .java files under that directory (including subdirectories.)

The left-hand side of :JAVA: takes a single argument of ".". This argument is not currently required but is strongly recommended as it provides compatibility with future enhancements we are planning for java support. Future releases will require the left-hand side argument.

So to build all the java code in all the packages under the com/ directory we simply use the following assertion:

    . :JAVA: com

What else?

If you want your .class files to be placed under a different directory specify the destination directory with the JAVACLASSDEST variable.

Use .SOURCE.class to specify directories, jar files, and zip files in which to find required class files. .SOURCE.class is used the same way .SOURCE.h and .SOURCE.a are used to find headers and libraries. It is important to use .SOURCE.class rather than hard-coding the CLASSPATH variable or the -classpath javac command line argument because relative directories specified in .SOURCE.class will be expanded through the viewpath. In fact nmake uses .SOURCE.class to define a -classpath argument for the java compiler, so there should be no need to set your own -classpath argument.

The basic features of :JAVA: are outlined above. For a complete view of :JAVA: features and usage see the :JAVA: man page.

A final example

If we put everything together above a working makefile will look something like this:

JAVACLASSDEST = $(VROOT)/class

.SOURCE.class : \
	$(VROOT)/external/db.jar	\
	/opt/exp/java/lib/classes.zip

. :JAVA: com

Updating dependencies

The first time you build your java code the files globaljavadeps and localjavadeps will be created by JavaDeps in the current directory. globaljavadeps contains the global dependency table and is used by JavaDeps to merge incremental updates to the dependencies. localjavadeps contains the dependencies using nmake assertions and is read by nmake to define the java dependencies.

In order to save time this version of java support does not automatically update the dependencies each time nmake is run. The user is responsible for updating the dependencies when necessary. The dependency information can be updated by setting the variable javadeps=1, and of course the dependency information will always be generated if the localjavadeps file does not already exist in the viewpath. The following command will update the dependencies and build the necessary java code:

nmake javadeps=1

We recommend official builds and nightly builds be run with javadeps=1. Developers will then have access to the latest dependency information and can do most of their builds without regenerating the dependencies. The developers need to use javadeps=1 only when they make code changes which may impact the dependencies.

The future

The nmake team is continuing to make enhancements for java development. It is likely the user interface to :JAVA: will change for these enhancements by adding a required argument to the left-hand side. We will talk in detail about the future enhancements in the next issue.

lu3.4 Probe Hints

In issue 5 we discussed the idea of probe hints, a way for nmake users to define overrides for specific probe variables. nmake lu3.4 includes our new probe hints feature. This feature allows projects to override the default probe values for probe information any time a compiler is probed (the cpp probe information is not supported.) This allows projects to eliminate manual editing of probe files by installing a probe_hints script.

For project wide use the probe_hints file should be installed as file <nmake_root>/lib/probe/C/make/probe_hints. When using localprobe it may be installed as <localprobe_root>/lib/probe/C/make/probe_hints (the probe_hints under <nmake_root> will be used if one does not exist under <localprobe_root>.) The probe_hints script should have its permissions set to 755.

The probe_hints script should never call 'exit' as this can result in an empty probe file.

The probe_hints script has access to the probed variable values and should override only the variables necessary. Variable names are mapped as follows (for a complete list of variable mapping please see page A-19 of the lu3.4 nmake User's Guide):

Probe Variable	Probe Hints Shell Variable
CC.CC	CC_CC
CC.PIC	CC_PIC
CC.STDINCLUDE	CC_STDINCLUDE
CC.STDLIB	CC_STDLIB
etc.

The variables overridden by probe_hints will come at the end of the probe file following the variable CC.PROBE.HINTS which is defined to the path of the hints script that was used. The probe file will contain both the originally probed values in their normal position in the probe file and the probe_hints values at the end. Since the probe_hints settings are last they will override the probed values.

A simple probe_hints can just reset the necessary variables. For example, if CC.PIC needs to be changed the probe script might look as follows:

#
# probe hints version 1.0
#
CC_PIC="+Z"

In practice this script is probably too simple. If someone uses a compiler that does not use +Z as the pic flag then it will be set wrong. It is a better idea to check the compiler being used and then set the variables appropriately. For example, if we need to change CC.PIC for the aCC compiler we can do the following:

#
# probe hints version 1.1
#
case $CC_CC in
    */aCC) 
	CC_PIC="+Z"
    ;;
esac

Using this method you can easily modify different probe variables for different compilers. It is also possible to use the probed value as part of your modified value. Say we need to add a standard include directory when gcc is used. The script can use "$CC_STDINCLUDE" as part of the definition of CC_STDINCLUDE. For example:

#
# probe hints version 1.2
#
case $CC_CC in
    */aCC) 
	CC_PIC="+Z"
    ;;
    */gcc|*/g++)
	CC_STDINCLUDE="/tools/include/gcc $CC_STDINCLUDE"
    ;;
esac

You may also want to use an nmake variable in one of your variable definitions. In order to get the nmake variable in the probe file you must escape the variable with a backslash so the shell does not interpret it. For example, if we want to add $(MYLIB) to the standard library directories for all compilers we can do the following:

#
# probe hints version 1.3
#
case $CC_CC in
    */aCC) 
	CC_PIC="+Z"
    ;;
    */gcc|*/g++)
	CC_STDINCLUDE="/tools/include/gcc $CC_STDINCLUDE"
    ;;
esac
CC_STDLIB="$CC_STDLIB \$(MYLIB)"

This should give you a good starting point in using the new probe hints feature. For details see page A-16 of the lu3.4 nmake User's Guide.

Proposal for PC Viewpathing

Many projects are using Windows based IDE's for a portion of their software development. This creates some difficultly when using Sablime® and nmake on UNIX for source code management and builds. How do you access your source code in a viewpathing environment from the PC? To answer this question we are experimenting with the idea of using 3d file system and Samba on UNIX to provide a "viewpath" for PC development. 3dfs provides a viewpathing shell environment and Samba allows file sharing using the SMB protocol used by Windows. By running Samba under 3dfs on UNIX and mapping the corresponding drive to Windows the Windows machine should have transparent access to the viewpath defined with 3dfs. As far as Windows can tell it is accessing just another remote drive, while Samba provides files based on the 3dfs viewpath.

This idea is not yet fully tested and we would like to gauge interest in such a solution. Please let us know if you interested in seeing further development of this idea and/or would find it useful.

Tidbits

Clobber

The clobber common action removes generated files such as targets, intermediate object files, and the makefile .mo and .ms files. It is used to clean up the stuff that generated during a build. clobber.install is used to clean up the installed files. Under certain conditions clobber may produce confusing results. In this article we hope to clarify some of its actions.

Here is our example makefile. Note the current directory is the same as BINDIR and LIBDIR.

BINDIR=/home/nmakehome/sds/ygu/test              
LIBDIR=/home/nmakehome/sds/ygu/test              

.MAIN: target2 $(LIBDIR)/target1 /home/nmakehome/target3 ../target4      

$(BINDIR)/target1 :: a.c                         

target2 :LIBRARY: b.c    

/home/nmakehome/sds/ygu/target3 :: c.c                        

../target4 :: d.c

$(LIBDIR) and $(BINDIR) are the default installation directories for libraries and executables. nmake will assume anything under these two directories are installed by running with the install common action - 'nmake install'.

In this example, target1 is generated under $(BINDIR) directly and target2 is under the current directory which happens to be the same location as $(LIBDIR). So both targets need to be cleaned by using clobber.install.

/home/nmakehome/sds/ygu/target3 is not clobbered because targets with absolute pathnames are excluded. This is to prevent global files or directories, such as probe files, from being cleaned by accident. Since /full_path/target format does not work in the viewpath we don't recommend using it.

File ../target4 will be clobbered as expected.

$ pwd
/home/nmakehome/sds/ygu/test              

$ nmake
+ cc -O -I- -c a.c
+ cc -O -o /home/nmakehome/sds/ygu/test/target1 a.o
+ cc -O -I- -c b.c
+ ar r libtarget2.a b.o
ar: creating libtarget2.a
+ rm -f b.o
+ cc -O -I- -c c.c
+ cc -O -o /home/nmakehome/sds/ygu/target3 c.o
+ cc -O -I- -c d.c
+ cc -O -o ../target4 d.o

$ nmake clobber
+ ignore rm -f d.o c.o a.o ../target4 Makefile.mo Makefile.ms

Note that only target4 is clobbered above. Now lets try clobber.install

$ nmake clobber.install
+ ignore rm -f -r /home/nmakehome/sds/ygu/test/target1 /home/nmakehome/s
ds/ygu/test/libtarget2.a

You can see clobber.install deletes target1 and target2 but not target3.

Viewing Output

Reading through build output files can be difficult when the lines become too long for some text editors, such as vi. Here are some other tools you can use to make things easier.

fold: fold is probably the most common solution. fold is a standard unix tool from most unix vendors and it it simply breaks the lines into manageable lengths. One disadvantage of this is when searching for strings in the output; if the line was broken in the middle of your search string then your search will skip it. Try the -s flag to break the lines at the nearest white space when possible.
less: less is a pager similar to more but with more features. It allows paging forwards and backwards in the document and supports vi style navigation and searching. If you know vi you should feel at home using less. less is available in exptools and should be found at /opt/exp/bin/less if your machine has the standard exptools distribution.
vim: vim stands for vi improved and is an enhanced vi clone. vim can handle lines much longer than the standard vi and generally has no problem dealing with large build logs (plus there are many other reasons to use vim over vi.) vim is also available in exptools and should be found at /opt/exp/bin/vim if your machine has the standard exptools distribution.
gnu emacs and xemacs: The popular editors GNU Emacs and its variant xemacs both support arbitrarily long lines and should have no problem reading build logs. Both are available in exptools as /opt/exp/bin/gnuemacs and /opt/exp/bin/xemacs on machines with the standard exptool distribution.

Newsletter Feedback

We are always interested in feedback! Let us know what you think of the newsletter, how we may improve, or ideas you have for future issues. Send us a note at nmake@alcatel-lucent.com. All ideas, suggestions, and comments are welcome.