How Make build software

This weekend I went through the GNU Make manual http://www.gnu.org/software/make/manual/make.html and basically I have already understood the syntax of Make, Make’s implicit/explicit rules, makefile automatic/defined variables, data structure and functions, and also understood Make’s execute phases (parse makefile to construct directed acyclic graph and execute rules).

As I have already understood the basic of Make, I couldn’t wait to look into OpenJDK’s make system! 🙂 I found a directories diagram about the source codes of OpenJDK ( They called it OpenJDK Mercurial Forest). https://blogs.oracle.com/kto/entry/openjdk_mercurial_forest, It was posted in Oct. 2007 and it was a bit outdated however overall it still reflects the current OpenJDK 8.

Before I describe my understanding of how OpenJDK builds, I want to introduce how Make builds source codes under multi-directories with multi directories levels. For a simply make example (all of the source codes under the same directory), refer to my post Hello World in MAKE. It is not easy if we have to construct a build system with Make to build software with multi-directories. Basically in Make, we have two methods.

Method 1 – Recursive Make:

Take below structure as example, it has one Makefile under the root directory and one Makefile under each sub-directory. This hierarchy of modules can be nested arbitrarily deep. Its method is, makefile will use $(make) to invoke its child makefiles recursively.

The top-level Makefile ofen looks a lot like a shell script:

MODULES = ant bee
all:
for dir in $(MODULES); do \
(cd $$dir; ${MAKE} all); \
done

The ant/Makefile looks like this:

all: main.o
main.o: main.c ../bee/parse.h
$(CC) -I../bee -c main.c

Its directed acyclic graph generated by make in memory should be like below,

So here the logic here is very frank, to build prog, it will recursively build main.o and parse.o. For more details, you might refer to http://aegis.sourceforge.net/auug97.pdf (Recursive Make Considered Harmful), in this thesis, it lists the harmful of recursive make. My understanding is, as in above example it has 2 sub-directories then everything here is fine, however considering we have hundreds of directories and their dependencies are very complicated – in other word, we have to tweak their build sequences, that will be painful. And another pitfall, supposed here main.o needs to invoke a lib generated from another makefile, it will be in risk that main.o will be not built properly as lib might be outdated as here lib is not in the makefile of main.o.

Method 2 – Inclusive Make:

Let’s use below example to explain what is Inclusive Make:

You might have found that, under each subdirectory, it has a .mk file. Inclusive means, in the main entrance of GNU make, it will include the .mk files. In this method, it can be ensure that it always has only one GNU make process (in above Recursive method, it might has more than one process). And the best is, it can manage the dependencies relationship together – that means we will not miss any dependencies and less risk in building outdated or improper software.

The advantages:

1. It has only one GNU make process running and its start up will be faster while in the recursive method, it might invoke hundreds of processes.

2. It still has one makefile to describe the rules for all of the files under each directory. Maintaining only one makefile is nightmare.

3. It maintains all of the dependencies relationship together and reduce the risk of generating improper build artfacts.

4. As said in point 3, its dependencies relationship is maintained by make together hence we don’t need to maintain the $(MAKE) sequences.

Well, let’s talk about its disadvantages 😦

1. It’s more difficult and complicated to compose makefiles as in make, a ‘include’ directive means including the text literal and we have to take care of variables declare, value assign, goals define more carefully!

How OpenJDK maintains makefiles

We have looked into two methods how make builds source codes in different directories with many directories levels. Let’s see into OpenJDK and experience how it maintains its build system structure.

In my post Let’s build openjdk, OpenJDK can be built just in one word: make. Below is its targets http://hg.openjdk.java.net/jdk8/jdk8/raw-file/tip/README-builds.html:

Make Target Description
empty build everything but no images
all build everything including images
all-conf build all configurations
images create complete j2sdk and j2re images
install install the generated images locally, typically in /usr/local
clean remove all files generated by make, but not those generated by configure
dist-clean remove all files generated by both and configure (basically killing the configuration)
help give some help on using make, including some interesting make targets

Let’s see how it works.

1. under common/makefiles/, it has one file Makefile with only one line

include ../../NewMakefile.gmk

2. In NewMakefile.gmk, it has below include snippets:

# ... and then we can include our helper functions
include $(root_dir)/common/makefiles/MakeHelpers.gmk
...
    ifeq ($(words $(SPEC)),1)
        # We are building a single configuration. This is the normal case. Execute the Main.gmk file.
        include $(root_dir)/common/makefiles/Main.gmk
    else
...
include $(root_dir)/common/makefiles/Jprt.gmk
...
help:
	$(info )
	$(info OpenJDK Makefile help)
...
	$(info )

.PHONY: help

3. Let’s see into $(root_dir)/common/makefiles/Main.gmk

In Main.gmk, it will inlcude MakeBase.gmk and others. in the right panel, it lists all of its targets and matches with the table I listed above.

 

Conclusion:

OpenJDK is using Inclusive make to build its source codes. It has one .gmk file under each source directory.

However to support build components individually, OpenJDK also provides Makefile for each component. OpenJDK consists of below components.

Repository Contains
. (root) common configure and makefile logic
hotspot source code and make files for building the OpenJDK Hotspot Virtual Machine
langtools source code for the OpenJDK javac and language tools
jdk source code and make files for building the OpenJDK runtime libraries and misc files
jaxp source code for the OpenJDK JAXP functionality
jaxws source code for the OpenJDK JAX-WS functionality
corba source code for the OpenJDK Corba functionality
nashorn source code for the OpenJDK JavaScript implementation

For example, for corba, under corba/ directory, it has a Makefile,

...
#
# Makefile for building the corba workspace.
#

BUILDDIR=.
include $(BUILDDIR)/common/Defs.gmk
include $(BUILDDIR)/common/CancelImplicits.gmk

#----- commands

CHMOD = chmod
CP = cp
ECHO = echo # FIXME
...
# Default target
default: all

#----- classes.jar

CLASSES_JAR = $(LIB_DIR)/classes.jar
$(CLASSES_JAR):
	$(MKDIR) -p $(@D)
	$(BOOT_JAR_CMD) -cf $@ -C $(CLASSES_DIR) .

#----- src.zip

SRC_ZIP_FILES = $(shell $(FIND) $(SRC_CLASSES_DIR) \( -name \*-template \) -prune -o -type f -print )
...
jprt_build_product jprt_build_debug jprt_build_fastdebug: all
	( $(CD) $(OUTPUTDIR) && \
	  $(ZIP) -q -r $(JPRT_ARCHIVE_BUNDLE) build dist )

#-------------------------------------------------------------------
...
#
# Phonies to avoid accidents.
#
.PHONY: all build clean clobber debug jprt_build_product jprt_build_debug jprt_build_fastdebug

Basically, I am clear about how Make works! Awesome!!! 😉