I'm Tired of Makefiles

At Cesanta, we use Makefiles to build our firmwares, libraries, and perform other things. But the more I use make, the less I like it. I want my incremental builds to be reliable, and I want to be able to reuse my Makefiles as needed. make can't provide that, and builds just kinda-sorta work most of the time.

Most annoying issues are:

Makefile reuse is a problem

Consider: we have a library mylib, which is a separate project on its own, and it has its own Makefile. The end product is a file mylib.a. So, among others, there is a rule which looks like:

mylib.a: src1.c src2.c src3.c 
    .... some recipe to build mylib.a ....

Now, we have a project app with a separate Makefile, and we want to use mylib.a there, and of course we want it to be up to date. How would we do that?

We can't just include ../mylib/Makefile, because it has a lot of irrelevant stuff such as unit tests, etc, and we don't want variable names to clash.

We could add a target wich a simple recipe which just invokes make properly:

mylib.a:
    $(MAKE) -C ../mylib mylib.a

The obvious problem here is that there are no prerequisites, so make won't be invoked when we change some mylib's sources. That's not acceptable.

We could factor out prerequisites in a third Makefile, specifically and carefully designed for inclusion, and include it in both mylib/Makefile and app/Makefile, and we'll make sure all the paths are correct, etc, etc. But this is really too much work here, in the app's Makefile: we want to just use the lib. We don't want to care about how to build it.

My next idea was to make mylib.a target phony, so that make will get invoked every time (and will be a no-op if no prerequisites changed), but is's also bad since a target which depends on a phony target will be rebuilt every time. We don't want the app target to always get rebuilt.

Ok, something that would really work is to add the make invocation to the recipe for every target which depends on mylib, like:

app: prereq1 prereq2 ...
    $(MAKE) -C ../mylib mylib.a
    ..... the rest of actual recipe for app .....

But this is just ugly. We actually end up specifying prerequisites in a recipe. Still, this is the easiest way that I'm aware of.

Change of compilation flags doesn't cause targets to rebuild

Assume we have a Makefile which accepts CFLAGS_EXTRA variable:

Makefile
#.....
CFLAGS_EXTRA ?=
 
CFLAGS = -DFOO -DBAR $(CFLAGS_EXTRA)
 
%.o: %.c
	gcc -c $(CFLAGS) $*.c -o $*.o
#.....

Now, build an app:

$ make

Now we decide to build it with one more macro defined:

$ make CFLAGS_EXTRA='-DMYMACRO=123'

The app won't be rebuilt, even though we want it to. We have to remember to do a clean build.

Well, we could apply some trick to make the value of CFLAGS_EXTRA variable actually affect the prerequisites, but here we begin fighting with make, rather than embracing it.

Incremental builds fail after renamed headers

UPD: as tavainator pointed out at reddit, this one is easy to solve with -MP flag of gcc, so this issue is no longer relevant for me.

Assume we have a C source file src1.c:

src1.c
#include "some_header.h"
 
int main(void) {
  /* do something */
  return 0;
}

And the following simple Makefile:

OBJS = src1.o
 
all: app
 
# link app
app: $(OBJS)
	gcc $(OBJS) -o app
 
# use autogenerated dependencies
-include $(OBJS:.o=.d)
 
# compile and generate dependency info
%.o: %.c
	gcc -c $(CFLAGS) $*.c -o $*.o
	gcc -MM $(CFLAGS) $*.c > $*.d
 
# cleanup generated files
clean:
	rm -f app *.o *.d

So when we type make, then, among other things, src1.d file will be generated:

src1.d
src1.o: src1.c some_header.h

Then we do some refactor during which some_header.h is renamed or deleted, or we just switch to another Git branch which doesn't have some_header.h, and incremental build fails with:

make: *** No rule to make target 'some_header.h', needed by 'src1.o'.  Stop.

We have to do a clean build.

Well, technically, it's not a problem of make as such: we could maintain all the header dependencies manually, instead of using dependency-generation facility of gcc. But it's a very common pattern, and it has severe flaws.

Timestamp-based judgement on what needs to be rebuilt is unreliable

Again, it works “most of the time”, probably for the most people, but it's not perfect e.g. when one switches back and forth between Git branches. And I do that very often.

Usually it only causes unnecessary builds (which is annoying but probably not critical on small-ish projects), but sometimes it can be a bigger issue, if for whatever reason some sources were copied with timestamps preserved.

Conclusion

I actually can cope with a lot of shortcomings which stuck for historical reasons. For example, I dislike that make does not fail if the recipe of a non-phony target did not actually create the target.

I also dislike that when we declare a variable like that: FOO = foo, the variable FOO can still be overridden from the command line. My point is that if the author of makefile wants some variable to be overriddable, they should just use FOO ?= foo. But this is also not critical: we'll write override FOO = foo when we want (even though we want that most of the time).

There are plenty of other issues, some of them require ancient wisdom to write Makefiles which are correct, but I got used to most of them. But the points I elaborated above seem too much for me.

So I'm learning Bazel. Let's see how far I can go with it.

Discuss on reddit:

Discussion

C. Ranky., 2016/12/12 16:49

Methinks you need to learn how to use Make properly.

Dmitry Frank, 2016/12/12 17:17

Surely I need, and I try hard actually. I've read the most of GNU Make Manual about a year ago; I was hoping this is enough to use Make properly.

I'd appreciate if you could share any good resources on a proper usage of Make, and in particular the ones which could resolve the points I indicated in the article.

And sadly, even the most proper usage of make won't help with the latest point: things will get rebuilt based on the timestamps. I'd really like it to be a checksum instead.

Michael Stapelberg, 2016/12/12 18:59

I think the correct way to use make is to use the GNU build system (autotools) ;)

Of course, this requires that you follow a certain amount of GNU coding conventions, but what you get is a rather pleasant experience compared to the amount of work/knowledge required to accomplish the same with standard make. Features such as parallel builds, out-of-tree builds, coverage reports, etc. just work. In addition, all the bits and features which Linux distribution package maintainers expect will be in the right place.

I used to dislike autotools, but recently came around and made the switch in i3: https://github.com/i3/i3/commit/4a52a7e9fb6fb2e1f0256b2e086cfa313f411cd8

All that being said, I do admire bazel! Not necessarily for its implementation choices (Java that looks heavy-weight to me), but for what it accomplishes: scaling a build tool to hundreds of thousands of targets.

Dmitry Frank, 2016/12/12 19:50

Thanks for sharing your experience.

Honestly I did not use autotools yet, so I don't know if I like it or not; but our company does have people who hate autotools with a passion, so I'm afraid I won't even try proposing it :)

Plus, I'm not sure whether autotools play well with weird embedded compilers, which we have to use for our firmwares.

Bob, 2016/12/13 16:07

Snarky comment + zero constructive input = brilliant contribution! Good work, guy.

Shaun Jackman, 2016/12/19 18:41
And sadly, even the most proper usage of make won't help with the latest point: things will get rebuilt based on the timestamps. I'd really like it to be a checksum instead.

Biomake can interpret standard Makefiles and implements this feature of using MD5 hashes rather than timestamps. Two other useful features are multi-pattern rules and the ability to run jobs on a scheduled cluster.

See https://github.com/evoldoers/biomake or brew install homebrew/science/biomake with http://brew.sh or http://linuxbrew.sh

Dmitry Frank, 2016/12/19 18:51

Hm, that looks interesting! Shaun, thanks for the comment, I'll definitely take a closer look at Biomake!

Bill Torpey, 2016/12/12 21:40

I feel your pain – I really do. My favorite make fail is the requirement that dependent rule lines must start with a tab, not a space, or “n” spaces, but a frickin' tab, dammit! I can't tell how many times that has bitten me in the backside.

I also agree with the autotools haters (and for them, here's a tweet that never fails to make me laugh).

Allow me to suggest cmake (cmake.org). It has a bit of a learning curve, but nothing like either make or autotools.

It seems like more and more projects (incl. e.g. LLVM & clang) are switching to cmake, so there's a body of knowledge out there that can help you get up to speed.

Good luck!

Dmitry Frank, 2016/12/12 21:51

Hi Bill, thanks for the comment!

Yeah, I've also got a few suggestions to use cmake in the reddit thread, so I'll definitely look into it and will try to decide which one fits better for me: Bazel or cmake.

The requirement to have a tab character in front of a recipe line drives me crazy sometimes as well, but it is one of the issues I could cope with, provided that there are no severe architectural problems (sadly, there are). My editor highlights the lines in red when it sees that what should be a recipe starts from the space.

Autotool tweet is awesome, it didn't fail to make me laugh as well! :)

Daniel Lyons, 2016/12/12 23:09

Make isn't perfect, but there are a few things you can do that do not exactly constitute “fighting with it” that may make it work a little better for you. The key is just that the only tool in Make's toolbox is the timestamp of files. So if your build problems involve more than files being out-of-date, then you have to “push” that information out to the filesystem so that Make can perceive it.

  1. Ensure every file you touch in the target is represented in the rule pattern
  2. Make sure that every change you want to cause a rebuild, causes a file to be modified

In your third example, if you make the rule pattern %.o %.d: %.c you may find it works better. If not, you'll need to separate them into two rules, one like %.d: %.c and one like %.o: %.c %.d.

For your CFLAGS_EXTRA problem, you would probably make it write to a file with a name like .extra-flags or something, and then make sure to depend on that in all your rules, like %.o: %.c .extra-flags. Then when you cause the .extra-flags file to become newer than your object files, Make will realize it has to rebuild them. Hopefully.

Make may not be the right tool for your problem, but I have usually managed to beat into shape for me, using tricks like these.

Dmitry Frank, 2016/12/13 08:07

Thanks; yeah there are some hacks possible, probably I'll have to resort to them. Let's see.

As to the third example: as someone pointed out in the reddit thread, it could be solved by GCC flag -MP.

Samuel Williams, 2016/12/13 01:12

I have some similar feelings and made http://teapot.nz/

Dmitry Frank, 2016/12/13 08:08

Thank you, looks interesting!

Bryan Elliott, 2016/12/13 01:15

You might also check out Gradle's 'cpp' plugin, if you've got the time.

Yawar Amin, 2016/12/13 03:44

The question of multi-project builds with dependency management is a complex one, and imho not especially suited to make-like build tools, which excel at building a single project. Conceptually, inter-package dependency management needs a specialised tool, like https://www.conan.io/ or Nix or (with some work) 0install. What these give you is a reproducible multi-project build.

Now, apart from the above problem, I think tup ( http://gittup.org/tup/ ) fits your needs pretty well. Tup tries to guarantee fully reproducible builds which behave exactly like clean builds–it even removes output files which are no longer being built. It's been working very well for me, albeit on a small C project. I think it's worth a look for you.

Rob, 2016/12/13 21:03

Conan actually solves this quite well and includes generators for many build systems including cmake

Kat Marsen, 2016/12/13 04:53

Indeed, makefiles suck.

Sébastien Boisvert, 2016/12/13 18:31

CMake is better than Makefiles.

Thomas Woolford, 2016/12/14 06:50

Have you tried Shake? http://shakebuild.com/why

Dmitry Frank, 2016/12/14 08:53

Not yet, thanks for the link, I'll take a look!

Zsolt Udvari, 2016/12/18 11:33

Change of compilation flags doesn't cause targets to rebuild

The make checks the timestamp of required files. If you want a clean build after change of Makefile please add it to dependencies list! But it's better than you create a cflags.mk and include it from your Makefile and add cflags.mk as dependency.

Dmitry Frank, 2016/12/18 11:44

First of all, there are disadvantages of this hack of adding a makefile itself to the dependencies list.

Secondly, you misunderstood the problem: in the article I consider the case when we just provide a custom CFLAGS_EXTRA in the command line. Makefile is not changed, but corresponding targets need to be rebuilt.

Zsolt Udvari, 2016/12/18 13:29

Sorry, you've right. I was unperceptive. I think in your EXTRA_CFLAGS-case the Makefile doesn't ship (simple) solution - you should choose another build system.

But: the Makefile-system it's a very good and simple system - with limitations. You can choose autotools eg. - it's a more complex system, and slower (check the run of ./configure). You should decide what do you want and what do you need: simple, fast and less features or complex (see the heaps of cmake's files), not too fast and more features.

Enter your comment (please, English only). Wiki syntax is allowed:
  _   __  ____    ____  _____ __  __
 | | / / / __ \  / __/ / ___/ \ \/ /
 | |/ / / /_/ / / _/  / /__    \  / 
 |___/  \____/ /___/  \___/    /_/
 
blog/2016/1211_i_am_tired_of_makefiles.txt · Last modified: 2016/12/12 16:07 by dfrank
Driven by DokuWiki Recent changes RSS feed Valid CSS Valid XHTML 1.0