How to integrate pretty-printing as part of build in bazel - c++

Question

Right now, I have a really dumb pretty-print script which does a little git-fu to find files to format (unconditionally) and then runs those through clang-format -i. This approach has several shortcomings:
There are certain files which are enormous and take forever to pretty print.
The pretty printing is always done, regardless of whether or not the underlying file actually changed or not.
In the past, I was able to do things with CMake that had several nice properties which I would like to reproduce in bazel:
Only ever build code after it has gone through linting / pretty printing / etc.
Only lint / pretty print / etc. stuff that has changed
Pretty print stuff regardless of whether or not it is under VC or not
In CMake-land, I used this strategy, inspired by SCons proxy-target trickery:
Introduce a dummy target (e.g. source -> source.formatted). The action associated with this target does two things: a) run clang-format -i source, b) output/touch a file called source.formatted (this guarantees that for reasonable file systems, if source.formatted is newer than source, source doesn't need to be reformatted)
Add a dummy target (target_name.aggregated_formatted) which aggregates all the .formatted files corresponding to a particular library / executable target's sources
Make library / executable targets depend on target_name.aggregated_formatted as a pre-build step
Any help would be greatly appreciated.

Answer 1

#abergmeier is right. Let's take it one step further by implementing the macro and its components.
We'll use the C++ stage 1 tutorial in bazelbuild/examples.
Let's first mess up hello-world.cc:
#include <ctime>
#include <string>
#include <iostream>
std::string get_greet(const std::string& who) {
return "Hello " + who;
}
void print_localtime() {
std::time_t result =
std::time(nullptr);
std::cout << std::asctime(std::localtime(&result));
}
int main(int argc, char** argv) {
std::string who = "world";
if (argc > 1) {who = argv[1];}
std::cout << get_greet(who) << std::endl;
print_localtime();
return 0;
}
This is the BUILD file:
cc_binary(
name = "hello-world",
srcs = ["hello-world.cc"],
)
Since cc_binary doesn't know anything about clang-format or linting in general, let's create a macro called clang_formatted_cc_binary and replace cc_binary with it. The BUILD file now looks like this:
load(":clang_format.bzl", "clang_formatted_cc_binary")
clang_formatted_cc_binary(
name = "hello-world",
srcs = ["hello-world.cc"],
)
Next, create a file called clang_format.bzl with a macro named clang_formatted_cc_binary that's just a wrapper around native.cc_binary:
# In clang_format.bzl
def clang_formatted_cc_binary(**kwargs):
native.cc_binary(**kwargs)
At this point, you can build the cc_binary target, but it's not running clang-format yet. We'll need to add an intermediary rule to do that in clang_formatted_cc_binary which we'll call clang_format_srcs:
def clang_formatted_cc_binary(name, srcs, **kwargs):
# Using a filegroup for code cleaniness
native.filegroup(
name = name + "_unformatted_srcs",
srcs = srcs,
)
clang_format_srcs(
name = name + "_formatted_srcs",
srcs = [name + "_unformatted_srcs"],
)
native.cc_binary(
name = name,
srcs = [name + "_formatted_srcs"],
**kwargs
)
Note that we have replaced the native.cc_binary's sources with the formatted files, but kept the name to allow for in-place replacements of cc_binary -> clang_formatted_cc_binary in BUILD files.
Finally, we'll write the implementation of the clang_format_srcs rule, in the same clang_format.bzl file:
def _clang_format_srcs_impl(ctx):
formatted_files = []
for unformatted_file in ctx.files.srcs:
formatted_file = ctx.actions.declare_file("formatted_" + unformatted_file.basename)
formatted_files += [formatted_file]
ctx.actions.run_shell(
inputs = [unformatted_file],
outputs = [formatted_file],
progress_message = "Running clang-format on %s" % unformatted_file.short_path,
command = "clang-format %s > %s" % (unformatted_file.path, formatted_file.path),
)
return struct(files = depset(formatted_files))
clang_format_srcs = rule(
attrs = {
"srcs": attr.label_list(allow_files = True),
},
implementation = _clang_format_srcs_impl,
)
This rule goes through every file in the target's srcs attribute, declaring a "dummy" output file with the formatted_ prefix, and running clang-format on the unformatted file to produce the dummy output.
Now if you run bazel build :hello-world, Bazel will run the actions in clang_format_srcs before running the cc_binary compilation actions on the formatted files. We can prove this by running bazel build with the --subcommands flag:
$ bazel build //main:hello-world --subcommands
..
SUBCOMMAND: # //main:hello-world_formatted_srcs [action 'Running clang-format on main/hello-world.cc']
..
SUBCOMMAND: # //main:hello-world [action 'Compiling main/formatted_hello-world.cc']
..
SUBCOMMAND: # //main:hello-world [action 'Linking main/hello-world']
..
Looking at the contents of formatted_hello-world.cc, looks like clang-format did its job:
#include <ctime>
#include <string>
#include <iostream>
std::string get_greet(const std::string& who) { return "Hello " + who; }
void print_localtime() {
std::time_t result = std::time(nullptr);
std::cout << std::asctime(std::localtime(&result));
}
int main(int argc, char** argv) {
std::string who = "world";
if (argc > 1) {
who = argv[1];
}
std::cout << get_greet(who) << std::endl;
print_localtime();
return 0;
}
If all you want are the formatted sources without compiling them, you can run build the target with the _formatted_srcs suffix from clang_format_srcs directly:
$ bazel build //main:hello-world_formatted_srcs
INFO: Analysed target //main:hello-world_formatted_srcs (0 packages loaded).
INFO: Found 1 target...
Target //main:hello-world_formatted_srcs up-to-date:
bazel-bin/main/formatted_hello-world.cc
INFO: Elapsed time: 0.247s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action

Answer 2

You might be able to use aspects for that. Being not certain, a Bazel-dev will probably point that out if it indeed is possible.
If you are familiar with Rules and Actions and the like, the quick and dirty way (which is similar to the CMake hackery) is to write a Macro. For e.g. cc_library you would do:
def clean_cc_library(name, srcs, **kwargs):
lint_sources(
name = "%s_linted" % name,
srcs = srcs,
)
pretty_print_sources(
name = "%s_pretty" % name,
srcs = ["%s_linted"],
)
return native.cc_library(
name = name,
srcs = ["%s_pretty"],
**kwargs
)
Then you of course need to replace every cc_library with clean_cc_library. And lint_sources and pretty_print_sources are rules that you have to implement yourself and need to produce the list of cleaned up files.

Answer 3

#abergmeier mentions maybe being able to use Aspects. You can, and I've made a prototype of a general linting system that leverages Aspects functionality so that BUILD files do not need to be modified to use Macros like clang_formatted_cc_library in-place of the core rules.
The basic idea is to have a bazel build step that is a pure function f(linter, sources) -> linted_sources_diff and a subsequent bazel run step that takes those diffs and applies them back to your source code to fix lint errors.
The prototype implementation is available at https://github.com/thundergolfer/bazel-linting-system.