Use a Rust macro to generate and compile a shader - opengl

Question

My origin of this approach comes from OpenGL shader programming, but the problem is more abstract. I will write some pseudo code to clarify what I mean.
In OpenGL, rendering is done in so called "shaders". A shader is a calculation kernel that is applied to each element of a data set, but with the advantage that the calculation is executed on the GPU and therefore takes the advantage of concurrent nature of the GPU to calculate as much at the same time as possible.
The problem is that shaders are presented as text at compile time, and the shader needs to be compiled at runtime by the driver of the GPU. This means that at the start of each program an init function needs to compile each shader source file into a program before the shader can be called. Here is an example, keep in mind it is simplified pseudocode:
let shader_src_A = r#"
attribute float a;
attribute float b;
out float b;
void main() {
b = a * b;
}
"#;
let shader_src_B = r#"
attribute float a;
attribute float b;
out float b;
void main() {
b = a + b;
}
"#;
let mut program_A : ShaderProgram;
let mut program_B : ShaderProgram;
fn init() {
initGL();
program_A = compile_and_link(shader_src_A);
program_B = compile_and_link(shader_src_B);
}
fn render() {
let data1 = vec![1,2,3,4];
let data2 = vec![5,6,7,8];
// move data to the gpu
let gpu_data_1 = move_to_gpu(data1);
let gpu_data_2 = move_to_gpu(data2);
let gpu_data_3 : GpuData<float>;
let gpu_data_4 : GpuData<float>;
program_A(
(gpu_data_1, gpu_data_2) // input
(gpu_data_3,) // output
);
program_B(
(gpu_data_1, gpu_data_2) // input
(gpu_data_4,) // output
);
let data_3 = move_to_cpu(gpu_data_3);
let data_4 = move_to_cpu(gpu_data_4);
println!("data_3 {:?} data_4 {:?}", data_3, data_4);
// data_3 [5, 12, 21, 32] data_4 [6, 8, 10, 12]
}
The goal for me to to be able to write something like this:
fn init() {
initGL();
mystery_macro!();
}
fn render() {
let data1 = vec![1,2,3,4];
let data2 = vec![5,6,7,8];
// move data to the gpu
let gpu_data_1 = move_to_gpu(data1);
let gpu_data_2 = move_to_gpu(data2);
let gpu_data_3 : GpuData<float>;
let gpu_data_4 : GpuData<float>;
shade!(
(gpu_data_1, gpu_data_2), // input tuple
(gpu_data_3,), // output tuple
"gpu_data_3 = gpu_data_1 * gpu_data_2;" // this is the shader source, the rest should be generated by the macro.
);
shade!(
(gpu_data_1, gpu_data_2), // input tuple
(gpu_data_3,), // output tuple
"gpu_data_4 = gpu_data_1 + gpu_data_2;" // this is the shader source, the rest should be generated by the macro.
);
let data_3 = move_to_cpu(gpu_data_3);
let data_4 = move_to_cpu(gpu_data_4);
println!("data_3 {:?} data_4 {:?}", data_3, data_4);
}
The key difference is that I do not have a common place where all my shaders are written. I write my shaders where I call them, and I do not write the part of the shader that can be inferred by the other arguments. Generating the part of the shader that is missing should be straight forward, the problem is the compilation of the shader. A renderer that calls the compilation of each shader on each call is far too slow to be useful at all. The idea is that the macro should generate this common place with all the shader sources and programs, so that the init function can compile and link all programs at program start.
Despite the title, I am also ok with a solution that solves my problem differently, but I would prefer a solution where all programs can be compiled in the init function.
EDIT:
I could also imagine, that shade is not a macro, but a placeholder no-op function, the macro will then operate on the shade function, and by traversing the AST, it can find all calls to shade, and create everything that needs to be done in the init function.

Answer 1

From The Rust Programming Language section on macros (emphasis mine):
Macros allow us to abstract at a syntactic level. A macro invocation is shorthand for an "expanded" syntactic form. This expansion happens early in compilation, before any static checking. As a result, macros can capture many patterns of code reuse that Rust’s core abstractions cannot.
Said another way, macros are only useful when you already have some code that has appreciable boilerplate. They cannot do something beyond what the code itself does.
Additionally, Rust macros work at a level above C macros. Rust macros are not presented with the raw text, but instead have some pieces of the AST of the program.
Let's start with this simplified version:
struct Shader(usize);
impl Shader {
fn compile(source: &str) -> Shader {
println!("Compiling a shader");
Shader(source.len())
}
fn run(&self) {
println!("Running a shader {}", self.0)
}
}
fn main() {
for _ in 0..10 {
inner_loop();
}
}
fn inner_loop() {
let shader_1_src = r#"add 1 + 1"#;
let shader_1 = Shader::compile(shader_1_src);
let shader_2_src = r#"add 42 + 53"#;
let shader_2 = Shader::compile(shader_2_src);
shader_1.run();
shader_2.run();
}
The biggest problem here is the repeated compilation, so we can lazily compile it once using the lazy_static crate:
#[macro_use]
extern crate lazy_static;
// Previous code...
fn inner_loop() {
const SHADER_1_SRC: &'static str = r#"add 1 + 1"#;
lazy_static! {
static ref SHADER_1: Shader = Shader::compile(SHADER_1_SRC);
}
const SHADER_2_SRC: &'static str = r#"add 42 + 53"#;
lazy_static! {
static ref SHADER_2: Shader = Shader::compile(SHADER_2_SRC);
}
SHADER_1.run();
SHADER_2.run();
}
You can then go one step further and make another macro around that:
// Previous code...
macro_rules! shader {
($src_name: ident, $name: ident, $l: expr, $r: expr) => {
const $src_name: &'static str = concat!("add ", $l, " + ", $r);
lazy_static! {
static ref $name: Shader = Shader::compile($src_name);
}
}
}
fn inner_loop() {
shader!(S1, SHADER_1, "1", "2");
shader!(S2, SHADER_2, "42", "53");
SHADER_1.run();
SHADER_2.run();
}
Note that we have to provide a name for the inner source constant because there's currently no way of generating arbitrary identifiers in macros.
I'm no game programmer, but this type of code would make me wary. At potentially any point, you might execute some shader compilation, slowing down your program. I agree that pre-compiling all your shaders at program startup makes the most sense (or at Rust compile time, if possible!), but it simply doesn't make sense with your desired structure. If you can write plain Rust code that does what you want, then you can make a macro that makes it prettier. I just don't believe it's possible to write Rust code that does what you want.
There is a possibility that a syntax extension may be able to do what you want, but I don't have enough experience with them yet to soundly rule it in or out.