I'm woking on a scientific visualisation using openGL where I try to basically create a "coloured fog" to illustrate a multidimensional field. I want the hue of the color to represent the direction of the field and the luminosity of the color to correspond to the intensity of the field.
I use GL_BLEND with glBlendFunc(GL_ONE, GL_ONE) to achieve additive blending of colors of the surfaces I create. The problem is: in some places the colors get saturated. For example:
(1, 0, 0) + (1, 0, 0) = (2, 0, 0), and when this is rendered it just becomes (1, 0, 0) (i.e. the "overflow" is just chopped off). And this is not the way I would like it to be handled. I would like to handle the overflow by preserving hue and luminocity, i.e.
(2, 0, 0) should be translated into (1, 0.5, 0.5) (i.e. a lighter red, red with twice the luminocity of "pure" red).
Is there any way to achieve this (or something similar) with OpenGL?
The output to the fragment shader will be clamped to [0, 1] if the image format of the destination buffer has a normalized format (e.g. UNSIGNED_BYTE). If you use a floating point format, then the output is not clamped. See Blending and Image Format.
It is tricky to blend the target buffer and a color, by a function which is not supported by blending. A possible solution may be to withe a shader program and to use the extension EXT_shader_framebuffer_fetch.
Furthermore, the extension KHR_blend_equation_advanced adds a number of "advanced" blending equations, like HSL_HUE_KHR, HSL_SATURATION_KHR, HSL_COLOR_KHR and HSL_LUMINOSITY_KHR.
Related
I am trying to implement in my shader a way of reading normals from a normal map. However, I found a problem when reading colors that prevents it.
I thought that one color such as (0, 0, 255) (blue) was equivalent to (0, 0, 1) in the shader. However, recently I found out that, for instance, if I pass a texture with the color (128, 128, 255), it is not equivalent to ~(0.5, 0.5, 1) in the shader.
In a fragment shader I write the following code:
vec3 col = texture(texSampler[0], vec2(1, 1)).rgb; // texture with color (128, 128, 255)
if(inFragPos.x > 0)
outColor = vec4(0.5, 0.5, 1, 1); // I get (188, 188, 255)
else
outColor = vec4(col, 1); // I get (128, 128, 255)
In x<0 I get the color (128, 128, 255), which is expected. But in x>0 I get the color (188, 188, 255), which I didn't expect. I expected both colors to be the same. What do I not know? What am I missing?
But in x>0 I get the color (188, 188, 255), which I didn't expect.
Did you render these values to a swapchain image, by chance?
If so, swapchain images are almost always in the sRGB colorspace. Which means that all floats written to them will be expected to be in a linear colorspace and therefore will be converted into sRGB.
If the source image was also in the sRGB colorspace, reading from it will reverse the transformation into a linear RGB colorspace. But since these are inverse transformations, the overall output you get will be the same as the input.
If you want to treat data in a texture as data rather than as colors, you must not use image formats that use the sRGB colorspace. And swapchain images are almost always sRGB, so you'll have to use a user-created image for such outputs.
Also, 128 will never yield exactly 0.5. 128/255 is slightly larger than 0.5.
After some research, I could solve it, so I will explain the solution. Nicol Bolas' answer shed some light on the problem too (thank you!).
In the old days, images were in (linear) RGB. Today, images are expected to be in (non-linear) sRGB. The sRGB color space gives more resolution to darker colors and less to lighter colors, because human eye distinguishes darker colors better.
Internet images (including normal maps) are almost always in sRGB by convention. When I analyze the colors of an image with Paint, I get the sRGB colors. When I pass that image as a texture to the shader, it is automatically converted to RGB (if you told Vulkan to do so), because the RGB color space is more appropriate for making operations with colors. Then, when the shader outputs the result, it automatically converts it back to sRGB.
My mistake was to consider the color information I got from the source image (using Paint) to be RGB, while it was really sRGB. When the color was converted to RGB in the shader, I was confused because I expected the same color I got in Paint. Since I want to use the texture as data rather than as color, I see 3 ways to solve this:
Save normals in a RGB image (tell Vulkan about this) (most correct option).
Transform the image to sRGB in the shader (my solution). Since the data was saved in an image as sRGB colors, it should be read in the shader as sRGB in order to get the correct data.
Now, talking about Vulkan, we have to specify the color space for the surface format and the swap chain (for instance: VK_COLOR_SPACE_SRGB_NONLINEAR_KHR). This way, the swapchain\display interprets the values when the image is presented. Also, we have to specify the color space of the Vulkan images we create.
References
Linear Vs Non-linear RGB: Great answer from Dan Hulme
Vulkan color space: Vulkan related info
Normal mapping 1 & Normal mapping 2
Demo https://codepen.io/Andreslav/pen/wvmjzwe
Scheme: Top left - the original.
Top right - the result.
Bottom right - rounding coordinates when extracting color.
The problem can be solved this way, but then the result is less smoothed:
coord = floor(coord) + .5;
How to make it better? Make it so that when calculating the average color, the program ignores the color of transparent pixels?
Maybe there are some settings that I haven't figured out..
Updated the demo
The result is even better after such an adjustment:
vec4 color = texture2D(texture, coord / texSize);
vec4 color_ = texture2D(texture, coordNoRound / texSize);
if(color_.a != color.a) {
color.a *= color_.a;
}
On the preview: bottom left. But this is not an ideal option, the correction is partial. The problem is relevant.
This appears to be a premultiplied alpha problem. And it's not as much of a glsl problem as it is a glfx problem.
Here's what happens:
Consider the RGBA values of two adjacent pixels at the edge of your source image. It would be something like this:
[R G B A ] [R, G, B, A]
[1.0, 1.0, 1.0, 1.0] [?, ?, ?, 0]
Meaning that there is a fully opaque, fully-white pixel to the left, and then comes a fully-transparent (A=0) pixel to the right.
But what are the RGB values of a completely transparent pixel?
They are technically ill-defined (this fact is the core problem which needs to be solved). In practice, pretty much every image processing software will put [0, 0, 0] there.
So the pixels are actually like this:
[R G B A ] [R, G, B, A]
[1.0, 1.0, 1.0, 1.0] [0, 0, 0, 0]
What happens if your swirl shader samples the texture halfway between those 2 pixels? You get [0.5, 0.5, 0.5, 0.5]. That's color [0.5 0.5 0.5], with 0.5 Alpha. Which is gray, not white.
The generally chosen solution to this problem is premultiplied alpha. Which means that, for any given RGBA color, the RGB components are defined so that they don't range from 0 .. 1.0, but instead from 0 .. A. With that definition, color [0.5 0.5 0.5 0.5] is now "0.5 A, with maximum RGB, which is white". One side effect of this definition is that the RGB values of a fully transparent pixel are no longer ill-defined; they must now be exactly [0, 0, 0].
As you can see, we didn't really change any values, instead, we just defined that our result is now correct. Of course, we still need to tell the other parts of the graphics pipeline of our definition.
Premultiplied alpha is not the only solution to the problem. Now that you know what's happening, you might be able to come up with your own solution. But pretty much all modern graphics pipelines expect that you are working with premultiplied alpha all the time. So the correct solution would be to make that true. That means:
(1) You need to make sure that your input texture also has premultiplied alpha, i.e. all its RGB values must be multiplied with their alpha value. This is generally what game engines do, all their textures have premultiplied alpha. Either every pixel must already be edited in the source file, or you do the multiplication once for each pixel in the texture after loading the image.
AND
(2) You need to convince every alpha blending component in your rendering pipeline to use premultiplied alpha blending, instead of "normal" alpha blending. It seems you use the "glfx" framework, I don't know glfx, so I don't know how you can make it blend correctly. Maybe check the docs. In case you are using raw OpenGL/WebGL, then this is the way to tell the pipeline that it should assume premultiplied alpha values when blending:
gl.blendEquation(gl.FUNC_ADD); // Normally not needed because it's the default
gl.blendFunc(gl.ONE, gl.ONE_MINUS_SRC_ALPHA);
(This can be derived from the analyzing the formula for source-over alpha blending, but without the last division step.)
The above code tells OpenGL/WebGL that every time it's blending two pixels on top of another, it should calculate the final RGBA values in a way that's correct assuming that both the "top" and the "bottom" pixel has premultiplied alpha applied to it.
For higher level APIs (for example, GDI+), you can typically specify the pixel format of images, where there is a separation between RGBA and RGBPA, in which case the API will automatically choose correct blending. That may not be true for glfx though. In essence, you always need to be aware whether the pixel format of your textures and drawing surfaces have premultiplied alpha or not, there is no magic code that always works correctly.
(Also note that using premultiplied alpha has other advantages too.)
For a quick fix, it appears that the framework you're using performs alpha blending so that it expects non-premultiplied alpha values. So you could just undo the premultiplication by adding this at the end:
color.rgb /= color.a;
gl_FragColor = color;
But for correctness, you still need to premultiply the alpha values of your input texture.
Because at the rounded corners, your input texture contains pixels which are fully white, but semi-transparent; their RGBA values would look like this:
[1.0, 1.0, 1.0, 0.8]
For the blending code to work correctly, the values should be
[0.8, 0.8, 0.8, 0.8]
,
because otherwise the line color.rgb /= color.a; would give you RGB values greater than 1.
Assume I have a C_1 = (255, 0, 0, 127) filled sf::Texture and a sf::RenderTexture filled with C_2 = (0, 0, 0, 0). Then I call
render_texture.draw(texture);
The result given by sf::BlendAlpha is a texture filled with (127, 0, 0, 127), but my goal (for the small graphics editor app I'm writing) is blending those two colors to get (255, 0, 0, 127), because, intuitively, drawing on perfectly transparent texture is "copying color of your brush to it". I've checked formulas of sf::BlendAlpha, and it appeared to work like this:
C_res = C_src * aplha_src + C_dst * (1 - alpha_src)
Where C_src is C_1 and C_dst is C_2. But wait, does it totally ignore aplha_dst?
If I understand blending right, the formula I need is something like
alpha_sum = alppha_src + alpha_dst
C_res = C_src * alpha_src / alpha_sum + C_dst * alpha_dst / alpha_sum
Which gives adequate colors and does not ignore alpha_dst. But with BlendingMode from SFML, using different Factors and Equations, there's no chance to get division (that also introduces troubles when dealing with alpha_sum = 0). Any suggestions on how actually I should think about blending colors and painting? At least, did I get it right, that usual alpha blending used by SFML is not what graphics editors do? (tested on Aseprite, Krita, Photoshop, they blend color from the brush to (0, 0, 0, 0) as I expect)
The solution I ended up with: writing your own blend mode with glsl and using shaders. This way you can create and formulas without the need to think about how to bring it to sfml, because it already supports shaders, that can be easily passed to any draw call.
I'm loading a texture using OpenGL like this
glTexImage2D(
GL_TEXTURE_2D,
0,
GL_RGBA,
texture.width,
texture.height,
0,
GL_RGBA,
GL_UNSIGNED_BYTE,
texture.pixels.data());
The issue is that the color of the image looks different from the one I see when I open the file on the system image viewer.
On the screenshot you can see the yellow on the face displayed on the system image viewer has the color #FEDE57 but the one that is displayed in the OpenGL window is #FEE262
Is there any flag or format I could use to match the same color calibration?
Displaying this same image as a Vulkan texture looks fine, so I can discard there is not an issue in how I load the image data.
In the end it seems like the framebuffer in OpenGL doesn't gets color corrected, so you have to tell the OS to do it for you
#include <Cocoa/Cocoa.h>
void prepareNativeWindow(SDL_Window *sdlWindow)
{
SDL_SysWMinfo wmi;
SDL_VERSION(&wmi.version);
SDL_GetWindowWMInfo(sdlWindow, &wmi);
NSWindow *win = wmi.info.cocoa.window;
[win setColorSpace:[NSColorSpace sRGBColorSpace]];
}
I found this solution here https://github.com/google/filament/blob/main/libs/filamentapp/src/NativeWindowHelperCocoa.mm
#Tokenyet and #t.niese are pretty much correct.
You need to approximately power you final colour's rgb values by 1.0/2.2. Something on the line of this:
FragColor.rgb = pow(fragColor.rgb, vec3(1.0/gamma)); //gamma --> float = 2.2
Note: this should be the final/last statement in the fragment shader. Do all your lighting and colour calculations before this, or else the result will be weird because you will be mixing linear and non-linear lighting (calculations).
The reason you need to do gamma correction is because the human eye perceives colour differently to what the computer outputs.
If the light intensity (lux) increases by twice the amount, your eye indeed sees it twice as bright. However, the actual brightness, when increased by twice the amount, increases in a logarithmic (or exponential?, someone please correct me here) relationship. The constant of proportionality between the two lighting spaces is ^2.2 (or ^(1.0/2.2) if you want to go the inverse (which is what you are looking for.)).
For more info: Look at this great tutorial on gamma correction!
Note 2: This is an approximation. Each computer, program, API have their own auto gamma correction method. You system image viewer may have different gamma correction methods (or not even have any for that matter) compared to OpenGL
Note 3: Btw, if this does not work, there are manual methods to adjust the colour in the fragment shader, if you know.
#FEDE57 = RGB(254, 222, 87)
which converted into OpenGL colour coordinates is,
(254, 222, 87) / 255 = vec3(0.9961, 0.8706, 0.3412)
Both images and displays have a gamma value.
If GL_FRAMEBUFFER_SRGB is not enabled then:
the system assumes that the color written by the fragment shader is in whatever colorspace the image it is being written to is. Therefore, no colorspace correction is performed.
( khronos: Framebuffer - Colorspace )
So in that case you need to figure out what the gamma value of the image you read in is and what the one of the output medium is and do the corresponding conversion between those.
To get the one of the output medium is however not always easy.
Therefore it is preferred to enable GL_FRAMEBUFFER_SRGB
If GL_FRAMEBUFFER_SRGB is enabled however, then if the destination image is in the sRGB colorspace […], then it will assume the shader's output is in the linear RGB colorspace. It will therefore convert the output from linear RGB to sRGB.
( khronos: Framebuffer - Colorspace )
So in that case you only need to ensure that the colors you set in the fragment shader don't have gamma correction applied but are linear.
So what you normally do is to get the gamma information of the image, which is done with a certain function of the library you use to read the image.
If the gamma of the image you read is gamma you can calculate the value to invert it with inverseGamme = 1. / gamma, and then you can use pixelColor.channel = std::pow(pixelColor.channel, inverseGamme) for each of the color channels and each pixel to make the color space linear.
You will use this values in the linear color space as texture data.
You could also use something like GL_SRGB8 for the texture, but then you would need to convert the values of the pixels you read form the image to sRGB colorspace, which roughly is done by first linearizing it and then applying a gamma of 2.2
As in Bilinear filtering, sampled color is calculated based on the weighted average of 4 closest texels, then why corner texels get the same color when magnified?
Eg:
In this case (image below) when a 3x3 image is magnified/scaled to 5x5 pixel image (using Bilinear filtering) corner 'Red' pixels get exact same color and border 'Green' as well?
In some documents, it is explained that corner texels are extended with the same color to give 4 adjacent texels which explains why corner 'Red' texels are getting the same color in 5x5 image but how come border 'Green' texels are getting same color (if they are calculated based on weighted average of 4 closest texels)
When you are using bilinear texture sampling, the texels in the texture are not treated as colored squares but as samples of a continuous color field. Here is this field for a red-green checkerboard, where the texture border is outlined:
The circles represent the texels, i.e., the sample locations of the texture. The colors between the samples are calculated by bilinear interpolation. As a special case, the interpolation between two adjacent texels is a simple linear interpolation. When x is between 0 and 1, then: color = (1 - x) * leftColor + x * rightColor.
The interpolation scheme only defines what happens in the area between the samples, i.e. not even up to the edge of the texture. What OpenGL uses to determine the missing area is the texture's or sampler's wrap mode. If you use GL_CLAMP_TO_EDGE, the texel values from the edge will just be repeated like in the example above. With this, we have defined the color field for arbitrary texture coordinates.
Now, when we render a 5x5 image, the fragments' colors are evaluated at the pixel centers. This looks like the following picture, where the fragment evaluation positions are marked with black dots:
Assuming that you draw a full-screen quad with texture coordinates ranging from 0 to 1, the texture coordinates at the fragment evaluation positions are interpolations of the vertices' texture coordinates. We can now just overlay the color field from before with the fragments and we will find the color that the bilinear sampler produces:
We can see a couple of things:
The central fragment coincides exactly with the red texel and therefore gets a perfect red color.
The central fragments on the edges fall exactly between two green samples (where one sample is a virtual sample outside of the texture). Therefore, they get a perfect green color. This is due to the wrap mode. Other wrap modes produce different colors. The interpolation is then: color = (1 - t) * outsideColor + t * insideColor, where t = 3 * (0.5 / 5 + 0.5 / 3) = 0.8 is the interpolation parameter.
The corner fragments are also interpolations from four texel colors (1 real inside the texture and three virtual outside). Again, due to the wrap mode, these will get a perfect red color.
All other colors are some interpolation of red and green.
You're looking at bilinear interpolation incorrectly. Look at it as a mapping from the destination pixel position to the source pixel position. So for each desintation pixel, there is a source coordinate that corresponds to it. This source coordinate is what determines the 4 neighboring pixels, as well as the bilinear weights assigned to them.
Let us number your pixels with (0, 0) at the top left.
Pixel (0, 0) in the destination image maps to the coordinate (0, 0) in the source image. The four neighboring pixels in the source image are (0, 0), (1, 0), (0, 1) and (1, 1). We compute the bilinear weights with simple math: the weight in the X direction for a particular pixel is 1 - (pixel.x - source.x), where source is the source coordinate. The same goes for Y. So the bilinear weights for each of the four neighboring pixels are (respective to the above order): (1, 1), (0, 0), (0, 0) and (0, 0).
In short, because the destination pixel mapped exactly to a source pixel, it gets exactly that source pixel's value. This is as it should be.