GLSL 2D Rounded corners - opengl

Question

I want to add some black outline to my game screen to make it look like the corners are rounded.
This is the effect I want to achieve:
I figured this effect was probably quite easy to create using a shader, instead of drawing a giant bitmap on top of everything.
Can someone help me with the GLSL shader code for this effect? I have 0 experience with shaders and was unable to find anything like this on the internet.

Answer 1

I've accidentaly found a nice solution for this. Not exactly what you've asked for, but in fact it looks even better.
// RESOLUTION is a vec2 with your window size in pixels.
vec2 pos = fragCoord.xy / RESOLUTION;
// Adjust .2 (first pow() argument) below to change frame thickness.
if (pos.x * pos.y * (1.-pos.x) * (1.-pos.y) < pow(.2,4.))
fragColor = vec4(0,0,0,1);
It gives following result:
If you don't like those thin lines, you can remove them just by upscaling the image. It can be done by adding this line:
// The .985 is 1/scale_factor. You can try to change it and see how it works.
// It needs to be adjusted if you change frame thickness.
pos = (pos - .5) * .985 + .5;
While this effect looks good, it may be smarter to add just a faint shadow instead.
It's easy to implement using the same equation: pos.x * pos.y * (1.-pos.x) * (1.-pos.y)
The value of it ranges from 0.0 at window edges to 0.5^4 in the center.
You can use some easy math to do a shadow that becomes more thick closer to the window edge.
Here is an example of how it may look.
(A screenshot from Duality, my entry for Ludum Dare 35.)

Answer 2

Thanks to #HolyBlackCat my shader now works. I've improved the performance and made it look smoothed.
varying vec4 v_color;
varying vec2 v_texCoord0;
uniform vec2 u_resolution;
uniform vec2 u_screenOffset;
uniform sampler2D u_sampler2D;
const float max = pow(0.2, 4);
void main()
{
vec2 pos = (gl_FragCoord.xy - u_screenOffset) / u_resolution;
float vignette = pos.x * pos.y * (1.-pos.x) * (1.-pos.y);
vec4 color = texture2D(u_sampler2D, v_texCoord0) * v_color;
color.rgb = color.rgb * smoothstep(0, max, vignette);
gl_FragColor = color;
}
Set the uniforms as follows in the resize event of libGDX:
shader.begin();
shader.setUniformf("u_resolution", viewport.getScreenWidth(), viewport.getScreenHeight());
shader.setUniformf("u_screenOffset", viewport.getScreenX(), viewport.getScreenY());
shader.end();
This will make sure the shader works with viewports (only tested with FitViewport) aswell.