Shader that replaces colors - opengl

Question

I want to make a shader that replace a color to be applied to a plain color character, but I can't just replace the color because the image contains pixels that are an average of two border colors.
For example the image looks like this:
Assuming that I want to change the color of the shirt, I want to replace the red color for a green one, but at the edges there are pixels that are not red:
Any ideas how to calculate the resultant color of one of those pixels?

Answer 1

Do you know which are the major colours in advance?
If not then a simple solution for finding them is to generate a histogram — scan the entire image and for each pixel that is the same as all four of its neighbours, add one to a count for the colour it contains. At the end, keep only those colours that fill at least a non-negligible portion of the display, e.g. at least 5% of those pixels that are not transparent.
Dealing with black borders is easy: use a luminance/chrominance colour space, and always leave luminance alone, remapping only chrominance. Factoring out brightness has a bonus: it collapses colour substitution from a 3d problem to a 2d problem.
If this weren't GLSL then a solid solution might be for each pixel that is not one of the selected major colours might be (i) find the nearest pixel that is a major colour; (ii) then find the nearest pixel that is a major colour but not the one found in (i). Use normal linear algebra to figure out the distance of that pixel on the 2d line from the one colour to the other. Substitute the colours, reinterpolate and output.
Being that it is GLSL, so "find the nearest" isn't especially realistic, assuming the number of major colours is small then just do it as distance from those lines. E.g. suppose you have five colours. Then that's 10 potential colour transitions in total — from each of the five colours there are four other options, suggesting twenty transitions, but half of them are exactly the same as the other half because they're just e.g. red to blue instead of blue to red. So ten.
Load those up as uniforms and just figure out which transition gradient the colour is closest to. Substitute the basis colours. Output.
So, in net:
transform (R, G, B) to (Y, x, y) — whether YUV or YIQ or Y doesn't matter, just pick one;
perform distance from a line for (x, y) and the colour transition gradients identified for this image;
having found the transition this pixel is closest to and its distance along that transition, substitute the end points, remap;
recombine with the original Y, convert back to RGB and output.
That's two dot products per colour transition gradient to establish closest, then a single mix to generate the output (x, y)/

Answer 2

Let Rx, Gx, Bx = Pixel values of color X (Red in your case) to be removed/replaced.
Let Ry, Gy, By = Pixel values of color Y (Green in your case) to be used as new color.
Then you will iterate over all pixels and using clever condition (below), identify the pixel that needs to be processed.
If Rc is current value of the selected pixel color (does not matter what combination of red and yellow is), then final values of the pixel are:
Rf = Rc - Rx + Ry
Gf = Gc - Gx + Gy
Bf = Bc - Bx + By
Of course, this processing should NOT happy for all pixels. Clever condition to identify only relevant pixels could be : If pixel color is Red or least one adjacent pixel is Red/Yellow.
UPDATE: Another clever condition using current pixel only:
This involves removing border colors YELLOW or BLACK color from the current color and checking if it is RED.
Rc - R(yellow) == R(RED) AND
Gc - G(yellow) == G(RED) AND
Bc - B(yellow) == B(RED)
OR
Rc - R(black) == R(RED) AND
Gc - G(black) == G(RED) AND
Bc - B(black) == B(RED)