GBEmu gets HQX upscaling

The latest update of GBEmu (that is, the GBEmu-module-io branch) includes a new upscaler, based on the concept of the hqnx-upscalers, using GLSL.

The images show the quality differences between hqnx, bilinear and nearest upscaler. When looking closely, one can see that my implementation is not bug free yet, showing gradients where there should be solid areas.

These filters analyse the similarity of the 8 neighbors to the center pixel, as well as the similarity of the pixel-pairs on the four secondary diagonals. Using a threshold, each of these similarities is reduced to a single bit of information, resulting in 12 bits of similarity information.

The original hqnx upscalers use a switch block with code for each of these similarity codes. (To be exact, for the 8 bits from the similarities to the center points, the secondary diagonals are conditionals in the case blocks.) Each of these code blocks creates the needed output pixels directly using the 9 input pixels.

My implementation, on the other hand, uses small factor maps encoding the scaling factors for each of the four neighbor pixels in each given quadrant. The four factors are encoded in the RGBA components in a texture. Each map is 16 by 16 pixels in size, and there are 64 by 64 factor maps in the final texture, giving a modest 1024 by 1024 RGBA texture.

The 12 bit of similarity information extracted from the original image are put into yet another texture of the same size as the original. Since the shader needs to blend two frames together(to enable flicker-free transparency effects for example in Zelda 4), this index map contains the indexes for the current and the last frame. That way, each component in the index map carries 6 bit of similarity information, which then is directly used to address the correct factor map in the factor texture.

Since the factor map is undersampled at lower magnifications, the factor texture is kept in 5 resolutions. That way, the antialiasing of lines is preserved.

While implementing the fragment shader, I noticed that the MESA GLSL- parser is a lot more forgiving of small coding mistakes than its NVIDIA counterpart. For example, the NVIDIA parser complains when one uses "2" at places where floats are expected, forcing the more correct "2.0"(or a cast). On the other hand, the MESA parser seems to be unable to parse the fractional part of floating point numbers(maybe only in my locale), so i needed to work around this by using "1.0/2.0" for "0.5".