How does a PenTile display controller interpret its RGB framebuffer? On a PenTile display, the mapping from pixels in the framebuffer to individual subpixels is not trivial. Each pixel in the framebuffer may only have RG and BG subpixels, in some interesting geometric arrangement.

I'm trying to do some subpixel antialiasing of content that will be displayed on a PenTile display, so to get best results I'd like to understand how I can set the values of each pixel in the framebuffer to control the subpixels individually. (N.B. I know the subpixel layout of the display I'm targeting.)
I can't say for certain, but I was interested in this myself and trying to look it up (with little success). I did analyze the many macro photos you can find on the web of these kinds of displays, though.

RGB pixels      Pentile Subpixels (RGBG Layout)

--+---+--           --+---+--
X | X | X           G r G b G
--+---+--   --&gt;     --+---+--
X | X | X           G b G r G
--+---+--           --+---+--


As far as I can tell, it looks like the green subpixels' locations exactly match the framebuffer pixel locations (X) and the intensity is transferred 1:1. The red and blue subpixels are shared between the two neighbouring pixels horizontally and their intensity seems to be the average of the red/blue channels of the two neighbouring framebuffer pixels left and right, effectively halving the horizontal resolution of R and B.

This strategy seems to carry over to the diamond Pentile layout even though the the red and blue subpixels are shifted downwards (or upwards) by half a pixel:

RGB Pixels       Diamond Pentile subpixels
                         _________
X   X   X                G | G | G
              --&gt;        __r___b__ 
X   X   X                G | G | G
                         __b___r__


Due to the additional half pixel vertical shift for red and blue of this layout that isn't accounted for in driving the red and blue subpixels, you might be able to notice that a horizonal edge between a black area and a white area might appear green or magenta. You can see this effect in macro photos of diamond Pentile subpixel matrices rendering text. But there is actually no consistent shift. In some photos it's downwards and in others it's upwards.  I don't know if the displays were manufactured differently or just installed upside-down in some devices.

So, yes, you can easily control all subpixels of a Pentile display. Each color channel value in your frame buffer is used exactly once and for only one pixel. It's just that for the red and blue subpixels two frame buffer pixels' red and blue values are averaged.

Knowing this, you could try to do a more sophisticated subpixel rendering. But keep in mind that your typical color space like sRGB is nonlinear. The filtering should be done in a linear domain. For example, a checker board of alternating 0 and 255 will have the same brightness as a flat area with value around 185 (if I remember correctly) instead of 128.

And finally, I want to point out that it's possible to come up with anti-alias filters for non-rectangular sampling lattices. For example, for this checkerboard sampling lattice you have for the red and blue subpixels, the ideal lowpass filter would have a pass band in form of a diamond shape in the spectral domain. If you tried to reduce the resolution by dropping every 2nd sample in a checkerboard fashion, reasonable anti-alias filters would be these 5x5 and 3x3 kernels while the 5x5 version retains more detail:

 0   -1    0   -1    0
-1    0   10    0   -1              0  1  0
 0   10   32   10    0              1  4  1
-1    0   10    0   -1              0  1  0
 0   -1    0   -1    0


These can be applied in-place due to the zeros' locations.

You might also want to include a half-pixel shift for one or two dimensions in there to account for the relative positioning of the red blue and green subpixels. For example, for the diamond pentile layout you might want to use one of these filters:

  3    0    0    0    0    3
  0  -25    0    0  -25    0          -1   0   0  -1
  0    0  150  150    0    0           0   9   9   0
  0    0  150  150    0    0           0   9   9   0
  0  -25    0    0  -25    0          -1   0   0  -1
  3    0    0    0    0    3


to come up with the right red/blue subpixels instead of just averaging four values.

The RGBG (non-diamond) Pentile pattern would require only a half-pixel shift horizontally, leading to a filter like this:

 -3   3   3  -3
  2  30  30   2
 -3   3   3  -3


Some care must be taken with clipping due to the negative filter coefficients. For example, you could try to redistribute the residual to surrounding pixels to keep the average brightness.