Betaflight OpenGL OSD rendering

Part 1 - Interfacing with Betaflight

So we want a way of displaying the Betaflight OSD on the frontend. The OSD is being generated by the “engine”, and this must therefore be communicated to the frontend. We can do this by utilizing the shared memory segment. The idea is like follows:

  1. Betaflight “renders” characters to the OSD backbuffer
  2. The backbuffer is copied into our shared memory segment
  3. The frontend then reads the OSD out the memory segment, and renders it to the display.

First we need a space in our shared memory segment that will hold the OSD data. In our memdef.h file we will add the following:

#define OSD_WIDTH 30
#define OSD_HEIGHT 16

struct memory_s {
...
    // Osd Array.
    unsigned char osd[OSD_HEIGHT*OSD_WIDTH];
...
};

We now have allocated 480 bytes to our OSD data. Let us now look at how Betaflight writes text to the OSD, so that we can modify it to do what we want!

How Betaflight does it

If Betaflight is compiled with OSD support definitions (USE_OSD), then Betaflight will call OSD related functions, below are a few of the important ones:

// This will write a string to the OSD at the specified
// location, which is done by iteratively setting individual
// characters in the backbuffer array.
static int writeString(displayPort_t *displayPort,
                       uint8_t x,
                       uint8_t y,
                       const char *s)

// This will write a character to the OSD at the specified
// location, by setting a character in the backbuffer array.
static int writeChar(displayPort_t *displayPort,
                     uint8_t x,
                     uint8_t y,
                     uint8_t c)

// This will draw what is currently in the back buffer to
// the display. In real betaflight this is done with `mspSerialPush`,
// we would however like it to copy the backbuffer to our
// shared memory segment.
static int drawScreen(displayPort_t *displayPort)

Modifying this so it works for us

We will now have to create our own fakeosd.c file that can be linked with Betaflight in place of the real OSD source code.

We can keep most of the functions the same, but some we need to change. I will also create a fakeosd.h file, which will have the following contents:

#define CHARS_PER_LINE 30
#define VIDEO_LINES 16

extern unsigned char* osdPntr;

The last line is a declaration of the pointer to the OSD array in the shared memory segment.

The key part of the new fakeosd.c file is as follows:

...
uint8_t *osdPntr = NULL;
static uint8_t osdBackBuffer[VIDEO_LINES][CHARS_PER_LINE];
...
static int drawScreen(displayPort_t *displayPort) {
    UNUSED(displayPort);
    memcpy(osdPntr, osdBackBuffer, CHARS_PER_LINE * VIDEO_LINES);
    return 0;
}

And I already know what you are asking - osdPntr is set to NULL, how can this be! Well, we will set *osdPntr as follows:

In our Interface.cpp file we will add a new very straightforward function:

void Interface::setOsdLocation(uint8_t* pnt) {
    bf::osdPntr = pnt;
}

Which is in turn called in our main.cpp file as follows:

bf_interface.setOsdLocation(shmem->osd);

Awesome, now betaflight is copying the OSD data into our shared memory segment ready for rendering!

Part 2 - OpenGL rendering

So we now have a pointer to unsigned char osd[OSD_HEIGHT*OSD_WIDTH]; which contains the OSD characters - how do we now render these characters to the display?

MCM format

First I must tell you about the MAX7456. This is a chip designed by Maxim which is a monochrome on-screen display (OSD) generator, used to overlay characters onto an analog video stream. This is commonly used on drones FPV video feeds to overlay information, such as battery voltages, headings, and lots of other info. The MAX7456 also allows the user to upload a custom font with their own glyphs, which Betaflight makes use of with characters, such as battery icons for example. Below are all the glyphs in the default font that is shipped with betaflight:

Array of the 256 glyphs in the Betaflight OSD font (If you are wondering about the chopped up logo at the bottom, that is for the boot logo!)

The MAX7456 accepts fonts in the MCM format, which I will describe in more detail below. In our case however, there is no MAX7456, but we rather wish to render these characters using OpenGL. This means we will need to take the MCM file, and turn it into a texture that OpenGL can understand - effectively a raster image.

MCM -> TEX conversion

These MCM files used with the MAX7456 are like follows:

 1MAX7456
 201010101
 301010101
 401010101
 501010101
 601010101
 701010101
 801010101
 901010101
1001010101
11( 16375 more lines )

We will first look at the MAX7456 documentation (https://www.analog.com/media/en/technical-documentation/data-sheets/max7456.pdf) I will not get too much into the details, but what you need to is the following:

Each character consists of 12 horizontal x 18 vertical pixels. Each pixel is represented by two bits:

  • 00 = Black, opaque
  • 10 = White, opaque
  • X1 = Transparent, usually 01.

There are therefore 12 x 18 = 216 pixels per character. Since one byte encodes 4 pixels, each character requires 216/4 = 54 bytes of data.

Since the memory is orgaized in blocks of 64 bytes, this means the next 10 bytes are unused.

Image from specification of how a character is encoded

I wanted to convert this to a texture where all the characters are laid out side by side, below is the code to do so:

#include <fstream>

#define CHAR_WIDTH 12
#define CHAR_HEIGHT 18
#define TEXTURE_WIDTH CHAR_WIDTH * NUM_CHARS

#define NUM_CHARS 256

#define BITDEPTH 4

struct color_s {
  uint8_t r;
  uint8_t g;
  uint8_t b;
  uint8_t a;
};

struct color_s black = color_s{0b00000000, 0b00000000, 0b00000000, 0b11111111};
struct color_s white = color_s{0b11111111, 0b11111111, 0b11111111, 0b11111111};
struct color_s transp = color_s{0b11111111, 0b11111111, 0b11111111, 0b00000000};

// Converts a MAX7456 MCM file to a array that can be used by OpenGL
// You should free the image memory after loading it into the GPU!
auto *loadMCM(char const *fpath, int &ret_texwidth, int &ret_texheight) {
  auto *image = new uint8_t[CHAR_HEIGHT][TEXTURE_WIDTH][BITDEPTH];
  std::ifstream is(fpath);
  std::string byte;
  std::string crumb;

  struct color_s color {};

  int counter = 0;

  int pixacross = 0;
  int pixdown = 0;

  getline(is, byte); // MAX7456 - can be ignored
  while (getline(is, byte)) {
    if (!(counter % 64 > 53)) {
      for (int pos = 0; pos < 8; pos += 2) {
        crumb = byte.substr(pos, 2);
        if (crumb == "00")
          color = black;
        else if (crumb == "10")
          color = white;
        else
          color = transp;

        image[pixdown][pixacross][0] = color.r;
        image[pixdown][pixacross][1] = color.g;
        image[pixdown][pixacross][2] = color.b;
        image[pixdown][pixacross][3] = color.a;

        pixacross += 1;
      }
      if (pixacross % CHAR_WIDTH == 0) {
        pixacross -= CHAR_WIDTH;
        if (pixdown == CHAR_HEIGHT - 1) {
          pixdown = 0;
          pixacross += CHAR_WIDTH;
        } else {
          pixdown += 1;
        }
      }
    }
    ++counter;
  }
  ret_texwidth = TEXTURE_WIDTH;
  ret_texheight = CHAR_HEIGHT;
  return image;
};

(Yes, apparently a crumb (2 bits) is a real thing!)

Now that we have our font loaded into a texture, we are now able to use it!

Using the texture in OpenGL

I decided that all OSD rendering functionality would be managed by an osdRenderer class. Let us create it:

class osdRenderer {
private:
    Shader osdShader = Shader("../resources/shaders/osd.vert",
                              "../resources/shaders/osd.frag");
    // Shader will be discussed later!

    unsigned int VAO_osd; // Vertex Array Object handle for the OSD
    unsigned int VBO_osd; // Vertex Buffer Object handle for the OSD
    unsigned int EBO_osd; // Element Buffer Object handle for the OSD
    unsigned int OSDTEX_osd; // Texture handle for the OSD

    // function that will load the data into our texture
    void loadTex(std::string fontname);

public:
    // constructor, initalizes everything
    osdRenderer(std::string fontname);

    ~osdRenderer();

    // returns available provided+user fonts
    std::vector<std::string> getOSDFonts();

    // allows dynamic changing of fonts
    // (really just a loadTex wrapper)
    void changeOSDFont(std::string fontname);

    // actaully renders the OSD, called every frame
    void renderOSD(memory_s *shmem, int width, int height);
    // other functions...
};

In the constructor, we will do the following: (Most of it should hopefully be somewhat self-explanatory)

osdRenderer::osdRenderer(std::string fontname) {
    glGenVertexArrays(1, &VAO_osd);
    glGenBuffers(1, &VBO_osd);
    glGenBuffers(1, &EBO_osd);

    glBindVertexArray(VAO_osd);
    glBindBuffer(GL_ARRAY_BUFFER, VBO_osd);
    glBufferData(GL_ARRAY_BUFFER, 64, NULL, GL_DYNAMIC_DRAW);
    // 64 is the size of 4 floats (size=4) * 4 of them, one for each corner
    // more on this later

    // indices for two tris forming a rect
    unsigned int indices[] = {0, 1, 3, 1, 2, 3};
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, EBO_osd);
    glBufferData(GL_ELEMENT_ARRAY_BUFFER, sizeof(indices), indices,
    GL_STATIC_DRAW);

    // vertex position (x,y,z), float
    glVertexAttribPointer(0, 2, GL_FLOAT, GL_FALSE, 4 * sizeof(float), 0);
    glEnableVertexAttribArray(0);

    // texture co-ordinates (x,y), float
    glVertexAttribPointer(1, 2, GL_FLOAT, GL_FALSE, 4 * sizeof(float),
                            (GLvoid *)(2 * sizeof(float)));

    glGenTextures(1, &OSDTEX_osd);

    loadTex(fontname);
};

Now we get to the interesting parts: loadTex. We wish to load the font data into our texture, which we will do as shown:

void osdRenderer::loadTex(std::string fontName) {
    glBindTexture(GL_TEXTURE_2D, OSDTEX_osd);

    // Any scaling would look horrible, so use nearest neighbor (GL_NEAREST):
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);

    int width, height;
    auto *data = loadMCM(("../resources/osdfonts/" + fontName + ".mcm").c_str(),
    width, height);
    if (data) {
        glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, width, height,
                     0, GL_RGBA, GL_UNSIGNED_BYTE, data);
        // we certainly need the alpha channel!
    } else {
        std::cout << "Failed to load texture" << std::endl;
    }
    // don't forget to free memory!
}

Now that our font data is loaded into the texture, how do we now render the contents of the OSD on to the display? Every frame, we will call osdRenderer::renderOSD, which will do just this:

void osdRenderer::renderOSD(memory_s *shmem, int width, int height) {
    // note: paramenters width and height are that of the window
    // *shmem is a pointer to the shared memory address (read pt1!)

    // MAX7456 height = 288, width = 360
    // we therefore wish to rescale
    float scale = ((float)height / (float)width) / (288.0f / 360.0f);
    float wscale = 1;
    float hscale = 1;
    if (scale < 1) {
        wscale = scale;
    } else {
        hscale = 1 / scale;
    }

    // OpenGL stuff
    glBindVertexArray(VAO_osd);
    glBindBuffer(GL_ARRAY_BUFFER, VBO_osd);
    glActiveTexture(GL_TEXTURE0 + 0);
    glBindTexture(GL_TEXTURE_2D, OSDTEX_osd);
    osdShader.use();


    const int width = 30;
    const int height = 16;
    for (char y = 0; y < 16; ++y) { //
        for (char x = 0; x < 30; ++x) {
            unsigned char curchar = shmem->osd[(15 - y) * width + x];
            if (curchar == 0 || curchar == 32) {
            // no need to render if 0 or null!
                continue;
            }

            // SELECTS THE CHARACTER WE WANT
            tex_xl = curchar / 256.0f;
            tex_xr = (curchar + 1) / 256.0f;

            // SELECTS THE RENDERING LOCATION
            ryu = (-1 + 2 * ((float)(y) / height)) * hscale;
            ryd = (-1 + 2 * ((float)(y + 1) / height)) * hscale;
            rxl = (-1 + 2 * ((float)(x) / width)) * wscale;
            rxr = (-1 + 2 * ((float)(x + 1) / width)) * wscale;

            float vertices[] = {
                // apos(2), texcoord(2)
                rxr, ryu, tex_xr, 1.0f, // top right
                rxr, ryd, tex_xr, 0.0f, // bottom right
                rxl, ryd, tex_xl, 0.0f, // bottom left
                rxl, ryu,  tex_xl, 1.0f, // top left
            }
            glDrawElements(GL_TRIANGLES, 6, GL_UNSIGNED_INT, 0);
        }
    }
}

And of course our two shaders:

Vertex Shader:

#version 330 core
layout (location = 0) in vec2 aPos;
layout (location = 1) in vec2 aTexCoord;

out vec2 TexCoord;

void main()
{
    gl_Position = vec4(aPos,0.0 ,1.0);
    TexCoord = aTexCoord;
}

Fragment Shader:

#version 330 core
out vec4 FragColor;

in vec2 TexCoord;

uniform sampler2D osdtex;

void main()
{
    vec4 texColor = texture(osdtex, TexCoord);
    if (texColor.a == 0.){
        discard;
    }else{
        FragColor = texColor;
    }
}

Wrapping up

Well, let us see this in action:

Demo of the OSD functionality in the client

Works beautifully and is extremely low-latency. There are however of course further optimizations that can be made!