So, I decided to write a BUFR decoder... Part 3

Contents

• Introduction and Recap
• BUFR Tables and Descriptors
  • FXY Descriptors
  • Table B: Element Descriptors
  • Table D: Sequence Descriptors
  • Table Lookup
• Decoding with Descriptors
• BUFR Query API
• End of Quarter Waffle Party

Introduction and Recap

This is the final part of a series where I discuss writing a WMO BUFR decoder with the expressed intention of using it to read NWS and IMET BUFR upperair soundings at their full resolution, inside of an application that can be compiled to Web Assembly and executed in a browser. In part 1, I go over the full motivation as to why on Earth I would do something this stupid. In part 2, I discuss the structure of BUFR files and messages, how their header metadata is structured, why I chose Zig as the language to tackle this challenge, and the wonders of compile-time reflection. These are absolutely required reading if you don’t want to feel lost, and I also put a lot of work into writing them — so if you haven’t yet, please go give them a read! I tried to make it so that there’s some interesting stuff to learn. However, I’ll still leave you with a TLDR:

• Existing BUFR decoders weren’t viable solutions for my end goal.
• The IMET community needs to be able to view raw BUFR upperair data in the field with limited to no internet connectivity.
• BUFR sections are highly structured templates with metadata having compile-time known bit-widths.
• Zig is an awesome systems programming language with compile-time reflection.
• I like learning new things and facing complex challenges.

I won’t beat around the bush — after about 10 weeks working on this project, I finally got things working. At least, for the minimum viable product I was targeting. If you follow me on BlueSky, you will have already seen the fruits of this labor!

Yeah, that timestamp is mid-August. I wanted to write about this much sooner and finish off this deep dive series, but I had to immediately shift gears onto other projects. Then there was the government shutdown, and I wasn’t in a particularly good mood for writing. The BUFR decoder is not yet fully spec-compliant, either… there are lots of complex features defined by “Operator Descriptors” that I have yet to implement, and I do plan to implement. However, the primary goal of this endeavor was to be able to decode BUFR upperair messages and display them in an application compiled to Web Assembly. Mission (mostly) accomplished.

So, with the initial project goals finished, here in part 3, I want to:

• Discuss BUFR Descriptors and BUFR Table B and D
• Talk about some of the challenges and problems solved along the way
• Demonstrate the query API
• Discuss future plans, perhaps show a demo, and hopefully wrap this series up!

Once again, this likely will not be a short read. At this point in the process, you know what you’re getting though, and you’re not here for brief overviews and summaries. You’re here for the deep, gritty details of how stuff gets done and problems get solved. So with that out of the way, let us descend into madness together, one last time…

BUFR Tables and Descriptors

BUFR tables are at the core of what makes the BUFR format considered “flexible”. It is also, in my humble opinion, one of the more frustrating parts of the format. However, by using tables to define variables, their units, scales, and bit-widths within a BUFR message, variables can be added or modified without breaking the spec. These BUFR tables are released on a semi-regular basis (around twice annually) by the World Meteorological Organization, and as of December 2025, BUFR4 Version 45 has been released. I even provided a very minor contribution in the form of a bug report for some (extremely minor) formatting inconsistencies encountered while writing this decoder! While these versioned BUFR tables do provide flexibility to the standard, it does mean that BUFR decoders have to account for each and every table version within the spec. In a typical installation environment on a computer, these tables would be stored in a known path that the library looks for, allowing for new tables to be dropped into the environment without a need to recompile. For a Web Assembly based environment, however, I really was not sure how to ship the tables in a portable way. Perhaps there is a way, and I’m just stupid (very likely, actually), but I had an idea nagging at me in the back of my mind…

What if I turned the versioned BUFR tables into compiled Zig code? It would potentially lead to some larger binary sizes (though binary + tables would theoretically be smaller, overall), but it also would mean I don’t have to manage the decoder environment and figure out how to “install” BUFR tables in a WASM environment. I could always extend the library functionality to support dynamic table lookup later, but static table lookup was guaranteed to be a viable solution for my problem. So, that’s what I did.

FXY Descriptors

FXY descriptors are 16-bit codes (FXXYYY) that describe data within a BUFR message. The Section 3 headers within a BUFR message encode a list of “unexpanded descriptors” (of type FXY) that describe the message in totality, with the values of F, X, and Y encoding what the descriptor is and what it points to in a BUFR table. For example, an upperair BUFR message has the following unexpanded descriptors: [301128, 309052, 205060, 13047, 11044, 11045, 11054, 11055]. In short, FXY descriptors are a coded map on how to interpret the data and where to look up the necessary information about the data.

A value of F == 0 is an Element Descriptor, which is defined by a single entry within BUFR Table B. The values of X and Y correspond to the Table B class and the element within that class, respectively. A value of F == 1 is a Replication Descriptor. The X value of a Replication Descriptor determines how many of the following FXY descriptors shall be repeated, and Y determines how many replications of the sequence shall be done. Sometimes replication can be “delayed” and defined by another Element Descriptor. If F == 2, then it is an Operator Descriptor. Operations, as defined in BUFR Table C, can define operators and operands for modification (i.e. extend the bit-width of a value in Table B). We won’t mention Operator Descriptors further, as this is an area this prototype library has yet to fully delve into. Finally, if F == 3, it is a Sequence Descriptor. These define sequences of Element Descriptors, Operator Descriptors, Replication Descriptors, or even other Sequence Descriptors. They are defined in BUFR Table D, with the values of X and Y corresponding to category and element, respectively.

These FXY descriptors can be represented in code fairly simply:

pub const FXY = packed struct {
    F: u2,
    X: u6,
    Y: u8,

    pub fn eql(self: @This(), other: FXY) bool {
        return (self.F == other.F) and (self.X == other.X) and (self.Y == other.Y);
    }

    pub fn to_string(self: @This(), buf: []u8) ![]const u8 {
        const val = std.fmt.bufPrint(buf, "{d}{d:02}{d:03}", .{
            self.F,
            self.X,
            self.Y,
        });
        return val;
    }

    pub fn from_string(fxy: []const u8) !@This() {
        const F = try std.fmt.parseInt(u2, fxy[0..1], 10);
        const X = try std.fmt.parseInt(u6, fxy[1..3], 10);
        const Y = try std.fmt.parseInt(u8, fxy[3..fxy.len], 10);

        return .{
            .F = F,
            .X = X,
            .Y = Y,
        };
    }
};

Though FXY descriptors can be represented easily, knowing what to do with them and how is a whole different issue, and so is accessing the metadata they reference within a given BUFR table!

Table B: Element Descriptors

BUFR Table B is simply a large table that describes variables. If we were to look up a descriptor with the format 012003, it would be interpreted as “An Element Descriptor describing an entry in Table B, Class 12 (Temperature), entry 3”.

ClassNo,ClassName_en,FXY,ElementName_en,BUFR_Unit,BUFR_Scale,BUFR_ReferenceValue,BUFR_DataWidth_Bits
12,Temperature,012001,Temperature/air temperature,K,1,0,12
12,Temperature,012002,Wet-bulb temperature,K,1,0,12
12,Temperature,012003,Dewpoint temperature,K,1,0,12
12,Temperature,012004,Air temperature at 2 m,K,1,0,12

Looking at the corresponding entry in Table B, we see it is the Dewpoint Temperature in Kelvin with a data-width of 12-bits! In a more traditional install, these CSV files describing the tables would be installed in a library path and looked up at runtime based on the value of FXY. However, with the previously stated goal of static table lookup rather than dynamic, these tables need to be transformed into code. A very simple representation of a Table B entry might look something like the following:

pub const ElementDescriptor = struct {
    fxy: FXY,
    name: []const u8 = undefined,
    unit: Unit,
    unit_name: []const u8 = undefined,
    scale: i32 = undefined,
    reference_value: i32 = undefined,
    width_bits: usize = undefined,

    const Unit = enum {
        CCITT_IA5,
        Code,
        Flag,
        Numeric,
    };
};

Encoding all of this information for every Table B entry for all 45 versions of the tables would be a tremendous effort. So, I wrote a preprocessor that runs at the time of compilation to parse the official WMO tables and write formatted, valid Zig code! Each of these tables are versioned based on their import paths so that BUFR4 version 44 Table B ends up in tables/{ver}/TableB.zig. Within that Zig file, the Table B entries are merely stored as entries within an array.

pub const TableB = &[_]ElementDescriptor{ 
...
.{
        .fxy = .{ .F = 0, .X = 12, .Y = 1 },
        .name = "Temperature/air temperature",
        .unit = .Numeric,
        .unit_name = "K",
        .scale = 1,
        .reference_value = 0,
        .width_bits = 12,
    },
    .{
        .fxy = .{ .F = 0, .X = 12, .Y = 2 },
        .name = "Wet-bulb temperature",
        .unit = .Numeric,
        .unit_name = "K",
        .scale = 1,
        .reference_value = 0,
        .width_bits = 12,
    },
...
};

I am fairly confident that there are more efficient ways to encode this information. For one, the string value of units (i.e. “K”) get duplicated a lot and that duplicated memory is effectively wasted. One improvement here would be to store an index into a global units array of strings. Additionally, linear lookup within a large array is going to scale with the number of table entries, though linear memory does benefit from vectorization. The uniqueness of FXY descriptors make maps/dictionaries an attractive solution due to their constant lookup time, though they would have to be constructed/initialized at runtime by the library. I feel like this is the direction I want to go, but given the time constraints I was under and the desire to reach the goals of my deliverable application, this was good enough to make something work. If any astute Ziglings happen to read this, it should also trigger thoughts of the Zig Object Notation (ZON), which, if I understand correctly, can be used both at runtime and compile time! With future goals of supporting both dynamic and static tables, ZON is definitely on my radar.

A quick aside about knowing when to move on with something in programming. Sometimes in development, good enough is just that — good enough — especially when you know that you will be able to revisit something in the future. I know that it can be risky to put things off until an unspecified “later”, but the core guts are likely to remain the same, the pipeline is mostly automated, and I am guaranteed to have to revisit some of this at a later date. It’s not worth getting stuck in a design, iterate, and optimize loop if it stops you from reaching your end goal… especially when it turns out the “good enough” runs plenty fast, anyway!

With BUFR Table B able to be translated to compiled code and versioned based on the WMO version number, it is now possible to perform FXY lookups for unexpanded descriptors within a BUFR message, and eventually, use it to decode the binary data. Unfortunately, there’s plenty more work before that happens.

Table D: Sequence Descriptors

As previously mentioned, Sequence Descriptors can define sequences of any arbitrary set of descriptors, including other sequences. The provided example of a BUFR Table D sequence is for information regarding radiosonde launch location metadata, which includes both Element Descriptors from Table B, and Sequence Descriptors from elsewhere in Table D.

CategoryOfSequences_en,FXY1,Title_en,SubTitle_en,FXY2,ElementName_en
Location and identification sequences,301121,(Radiosonde launch point location),,008041,Data significance
Location and identification sequences,301121,(Radiosonde launch point location),,301122,Date/time (to hundredths of second)
Location and identification sequences,301121,(Radiosonde launch point location),,301021,Latitude/longitude (high accuracy)
Location and identification sequences,301121,(Radiosonde launch point location),,007031,Height of barometer above mean sea level
Location and identification sequences,301121,(Radiosonde launch point location),,007007,Height

The data type used to store a sequence is going to seem incredibly anticlimactic. I feel the need to mention that I spent a lot of time spinning my wheels on how to represent sequences, particularly, nested sequences. I thought that maybe I could leverage Zig’s “comptime” to help unwind some of the nesting, I tried various parsing methods of the CSVs and preprocessing steps, I tried different formats for both elements and sequences, and I just could not get it to work the way I wanted. Really, what I wanted was a list of some union type of all descriptors that could be iterated over. Perhaps it is possible, and I’m just not that bright! The main problem I encountered is that sequences can be pretty heavily nested, and even defined out-of-order from within the table. After spending too long trying all of the wrong solutions, I finally settled on a simple array of FXY descriptors that could be looked up on their own.

pub const SequenceDescriptor = struct {
    fxy: FXY,
    category: []const u8 = undefined,
    title: []const u8 = undefined,
    subtitle: []const u8 = undefined,
    sequence: []const FXY = undefined,
};
pub const TableD = &[_]SequenceDescriptor{ 
    ...
    .{
        .fxy = .{ .F = 3, .X = 1, .Y = 121 },
        .category = "Location and identification sequences",
        .title = "(Radiosonde launch point location)",
        .subtitle = "",
        .sequence = &[_]FXY{
            .{ .F = 0, .X = 8, .Y = 41 },
            .{ .F = 3, .X = 1, .Y = 122 },
            .{ .F = 3, .X = 1, .Y = 21 },
            .{ .F = 0, .X = 7, .Y = 31 },
            .{ .F = 0, .X = 7, .Y = 7 },
        },
    },
    ...
};

Much like Table B, Table D is just a flat array of SequenceDescriptors, and much like Table B, has some room for improvement on string/memory usage and lookup efficiency. However, it gets the job done and helps satisfy the requirement of not having to deal with dynamic tables!

Table Lookup

The last step in this process is to handle table version and entry lookup. I have mentioned already that the source code containing all of the versioned tables are in a structured directory tree. We can leverage this ordered directory tree to take a runtime value (encoded in the BUFR message header) about which table version to use, and resolve an import to that specific table.

pub fn get_table(version: u8) !BUFRTable {
    return switch (version) {
        0...17 => error.UnsupportedTableVersion,
        18 => @import("./tables/0/18/root.zig").table,
        ...
        42 => @import("./tables/0/42/root.zig").table,
        43 => @import("./tables/0/43/root.zig").table,
        44 => @import("./tables/0/44/root.zig").table,
        else => error.InvalidTableVersion,
    };
}

The downside to this approach is that, technically speaking, all table versions are retained in program memory. While we’re talking about MB order storage in a design philosophy where all file bytes are also read and stored in memory, it is still something to be aware of considering the growing nature of the BUFR standard and its tables. Having dynamic tables parsed and constructed at runtime technically alleviates this, though the raw CSV tables are significantly larger than the compiled versions!

Even the worst-case scenario of using the binary with Debug symbols and no optimizations is smaller than the totality of the raw CSVs describing the BUFR tables. So, overall, the translation from CSV -> compiled code is (expectedly) a net storage savings… though a dynamic decoder would only include a single table version, hence the inclusion of only the version 44 CSVs size above. There are some tricks that can be employed to help further reduce memory footprint of the tables, primarily through string reuse, but I felt it was prudent to mention that there are in fact some down sides to intentionally trying to stretch and break the spirit of the BUFR standard. However, one could still argue that this is an overall net benefit!

Once the table version lookup has been done, the returned BUFRTable can be used to perform Table B and Table D queries based on an FXY descriptor.

fn Order(comptime Descriptor: type) type {
    return struct {
        pub fn order(context: FXY, item: Descriptor) std.math.Order {
            comptime if (Descriptor != ElementDescriptor and Descriptor != SequenceDescriptor) {
                @compileError("Must be an ElementDescriptor or SequenceDescriptor");
            };
            const context_val: u16 = @bitCast(context);
            const item_val: u16 = @bitCast(item.fxy);

            const context_swapped = std.mem.nativeToBig(u16, context_val);
            const item_swapped = std.mem.nativeToBig(u16, item_val);

            return std.math.order(context_swapped, item_swapped);
        }
    };
}

pub const BUFRTable = struct {
    TableB: []const ElementDescriptor,
    TableD: []const SequenceDescriptor,

    pub fn get_table_B(self: @This(), find: FXY) !ElementDescriptor {
        const order = comptime Order(ElementDescriptor).order;
        const maybe_idx = std.sort.binarySearch(ElementDescriptor, self.TableB, find, order);
        if (maybe_idx) |idx| return self.TableB[idx];
        return error.TableBEntryNotFound;
    }

    pub fn get_table_D(self: @This(), find: FXY) !SequenceDescriptor {
        const order = comptime Order(SequenceDescriptor).order;
        const maybe_idx = std.sort.binarySearch(SequenceDescriptor, self.TableD, find, order);
        if (maybe_idx) |idx| return self.TableD[idx];
        return error.TableDEntryNotFound;
    }
};

Since Table B and Table D are actually ordered by FXY on construction, this fact can be leveraged to assist with lookup performance and bring table lookup from O(N) to O(log(N)), where a hashmap/dictionary implementation would be O(1) (constant time). It’s a bit of a bandaid on a previously mentioned wart of storing the tables as flat arrays, but it serves its purpose well until I can revisit the tables at a later date! The Order method provided is just for completeness… the standard library binary search expects a struct with an order method that tells the algorithm how to do the comparisons in the binary search. In this case, the FXY values are cast into an unsigned 16 bit integer and swapped to Big Endian bytes before determining the order.

Decoding with Descriptors

With the core infrastructure of BUFR Tables and Descriptors in place, including some complexities of the Replication Descriptor that I am skipping over for the sake of time, the decoding of binary data in BUFR messages can be achieved. Going into the full details of that process, with code, would be quite the task to write up, but I did at least want to provide a high-level description of how I tackled this task.

The initial unexpanded descriptors in the Section 3 headers define an ordered list of “things” and instructions within the linear set of encoded bytes, meaning that everything has to be ordered. However, the expansion of descriptors can involve expanding sequences, which in and of themselves, can also have additional expansions and replications. While I am not exactly writing software for spaceships and satellites, lately I have been challenging myself to try and write code in the style of the NASA/JPL Rules for Developing Safety Critical Code. Particularly, the rule on control flow that suggests avoiding the usage of recursion.

“Simpler control flow translates into stronger capabilities for verification and often results in improved code clarity. The banishment of recursion is perhaps the biggest surprise here. Without recursion, though, we are guaranteed to have an acyclic function call graph, which can be exploited by code analyzers, and can directly help to prove that all executions that should be bounded are in fact bounded.” - Gerard Holzmann, NASA JPL

Recursion is a fairly attractive solution to BUFR decoding, but instead of going with that approach, I wrote a stack queue based system that pushes and pops from the top of the stack to preserve the required order of operations. For example, if the first descriptor to be expanded is a sequence, the decoder process will:

• Pop the sequence from the top of the stack
• Perform the Table D lookup for the sequence
• Append the list of FXY sequences for that descriptor to the top of the stack
• Begin processing again from the top of the stack

This ensures that the ordered nature of descriptors and data are retained while also avoiding recursion, with the added bonus of allowing for me to also follow the guidelines (and a general best practice) of not dynamically allocating memory after program initialization.

BUFR Query API

It’s one thing to be able to decode a BUFR message in a big text dump, and another entirely to be able to query data dynamically for use in an application. This is absolutely the area I have spent the least amount of time and effort, and the area most likely to see significant changes in the future. With the limited amount of time I had available and the relatively narrow requirements for my application, I relied heavily on a query API design used by PyBufrKit. Specifically, the ability to query based on descriptors with some sense of parent/child relationship:

# Query for 001002 that is a direct child of 301001
pybufrkit query /301001/001002 BUFR_FILE

In the case of BUFR radiosonde data, each vertical level is a sequence descriptor with the variables measured at that particular level. By using an ArrayList type data structure that can grow with the number of vertical levels in the BUFR message, each array within the radiosonde profile can be queried and stored for efficient access. This is far from generalized for every single BUFR use case, but it seemed to make good sense for what I needed it to do! Below is an example program that reads BUFR upperair data and queries specific values into arrays of vertical profile data.

...
var bufr_iter = BUFRMessage.BUFRMessageIterator{
    .bytes = bufr_data,
};

// the fields desired from the BUFR file
const query_fields = comptime &[_][]const u8{
    "005001", // launch latitude
    "006001", // launch longitude
    "303054/007004", // pressure
    "303054/010009", // height
};

// Iterate over the number of BUFR Messages. 
// This can often be just one message.
while (try bufr_iter.next()) |msg| {
    var data_store = try msg.query(alloc, query_fields);
    defer data_store.deinit();
    ...
    // these are arrays of size 1
    const lon_entry = data_store.get("006001") catch @panic("Failed to load 'lon' from datastore");
    const lat_entry = data_store.get("005001") catch @panic("Failed to load 'lat' from datastore");
    // these are arrays of size n_levels
    const pres_entry = data_store.get("303054/007004") catch @panic("Failed to load 'pres' from datastore");
    const hght_entry = data_store.get("303054/010009") catch @panic("Failed to load 'hght' from datastore");
}

End of Quarter Waffle Party

A few more steps were required to get the working sounding decoding in the browser showcased at the beginning, such as constructing an interface between the C++ and Zig code and handling memory lifetimes/ownership of the allocated data in order to prevent memory leaks. However, I ultimately deemed that content/information to be tertiary to the overall goal of talking about BUFR decoding, not how to manage foreign function interfaces. As previously mentioned, the library is not quite feature-complete, either… some of the more nuanced parts of the BUFR spec such as Operator Descriptors, or local tables defined within BUFR messages themselves, still need to be worked out. I’m pretty optimistic about getting to revisit those finer points, however. For one, point forecast soundings distributed in BUFR by NCEP make use of local tables that are defined by the first BUFR message in the file. If I want to load the model forecast profiles, then I need to be able to read those local tables and work with them!

Still, for the most pressing requirement of reading and displaying observed BUFR profiles, I’m proud of the result. It’s at least a step in the right direction, and the hope is that it proves useful to the IMET community during their fire deployments this next year. I’ll be presenting and training on the usage of this new sounding viewer during this year’s IMET training, so there’s also some good reasons there to shore things up a little bit more. I also hope to be able to make a more public version of this available in the future, so stay tuned for that!

I hope you’ve enjoyed, or at least learned something from, this deep dive series into tackling BUFR decoding. I’m pretty proud of being able to meet the challenge, no matter how stupid or insane it may have seemed at the time. I know I certainly got a lot out of the experience, and feel like I learned and grew as a programmer. I also have enjoyed taking some time to write about it and document the journey, even if it’s just for my future self. One thing this experience has proven to me is that any problem can be solved, no matter how daunting, if you can break it up into small enough discrete chunks.