Converting / Parsing C++ byte struct to Go - c++

Question

I am reading some data packets in Go, where the fields are C++ data types. I tried parsing the data but I am reading garbage values.
Here is a small example - the data spec sheet for a particular datatype is as follows in C++,
struct CarTelemetryData
{
uint16 m_speed;
uint8 m_throttle;
int8 m_steer;
uint8 m_brake;
uint8 m_clutch;
int8 m_gear;
uint16 m_engineRPM;
uint8 m_drs;
uint8 m_revLightsPercent;
uint16 m_brakesTemperature[4];
uint16 m_tyresSurfaceTemperature[4];
uint16 m_tyresInnerTemperature[4];
uint16 m_engineTemperature;
float m_tyresPressure[4];
};
And below is what I have defined in Go
type CarTelemetryData struct {
Speed uint16
Throttle uint8
Steer int8
Brake uint8
Clutch uint8
Gear int8
EngineRPM uint16
DRS uint8
RevLightsPercent uint8
BrakesTemperature [4]uint16
TyresSurfaceTemperature [4]uint16
TyresInnerTemperature [4]uint16
EngineTemperature uint16
TyresPressure [4]float32
}
For the actual un-marshalling, I am doing this -
func decodePayload(dataStruct interface{}, payload []byte) {
dataReader := bytes.NewReader(payload[:])
binary.Read(dataReader, binary.LittleEndian, dataStruct)
}
payload := make([]byte, 2048)
s.conn.ReadFromUDP(payload[:])
telemetryData := &data.CarTelemetryData{}
s.PacketsRcvd += 1
decodePayload(telemetryData, payload)
I suspect that this is because the datatypes are not equivalent and there is some conversion issue while reading the bytes into Go data-types, whereas they have been originally packages as C++. How can I deal with this?
Note: I don't have any control over the data that is sent, this is sent by a third party service.

Answer 1

The issue you're facing has to do with the alignment of struct members. You can read more about it here but, in short, the C++ compiler will sometimes add padding bytes in order to maintain the natural alignment expected by the architecture. If that alignment is not used, it may cause degraded performance or even an access violation.
For x86/x64, for example, the alignment of most types will usually (but not necessarily guaranteed to) be the same as the size. We can see that
#include <cstdint>
#include <type_traits>
std::size_t offsets[] = {
std::alignment_of_v<std::uint8_t>,
std::alignment_of_v<std::uint16_t>,
std::alignment_of_v<std::uint32_t>,
std::alignment_of_v<std::uint64_t>,
std::alignment_of_v<__uint128_t>,
std::alignment_of_v<std::int8_t>,
std::alignment_of_v<std::int16_t>,
std::alignment_of_v<std::int32_t>,
std::alignment_of_v<std::int64_t>,
std::alignment_of_v<__int128_t>,
std::alignment_of_v<float>,
std::alignment_of_v<double>,
std::alignment_of_v<long double>,
std::alignment_of_v<void*>,
};
compiles to
offsets:
.quad 1
.quad 2
.quad 4
.quad 8
.quad 16
.quad 1
.quad 2
.quad 4
.quad 8
.quad 16
.quad 4
.quad 8
.quad 16
.quad 8
Due to these (and other) implementation details, it may be advisable to not rely on the internal representation. In some cases, however, other methods may not be fast enough (such as serializing field by field), or you may not be able to change the C++ code, like OP.
binary.Read expects packed data, but C++ will use padding. We need to either use a compiler-dependent directive such as #pragma pack(1) or add padding the Go struct. The first is not an option for OP, so we'll use the second.
We can use the offsetof macro to determine the offset of a struct member relative to the struct itself. We can do something like
#include <array>
#include <cstddef>
#include <cstdint>
using int8 = std::int8_t;
using uint8 = std::uint8_t;
using uint16 = std::uint16_t;
struct CarTelemetryData {
uint16 m_speed;
uint8 m_throttle;
int8 m_steer;
uint8 m_brake;
uint8 m_clutch;
int8 m_gear;
uint16 m_engineRPM;
uint8 m_drs;
uint8 m_revLightsPercent;
uint16 m_brakesTemperature[4];
uint16 m_tyresSurfaceTemperature[4];
uint16 m_tyresInnerTemperature[4];
uint16 m_engineTemperature;
float m_tyresPressure[4];
};
// C++ has no reflection (yet) so we need to list every member
constexpr auto offsets = std::array{
offsetof(CarTelemetryData, m_speed),
offsetof(CarTelemetryData, m_throttle),
offsetof(CarTelemetryData, m_steer),
offsetof(CarTelemetryData, m_brake),
offsetof(CarTelemetryData, m_clutch),
offsetof(CarTelemetryData, m_gear),
offsetof(CarTelemetryData, m_engineRPM),
offsetof(CarTelemetryData, m_drs),
offsetof(CarTelemetryData, m_revLightsPercent),
offsetof(CarTelemetryData, m_brakesTemperature),
offsetof(CarTelemetryData, m_tyresSurfaceTemperature),
offsetof(CarTelemetryData, m_tyresInnerTemperature),
offsetof(CarTelemetryData, m_engineTemperature),
offsetof(CarTelemetryData, m_tyresPressure),
};
constexpr auto sizes = std::array{
sizeof(CarTelemetryData::m_speed),
sizeof(CarTelemetryData::m_throttle),
sizeof(CarTelemetryData::m_steer),
sizeof(CarTelemetryData::m_brake),
sizeof(CarTelemetryData::m_clutch),
sizeof(CarTelemetryData::m_gear),
sizeof(CarTelemetryData::m_engineRPM),
sizeof(CarTelemetryData::m_drs),
sizeof(CarTelemetryData::m_revLightsPercent),
sizeof(CarTelemetryData::m_brakesTemperature),
sizeof(CarTelemetryData::m_tyresSurfaceTemperature),
sizeof(CarTelemetryData::m_tyresInnerTemperature),
sizeof(CarTelemetryData::m_engineTemperature),
sizeof(CarTelemetryData::m_tyresPressure),
};
constexpr auto computePadding() {
std::array<std::size_t, offsets.size()> result;
std::size_t expectedOffset = 0;
for (std::size_t i = 0; i < offsets.size(); i++) {
result.at(i) = offsets.at(i) - expectedOffset;
expectedOffset = offsets.at(i) + sizes.at(i);
}
return result;
}
auto padding = computePadding();
which compiles to (constexpr FTW)
padding:
.quad 0
.quad 0
.quad 0
.quad 0
.quad 0
.quad 0
.quad 1
.quad 0
.quad 0
.quad 0
.quad 0
.quad 0
.quad 0
.quad 2
So, on x86, we need one byte before EngineRPM and two bytes before TyresPressure.
So, let's check if that works.
C++:
#include <cstddef>
#include <cstdint>
#include <iomanip>
#include <iostream>
#include <span>
using int8 = std::int8_t;
using uint8 = std::uint8_t;
using uint16 = std::uint16_t;
struct CarTelemetryData {
uint16 m_speed;
uint8 m_throttle;
int8 m_steer;
uint8 m_brake;
uint8 m_clutch;
int8 m_gear;
uint16 m_engineRPM;
uint8 m_drs;
uint8 m_revLightsPercent;
uint16 m_brakesTemperature[4];
uint16 m_tyresSurfaceTemperature[4];
uint16 m_tyresInnerTemperature[4];
uint16 m_engineTemperature;
float m_tyresPressure[4];
};
int main() {
CarTelemetryData data = {
.m_speed = 1,
.m_throttle = 2,
.m_steer = 3,
.m_brake = 4,
.m_clutch = 5,
.m_gear = 6,
.m_engineRPM = 7,
.m_drs = 8,
.m_revLightsPercent = 9,
.m_brakesTemperature = {10, 11, 12, 13},
.m_tyresSurfaceTemperature = {14, 15, 16, 17},
.m_tyresInnerTemperature = {18, 19, 20, 21},
.m_engineTemperature = 22,
.m_tyresPressure = {23, 24, 25, 26},
};
std::cout << "b := []byte{" << std::hex << std::setfill('0');
for (auto byte : std::as_bytes(std::span(&data, 1))) {
std::cout << "0x" << std::setw(2) << static_cast<unsigned>(byte)
<< ", ";
}
std::cout << "}";
}
results in
b := []byte{0x01, 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x00, 0x07, 0x00, 0x08, 0x09, 0x0a, 0x00, 0x0b, 0x00, 0x0c, 0x00, 0x0d, 0x00, 0x0e, 0x00, 0x0f, 0x00, 0x10, 0x00, 0x11, 0x00, 0x12, 0x00, 0x13, 0x00, 0x14, 0x00, 0x15, 0x00, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb8, 0x41, 0x00, 0x00, 0xc0, 0x41, 0x00, 0x00, 0xc8, 0x41, 0x00, 0x00, 0xd0, 0x41, }
Let's use that in Go:
// Type your code here, or load an example.
// Your function name should start with a capital letter.
package main
import (
"bytes"
"encoding/binary"
"fmt"
)
type CarTelemetryData struct {
Speed uint16
Throttle uint8
Steer int8
Brake uint8
Clutch uint8
Gear int8
_ uint8
EngineRPM uint16
DRS uint8
RevLightsPercent uint8
BrakesTemperature [4]uint16
TyresSurfaceTemperature [4]uint16
TyresInnerTemperature [4]uint16
EngineTemperature uint16
_ uint16
TyresPressure [4]float32
}
func main() {
b := []byte{0x01, 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x00, 0x07, 0x00, 0x08, 0x09, 0x0a, 0x00, 0x0b, 0x00, 0x0c, 0x00, 0x0d, 0x00, 0x0e, 0x00, 0x0f, 0x00, 0x10, 0x00, 0x11, 0x00, 0x12, 0x00, 0x13, 0x00, 0x14, 0x00, 0x15, 0x00, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb8, 0x41, 0x00, 0x00, 0xc0, 0x41, 0x00, 0x00, 0xc8, 0x41, 0x00, 0x00, 0xd0, 0x41}
var dataStruct CarTelemetryData
dataReader := bytes.NewReader(b[:])
binary.Read(dataReader, binary.LittleEndian, &dataStruct)
fmt.Printf("%+v", dataStruct)
}
which prints
{Speed:1 Throttle:2 Steer:3 Brake:4 Clutch:5 Gear:6 _:0 EngineRPM:7 DRS:8 RevLightsPercent:9 BrakesTemperature:[10 11 12 13] TyresSurfaceTemperature:[14 15 16 17] TyresInnerTemperature:[18 19 20 21] EngineTemperature:22 _:0 TyresPressure:[23 24 25 26]}
Take the padding bytes out and it fails.