314 lines
12 KiB
Markdown
314 lines
12 KiB
Markdown
<p align="center">
|
||
<img src="images/logo.png" alt="arpack logo" width="240">
|
||
</p>
|
||
|
||
# ArPack
|
||
|
||
|
||
|
||
|
||
|
||
|
||
Binary serialization code generator for Go, C#, TypeScript, and Lua. Define messages once as Go structs — get zero-allocation `Marshal`/`Unmarshal` for Go, `unsafe` pointer-based `Serialize`/`Deserialize` for C#, `DataView`-based serialization for TypeScript/browser, and pure Lua implementation for Defold/LuaJIT.
|
||
|
||
## Features
|
||
|
||
- **Single source of truth** — define messages in Go, generate code for Go, C#, TypeScript, and Lua
|
||
- **Float quantization** — compress `float32`/`float64` to 8 or 16 bits with a `pack` struct tag
|
||
- **Boolean packing** — consecutive `bool` fields are packed into single bytes (up to 8 per byte)
|
||
- **Enums** — `type Opcode uint16` + `const` block becomes C#/TypeScript enums
|
||
- **Nested types, fixed arrays, slices** — full support for complex message structures
|
||
- **Cross-language binary compatibility** — Go, C#, TypeScript, and Lua produce identical wire formats
|
||
- **Browser support** — TypeScript target uses native DataView API for zero-dependency serialization
|
||
|
||
## When to use
|
||
|
||
ArPack is designed for real-time multiplayer games and other latency-sensitive systems where a Go backend talks to a C# client over a binary protocol.
|
||
|
||
Typical setups:
|
||
|
||
- **[Nakama](https://heroiclabs.com/nakama/) + Unity** — define all network messages in Go, generate C# structs for Unity. Both sides share the exact same wire format with no reflection or boxing.
|
||
- **Custom Go game server + Unity** — roll your own server without pulling in a serialization framework. ArPack generates plain `Marshal`/`Unmarshal` methods with zero allocations on the hot path.
|
||
- **Any Go service + .NET client** — works anywhere you control both ends and want a compact binary protocol without Protobuf's runtime overhead or code-gen complexity.
|
||
- **Go backend + Browser/WebSocket** — generate TypeScript classes for browser-based clients. Uses native DataView API with zero dependencies.
|
||
- **Go backend + Defold/Lua** — generate Lua modules for Defold game engine. Pure Lua implementation compatible with LuaJIT.
|
||
|
||
## Installation
|
||
|
||
```bash
|
||
go install github.com/edmand46/arpack/cmd/arpack@latest
|
||
```
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
# Generate Go + C# + TypeScript
|
||
arpack -in messages.go -out-go ./gen -out-cs ../Unity/Assets/Scripts -out-ts ./web/src/messages
|
||
|
||
# Generate only TypeScript
|
||
arpack -in messages.go -out-ts ./web/src/messages
|
||
|
||
# Generate only Lua (for Defold)
|
||
arpack -in messages.go -out-lua ./defold/scripts/messages
|
||
```
|
||
|
||
| Flag | Description |
|
||
|---|---|
|
||
| `-in` | Input Go file with struct definitions (required) |
|
||
| `-out-go` | Output directory for generated Go code |
|
||
| `-out-cs` | Output directory for generated C# code |
|
||
| `-out-ts` | Output directory for generated TypeScript code |
|
||
| `-out-lua` | Output directory for generated Lua code |
|
||
| `-cs-namespace` | C# namespace (default: `Arpack.Messages`) |
|
||
|
||
**Output files:**
|
||
- Go: `{name}_gen.go`
|
||
- C#: `{Name}.gen.cs`
|
||
- TypeScript: `{Name}.gen.ts`
|
||
- Lua: `{name}_gen.lua` (snake_case for Lua `require()` compatibility)
|
||
|
||
## v1 Contract
|
||
|
||
ArPack `v1` intentionally supports a narrow schema model:
|
||
|
||
- Input is a single Go source file.
|
||
- Message types must be defined in that same file.
|
||
- External package types, pointers, and platform-dependent integer aliases (`int`, `uint`, `uintptr`) are not supported.
|
||
- Wire format is stable within `v1.x` for unchanged schemas.
|
||
|
||
This is a deliberate product boundary for predictable code generation and cross-language compatibility.
|
||
|
||
## Schema Definition
|
||
|
||
Messages are defined as Go structs in a single `.go` file:
|
||
|
||
```go
|
||
package messages
|
||
|
||
// Quantized 3D vector — 6 bytes instead of 12
|
||
type Vector3 struct {
|
||
X float32 `pack:"min=-500,max=500,bits=16"`
|
||
Y float32 `pack:"min=-500,max=500,bits=16"`
|
||
Z float32 `pack:"min=-500,max=500,bits=16"`
|
||
}
|
||
|
||
// Enum
|
||
type Opcode uint16
|
||
|
||
const (
|
||
OpcodeUnknown Opcode = iota
|
||
OpcodeAuthorize
|
||
OpcodeJoinRoom
|
||
)
|
||
|
||
type MoveMessage struct {
|
||
Position Vector3 // nested type
|
||
Velocity [3]float32 // fixed-length array
|
||
Waypoints []Vector3 // variable-length slice
|
||
PlayerID uint32
|
||
Active bool // 3 consecutive bools →
|
||
Visible bool // packed into 1 byte
|
||
Ghost bool
|
||
Name string
|
||
}
|
||
```
|
||
|
||
### Supported Types
|
||
|
||
| Type | Wire Size | Lua Support |
|
||
|---|---|---|
|
||
| `bool` (packed) | 1 bit (up to 8 per byte) | ✓ (uses BitOp library) |
|
||
| `int8`, `uint8` | 1 byte | ✓ |
|
||
| `int16`, `uint16` | 2 bytes | ✓ |
|
||
| `int32`, `uint32`, `float32` | 4 bytes | ✓ |
|
||
| `int64`, `uint64` | 8 bytes | ✗ (LuaJIT limitation) |
|
||
| `float64` | 8 bytes | ✓ |
|
||
| `string` | 2-byte length prefix + UTF-8 | ✓ |
|
||
| `[N]T` | N × sizeof(T) | ✓ |
|
||
| `[]T` | 2-byte length prefix + N × sizeof(T) | ✓ |
|
||
|
||
**Note:** platform-dependent `int`, `uint`, and `uintptr` are not supported. Use explicit widths like `int32`, `uint32`, `int64`, or `uint64`.
|
||
|
||
**Note:** `int64`/`uint64` are not supported in Lua target. LuaJIT (used by Defold) represents numbers as double-precision floats, which can only safely represent integers up to 2^53. Use `int32`/`uint32` instead.
|
||
|
||
### Float Quantization
|
||
|
||
Use the `pack` struct tag to compress floats:
|
||
|
||
```go
|
||
X float32 `pack:"min=-500,max=500,bits=16"` // 2 bytes instead of 4
|
||
Y float32 `pack:"min=0,max=1,bits=8"` // 1 byte instead of 4
|
||
```
|
||
|
||
| Parameter | Description |
|
||
|---|---|
|
||
| `min` | Minimum expected value |
|
||
| `max` | Maximum expected value |
|
||
| `bits` | Target size: `8` (uint8) or `16` (uint16) |
|
||
|
||
Values are linearly mapped: `encoded = (value - min) / (max - min) * maxUint`.
|
||
|
||
Quantized values must stay within the declared `[min, max]` range. Generated serializers fail fast on out-of-range or `NaN` inputs instead of silently truncating them.
|
||
|
||
## Generated Code
|
||
|
||
### Go
|
||
|
||
```go
|
||
func (m *MoveMessage) Marshal(buf []byte) []byte
|
||
func (m *MoveMessage) Unmarshal(data []byte) (int, error)
|
||
```
|
||
|
||
`Marshal` appends to the buffer and returns it. `Unmarshal` reads from the buffer and returns bytes consumed.
|
||
|
||
**Failure behavior:** generated `Marshal` panics if a string/slice exceeds the `uint16` wire limit or if a quantized value is outside its declared range.
|
||
|
||
### C#
|
||
|
||
```csharp
|
||
public unsafe int Serialize(byte* buffer)
|
||
public static unsafe int Deserialize(byte* buffer, out MoveMessage msg)
|
||
```
|
||
|
||
Uses unsafe pointers for zero-copy serialization. Returns bytes written/consumed.
|
||
|
||
**Failure behavior:** generated `Serialize` throws if a string/slice exceeds the `uint16` wire limit or if a quantized value is outside its declared range.
|
||
|
||
### TypeScript
|
||
|
||
```typescript
|
||
export class MoveMessage {
|
||
position: Vector3 = new Vector3();
|
||
velocity: number[] = new Array<number>(3).fill(0);
|
||
waypoints: Vector3[] = [];
|
||
playerId: number = 0;
|
||
active: boolean = false;
|
||
visible: boolean = false;
|
||
ghost: boolean = false;
|
||
name: string = "";
|
||
|
||
serialize(view: DataView, offset: number): number
|
||
static deserialize(view: DataView, offset: number): [MoveMessage, number]
|
||
}
|
||
```
|
||
|
||
Uses native DataView API for browser-compatible serialization with zero dependencies. Returns bytes written/consumed.
|
||
|
||
**Note:** TypeScript field names are converted to camelCase (e.g., `PlayerID` → `playerId`).
|
||
|
||
**Failure behavior:** generated `serialize(...)` throws `RangeError` if a string/slice exceeds the `uint16` wire limit or if a quantized value is outside its declared range.
|
||
|
||
### Lua
|
||
|
||
```lua
|
||
local messages = require("messages.messages_gen")
|
||
|
||
-- Create message
|
||
local msg = messages.new_move_message()
|
||
msg.player_id = 123
|
||
msg.active = true
|
||
|
||
-- Serialize
|
||
local data = messages.serialize_move_message(msg)
|
||
|
||
-- Deserialize
|
||
local decoded, bytes_read = messages.deserialize_move_message(data, 1)
|
||
```
|
||
|
||
Uses pure Lua with inline helper functions for byte manipulation. Compatible with LuaJIT (Defold). All identifiers use snake_case (e.g., `MoveMessage` → `move_message`, `PlayerID` → `player_id`).
|
||
|
||
**Requirements:** The generated Lua code requires the [BitOp library](https://bitop.luajit.org/) for bit manipulation. This library is included in LuaJIT (used by Defold).
|
||
|
||
**Limitations:**
|
||
- Lua target does not support `int64`/`uint64` types. Use `int32`/`uint32` instead. This is because LuaJIT represents numbers as double-precision floats, which can only safely represent integers up to 2^53.
|
||
- Variable-length fields use `uint16` length prefixes, so `string` byte length and `[]T` element count must not exceed `65535`. Serialization raises an error if the limit is exceeded.
|
||
- Quantized values must stay within the declared `[min, max]` range. Serialization raises a Lua error on out-of-range or `NaN` inputs.
|
||
- Deserialization raises Lua errors on malformed or truncated input. If you need a recoverable boundary, wrap decode calls in `pcall(...)`.
|
||
- Generated file uses snake_case naming (e.g., `messages_gen.lua`) for proper Lua `require()` resolution.
|
||
|
||
## Wire Format
|
||
|
||
- Little-endian byte order
|
||
- No message framing — fields are written in declaration order
|
||
- Variable-length fields (`string`, `[]T`) prefixed with `uint16` length
|
||
- Booleans packed as bitfields (LSB first, up to 8 per byte)
|
||
- Quantized floats stored as `uint8` or `uint16`
|
||
|
||
## Compatibility Guarantees
|
||
|
||
Within `v1.x`, the following are considered compatibility guarantees for a fixed schema:
|
||
|
||
- Same field declaration order produces the same wire layout.
|
||
- Go, C#, TypeScript, and Lua generators produce identical wire bytes for supported types.
|
||
- `string` and `[]T` always use `uint16` length prefixes.
|
||
- Consecutive `bool` fields are bit-packed in declaration order, least-significant bit first.
|
||
- Enum fields use their declared underlying integer type on the wire.
|
||
|
||
The following are breaking changes:
|
||
|
||
- changing field order
|
||
- changing a field type
|
||
- changing quantization parameters
|
||
- changing enum underlying types
|
||
- changing how booleans are grouped or how lengths are encoded
|
||
|
||
## Benchmarks
|
||
|
||
### Go Results (M3 Max)
|
||
```
|
||
BenchmarkArPack_Marshal-16 382568360 9.5 ns/op 5065 MB/s 0 B/op 0 allocs/op
|
||
BenchmarkArPack_Unmarshal-16 98895892 34.6 ns/op 1388 MB/s 40 B/op 2 allocs/op
|
||
BenchmarkProto_Marshal-16 21989466 163.6 ns/op 416 MB/s 0 B/op 0 allocs/op
|
||
BenchmarkProto_Unmarshal-16 13950333 256.9 ns/op 265 MB/s 248 B/op 7 allocs/op
|
||
BenchmarkFlatBuffers_Marshal-16 16297458 221.4 ns/op 687 MB/s 0 B/op 0 allocs/op
|
||
BenchmarkFlatBuffers_Unmarshal-16 56095480 64.8 ns/op 2345 MB/s 24 B/op 1 allocs/op
|
||
```
|
||
|
||
| Format | Size |
|
||
|---|---|
|
||
| ArPack | 48 bytes |
|
||
| Protobuf | 68 bytes |
|
||
| FlatBuffers | 152 bytes |
|
||
|
||
```bash
|
||
go test ./benchmarks/... -bench=. -benchmem
|
||
```
|
||
|
||
### Unity Mono (M3 Max)
|
||
|
||
```
|
||
ArPack Serialize: 96.7 ns/op | 0 B/op
|
||
ArPack Deserialize: 205.4 ns/op | 0 B/op
|
||
Proto Serialize (alloc): 930.2 ns/op | 0 B/op
|
||
Proto Deserialize (alloc): 1621.2 ns/op | 29 B/op
|
||
Proto Serialize (reuse): 652.7 ns/op | 0 B/op
|
||
```
|
||
|
||
ArPack serialize is ~10× faster than Protobuf in Unity. Protobuf deserialize allocates on every call — a GC pressure source in hot game loops. ArPack deserialize is zero-alloc.
|
||
|
||
```bash
|
||
make gen-unity
|
||
# then attach BenchmarkRunner to any GameObject in SampleScene and press Play
|
||
```
|
||
|
||
## Running Tests
|
||
|
||
```bash
|
||
# Full test suite
|
||
make test
|
||
|
||
# Benchmarks
|
||
go test ./benchmarks/... -bench=. -benchmem
|
||
```
|
||
|
||
## Troubleshooting
|
||
|
||
- `unknown type "..."`
|
||
The field type is not a supported primitive and is not defined in the same schema file.
|
||
- `external package types not supported`
|
||
Copy the wire-facing type definition into the schema file instead of referencing another package.
|
||
- `... exceeds uint16 limit`
|
||
A `string` encoded to more than `65535` bytes, or a slice contains more than `65535` elements.
|
||
- `quantized value out of range`
|
||
The runtime value does not satisfy the declared `pack:"min=...,max=..."` bounds.
|