rawproto
Version:
Guess structure of protobuf binary from raw data
111 lines (71 loc) • 5.34 kB
Markdown
# rawproto
[](https://github.com/konsumer/rawproto/actions/workflows/test.yml) [](https://codecov.io/gh/konsumer/rawproto) [](https://www.npmjs.com/package/rawproto)
Guess structure of protobuf binary from raw data, query binary protobuf without the schema, and output guessed JSON or schema, some CLI utils, and a web tool for exploring raw protobuf.
You can explore your proto binary data [here](https://konsumer.js.org/rawproto/). Use it to view, generate proto/json files, or select how to parse fields.
If you are coming form an older version, or anothe rlibrary, check out [migration instructions](#migration).
## installation
`npm i rawproto` will add this to your project.
You can also use `npx rawproto` to run the CLI.
If you just want the CLI, and don't use node, you can also find standalone builds [here](https://github.com/konsumer/rawproto/releases).
## usage
### CLI
Install it in your path with `npm i -g rawproto` or use it 1-off with `npx rawproto`. Get help with `rawproto --help`
### code
You can use it in code like this:
```js
import { readFile } from 'fs/promises'
import RawProto from 'rawproto'
// load proto
const proto = new RawProto(await readFile('data.pb'))
// get a single field, without parsing the whole tree
console.log(proto.query('1.2.4.10.5:string'))
// you can also pull things like they are arrays/values
console.log(proto['1'][0]['2'][0]['4'][0]['10'].map(r => r['5'][0].string ))
// guess to decode as JS object
console.log(proto.toJS())
// guess to generate .proto file string
console.log(proto.toProto())
// walk over all fields recursively, calling your callback.
const mydata = proto.walk((field) => {
console.log(field)
// just do whatever it normally does to make JS-object
return walkerJS(field)
})
```
## types
Protobuf encodes several different possible types for every wire-type. In this lib, we guess the type based on some context-clues, but it will never be perfect, without hand-tuning. Here are the possible types we support:
```
VARINT - int, bool, string
FIXED64 - uint, int, bytes, float, string
LEN - string, bytes, packedIntVar, packedInt32, packedInt64, string
FIXED32 - int, uint, bytes, float, string
```
- You can use any [protobuf scalar type-name](https://protobuf.dev/programming-guides/proto3/#scalar).
- You can use `raw` for any type to get the raw field with bytes + meta.
- Groups are treated as repeated `LEN` message-fields
- `LEN` will try to be parsed as sub-tree, but you can override with other types in `query` (for example if it tries to make a sub-message with part of a string)
## query-map
Many things (ui, `toJS`, `toProto`, cli) use `queryMap` which is just a map of `name` to `path:type`. Here is one that works well with [hearthstone test data](https://github.com/konsumer/rawproto/raw/master/test/hearthstone.bin):
```json
{
"id": "1.2.4.1:string",
"title": "1.2.4.5:string",
"company": "1.2.4.6:string",
"description": "1.2.4.7:string",
"media": "1.2.4.10",
"dimensions": "1.2.4.10.2",
"width": "1.2.4.10.2.3:uint",
"height": "1.2.4.10.2.4:uint",
"url": "1.2.4.10.5:string",
"type": "1.2.4.10.1:uint",
"bg": "1.2.4.10.15:string"
}
```
You can use any types, from above, and set the name to whatever you want.
## migration
I used to have the functionality of this lib split up into several other projects. Here is migration instructions, if you want to update to this one (recommended):
- [protobuf-decoder](https://github.com/konsumer/protobuf-decoder) - just use [site](https://konsumer.js.org/rawproto/). The code is [here](https://github.com/konsumer/rawproto/tree/master/ui)
- [rawprotoparse](https://github.com/konsumer/rawprotoparse) - this originally would create JSON from protobuf binary. If you were using this as-is, it had a lot of options, which have been merged into either `toJS` (see [tests](https://github.com/konsumer/rawproto/blob/master/test/json.test.js) for examples.) It may require a little bit more custom-code, if you were not using it with defaults, but overall should work better, and merges shared code that was in both libs. Main thing is that regular `toJS`, without a custom-mapper, will make all values an array, since it's possible for any field ID to be found multiple times.
- [newrawprotoparser](https://github.com/konsumer/newrawprotoparser) - this was some of the start of ideas for this. No one is probly using this. Essentially, it's the same stuff in [path](https://github.com/konsumer/rawproto/blob/master/test/path.test.js)
- [protoquery](https://github.com/konsumer/protoquery) - this was some of the start of ideas for this. No one is probly using this. Essentially it's the same stuff in [query](https://github.com/konsumer/rawproto/blob/master/test/query.test.js)
- rawproto - This lib used to be able to do JSON and generate proto, and provided a different CLI. You should be able to use the new APIs to accomplish all the same stuff, but it may require a bit of a change to your code. Have a look at the [unit-tests](https://github.com/konsumer/rawproto/tree/master/test), to get an idea of how it works.