Edit on GitHub

# Weights file format

The NN weights file is in Google Protocol Buffers1 format.

The schema definition is located in the lczero-common repository.

### NN format description

The network format description is contained in `weights.format().network_format()` submessage.

• It’s safe to assume that `NetworkStructure` 0, 1 or 2 will never be passed to the backend. I.e.
• if `weights.format().network_format().network() == NETWORK_CLASSICAL_WITH_HEADFORMAT` (3), residual blocks don’t have SE layers.
• if `weights.format().network_format().network() == NETWORK_SE_WITH_HEADFORMAT` (4), residual blocks has SE layers.
• Otherwise the backend should throw an exception.

`weights.format().network_format().policy()`, `weights.format().network_format().value()` and `weights.format().network_format().moves_left()` must be checked, and in the case of the unsupported value, throw an exception.

`weights.format().input()` should be copied to backend’s `NetworkCapabilities`.

### Weights encoding

Number of `FILTERS` is `LayerAdapter(weights.weights().input().biases()).size() / 112`. Number of `BLOCKS` is `weights.weights().residual_size()`.

#### NN parameters encoding

Internally, the parameters are encoded as 16-bit quantized floating point numbers, in `Weights::Layer` protobuf submessage.

To use them as an array of floats, use `LayerAdapter` class from utils/weights_adapter.h.

For example here is how to access biases of the input convlayer:

``````const auto& input = weights.weights().input();
// Now use biases as if it was a vector<float>, e.g.
std::cout << biases.size();
for (auto x: biases) { /* do something */ }
``````

Tensors are stored linearized to a vector. Noone remembers anymore which transposition is used for it, but most likely it’s `HWC` (e.g. the conv kernel is 3×3×`FILTERS`). When you figure that out, feel free to update this document.

#### Convolution layers

Convolution layers are encoded in `Weights::ConvBlock` for example, input convolution is `weights.weights().input()` of this type:

``````  message ConvBlock {
optional Layer weights = 1;
optional Layer biases = 2;
optional Layer bn_means = 3;
optional Layer bn_stddivs = 4;
optional Layer bn_gammas = 5;
optional Layer bn_betas = 6;
}
``````

`weights` contains weights, and `biases` contains biases. All `bn_` are not used anymore, always zero (they were used for batch normalization but now it’s fused).

#### Residual blocks

In `weights.weights().residual()` repeated submessage field, this is pretty self-describing.

``````  message Residual {
optional ConvBlock conv1 = 1;
optional ConvBlock conv2 = 2;
optional SEunit se = 3;
}
``````

`SEunit` is also quite clear:

``````  message SEunit {
optional Layer w1 = 1;
optional Layer b1 = 2;
optional Layer w2 = 3;
optional Layer b2 = 4;
}
``````

`SE_CHANNELS` can be compute as `LayerAdapter(se_unit.b1()).size()`;

For `POLICY_CLASSICAL`, `POLICY_CONV_SIZE` is computed as `LayerAdapter(weights.policy().biases()).size()`. Also for `POLICY_CLASSICAL`, the `policy1` field is unused.

``````  // Policy head
// Extra convolution for AZ-style policy head
optional ConvBlock policy1 = 11;
optional ConvBlock policy = 3;
optional Layer ip_pol_w = 4;
optional Layer ip_pol_b = 5;
``````

Probably clear..

``````  // Moves left head
optional ConvBlock moves_left = 12;
optional Layer ip1_mov_w = 13;
optional Layer ip1_mov_b = 14;
optional Layer ip2_mov_w = 15;
optional Layer ip2_mov_b = 16;
``````

`MLH_CHANNELS` = `LayerAdapter(weights.moves_left().biases()).size()`

`FC_SIZE` = `LayerAdapter(weights.ip1_mov_b()).size()`

## LegacyWeights wrapper

There is a `LegacyWeights` struct which was used to contain weights before Protocol Buffer weights format was introduced. This struct was planned to be deprecated and later removed.

However, it seems that it adds considerable amount of convenience (as working with `std::vector<float>` is more familiar) so it’s probably here to say.

Last Updated: 2020-06-04
{"ft_lang_label":"__label__en","ft_lang_prob":0.72883195,"math_prob":0.9470568,"size":3782,"snap":"2020-24-2020-29","text_gpt3_token_len":909,"char_repetition_ratio":0.17337215,"word_repetition_ratio":0.0,"special_character_ratio":0.24775252,"punctuation_ratio":0.19732937,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9804446,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2020-07-09T12:23:44Z\",\"WARC-Record-ID\":\"<urn:uuid:dbe538d8-ea21-403d-b51d-91541a457161>\",\"Content-Length\":\"25197\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:f597ccd0-28e7-415e-a96a-b08e50f4faa7>\",\"WARC-Concurrent-To\":\"<urn:uuid:8fcbca41-ade8-4d35-9996-520941331590>\",\"WARC-IP-Address\":\"104.24.114.1\",\"WARC-Target-URI\":\"http://draft.lczero.org/dev/backend/weights/\",\"WARC-Payload-Digest\":\"sha1:U65GEDSBQLKW75XJ2SM4LNVC2BYDMMMQ\",\"WARC-Block-Digest\":\"sha1:ACSWGE3TDKPEARK25DYYLQ5BYOA5PI3Q\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2020/CC-MAIN-2020-29/CC-MAIN-2020-29_segments_1593655899931.31_warc_CC-MAIN-20200709100539-20200709130539-00477.warc.gz\"}"}