## digitalmars.D.learn - Bitfields

• Russel Winder (17/17) May 21 2019 Hi,
• Boris-Barboris (5/14) May 21 2019 Never used it myself, but BitArray with careful handling of
• Russel Winder (19/25) May 22 2019 On Tue, 2019-05-21 at 18:22 +0000, Boris-Barboris via Digitalmars-d-lear...
• Era Scarecrow (14/16) May 21 2019 I worked on/with bitfields in the past, the limit sizes is more
• Russel Winder (26/42) May 22 2019 [=E2=80=A6]
• Basile B. (9/47) May 22 2019 If the type of operations you need on your 128 bit container is
```Hi,

Has anyone used D to work with arbitrary length bitfields with multiple
occurences of a sub-bitfield. I am working with DVB Sections and EIT packet=
s
are defined as bitfields with loops in them and the header is 112 bits. The
loops are handleable with subfields obviously, assuming you can work out ho=
w
the bigendian works on the byte sequence.

As far as I can see std.bitmanip only caters for 8, 16, 32, and 64 bit long
bitfields.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
```
May 21 2019
```On Tuesday, 21 May 2019 at 17:16:05 UTC, Russel Winder wrote:
Hi,

Has anyone used D to work with arbitrary length bitfields with
multiple occurences of a sub-bitfield. I am working with DVB
Sections and EIT packets are defined as bitfields with loops in
them and the header is 112 bits. The loops are handleable with
subfields obviously, assuming you can work out how the
bigendian works on the byte sequence.

As far as I can see std.bitmanip only caters for 8, 16, 32, and
64 bit long bitfields.

Never used it myself, but BitArray with careful handling of

https://dlang.org/phobos/std_bitmanip.html#.BitArray.this.2
https://dlang.org/phobos/std_bitmanip.html#.peek
```
May 21 2019
```On Tue, 2019-05-21 at 18:22 +0000, Boris-Barboris via Digitalmars-d-learn
wrote:
[=E2=80=A6]
=20
Never used it myself, but BitArray with careful handling of=20
=20
https://dlang.org/phobos/std_bitmanip.html#.BitArray.this.2
https://dlang.org/phobos/std_bitmanip.html#.peek

I'll have to mull that one over. The incoming data is a sequence of bytes t=
hat
is treated as a bitfield. Although the standard says "big-endian" it is isn=
't
entirely clear how this relates to the bitfields, I guess I need to read th=
e
standard more. :-(

I guess the question is whether BitArray can work with bytes rather than
size_t as elements of the backing array.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
```
May 22 2019
```On Tuesday, 21 May 2019 at 17:16:05 UTC, Russel Winder wrote:
As far as I can see std.bitmanip only caters for 8, 16, 32, and
64 bit long bitfields.

I worked on/with bitfields in the past, the limit sizes is more
or less for natural int types that D supports.

However this limitation is kinda arbitrary, as for simplicity it
relies on shifting bits, going larger or any byte size is
possible depending on what needs to be stored, but ti's the speed
that really takes a penalty when you aren't using native types or
you have to do a lot of shifting to get the job done.

What's the layout of what you need? I'll see if i can't make
something that would work for you.

Would be better if you can use a object that breaks the parts
down and you can actually fully access those parts, then just
re-store it into the limited space you want for storage, which
then would be faster than bitfields (although not by much)
```
May 21 2019
```On Tue, 2019-05-21 at 19:14 +0000, Era Scarecrow via Digitalmars-d-learn
wrote:
=20

[=E2=80=A6]
I worked on/with bitfields in the past, the limit sizes is more=20
or less for natural int types that D supports.

Rust bitfield crate and it's macros are the same, the underlying type for a
bitfield must be a primitive integer type. Fortunately, Rust has i128 and u=
128
which is enough for my 112 bit EIT header.

Boris Barboris suggested using BitArray and I willinvestigate but the
size_t/byte problem would need to go away.

However this limitation is kinda arbitrary, as for simplicity it=20
relies on shifting bits, going larger or any byte size is=20
possible depending on what needs to be stored, but ti's the speed=20
that really takes a penalty when you aren't using native types or=20
you have to do a lot of shifting to get the job done.
=20
What's the layout of what you need? I'll see if i can't make=20
something that would work for you.
=20
Would be better if you can use a object that breaks the parts=20
down and you can actually fully access those parts, then just=20
re-store it into the limited space you want for storage, which=20
then would be faster than bitfields (although not by much)

I found an interesting way forward in the source code of dvbsnoop. It
basically uses the byte sequence as a backing store and then has a function=
to
do the necessary accesses to treat it as a bit array.

If D's Bit Array can work with bytes instead of size_t then it is exactly w=
hat

The Rust solution using bitfield with a u128 backing it seems to work, but =
it
is all very clumsy even if it is efficacious.=20

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
```
May 22 2019
```On Wednesday, 22 May 2019 at 08:54:45 UTC, Russel Winder wrote:
On Tue, 2019-05-21 at 19:14 +0000, Era Scarecrow via
Digitalmars-d-learn wrote:

[…]
I worked on/with bitfields in the past, the limit sizes is
more
or less for natural int types that D supports.

Rust bitfield crate and it's macros are the same, the
underlying type for a bitfield must be a primitive integer
type. Fortunately, Rust has i128 and u128 which is enough for

Boris Barboris suggested using BitArray and I willinvestigate
but the size_t/byte problem would need to go away.

However this limitation is kinda arbitrary, as for
simplicity it
relies on shifting bits, going larger or any byte size is
possible depending on what needs to be stored, but ti's the
speed
that really takes a penalty when you aren't using native types
or
you have to do a lot of shifting to get the job done.

What's the layout of what you need? I'll see if i can't make
something that would work for you.

Would be better if you can use a object that breaks the parts
down and you can actually fully access those parts, then just
re-store it into the limited space you want for storage, which
then would be faster than bitfields (although not by much)

I found an interesting way forward in the source code of
dvbsnoop. It basically uses the byte sequence as a backing
store and then has a function to do the necessary accesses to
treat it as a bit array.

If D's Bit Array can work with bytes instead of size_t then it
is exactly what dvbsnoop does (in C) but adds writing as well

The Rust solution using bitfield with a u128 backing it seems
to work, but it is all very clumsy even if it is efficacious.

If the type of operations you need on your 128 bit container is
simple enough (bitwise op) you can implement them, backed by two
ulong in a custom struct.
I did something similar for the tokens if my language. There
are probably wrong stuff in there but for my usage (include and
bt) that works seamlessly.

:
https://github.com/Basile-z/styx/blob/master/src/styx/data_structures.d#L40
```
May 22 2019
{"ft_lang_label":"__label__en","ft_lang_prob":0.91208875,"math_prob":0.6266333,"size":7967,"snap":"2021-43-2021-49","text_gpt3_token_len":2446,"char_repetition_ratio":0.13776214,"word_repetition_ratio":0.67611027,"special_character_ratio":0.2962219,"punctuation_ratio":0.10520297,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9782715,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2021-10-16T23:39:19Z\",\"WARC-Record-ID\":\"<urn:uuid:699561c5-de24-4fd7-89e9-5902697ad65e>\",\"Content-Length\":\"21526\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:d64b8b87-64a0-4b1a-84f3-effb76b191c4>\",\"WARC-Concurrent-To\":\"<urn:uuid:afb1809c-5c34-40fb-8e47-c955b68c8320>\",\"WARC-IP-Address\":\"162.217.114.56\",\"WARC-Target-URI\":\"https://www.digitalmars.com/d/archives/digitalmars/D/learn/Bitfields_109807.html\",\"WARC-Payload-Digest\":\"sha1:B5LYQVAJCIUFFRW32GTZVCLU22AHOVNI\",\"WARC-Block-Digest\":\"sha1:ZSZF4B44AN4XHWF6LC2MKD5N25UOUECD\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2021/CC-MAIN-2021-43/CC-MAIN-2021-43_segments_1634323585045.2_warc_CC-MAIN-20211016231019-20211017021019-00198.warc.gz\"}"}