Hz.tools – Paultag's SDR Code
Announcing hz.tools
Paul Tagliamonte
2023-02-22
projectEver since 2019, I’ve been learning about how radios work, and trying to learn
about using them “the hard way” – by writing as much of the stack as is
practical (for some value of practical) myself. I wrote my first “Hello World”
in 2018, which was a simple FM radio player, which used librtlsdr to read in
an IQ stream, did some filtering, and played the real valued audio stream via
pulseaudio. Over 4 years this has slowly grown through persistence, lots of
questions to too many friends to thank (although I will try), and the eternal
patience of my wife hearing about radios nonstop – for years – into a number
of Go repos that can do quite a bit, and support a handful of radios.I’ve resisted making the repos public not out of embarrassment or a desire to
keep secrets, but rather, an attempt to keep myself free of any maintenance
obligations to users – so that I could freely break my own API, add and remove
API surface as I saw fit. The worst case was to have this project feel like
work, and I can’t imagine that will happen if I feel frustrated by PRs
that are “getting ahead of me” – solving problems I didn’t yet know about, or
bugs I didn’t understand the fix for.As my rate of changes to the most central dependencies has slowed, i’ve begun
to entertain the idea of publishing them. After a bit of back and forth, I’ve
decided
it’s time to make a number of them public,
and to start working on them in the open, as I’ve built up a bit of knowledge
in the space, and I and feel confident that the repo doesn’t contain overt
lies. That’s not to say it doesn’t contain lies, but those lies are likely
hidden and lurking in the dark. Beware.That being said, it shouldn’t be a surprise to say I’ve not published
everything yet – for the same reasons as above. I plan to open repos as the
rate of changes slows and I understand the problems the library solves well
enough – or if the project “dead ends” and I’ve stopped learning.It’s my sincere hope that my repos help to make Software Defined Radio (SDR)
code a bit easier to understand, and serves as an understandable framework to
learn with. It’s a large codebase, but one that is possible to sit down and
understand because, well, it was written by a single person. Frankly, I’m also
not productive enough in my free time in the middle of the night and on
weekends and holidays to create a codebase that’s too large to understand, I
hope!I remain wary of this project turning into work, so my goal is to be very
upfront about my boundaries, and the limits of what classes of contributions
i’m interested in seeing.Here’s some goals of open sourcing these repos:I do want this library to be used to learn with. Please go through it
all and use it to learn about radios and how software can control them!I am interested in bugs if there’s a problem you discover. Such bugs
are likely a great chance for me to fix something I’ve misunderstood or
typoed.I am interested in PRs fixing bugs you find. I may need a bit of a
back and forth to fully understand the problem if I do not understand
the bug and fix yet. I hope you may have some grace if it’s taking a long
time.Here’s a list of some anti-goals of open sourcing these repos.I’m not saying all this to be a jerk, I do it to
make sure I can continue on my journey to learn about how radios work without
my full time job becoming maintaining a radio framework single-handedly for
other people to use – even if it means I need to close PRs or bugs without
merging it or fixing the issue.With all that out of the way, I’m very happy to announce that the repos are now
public under github.com/hztools.Should you use this?Probably not. The intent here is not to provide a general purpose Go SDR
framework for everyone to build on, although I am keenly aware it looks and
feels like it, since that what it is to me. This is a learning project, so for
any use beyond joining me in learning should use something like
GNU Radio
or a similar framework that has a community behind it.In fact, I suspect most contributors ought to be contributing to GNU Radio, and
not this project. If I can encourage people to do so,
contribute to GNU Radio!
Nothing makes me happier than seeing GNU Radio continue to be the go-to, and
well supported. Consider
donating to GNU Radio!The hz.tools/rf library contains the abstract concept of frequency, and
some very basic helpers to interact with frequency ranges (such as helpers
to deal with frequency ranges, or frequency range math) as well as frequencies
and some very basic conversions (to meters, etc) and parsers (to parse values
like 10MHz). This ensures that all the hz.tools libraries have a shared
understanding of Frequencies, a standard way of representing ranges of
Frequencies, and the ability to handle the IO boundary with things like CLI
arguments, JSON or YAML.The git repo can be found at
github.com/hztools/go-rf, and is
importable as hz.tools/rf.    // Parse a frequency using hz.tools/rf.ParseHz, and print it to stdout.
	freq := rf.MustParseHz("-10kHz")

	fmt.Printf("Frequency: %s\n", freq+rf.MHz)
    // Prints: 'Frequency: 990kHz'

    // Return the Intersection between two RF ranges, and print
    // it to stdout.
	r1 := rf.Range{rf.KHz, rf.MHz}
	r2 := rf.Range{rf.Hz(10), rf.KHz * 100}

	fmt.Printf("Range: %s\n", r1.Intersection(r2))
    // Prints: Range: 1000Hz->100kHz
These can be used to represent tons of things - ranges can be used for things
like the tunable range of an SDR, the bandpass of a filter or the frequencies
that correspond to a bin of an FFT, while frequencies can be used for things
such as frequency offsets or the tuned center frequency.This… is the big one. This library represents the majority of the shared
types and bindings, and is likely the most useful place to look at when
learning about the IO boundary between a program and an SDR.The git repo can be found at
github.com/hztools/go-sdr, and is
importable as hz.tools/sdr.This library is designed to look (and in some cases, mirror) the Go io idioms
so that this library feels as idiomatic as it can, so that Go builtins interact
with IQ in a way that’s possible to reason about, and to avoid reinventing the
wheel by designing new API surface. While some of the API looks (and is even
called) the same thing as a similar function in io, the implementation is
usually a lot more naive, and may have unexpected sharp edges such as
concurrency issues or performance problems.The following IQ types are implemented using the sdr.Samples interface. The
hz.tools/sdr package contains helpers for conversion between types, and some
basic manipulation of IQ streams.IQ Formathz.tools NameUnderlying Go TypeInterleaved uint8 (rtl-sdr)sdr.SamplesU8[][2]uint8Interleaved int8 (hackrf, uhd)sdr.SamplesI8[][2]int8Interleaved int16 (pluto, uhd)sdr.SamplesI16[][2]int16Interleaved float32 (airspy, uhd)sdr.SamplesC64[]complex64The following SDRs have implemented drivers in-tree.SDRFormatRX/TXStatertlu8RXGoodHackRFi8RX/TXGoodPlutoSDRi16RX/TXGoodrtl kerberosu8RXOlduhdi16/c64/i8RX/TXGoodairspyhfc64RXExpThe following major packages and subpackages exist at the time of writing:ImportWhat is it?hz.tools/sdrCore IQ types, supporting types and implementations that interact with the byte boundaryhz.tools/sdr/rtlsdr.Receiver implementation using librtlsdr.hz.tools/sdr/rtl/kerberosHelpers to enable coherent RX using the Kerberos SDR.hz.tools/sdr/rtl/e4kHelpers to interact with the E4000 RTL-SDR dongle.hz.tools/sdr/fftInterfaces for performing an FFT, which are implemented by other packages.hz.tools/sdr/rtltcpsdr.Receiver implementation for rtl_tcp servers.hz.tools/sdr/plutosdr.Transceiver implementation for the PlutoSDR using libiio.hz.tools/sdr/uhdsdr.Transceiver implementation for UHD radios, specifically the B210 and B200minihz.tools/sdr/hackrfsdr.Transceiver implementation for the HackRF using libhackrf.hz.tools/sdr/mockMock SDR for testing purposes.hz.tools/sdr/airspyhfsdr.Receiver implementation for the AirspyHF+ Discovery with libairspyhf.hz.tools/sdr/internal/simdSIMD helpers for IQ operations, written in Go ASM. This isn’t the best to learn from, and it contains pure go implemtnations alongside.hz.tools/sdr/streamCommon Reader/Writer helpers that operate on IQ streams.The hz.tools/fftw package contains bindings to libfftw3 to implement
the hz.tools/sdr/fft.Planner type to transform between the time and
frequency domain.The git repo can be found at
github.com/hztools/go-fftw, and is
importable as hz.tools/fftw.This is the default throughout most of my codebase, although that default is
only expressed at the “leaf” package – libraries should not be hardcoding the
use of this library in favor of taking an fft.Planner, unless it’s used as
part of testing. There are a bunch of ways to do an FFT out there, things like
clFFT or a pure-go FFT implementation could be plugged in depending on what’s
being solved for.The hz.tools/fm and hz.tools/am packages contain demodulators for
AM analog radio, and FM analog radio. This code is a bit old, so it has
a lot of room for cleanup, but it’ll do a very basic demodulation of IQ
to audio.The git repos can be found at
github.com/hztools/go-fm and
github.com/hztools/go-am,
and are importable as
hz.tools/fm and
hz.tools/am.As a bonus, the hz.tools/fm package also contains a modulator, which has been
tested “on the air” and with some of my handheld radios. This code is a bit
old, since the hz.tools/fm code is effectively the first IQ processing code
I’d ever written, but it still runs and I run it from time to time.    // Basic sketch for playing FM radio using a reader stream from
    // an SDR or other IQ stream.

    bandwidth := 150*rf.KHz
    reader, err = stream.ConvertReader(reader, sdr.SampleFormatC64)
    if err != nil {
        ...
    }
    demod, err := fm.Demodulate(reader, fm.DemodulatorConfig{
        Deviation:       bandwidth / 2,
        Downsample:      8, // some value here depending on sample rate
        Planner:         fftw.Plan,
    })
    if err != nil {
        ...
    }
    speaker, err := pulseaudio.NewWriter(pulseaudio.Config{
        Format:     pulseaudio.SampleFormatFloat32NE,
        Rate:       demod.SampleRate(),
        AppName:    "rf",
        StreamName: "fm",
        Channels:   1,
        SinkName:   "",
    })
    if err != nil {
        ...
    }

    buf := make([]float32, 1024*64)
    for {
        i, err := demod.Read(buf)
        if err != nil {
            ...
        }
        if i == 0 {
            panic("...")
        }
        if err := speaker.Write(buf[:i]); err != nil {
            ...
        }
    }
The hz.tools/rfcap package is the reference implementation of the
rfcap “spec”, and is how I store IQ captures
locally, and how I send them across a byte boundary.The git repo can be found at
github.com/hztools/go-rfcap, and is
importable as hz.tools/rfcap.If you’re interested in storing IQ in a way others can use, the better approach
is to use SigMF – rfcap exists for cases
like using UNIX pipes to move IQ around, through APIs, or when I send
IQ data through an OS socket, to ensure the sample format (and other metadata)
is communicated with it.rfcap has a number of limitations, for instance, it can not express a change
in frequency or sample rate during the capture, since the header is fixed at
the beginning of the file.