Some days ago I was in a meeting talking about the E2E implementation for Fractal, to know the current implementation state and to talk with other developers. This meeting was promoted by Puri.sm people, because they want to use Fractal for the Librem5 phone and they want to have E2E. There's people working in the E2E and I think we can have this on Fractal at the end of the year, but this is not what I want to talk about today.
In this meeting there was Fractal developers but we've also other people, like the nheko developer, mujx (nheko is a Qt matrix client). During the meeting, mujx ask us about the cache storage that we're using, because for the E2E is too important to don't lose any key, because that'll be catastrophic, you won't be able to read room messages. So it's important to have a transactional database storage for this information. They are using LMDB, I didn't know nothing about LMDB so after this meeting I start to about it.
Lightning Memory-Mapped Database Manager (LMDB) is a key-value database, it's memory mapped so it's fast, and also uses filesystem storage so we've persistence. This database has transactions so it's safe to read/write from different threads or process.
In Fractal we're using a simple json file for cache, but this doesn't support transactions and if the app crash or if something bad happens, we can lose data. This method is simple, but is slow and insecure, so using LMDB will improve Fractal in several ways.
But LMDB is in memory and in Fractal we've a lot of interface code sharing the app state, so we're passing the state between threads with copies and complex data sharing. This can simplify the interface code because using LMDB for the application global state will make this state accesible from different threads.
Fractal is written in Rust so I want to write some tests before start to use this on Fractal. There's a simple LMDB rust crate, and I've been writting an example lib to test it.
The struct should be serializable/deserializable with serde because in this example I'm using bincode to convert structs to [u8].
The Cacheable trait only have two required methods, the db and the key. The db is the db name to use to store this struct instances and the key is the key to use when storing a concrete instance.
This is thread safe, so we can read from the cache from different threads and we'll always get the last version in the database.
We're thinking about moving the Fractal data model from the AppOp struct to a new crate, independent of the UI, to simplify the UI code and to be able to use the same data model from different UIs (we wan't to split fractal in two different apps), Julian wrote about this.
I think that we can use the LMDB cache to store this new app state that we want to have and this will simplify a lot our code, because we will share the same state and this state will be persisted in filesystem.
I'll start to write a new crate for Fractal to start to move all the app state to this new crate. I think I can write a generic tool to simplify the LMDB use, maybe I'll publish another crate in crates.io and use that in Fractal, but I need to think a little more about the pattern to follow.
I write a lot of web code in my day to day work, and I've been working with react+redux. This LMDB cache thing in Fractal reminds me a lot to the redux store and I want to follow a similar pattern so we can have only an app state and only a way to update this state.
Yesterday I read a blog post about the new Riot.im Stickers. This is not a matrix.org feature, it's implemented as a widget, when you send an sticker you're sending a new event with the type "m.sticker", which is similar to the "m.image" event.
The second day of Fractal hackfest was really productive, we talk about a lot of topics and takes some important decisions.