How to configure InnoDB/MariaDB to not fill ibdata1 with (rollback/undo?) data that is useless in an in-memory database?
Doing smaller transactions and keeping connections not open (Open Transactions) for long time will probably do the job.
And:
The undo log is usually part of the physical system tablespace, but from MariaDB 10.0, the innodb_undo_directory and innodb_undo_tablespaces system variables can be used to split into different tablespaces and store in a different location (perhaps on a different storage device).
https://mariadb.com/kb/en/innodb-undo-log/
For MEMORY: What about VARCHAR(65532)?