Large Sparse Matrix Data Saved To / Loaded From SQL vs. Numpy File Data
Use Parquet
Have you considered using the parquet file format? It stores tabular data really efficient and is very fast on reads and decently fast on writes. With Apache Drill or Spark you can directly use it with the SQL language. Dask and all major data processing frameworks support it, too.
To update a parquet file you would need to overwrite the whole file.
Don’t use a SQL server
I would not recommend using an SQL server database as they have different goals:

low latency
operation on just a subset of the data

In your case these don’t apply.
Large Sparse Matrix Data Saved To / Loaded From SQL vs. Numpy File Data
I am using a large sparse matrix (~20k x 100k) and vector(s) (~20K x 1) in my python application with great speed and success.
Over time, my large sparse matrix will grow. Growth management is easy and fast while the matrix is in memory.
I am wanting to find the fastest method to save and load my matrix when my app closes and opens, respectively.
I had planned to save and load using numpy save and load functions. SQL Server through pyodbc is lightning fast, but I have not ever thought to use SQL to manage and maintain sparse matrices before.
Is it crazy to think of managing long term storage of sparse matrices in a SQL Server DataBase table? I would store each column of the sparse matrix as a column in the sparse matrix SQL table, and each row would represent a shared aspect amongst the columns. I do believe that updating the SQL table would be faster than storing the whole sparse matrix as a numpy data object to file. I am more worried about loading the matrix into memory when the app starts. Would numpy always be faster at this?
Loading the matrix would be as simple as
SELECT * FROM SPARSE_MATRIX.
Those query results would then need to be formatted correctly and converted to a numpy sparse matrix. I am also open to some other SQL storage approach, but I cannot conceive of one yet.
Motivation for asking this? Management of my sparse matrix for my particular app would be VERY elegant in SQL with regard to saving, growing and maintaining the matrix.
IF I do not get an answer, I will likely do a speed test comparison and answer this question myself for members in the greater community that might have this same question, BUT I am hoping someone has had some experience with this.
This might all be more practical if I run things through pandas or dask between python and SQL. Eager to find an answer to this one way or another.