1. A system for offloading processing from a host to one or more storage processing units each comprising a non-volatile memory (NVM), the system comprising: a host having a processing task; a plurality of storage processing units (SPUs); a host interface configured to enable communications between the host and each of the plurality of SPUs; and an interconnection network coupled to at least three of the plurality of SPUs; wherein the host is configured to: command at least one of the plurality of SPUs to perform the processing task; and command the interconnection network to couple three or more of the plurality of SPUs, wherein the interconnection network enables any one of the coupled three or more of the plurality of SPUs to communicate with any other one of the coupled three or more of the plurality of SPUs, and wherein each of the plurality of SPUs comprises an NVM and a processing circuitry configured to perform the processing task.