How to use RNN With Attention Mechanism on Non Textual Data?
The article referenced doesn't highlight the most common terms, which is why you may not be finding what you want. We've found that not all the designs have public implementations in every language. Some we've had to translate from Python or Java to C++ to get the speed we needed for embedded applications. Attention mechanisms are not one thing either. There are attention mechanisms in the cells, before the layers, after the layers, as oversight mechanisms, and in push-pull balancing arrangements. The common cell using attention ideas use gates in the cell and the article only briefly mentions it. That's the long-short term memory cell, which is a type of RNN. Look for LSTM and B-LSTM for the first examples, which are gated. The GRU is also a type of RNN cell with gating. This paper is recent and proposes two mechanisms: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/16126/16099. If you search for DiSAN there are a few implementations. The newest kind, which has promise is not really an RNN strictly. That's the recursive network, which can have attention mechanisms but can build learned graph structures instead of tensors. That may also be what you want.