| **⚠Warning⚠ this is an experimental weight. It may not have practical performance.**<br> | |
| **Also, the model file must be manually rewritten or replaced to use this weight.**<br> | |
| The model file is available here.<br> | |
| https://github.com/lucidrains/BS-RoFormer | |
| The BS-Roformer has been updated in terms of architecture for the first time in a while.<br> | |
| In the 0.5.x update, a mechanism called "Value Residual Learning" was introduced. (https://arxiv.org/abs/2410.17897)<br> | |
| The paper argues that this mechanism can reduce the over-focus of attention and further reduce the vanishing gradient problem. |