Rotate every 2x2 block in a matrix The Challenge
Given a n x m matrix with n &gt; 1 and m &gt; 1  filled with integers

1 2 3
4 5 6

and a list of integers with exactly as many values as 2x2 blocks in the matrix ((n-1)*(m-1) if you need the exact number)

[1, 2]

Output the matrix with every 2x2 block rotated by the current value in the list in the given order. The example above would yield

4 6 2
5 3 1

The first block gets rotated one time to the right and the second block gets rotated two to the right.
Notes

A positive integer means you rotate right by that many steps.
A negative integer means you rotate left by that many steps.
A zero means that you don't rotate.
You rotate the blocks row-wise. That means that you start in the first row and go to the right. Once you rotated every block in that row you go to the next one. At the end every block was rotated exactly once.
Keep in mind that the blocks overlap each other. The first matrix above has the blocks [[1,2],[4,5]] and [[2,3],[5,6]] for example.
Each rotation of a block affects the rotation on the adjacent blocks. This is why you have to do the rotations in the pattern described above.

Rules

You may take the input in the most convenient format. Please specify in your answer which one you use. This does not allow you to read the matrix block-wise though.
Function or full program allowed.
Default rules for input/output.
Standard loopholes apply.
This is code-golf, so lowest byte-count wins. Tiebreaker is earlier submission.

Test cases
Input format here is a list of lists for the matrix and a normal list for the values.

[[1,2],[3,4]], [-3] -> [[4,1],[3,2]]
[[1,1,1],[1,1,1]], [-333, 666] -> [[1,1,1],[1,1,1]]
[[1,2,3],[4,5,6]], [1,2] -> [[4,6,2],[5,3,1]]
[[1,2,3],[4,5,6],[7,8,9]], [4,0,12,-20] -> [[1,2,3],[4,5,6],[7,8,9]]
[[1,2,3,4,5],[5,4,3,2,1],[1,2,3,4,5]], [2,-3,4,1,6,24,21,-5] -> [[4,1,5,2,4],[2,1,3,5,5],[3,2,4,3,1]]

Happy Coding!
CJam (42 40 bytes)

{(\@2$,(/.{@a@a+z(\@.{4,={+2/zW%~}*}z~}}


Online demo

Basically this applies the same technique twice in order to fold the block

{4,={+2/zW%~}*}


which operates on a 2x2 matrix and a number of times to rotate.

[first row] [[second row] [third row] ... [last row]] [value_0 value_1 ... value_n]


can be processed with

.{block}


and has the effect of

[first row]
[second row] value_0 {block}~
[third row] value_1 {block}~
...


because . (like % in CJam) doesn't gather the results into an array until it's finished.