Markov Model for a Traffic Intersection
Why not use a single agent to control the intersection with the following rules:


define 8 traffic lights
Each light has two possible values, 0 or 1, with 0 = Red and 1 = Green
Lights are as follows:


0 = North to South
1 = North to East (Left Turn)
2 = East to West
3 = East to South (Left Turn)
4 = South to North
5 = South to West (Left Turn)
6 = West to East
7 = West to North (Left Turn)

map lights into a bitmap of 8 bits, with the least significant bit (left bit) is the North to south 


Given the above, we have the following valid moves, which are both State Space  and Action Space :


State/Move 0 - Vertical Moves only   = 10001000
State/Move 1 - Horizontal Moves only = 00100010
State/Move 2 - Originated from North only = 11000000
State/Move 3 - Originated from East only = 00110000
State/Move 4 - Originated from South only = 00001100
State/Move 5 - Originated from West only = 00000011
State/Move 6 - Vertical Diagonal only = 01000100
State/Move 7 - Horizontal Diagonal only = 00010001


i.e. S = {10001000, 10001000, 00100010, 11000000, 00110000, 00001100, 00000011, 01000100, 00010001}

and  A = {10001000, 10001000, 00100010, 11000000, 00110000, 00001100, 00000011, 01000100, 00010001}

Now we have a &in; A and s &in; S and at any given s, a will cause the next state s' to have the same value of a

Final component to complete the moel, is the reward signal, which can be simply the negative sum of time spent per car in the intersection 

i.e.     rt+1 = -&Sum;ci &in; Cars time(ci)

a single agent, will be able to optimize the solution to avoid punishment (negative value caused by wait time) 

Update

You can increase the state space to double or triple its size to accommodate for time, by appending a time counter component to the bitmap. 
in addition to an extra (9th) action of do nothing, 

This should give a more subtle traffic controller. e.g. Vertical Traffic Only, at timer = 0 can be represented by 100010000000 or in hex 880, then at time 1 = 881, and so on until time 15 to be 88F. 

Update 2

You can find the source code here and here