How does Implicit Quantile-Regression Network (IQN) differ from QR-DQN?
Would like to thank Will Dabney and Georg Ostrovski (two of the paper's several authors),  and Massimiliano Tomassoli (Simple-Machine-Learning blog owner) for helping me understand IQN.

To understand this algorithm, you will need to know:


TD Learning
Value Functions of Reinforcement Learning
What a DQN is.
What C51 is
What a Cummulative Distribution Function is (CDF)
What an Inverse CDF is.


Just like C51 or QR-DQN, IQN allows us to estimate an entire distribution of possible rewards if we take some action $a$.  However, IQN doesn't output a distribution like those did.  Instead, it outputs a single sample every time you ask it. It outputs that red dot you see on the right, of the bottom image.

The final output of IQN is 1 scalar value per action. In other words, just a single sample per action.  We re-run IQN several times to get more samples for the "CurrentState".

IQN takes input at 2 different stages: 1)  and 2)


First, the IQN takes a current state S (which is a vector), transforms it into another vector V (for example, of dimension 10).
now, we take any random scalar value called $\tau$ (drawn from a uniform [0,1] range), and feed that scalar value into function $\phi(\tau)$.  It gives us a vector H,  which has the same dimension as V
then, vectors V and H get combined via concatenation. In fact, we perform multiplication instead of concatenation if there won't be many forward layers further on.
At the end of this forward-pass, our IQN outputs an |A| dimensional vector,  containing samples of the action-distributions we are trying to estimate. |A| is the other way to say the "number of actions". Notice: because IQN outputs a vector, and not a matrix (like QR-DQN does), we only get 1 scalar sample per action when performing a forward pass from the current State.
To get more samples, we have to return back to step 2)  and re-run the forward pass with some another scalar $\tau$  once again sampled from [0,1] range. Or, of course, we can re-use our IQN, to run the needed number of such forward-passes in parallel, if hardware allows. No need to start all the way from step 1) because we only care about the end-layers where $\tau$ actually enters the system. "Any preceding layers" would be re-computed to the same values anyway.


When training, we evaluate "CurrentState+1" as well. We bootstrap our results from "CurrentState" towards "CurrentState+1" via usual TD-learning, just like a usual DQN would do.  Don't forget, we will have to re-run forward pass several times (as step 5. has mentioned) both for "CurrentState" and for "CurrentState+1" to get several samples of the current and target distributions. This gives us a fairly good idea about what those two distributions look like. 


According to the paper, in most cases re-running the forward pass 8 times (thus getting 8 samples) suffices for approximating both the $Z_{\tau}$ and our target distribution $Z$ (the distribution of the best action for "CurrState+1").  So we perform 16 forward passes in total:  8 for  "CurrentState"  and 8 for "CurrentState+1" to get an idea about those two distributions.




Why does feeding a random $\tau$ into the network work?

You have to understand that a well-trained IQN network represents an Inverse CDF itself.  That is, given a request of some scalar 'amount' value (taken from a uniform range of 0 to 1), the Inverse CDF will output a value from the actual distribution, that sits at that amount.

Although distributions won't necessarily look like Bell-curves, have a look at this example:





Why do authors use cosine inside $\phi(\tau)$ ?  I still have no idea, but the Appendix section of the paper states it seems to work the best. Perhaps it works good with ReLU.





I also got confused by $w_{ij}$ or $b_j$. To me it seemed there is a matrix of weights and a vector of bias values. ...Do we input an entire vector of different $\tau$ values at once?

The answer is no. Notice, authors used $j$ only to remind that forward pass has to run several times. 
In total, there is only i number of weights in $\phi(\tau)$ and 1 bias.

Therefore, we get some scalar $\tau$, and use it with all those weights.  Basically $\phi(\tau)$ resembles a simple fully-connected layer, which only accept 1 scalar input value at a time.  We will re-use this same layer for different $\tau$ scalar values.