How multiple unfinished payments in single LN channel work? I think I understand HTLCs, at least if they are on chain. I also have a basic understanding of how off chain channel looks like. 

I understand it as follows. It is a 2-of-2 multisig transaction output, which is on chain and confirmed sufficiently. Before this transaction was put on chain, the parties created another transaction, which they haven't published yet. This second transaction is necessary in case either party would not be cooperative to provide its signature for the initial multisig.

This second transaction pays each party the agreed final balance as per current status of the channel. This transaction is ready to go on chain, it is fully signed, the parties just decided not to publish it yet because they may want to replace this second transaction with different finalising transaction.

Now the question is – When a payment is to be done through this channel using HTLC, what are the inputs to it? Is the output of the original multisig? If so, how would you do 2 or more HTLCs at the same time (i.e. you have enough liquidity on the channel to do multiple payments, but it takes time to finish current payment when someone wants to pay through you again)? 

What would be the inputs for those other HTLCs? Exactly when is the finalising transaction spending the initial multisig updated in the database of the participants?

And would you need to close the channel and put a transaction on chain in case a single HTLC payment fails?
The htlcs are just an additional output in the commitment tx which spends the funding transaction (as you described) 

So assume Alice and Bob have a channel of capacity 0.1 BTC in which alice owns 0.03 BTC. 

On chain you have a multisig wallet that has 0.1 BTC and offchain you have a commitment tx with two outputs. 

0.03 to alice
0.07 to Bob


Let's assume Bob wants to pay Alice 0.015 BTC or wants to forward an htlc through Alice of the same amount a new commitment tx is negotiated (by which the old one is invalidated) that has 3 outputs

0.03 to alice
0.055 to bob
0.015 htlc clsimable by Alice if she can show a preimage within a Timelock and afterwards by bob


Now if routing is successful alive provides the preimage and a new state is negotiated in which the htlc output is removed but added to slice balance resulting in

0.045 to alice
0.055 to bob


Otherwise the htlc is removed but the amount is added to bobs output. 

I omitted onchain fees for simplicity