what are the exact formulas for calculating the matrice elements Lo, Co, Ro, Go, Rs, Gd
The link provided in the comments points to HSPICE's user guide. At page 154 (194/1024) you'll see the description for the W element. The values are entered by the user, which means that these values are either calculated by the user, or provided based on some measurements, or datasheet. How you calculate these values is up to the user, since there is no formula to rule them all, they are all application-, and transmission-line-specific.
In other words, the user chooses to model a certain transmission line, and based on that, the values are chosen. So, what transmission line do you want to model? Does it have a datasheet? If yes, what does the datasheet specify?