Why do airlines not use direct routes?
You assume that airlines always fly the published SID or STAR. That's not true though. SIDs and STARS are used in procedural control to separate arriving and departing flights. When you have radar, then you don't necessarily need the procedures to do that.

When a flight is cleared for takeoff they are assigned a SID that is relevant to its destination. After takeoff and when the flight is handed off to approach (1) controller, the controller will give a vector (heading) to the flight or direct to a point on the TMA boundary. If for any reason (see below) they intend to keep the flight on the SID the controller says proceed as cleared(2) on first contact.

For approach, the flight is assigned a STAR and when it reaches the TMA boundary, the approach controller will give them a vector and the flight will deviate from the assigned STAR. See this video from NATS  and watch the arrivals to Gatwick. Compare the trajectories to what you see in the instrument procedure maps from their website. They don't seem to follow a route looking like that



source: UK AIP chart AD 2-EGKK-7-5

They seem to go direct to the holding stacks. And after exiting holding they don't seem to follow any of these routes



source: UK AIP chart AD 2-EGKK-7-12 

Now when will a flight actually do the SID or STAR you will ask. There are several cases like for example TMAs without radar service. Normally the control is procedural in that case and that includes flying the assigned SID or STAR. Another case is during unusual circumstances like for example equipment failure (radar, communications etc) that will make radar vectoring harder or even impossible.

It's worth noting that it's common for pilots to ask for shortcuts if that can be done legally. For example if the weather conditions are VMC an arriving flight might request a visual approach. In that case IFR is canceled and the pilot will deviate from the STAR or IAP.

During takeoff, a pilot might request direct to an en-route point and thus exit the SID. If the current controller cannot grant it (let's say no radar contact), they will ask the next controller (en-route controller) if they can grant it. If yes (the en-route radar has the flight) then the flight is handed over to en-route and they are granted the direct, exiting the SID. That might not always be the case. If for example en-route control has radar contact, but because of heavy military activity the direct cannot be given safely, then the flight stays on the SID until the end.



Notes

Links to pdfs unfortunately don't work; you need to open them manually from NATS AIS page, linked in main answer.

(1) Or departure, sometimes it's the same controller.

(2) This phraseology is used in Greece (and I hope I remember it correctly). Other jurisdictions might use something slightly different.