Twilight Imperium without the Agenda phase
Honestly, I think it would be pretty straight-forward to remove The Agenda Phase from the game. Not too many components actually rely on it. With that being said, none of the below suggestions have been play-tested and there may be things that I have overlooked in terms of gameplay. I've played this game quite a few times but am by no means an expert.
Here's a list of things I found, some of which are also noted in your question:
Factions
The Nekro Virus

Faction Ability: Galactic Threat

The problem with the Nekro Virus is that they cannot research technologies. When you would research technology you instead gain 3 command tokens. This can be super advantageous in the early game. You could instead allow the Nekro Virus to research technology as normal whilst keeping their combat-centric technology thievery intact. This faction has good starting units but not getting early command tokens may be slow their early game expansion. Their late game would depend on how they balance neighbor relations and stealing technology. Removing Galactic Threat and allowing Nekro to research technology makes The Nekro Virus marginally weaker but not terribly weaker.
The Xxcha Kingdom

Faction Ability: Quash
Faction Promissory: Political Favor

I feel like the main fun of Xxcha is being able to play around with the Agenda Phase and being able to sell off your faction promissory. Without these abilities I'm not sure Xxcha would ever be a better choice over any of the other factions. I would recommend removing them from selection if there is no Agenda Phase.
Action Cards
These action cards could be removed entirely from the action deck. My only concern is now the action deck is less saturated with &quot;later phase&quot; cards. What I mean is that you'll now have a better chance of getting things like Sabotage and Direct Hit cards that will directly effect the main Strategy Phase. Common action cards may need to be slimmed down.
15/80 (18.75%) of all Action Cards directly relate to the Agenda Phase. These cards include:

Ancient Burial Sites
Assassinate Representative
Bribery
Confusing Legal Text
Distinguished Councilor
Repel Law
Veto

All Riders:

Construction Rider
Diplomacy Rider
Leadership Rider
Imperial Rider
Politics Rider
Trade Rider
Technology Rider
Warfare Rider

There are no Objectives (public or secret) which directly rely on The Agenda Phase.

While I understand not enjoying the Agenda Phase it can provide a breather from a normally strategic and tense game. It also allows you to build relations with players who are not your neighbor. In my opinion, it's all part of the overall strategy of the game as half is playing the players and the other is playing the pieces.