Can I discard more than 1 card at a time if there are no draw cards left?
Looking at rules here I think your interpretation is correct.  Although I suspect from the way you say "its a brilliant move, and I don't see anything specifically preventing this" makes me think whoever you did this against didn't agree!

anyway the rules quotes....

On a players turn one of there options is


  Pass and discard 1-3 face-down cards from your hand..... then draw 1-3 cards.


The note underneath clarifies


  When the pile is used up, no cards are drawn.


So if there were a situation where a player is to draw a card and can't they simply don't get a card.  So yes you can discard cards knowing you can't draw back and deliberately shrinking your hand size.

The rules for swapping state :-


  The player who plays this card chooses one other player and exchanges hands with them. Discard the action card, then the other player gets to draw 1 card!


So you swap your remaining hand of zero cards with the other player.  They are unable to draw a card as non left.

So yes the move seems entirely legal based on that.