Image transform with either width or height
A similar question has already been asked here: Is there a way to find out if images (assets) are landscape or portrait?

You will need to create the two asset transforms you mentioned in your question, and then in your template code use an if/else statement to determine which transform to use, depending on if the asset is portrait or landscape.

Thanks to Josh for the code, this is taken from his answer:

{% set myImage = entry.imagesField.first %}

{% if myImage.getWidth() &lt; myImage.getHeight() %}
  {# portrait #}
{% elseif myImage.getWidth() &gt; myImage.getHeight() %}
  {# landscape #}
{% else %}
  {# square #}
{% endif %}