SQL Managment doesn't use all memory, queries fail due insufficiënt memory
I believe memory usage in Express edition is limited to 1Gb per instance: http://sqlknowledge.com/2014/04/sql-2014-express-edition-limitations/

So setting Max Memory any higher than that will be ignored by an Express instance. With that in mind I'm a bit surprised that you are seeing memory usage hitting 3.5Gb

You mention the ORDER BY in your question - does the query run ok without it?