Order implicit groups Given this set...

Site, Date
4   10/10/18
3   10/10/18
4   07/10/18
5   12/10/18
3   11/10/18
5   13/11/18


I want to order it such that the rows are ordered by site and the site groups are ordered by MIN date.

i.e.

Site, Date
4   07/10/18
4   10/10/18
3   10/10/18
3   11/10/18
5   12/10/18
5   13/11/18


I have tried using...

SELECT

    RANK() OVER (PARTITION BY Site ORDER BY (Date)) AS GroupRank,
    a.*
FROM

    (select * from
    (select 4 AS Site,   '10/10/18' as date union
    select 3,   '10/10/18' union
    select 4,   '07/10/18'  union
    select 5,   '12/10/18'  union
    select 3,   '11/10/18'  union
    select 5,   '13/11/18'  as date ) b) a
    order by GroupRank


The results don't make sense.

1   3   10/10/18
1   4   07/10/18
1   5   12/10/18
2   5   13/11/18
2   4   10/10/18
2   3   11/10/18


It's supposed to be partitioning by site but it's given the same rank to rows with different sites.
Something like this:

SELECT

    min(date) OVER (PARTITION BY Site) AS MinDate,
    a.*
FROM

    (select * from
    (select 4 AS Site,   '10/10/18' as date union
    select 3,   '10/10/18' union
    select 4,   '07/10/18'  union
    select 5,   '12/10/18'  union
    select 3,   '11/10/18'  union
    select 5,   '13/11/18'  as date ) b) a
    order by MinDate, Site