How to order by Levenshtein distance?
You didn't tell us what the "failed attempt" means. But something like this should work:

with ldist as (
   select name, 
          levenshtein(substring(name,1,200), lag(name) OVER (order by name))  as distance
   FROM books 
   WHERE name &lt;&gt; ''
) 
select * 
from ldist
order by distance;


you don't want partition by name because that essentially puts every name into a single group and thus there is no "previous" row to. 

As you you want the "previous" row based on the ordering of the name column, you need an order by name in the window definition.