Left join on a varchar column
OK, I'll stop trying to figure out why your query is not working and give you a technique for solving it...

SELECT  u.*, p.*
    FROM  
       ( SELECT  user_id   -- minimal data (just PK in this case)
            FROM  users
            ORDER BY  user_id
            LIMIT  100    -- now in subquery
        ) AS x
    JOIN  users AS u  ON u.user_id = x.user_id
    JOIN  phonedetails AS p  ON p.phone = u.phone
    ORDER BY  x.user_id   -- yes, need extra ORDER BY


The general principle:  When doing a LIMIT or GROUP BY and the query is even somewhat complex, it may be better to build a subquery that identifies which rows as simply as possible.  Then do the JOIN, etc, to get the rest of the data.


The subquery (called a "derived" table in this syntax) finds the user_ids -- and only the 100 needed.  Since you are doing LEFT, there is no need to involve phonedetails in this query.
Then reach out to the other tables to get the desired columns.


Caveat:  This version of the query may be slow for the same yet-to-be-discovered reason.  This reformulation is effective in some situations.