A lot of ad-hoc queries impact on server
It can impact the amount of memory available. Each query plan takes a little memory away from SQL Server. 

Also, each time a plan has to be created, SQL Server has to compile it. This means it uses CPU which could be used for other processes. If you have a high CPU contention than that could become a problem.

This query can help you see how large the query plans actually are:

SELECT TOP 1000
    st.text ,
    cp.cacheobjtype ,
    cp.objtype ,
    cp.refcounts ,
    cp.usecounts ,
    cp.size_in_bytes ,
    cp.bucketid ,
    cp.plan_handle
FROM    sys.dm_exec_cached_plans cp
    CROSS APPLY sys.dm_exec_sql_text(cp.plan_handle) st
WHERE   cp.cacheobjtype = 'Compiled Plan'
    AND cp.objtype = 'Prepared'
ORDER BY cp.usecounts DESC;