Full text query slow on first run
When I run a query for the first time, it takes 3-10 seconds to show me the results. On running same query again, it is in less than 1 second (very fast)


The second time you run the query, it is likely that data and index pages for the relational tables are coming from SQL Server's cache rather than being read in from disk. Likewise, the full-text portion of the processing will also benefit from caching.

As the comments on the question indicate, 4GB does not seem like very much memory for a SQL Server managing a 65GB database. More physical memory will allow SQL Server to cache a greater proportion of your relational tables and full-text indexed data.

The exact amount you need depends on the nature of your workload, and the response times your users expect. I would imagine that expanding the memory even modestly (say to 8 or 16GB) would result in a very noticeable performance improvement, and more predictable response times overall.

You would also want to configure the maximum memory for SQL Server to ensure there is enough for Windows as well. See the following links for more information on memory (and other important details):

How much memory does my SQL Server actually need?

Troubleshooting SQL Server: A Guide for the Accidental DBA (free eBook)