Conversion of a varchar data type to a datetime data type resulted in an out-of-range value in SQL query
I think you'll only get the out-of-range error if you try to convert, say a "June 31st" or a 13th month. If the text is just totally un-datelike (is that a word?) it'll give a different error.

With that inb mind, TRY_CONVERT() is a variant of CONVERT() which won't error if the format isn't quite right. You can use it to look for filenames which'll fail the convert.

Try running:

SELECT JobTasks.FileName
FROM   JobTasks 
WHERE  TRY_CONVERT(DATETIME, SUBSTRING(JobTasks.FileName, 2, 8), 20) is null