public files config in settings.php creates .htaccess file
The main culprit was extra trailing / in the files path in settings.php file. Which I changed to 

$conf['file_public_path'] = 'sites/default/files';
$conf['file_temporary_path'] = '/tmp';


And deleted the .htaccess file, now the files inside sites/default/files are readbale by server even after clearing the cache.