Drupal 7 CKEditor-CKFinder remove "Browse Server"
Well I have found that patch solution. So just answering my own question. 
In sites/all/modules/ckeditor/includes/ckeditor.lib.inc file, 

There is a section as 



if (user_access('allow CKFinder file uploads')) {
                $ckfinder_full_path = $host . ckfinder_path();
                //$settings['filebrowserBrowseUrl'] = $ckfinder_full_path . '/ckfinder.html?id=' . $profile_name;
                //$settings['filebrowserImageBrowseUrl'] = $ckfinder_full_path . '/ckfinder.html?Type=Images&id=' . $profile_name;
                //$settings['filebrowserFlashBrowseUrl'] = $ckfinder_full_path . '/ckfinder.html?Type=Flash&id=' . $profile_name;
                $settings['filebrowserBrowseUrl'] = '';
                $settings['filebrowserImageBrowseUrl'] = '';
                $settings['filebrowserFlashBrowseUrl'] = '';
                $settings['filebrowserUploadUrl'] = $ckfinder_full_path . '/core/connector/php/connector.php?command=QuickUpload&type=Files&id=' . $profile_name;
                $settings['filebrowserImageUploadUrl'] = $ckfinder_full_path . '/core/connector/php/connector.php?command=QuickUpload&type=Images&id=' . $profile_name;
                $settings['filebrowserFlashUploadUrl'] = $ckfinder_full_path . '/core/connector/php/connector.php?command=QuickUpload&type=Flash&id=' . $profile_name;
            }



So I commented the actual filebrowserBrowseUrl, filebrowserImageBrowseUrl &amp; filebrowserFlashBrowseUrl settings and made those as blank. That did the trick.