How can I improve performance of my selecting code
So pretty much everything will make a insignificant effect apart from the section where you are looping through each of the checkboxes. See my modifications and notes below.

$("a.action:not(.remove)").click(function (e) { // .remove =&gt; do not execute on download tasks page
     var selected = new Array(); // array to hold selections or clicks
     var obj = $(this),
         objData = obj.data(),

         // Moved these up here as they are not specific to the checkboxes
         packageId = objData.ajaxPackageid.removeSpecialChars().toUpperCase(),
         operation = objData.ajaxType;

     e.preventDefault();
     if (!obj.hasClass('disablelink')) {
         if (objData.ajaxMachineid) {
             selected.push(objData.ajaxMachineid);
         } else {

             // This seems a little convoluted to me, normally you would 
             // use [type=checkbox]
             $("input[type=checkbox]:checked:not(.checkall)").each(function () {

                 var checkbox = $(this),
                     machineId = checkbox.val();

                 // No need to call .removeSpecialChars().toUpperCase() on packageId
                 var $row = $("#" + machineId + packageId);

                 if ($row.length) {

                     // Not 100% whether you need .length here
                     // It could also be beneficial to add a class to the row instead
                     // of calling this complex selector
                     $row.has("a[data-ajax-type=" + operation + "]:not(.hide)").length 
                         ? selected.push(machineId) 
                         : $(this).attr('checked', false);

                 }
             });
         }
         if (selected.length &gt; 0) {
             // do something more here blah blah blah
         };
     }
     $("input.checkall").attr("checked", false);
 });


The obvious thing to do on top of my suggestions above is to cache the checkbox list, this means that every time it enters that section, the app will no longer need to go through every element in the DOM looking for all the checkboxes. Doing this will probably have the most meaningful impact on performance.

// in document ready
var checkboxes = $('input[type=checkbox]');

// When you need to find the checked check boxes
var checked = checkboxes.find(':checked:not(.checkall)');