Shorten a sorting function
The code for the two cases (sorting by name and sorting by price) have nearly nothing in common.  You may be better off writing two functions.  Even if you really like this interface, it might make sense to have the master function to call a sort-by-name function or a sort-by-price function to do the work.

I have no idea what aux.cat = malloc(sizeof *(aux.cat)) is for.  You never really store anything in the .cat member.  Furthermore, after all the swapping done in the sorting process, the aux.cat at the end of the function is not the same aux.cat that you started out with.  That's a memory leak.

Your sort algorithm is selection sort, which takes O(n2) time.  There are more efficient algorithms that work in O(n log n) time.

Your sort() function always returns 0.  You might as well make it a void function and not return any value.

For efficiency and brevity, you would be better off using sorting routines built into the C library.  All you have to do to use those routines is to define a comparator function, which tells the sorting function how to order any pair of books.

#include &lt;stdlib.h&gt;
#include &lt;string.h&gt;

static int compare_books_by_name_asc(const void *a, const void *b) {
    const book *book_a = (const book *)a,
               *book_b = (const book *)b;
    return strcmp(book_a-&gt;fname, book_b-&gt;fname);
}

static int compare_books_by_name_desc(const void *a, const void *b) {
    return compare_books_by_name_asc(b, a);
}

static int compare_books_by_price_asc(const void *a, const void *b) {
    int price_a = ((const book *)a)-&gt;cat-&gt;price,
        price_b = ((const book *)b)-&gt;cat-&gt;price;
    return price_a &lt; price_b ? -1 :
           price_a &gt; price_b ? +1 : 0;
}

static int compare_books_by_price_desc(const void *a, const void *b) {
    return compare_books_by_price_asc(b, a);
}

void sort(book *books, int num_books, char mode, char sortBy) {
    int (*comparator)(const void *, const void *);
    if (sortBy == 'n') {   /* Sort by name */
        comparator = (mode == 'a') ? compare_books_by_name_asc
                                   : compare_books_by_name_desc;
    } else {               /* Sort by price */
        comparator = (mode == 'a') ? compare_books_by_price_asc
                                   : compare_books_by_price_desc;
    }
    qsort(books, num_books, sizeof(*books), comparator);
}