Subsetting and Ordering in R
a$group=="a" gives 

# [1]  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE


and a$marks[order(a$marks,decreasing=T)] gives 

# [1] 8 7 6 5 4 3 2


Combining those to using a logical and (&amp;) does not make much sense here. TRUE &amp; 8 is TRUE - the subsetting may accidentally work. But you want order to look at your subset only, not at all rows in a.  

Try for instance

newa &lt;- a[a$group=="a", ] 
newa[order(newa$marks, decreasing = TRUE), ]
#   group marks  upd
# 7     a     8 down
# 5     a     6 down
# 3     a     4 down
# 1     a     2   up


or, more convenient and readable:

library(dplyr)
a %&gt;% filter(group=="a") %&gt;% arrange(desc(marks))