When is relation in 3NF but not in BCNF?
"I think we can correctly interpret this condition as β−α are key attributes."


No. β−α is a set of attributes. What that statement is saying is,

Every attribute in that set is part of some (at least one) candidate key for the relation.

Also, if by 'key' attribute, you mean an attribute that is a superkey, i.e., an attribute whose value uniquely determines the tuple, then NO, every element in β−α is NOT a key attribute. It is just a prime attribute (in other words, part of some candidate key).

For example, if we write your schema as:


  Dept_advisor : {A, B, C},


then the FD's can be written as:


B → C (α = {B}, β = {C}, β−α = {C} - {B} = {C})
AC → B (α = {A, C}, β = {B}, β−α = {B} - {A, C} = {B})


Here, [1] is neither a trivial dependency nor is α (here, B) a superkey of the relation. In other words, none of the two conditions for BCNF hold for this FD.
For BCNF, you need all the FD's to satisfy at least one of the two conditions, but here you have one FD which satisfies none. So this schema is not in BCNF. You got this right. 

Now, testing it for 3NF, you can confirm that both [1] and [2] satisfy at least one of the 3 conditions for 3NF. [2] satisfies the 2nd condition (α = {A,C} is a superkey), so we are good so far. [1] does not satisfy any of the first two conditions, which are also the conditions for BCNF. But, it does satisfy the third condition, i.e., each attribute in β−α (here, only one attribute, namely B) is part of some candidate key. 

What candidate key is that, you ask? You can easily find one.

Take {A, B}. It is a candidate key because you have the FD B → C, which means {A, B} → {A, B, C} gives all the attributes of the relation, which means it is a superkey. But no proper subset of {A, B}, i.e., neither A nor B alone, are superkeys themselves. In other words, neither {A} → {A, B, C}, nor {B} → {A, B, C}. So yes, you got this one right too, but for the wrong reasons.

As for the other statement, about 3NF tables being dependent only on the "primary key", you are better off just ignoring it and sticking to the definition and trying to understand and use it correctly.