Check if element is a member of all sets in Scheme
so this is not a question about code, but about conventions. Usually to say "a condition holds for all elements of a set" is the same as "there's no such element in the set that the condition doesn't hold for it". Which there clearly is no such element, in an empty set. So by convention all such "all-are-true" predicates hold for an empty set. 

But if you want your function to return #f in such a case, just add simple entry check for this, and convert what you have right now into an inner (nested) definition:

(define memberall
  (lambda (a l-set) 
    (and (not (null? l-set))
      (letrec ((alltrue 
                 (lambda (l-set)
                   (cond
                     ((null? l-set) #t)
                     ((not (member a (car l-set))) #f)
                     (else (alltrue (cdr l-set)))))))
        (alltrue l-set)))))


Or with named let:

(define memberall
  (lambda (a l-set) 
    (and (not (null? l-set))
      (let alltrue ((l-set l-set))
        (cond
          ((null? l-set) #t)
          ((not (member a (car l-set))) #f)
          (else (alltrue (cdr l-set))))))))