making jam with brown (raw) caster sugar
Yes, that's perfectly fine. I've done so myself (using raw cane sugar) because I like the taste, at least for some kinds of fruit. 

My health store has different brands, some lighter, some darker. If your store does, too, you might want to consider using the lighter variety - the darker ones sometimes can impart a slight bitterness (just an undertone, really), which can be "too much" for some fruit.