What type of behaviour is showing, but withholding, a reward? A specific example of what I mean is: when you go shopping at a supermarket with a loyalty card system, and you do not participate, the receipt say something along the lines of 


  If you had a loyalty card you would have earned 20 points.


My first impression was that this is some sort of punishment, i.e. because of your behaviour (not having a loyalty card) we're throwing these points that you earned away.

This is based on definition 1, for punishment, from the Penguin dictionary of psychology (2nd ed.):


  The administration of some aversive stimulus contingent upon a particular behaviour.


Then I wondered if it was just negative reinforcement, because the message would go away if I signed up for the card.

This is based on definition 2, for negative reinforcement, from the same book:


  Any event, stimulus or behaviour which, when its removal is made contingent upon a response, will increase the frequency or likelihood of that response.


Do either of these fit the bill? Compare that message with telling someone "You could had have this cake, if only you had been a few minutes earlier. Now I will throw the cake in the bin."

Is it possible to categorise this type of behaviour in behaviourism?
The problem is that this doesn't fall under any of these conditioning definitions in behaviourism because the store isn't really trying to condition a response.  They're just trying to get you to do one thing once.  The behavioural techniques you mention are designed so that you get an organism to make a conditioned response spontaneously whenever the conditioned stimulus occurs (or is anticipated to occur or a second order associate occurs.. ok, it's more complicated than that).  

It's also not really considered conditioning when it's done through insight.

That doesn't mean there's no science behind this.  There's lots and it falls more under behavioural economics than psychology, unfortunately.  (Consumer psychology hasn't really moved to be as complementary to behavioural economics as it probably should be.)

That said, these kinds of "nags" would be closest to negative reinforcement.  They're encouraging a behaviour that removes an aversive stimulus... being reminded of a gain you did not receive.  Punishment is typically more to extinguish a behaviour.  They certainly don't want you to stop shopping.  Sometimes it's hard to see what it is until you figure out the goal behaviour.

(As an aside, what they want more than anything is your personal information.  That's what they're trying to get.)