Is it allowed to log actual instrument under IFR in VMC when VFR would be prohibited?
14 CFR 61.51(g) says (my emphasis):


  (g) Logging instrument time. (1) A person may log instrument time
  only for that flight time when the person operates the aircraft
  solely by reference to instruments under actual or simulated
  instrument flight conditions.
  
  (2) An authorized instructor may log instrument time when conducting
  instrument flight instruction in actual instrument flight
  conditions.


Based on that, the answers to your questions are:


Are you technically even in VMC if you are that close to a cloud? Technically flight is not strictly by reference to instruments, but VFR would be illegal. You're in IMC (i.e. you're operating below the visual minimums in 91.155) but as you said, that doesn't mean you're flying "solely by reference to instruments"
Can an instructor log instrument time? (a PPL would not be allowed to fly through tiny holes in a cloud) If the conditions are IMC, yes (61.51 doesn't require an instructor to operate the aircraft using only instruments)
Does an instrument student need foggles or can it be logged as "actual"? Whether you're wearing foggles or not, you can log instrument time only when you're operating solely on instruments. Practically speaking, using foggles guarantees that you're doing that and can therefore log the time. Without them, some or all of the time might be loggable, i.e. only the part that you really spend flying only on instruments.
Are you still required to see and avoid (obviously still a good idea) Yes, absolutely. According to 91.113(b):


  When weather conditions permit, regardless of whether an operation is
  conducted under instrument flight rules or visual flight rules,
  vigilance shall be maintained by each person operating an aircraft so
  as to see and avoid other aircraft.



It's very common to fly cross country under IFR without logging any instrument time, or logging very little, simply because you're never solely on instruments.