Proper rounding when converting between digitized values and floating point I have some very large files that are unsigned 8 bit (0 to 255), real valued.  I need to do some simple tuning and decimation.  In the past, when I've dealt with low bit rate samples, I've run into issues with artifacts.  I want to make sure I get the rounding correct.

The data will be processed in Matlab using all native double precision.  The first question is should I subtract 127.5 or 128 when converting, or does it depend?

To keep things simple, I'm not doing anything fancy with the filter gain or multiple passes to make sure I don't clip.  Instead I'll notify the user if there was a problem.  The output also needs to be the same format, so I can't use any additional effective bits gained by decimation.  The next thing I'm unsure about is do I round 255.3 down to 255 or do I consider this an overflow condition?

EDIT

I thought I could just process the data with the offset in there doing something like this at the output.

y = y * scale + (1-scale)/2*255


The trouble is the tuning operation may filter out the offset before I correct for it.  So instead, I decided the simplest thing to do is subtract 127.5 and add it back in at the end.  Then round as I described above.

I'm not sure how the different methods of adjusting for the offset and scaling effect the resultant artifacts, but as arnfinn said there is probably nothing I can do about in any case (unless I'm willing to use dither).

EDIT 2

This is actually more along the lines of what I was thinking would be an issue.  In the distant past I have experienced this exact problem with requantization.  Certain combinations of quantization parameters can really mess up quality (worse artifacts).  My situation is a little different because I'm processing the data before requantization.  But I believe it can still be an issue.

A requantization algorithm for the transcoding of JPEG images 

Signal Processing: Image Communication.
Volume 21, Issue 1, January 2006, Pages 13–21.
Jae Won Moona, Jong Seok Leeb, Nam Ik Cho.
I think you don't have to worry too much about so-called 'transparency' (ability to represent accurately the integer value as a floating point value) unless you do normalization. See the libsndfile FAQ.

As quantization is a lossy operation (many-to-one, or surjective, mapping) you will always have artifacts when re-quantizing from floating point to integer values. The standard method for de-correlating the quantization error is to use dither. For small word lengths, such as 8 bit, the classic solution of adding noise with a triangular distribution might cause the dither noise to be unaccptably high, so you might have to look around for some semi-optimal (in terms of quantization error moments) dithering method that suits your application better. Or just re-quantize and accept the artifacts, they might be small, depending on your data and what you are planning to do with it.