How to determine a quantization table for image compression with DCT?
The standard JPEG quantization table is a actually an 8 x 8 matrix of weights whose elements divide the corresponding elements of the 2D-DCT coefficient matrix of an 8 x 8 image block.

The division effectively reduces the magnitude of the DCT coefficients and most of them are forced to be zero, due to the integer divison rounding. 

The standard table is determined to yield a minimum distortion under maximum compression applied to the most typical image (i.e., exhaustive average of large image data set) and typical viewer visual characteristics. 

As a result of this quantization, high frequency coefficients are reduced as much as possible.

Note that the JPEG quality parameter provides a means of scaling the standard quantization table weights, but no control over individual elements. 

You can also design and use a specific quantization table for your needs, according to the rate-distortion criteria you have in your mind. But this requires you to embed Q table information within the compressed data bitstream.