fix bad saturation order in QuantizeBlock

Saturation was done on input coeff, not quantized one.

This saturation is not absolutely needed: output of FTransformWHT
is in range [-16320, 16321]. At quality 100, max quantization steps is 8,
so the maximal range used by QuantizeBlock() is [-2040, 2040].
But there's some extra bias (mtx->bias_[] and mtx->sharpen_[]) so
it's better to leave this saturation check for now.

addresses issue #145

Change-Id: I4b14f71cdc80c46f9eaadb2a4e8e03d396879d28
This commit is contained in:
Pascal Massimino
2013-03-24 08:47:41 -07:00
parent 5f25c396ab
commit 3c8eb9a806
5 changed files with 14 additions and 11 deletions

View File

@ -18,7 +18,8 @@
extern "C" {
#endif
extern const uint16_t VP8LevelFixedCosts[2048]; // approximate cost per level
// approximate cost per level:
extern const uint16_t VP8LevelFixedCosts[MAX_LEVEL + 1];
extern const uint16_t VP8EntropyCost[256]; // 8bit fixed-point log(p)
// Cost of coding one event with probability 'proba'.