AllocateBuffer: fix int multiplication overflow check

after the check using 64-bit math we used a signed integer in the
multiplication. previously unsigned integer max was tested.

fixes cases like:
src/dec/buffer_dec.c:108:16: runtime error: signed integer overflow:
944731466 * 4 cannot be represented in type 'int'
    #0 0x55e56187dc1d in AllocateBuffer src/dec/buffer_dec.c:108:16
    #1 0x55e56187dc1d in WebPAllocateDecBuffer src/dec/buffer_dec.c:216:12
    ...

Bug: chromium:1196850
Change-Id: I6e5b3e5d1d5b50b5c98c39bbf9813a63fedc5ca7
This commit is contained in:
James Zern 2021-06-07 18:59:24 -07:00
parent 315abbd60b
commit 776983d427

View File

@ -102,7 +102,7 @@ static VP8StatusCode AllocateBuffer(WebPDecBuffer* const buffer) {
int stride;
uint64_t size;
if ((uint64_t)w * kModeBpp[mode] >= (1ull << 32)) {
if ((uint64_t)w * kModeBpp[mode] >= (1ull << 31)) {
return VP8_STATUS_INVALID_PARAM;
}
stride = w * kModeBpp[mode];