Commit Graph

33575 Commits

Author SHA1 Message Date
Przemysław Sobala 4818e074a0 avcodec/imgconvert: Support non-planar colorspaces while padding
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 0d097a869c)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-04-27 04:40:11 +02:00
Michael Niedermayer 82492c3a96 avcodec/mjpegdec: Fix decoding slightly odd progressive jpeg
Fixes: ebd58db6-dc86-11e5-91c2-59daeddf50c7.jpg

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit c6f4720b86)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-04-27 04:40:11 +02:00
James Almer 175110a041 libwebpenc_animencoder: print library messages in verbose log levels
Reviewed-by: James Zern <jzern@google.com>
Signed-off-by: James Almer <jamrial@gmail.com>
(cherry picked from commit f875ba4873)
2016-03-17 12:57:23 -03:00
James Almer 76c157cfd7 libwebpenc_animencoder: zero initialize the WebPAnimEncoderOptions struct
This zeroes the WebPAnimEncoderOptions.verbose field, silencing library info messages
printed to stderr.

Reviewed-by: James Zern <jzern@google.com>
Signed-off-by: James Almer <jamrial@gmail.com>
(cherry picked from commit 626b6b769c)
2016-03-17 12:57:20 -03:00
Michael Niedermayer 800334947d avcodec/avpacket: clear priv in av_init_packet()
This should fix leaving uninitialized pointers in priv which can confuse
user applications.
See: https://github.com/golang/go/issues/14426

Only for release branches

Reviewed-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-02-24 16:56:12 +01:00
Michael Niedermayer b3a64fc039 avcodec/h264: Execute error concealment before marking the frame as done.
Fixes race condition causing artifacts
Fixes Ticket4122

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 98a0053d0f)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-02-19 02:07:46 +01:00
Michael Niedermayer b15ae71305 avcodec/jpeg2000dec: More completely check cdef
Fixes out of array access
Fixes: j2k-poc.bin

Found-by: Lucas Leong <wmliang.tw@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 0aada30510)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-28 15:53:54 +01:00
Paul B Mahol 0dc379cfa6 avcodec/flacenc: fix calculation of bits required in case of custom sample rate
Sample rate of 11025 takes 16 bits but previous code would pick only 8.
Fixes assertion failure.

Reviewed-by: Rostislav Pehlivanov <atomnuker@gmail.com>
Signed-off-by: Paul B Mahol <onemda@gmail.com>
(cherry picked from commit 3e7d684912)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-28 15:53:54 +01:00
Michael Niedermayer 9819998342 avcodec/mjpegdec: Check for end for both bytes in unescaping
Fixes assertion failure
Fixes: c40c779601b77dc6e19aaea0b04b9751/signal_sigabrt_7ffff6ae7cb7_5769_b94f6ec70caecb2d3d76b4771b109ac1.avi

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 509c9e74e5)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-28 15:53:54 +01:00
Michael Niedermayer 85cfcb87ff avcodec/mpegvideo_enc: Check for integer overflow in ff_mpv_reallocate_putbitbuffer()
Fixes assertion failure
Fixes: 6568d187979ce17878b6fe5fbbb89142/signal_sigabrt_7ffff6ae7cb7_7176_564bbc6741bdcf907f5c4e685c9a77a2.mpg

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit b65efbc0f4)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-28 15:53:54 +01:00
Michael Niedermayer 00393c56da avcodec/wmaenc: Check ff_wma_init() for failure
Fixes null pointer dereference
Fixes: c4faf8280ba366bf00a79d425f2910a8/signal_sigsegv_1f96477_5177_1448ba7e4125faceb966f44ceb69abfa.qcp
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 19e456d48c)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-28 15:53:54 +01:00
Michael Niedermayer f77b656b6e avcodec/mpeg12enc: Move high resolution thread check to before initializing threads
Cleaner solution is welcome!

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit a53fbda9dc)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-28 15:53:54 +01:00
Michael Niedermayer c6f6829ce6 avcodec/ass_split: Fix null pointer dereference in ff_ass_style_get()
Fixes: 55d71971da50365d542ed14b65565fe1/signal_sigsegv_4765a4_8499_f146af090a94f591d6254515c7700ef5.mkv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Reviewed-by: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 158f0545d8)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-28 15:53:54 +01:00
Michael Niedermayer 828d85bf86 avcodec/gif: Fix lzw buffer size
Fixes out of array access
Fixes: aaa479088e6fb40b04837b3119f47b04/asan_heap-oob_e38c68_8576_9d653078b2470700e2834636f12ff557.tga

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 03d83ba34b)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-28 15:53:53 +01:00
Michael Niedermayer aa833e1a60 avcodec/put_bits: Assert buf_ptr in flush_put_bits()
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 3ef5de0f19)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-28 15:53:53 +01:00
Michael Niedermayer 61850f1c84 avcodec/tiff: Check subsample & rps values more completely
Fixes out of array access
Fixes: 83aedfb29af669c4d6e10f1bfad974d2/asan_heap-oob_1ab42fe_4984_9f6ec14462f8d8a00ea24b320572a963.tif

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 89f464e9c2)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-28 15:53:53 +01:00
Michael Niedermayer 8008a029ab avcodec/aacenc: Check both channels for finiteness
Fixes null pointer dereference
Fixes: 10412fc52ecc6eab40ed67f82ca7b372/signal_sigsegv_2618c99_2129_f808373959e46afb165593332799ffbc.aif

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 057549a9cc)

Conflicts:

	libavcodec/aacenc.c
2016-01-28 15:53:53 +01:00
Andreas Cadhalpun 1272b88d04 dca: fix misaligned access in avpriv_dca_convert_bitstream
src and dst are only 8-bit-aligned, so accessing them as uint16_t causes
SIGBUS crashes on architectures like sparc.

This fixes ubsan runtime error: load of misaligned address for type
'const uint16_t', which requires 2 byte alignment

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit 44ac13eed4)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
2016-01-27 23:45:44 +01:00
Andreas Cadhalpun 6ffaf40c02 vorbisdec: reject rangebits 0 with non-0 partitions
This causes non-unique elements in floor_setup->data.t1.list, which
makes the stream undecodable according to the specification.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit e7a7b3135a)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
2016-01-27 23:45:44 +01:00
Andreas Cadhalpun e3d7796336 vorbisdec: reject channel mapping with less than two channels
It causes the angle channel number to equal the magnitude channel
number, which makes the stream undecodable according to the
specification.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit b4b13848de)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
2016-01-27 23:45:44 +01:00
Paul B Mahol 7b0fb4fdf7 avformat/ipmovie: put video decoding_map_size into packet and use it in decoder
The size of decoding map can differ from one calculated
internally, producing artifacts while decoding video.

Signed-off-by: Paul B Mahol <onemda@gmail.com>
(cherry picked from commit c293ef258c)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
2016-01-27 23:28:49 +01:00
James Almer b8fa3ff95d avcodec/wavpackenc: print channel count in av_log call
Fixes a warning with -Wformat-extra-args
(cherry picked from commit 17e7fdf61a)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 17:53:58 +01:00
Michael Niedermayer 4676e20a9d avcodec/pngenc: Fix mixed up linesizes
Fixes out of array accesses
Fixes: 0cf176e6d3ab9fe924f39738e513f547/asan_generic_4a54aa_3431_aaa28be1cb32e307a9890cad06f84fba.avi

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 8f4c3e4b92)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 12:30:41 +01:00
Michael Niedermayer 0c5a71fece avcodec/pngenc: Replace memcpy by av_image_copy()
Fixes out of array access
Fixes: 0cf176e6d3ab9fe924f39738e513f547/asan_generic_4a54aa_3431_aaa28be1cb32e307a9890cad06f84fba.avi

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 7ec9c5ce8a)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 12:30:41 +01:00
Michael Niedermayer d408261b65 avcodec/mpeg4video: Check time_incr
Fixes assertion failure
Fixes out of memory access

Fixes: test_casex.ivf

Found-by: Tyson Smith <twsmith@mozilla.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 7c97946d61)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 12:30:40 +01:00
Michael Niedermayer d7f7586c3f avcodec/wavpackenc: Check the number of channels
They are stored in a byte, thus more than 255 is not possible

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 59c915a403)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 12:30:40 +01:00
Michael Niedermayer f05e0e76d6 avcodec/wavpackenc: Headers are per channel
Fixes: 1b8b83a53bfa751f01b1daa65a4758db/signal_sigabrt_7ffff6ae7cb7_7488_403f71d1a2565b598d01b6cb110fac8f.aiff
Fixes: assertion failure

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 26757b0279)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 12:30:40 +01:00
Michael Niedermayer e5567f2f69 avcodec/aacdec_template: Check id_map
Fixes index out of bounds error
Fixes: aac_index_out_of_bounds.wmv

Found-by: Piotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 590863876d)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 12:30:40 +01:00
Michael Niedermayer b8382a1fae avcodec/dvdec: Fix "left shift of negative value -254"
Fixes: dvdec_left_shift.avi

Found-by: Piotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 93ac72a98d)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 12:30:40 +01:00
Michael Niedermayer 9cad208223 avcodec/g2meet: Check for ff_els_decode_bit() failure in epic_decode_run_length()
Fixes invalid shift
Fixes: g2m_left_shift_2.wmv

Found-by: Piotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 030c7f0309)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 12:30:40 +01:00
Michael Niedermayer 34e06ce309 avcodec/mjpegdec: Fix negative shift
Fixes: mjpeg_left_shift.avi

Found-by: Piotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit d86d7b2486)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 12:30:40 +01:00
Michael Niedermayer df4ee166f1 avcodec/mss2: Check for repeat overflow
Fixes: mss2_left_shift.wmv

Found-by: Piotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit e273dade78)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 12:30:40 +01:00
Michael Niedermayer 36861557a7 avcodec/mpegvideo_enc: Clip bits_per_raw_sample within valid range
Fixes out of array read
Fixes: test_case-mdc.264 (b47be15a120979f5a1a945c938cbef33)

Found-by: Tyson Smith <twsmith@mozilla.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 13f266b50c)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 12:30:40 +01:00
Michael Niedermayer 6457cd17be avcodec/motion_est: Fix mv_penalty table size
Fixes out of array read

Found-by: Tyson Smith <twsmith@mozilla.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 5b4da8a38a)

Conflicts:

	libavcodec/motion_est.h
2016-01-15 12:30:40 +01:00
Michael Niedermayer aee9aa9216 avcodec/h264_slice: Fix integer overflow in implicit weight computation
Fixes mozilla bug 1230423

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 7cc01c2572)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 12:30:40 +01:00
Michael Niedermayer 709abc74f5 avcodec/put_bits: Always check buffer end before writing
This causes a overall slowdown of 0.1 % (tested with mpeg4 single thread encoding of matrixbench at QP=3)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit cccb0ffccc)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 12:30:40 +01:00
Andreas Cadhalpun 83c0df90b0 mjpegdec: extend check for incompatible values of s->rgb and s->ls
This can happen if s->ls changes from 0 to 1, but picture allocation is
skipped due to s->interlaced.

In that case ff_jpegls_decode_picture could be called even though the
s->picture_ptr frame has the wrong pixel format and thus a wrong
linesize, which results in a too small zero buffer being allocated.

This fixes an out-of-bounds read in ls_decode_line.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit 7ea2db6eaf)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 12:30:40 +01:00
Andreas Cadhalpun 778c8de40f xwddec: prevent overflow of lsize * avctx->height
This is used to check if the input buffer is large enough, so if this
overflows it can cause a false negative leading to a segmentation fault
in bytestream2_get_bufferu.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit 9d38f06d05)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
2015-12-20 13:39:37 +01:00
Andreas Cadhalpun 945ae04fab exr: fix out of bounds read in get_code
This macro unconditionally used out[-1], which causes an out of bounds
read, if out is the very beginning of the buffer.

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit 90b99a8107)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
2015-12-20 13:39:29 +01:00
Andreas Cadhalpun 6d7b4dbcb4 on2avc: limit number of bits to 30 in get_egolomb
More don't fit into the integer output.

Also use get_bits_long, since get_bits only supports reading up to 25
bits, while get_bits_long supports the full integer range.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit 4d5c3b02e9)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
2015-12-20 13:39:24 +01:00
Andreas Cadhalpun 9be3441c31 rawdec: only exempt BIT0 with need_copy from buffer sanity check
Otherwise the too samll buffer is directly used in the frame, causing
segmentation faults, when trying to use the frame.

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-20 02:58:21 +01:00
Michael Niedermayer bdc6ba4604 avcodec/mpeg4videodec: also for empty partitioned slices
Fixes assertion failure
Fixes: id_acf3e47f864e1ee4c7b86c0653e0ff31e5bde56e.m4v

Found-by: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 70f13abb4f)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-19 23:54:47 +01:00
Michael Niedermayer 3a3be02208 avcodec/h264_refs: Fix long_idx check
Fixes out of array read
Fixes mozilla bug 1233606

Found-by: Tyson Smith
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit b92b4775a0)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-19 22:19:21 +01:00
Michael Niedermayer cabd9ae5be avcodec/h264_mc_template: prefetch list1 only if it is used in the MB
Fixes ubsan warning
Fixes Mozilla bug 1230276

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit c8ea57664f)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-18 00:55:27 +01:00
Michael Niedermayer cfbf608c0a avcodec/h264_slice: Simplify ref2frm indexing
This also suppresses a ubsan warning
Fixes Mozilla bug 1230247

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit ef8f6464a5)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-18 00:55:15 +01:00
Michael Niedermayer 1dddd5371a Revert "avcodec/aarch64/neon.S: Update neon.s for transpose_4x4H"
The change was not correct and broke H264

This reverts commit cd83f899c9.
(cherry picked from commit 95b59bfb9d)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-17 21:50:57 +01:00
Andreas Cadhalpun 748d5fa2ed sonic: make sure num_taps * channels is not larger than frame_size
If that is the case, the loop setting predictor_state in
sonic_decode_frame causes out of bounds reads of int_samples, which has
only frame_size number of elements.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit 9637c2531f)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
2015-12-17 20:04:35 +01:00
Andreas Cadhalpun 796f1a24f5 opus_silk: fix typo causing overflow in silk_stabilize_lsf
Due to this typo max_center can be too large, causing nlsf to be set to
too large values, which in turn can cause nlsf[i - 1] + min_delta[i] to
overflow to a negative value, which is not allowed for nlsf and can
cause an out of bounds read in silk_lsf2lpc.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit f61d44b74a)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
2015-12-17 20:04:30 +01:00
Andreas Cadhalpun a79a5c32f9 golomb: always check for invalid UE golomb codes in get_ue_golomb
Also correct the check to reject log < 7, because UPDATE_CACHE only
guarantees 25 meaningful bits.

This fixes undefined behavior:
runtime error: shift exponent is negative

Testing with START/STOP timers in get_ue_golomb, one for the first
branch (A) and one for the second (B), shows that there is practically no
slowdown, e.g. for the cavs decoder:

With the check in the B branch:
    629 decicycles in get_ue_golomb B, 4194260 runs,     44 skips
    433 decicycles in get_ue_golomb A,268434102 runs,   1354 skips

Without the check:
    624 decicycles in get_ue_golomb B, 4194273 runs,     31 skips
    433 decicycles in get_ue_golomb A,268434203 runs,   1253 skips

Since the B branch is executed far less often than the A branch, this
change is negligible, even more so for the h264 decoder, where the ratio
B/A is a lot smaller.

Fixes: mozilla bug 1230239
Fixes: fbeb8b2c7c996e9b91c6b1af319d7ebc/asan_heap-oob_195450f_2743_e8856ece4579ea486670be2b236099a0.bit

Found-by: Tyson Smith
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit 22e960ad47)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
2015-12-17 20:03:56 +01:00
Andreas Cadhalpun b9087aa651 sbr_qmf_analysis: sanitize input for 32-bit imdct
If the input contains too many too large values, the imdct can overflow.
Even if it didn't, the output would be larger than the valid range of 29
bits.

Note that this is a very delicate limit: Allowing values up to 1<<25
does not prevent input larger than 1<<29 from arriving at
sbr_sum_square, while limiting values to 1<<23 breaks the
fate-aac-fixed-al_sbr_hq_cm_48_5.1 test.

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit fdc94db37e)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
2015-12-17 20:03:48 +01:00