Commit Graph

15171 Commits

Author SHA1 Message Date
Ronald S. Bultje ddb1149e25 tgq: convert to bytestream2 API.
This protects against input buffer overreads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 1255eed533)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 21:09:19 +01:00
Ronald S. Bultje f6778f58d4 algmm: convert to bytestream2 API.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit a55d5bdc6e)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 21:09:19 +01:00
Paul B Mahol e4e4d92641 jvdec: unbreak video decoding
The safe bitstream reader broke it since the buffer size was specified
in bytes instead of bits.

Signed-off-by: Janne Grunau <janne-libav@jannau.net>
CC: libav-stable@libav.org
(cherry picked from commit a1c036e961)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 21:02:23 +01:00
Michael Niedermayer de0ff4ce69 h264: Fix invalid interlaced/progressive MB combinations for direct mode prediction.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit 758ec11153)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 21:00:52 +01:00
Anton Khirnov 6548cb2578 libx264: add 'stats' private option for setting 2pass stats filename.
x264 always opens the file itself with fopen, so we cannot use the
standard lavc stats mechanism.

CC: libav-stable@libav.org
(cherry picked from commit d533e395e1)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 21:00:12 +01:00
Anton Khirnov f6257cf4b7 libx264: fix help text for slice-max-size option.
CC: libav-stable@libav.org
(cherry picked from commit 9d5c131ece)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 21:00:02 +01:00
Janne Grunau d94256d36c Revert "h264: clear trailing bits in partially parsed NAL units"
This reverts commit 729ebb2f18.

There was an off-by-one error in the bit mask calculation clearing
actually the last valid bit and causing
http://bugzilla.libav.org/show_bug.cgi?id=227

The broken sample (Mr_MrsSmith-h264_aac.mp4) the commit was fixing
does not work after correcting the off-by-one error.

CC: libav-stable@libav.org
(cherry picked from commit 8a6037c390)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 20:56:55 +01:00
Ronald S. Bultje 7bb97a61df mpc: pad mpc_CC/SCF[] tables to allow for negative indices.
MPC8 allows indices of mpc_CC up to -1, and mpc_SCF up to -6, thus pad
the tables by that much on the left end.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit d7eabd5042)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 20:48:29 +01:00
Ronald S. Bultje c65eadee5d xxan: protect against chroma LUT overreads.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit f77bfa8376)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 20:47:19 +01:00
Ronald S. Bultje a43f4bd601 xxan: convert to bytestream2 API.
Protects against overreads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 5518827816)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 20:47:19 +01:00
Ronald S. Bultje 8f881885c2 xxan: don't read before start of buffer in av_memcpy_backptr().
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit f1279e286b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 20:47:19 +01:00
Ronald S. Bultje 26521d87ba dsicinvideo: validate buffer offset before copying pixels.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit c95fefa042)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 20:47:19 +01:00
Ronald S. Bultje e1a4143793 cook: error out on quant_index values outside [-63, 63] range.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 97e48b2f54)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 20:47:19 +01:00
Ronald S. Bultje b9482a6efd cook: extend channel uncoupling tables so the full bit range is covered.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 37cc8600d0)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 20:47:19 +01:00
Ronald S. Bultje 88c3cc019c cook: expand dither_tab[], and make sure indexes into it don't overflow.
Fixes overflows in accessing dither_tab[].

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 442c3a8cb1)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-13 23:40:29 +01:00
Ronald S. Bultje 9980e4df3b huffyuv: add padding to classic (v1) huffman tables.
We slightly overread the input buffer, so we require
padding at the end of the buffer, as is documented in the
get_bits API. Without padding, we'll read uninitialized
data or beyond the end of the .rodata, which may crash.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 4ffe5e2aa5)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-13 23:36:39 +01:00
Ronald S. Bultje d4f2786cda avs: fix infinite loop on end-of-stream.
The codec would keep returning the last decoded frame if the stream
contains B-frames, since it wouldn't clear that frame from the list of
frames to be returned to the user.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 83f15a1228)

Conflicts:

	libavcodec/cavsdec.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-13 23:36:02 +01:00
Alex Converse 2744fdbd9e tiffdec: Prevent illegal memory access caused by recycled pointers.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit fd0be63049)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-13 23:30:55 +01:00
Ronald S. Bultje 1fcc2c6091 wma: fix off-by-one in array bounds check.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit b4bccf3e4e)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-13 23:30:39 +01:00
Ronald S. Bultje 74871ac70a dv: check buffer size before reading profile.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit e97efecec8)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-13 23:30:21 +01:00
Ronald S. Bultje 9cb7f6e54a raw: move buffer size check up.
This way, it protects against overreads for 4bpp/2bpp content also.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit cc5dd632ce)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-13 23:23:58 +01:00
Ronald S. Bultje ed6aaf579d dca: prevent accessing static arrays with invalid indexes.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit e6ffd997cb)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-13 23:22:32 +01:00
Ronald S. Bultje e1b4614ab4 lpcm: fix sample size calculation for 20bit LCPM.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit f1320dc3be)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-13 23:12:00 +01:00
Ronald S. Bultje 12247a13e0 Don't use ff_cropTbl[] for IDCT.
Results of IDCT can by far outreach the range of ff_cropTbl[], leading
to overreads and potentially crashes.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit c23acbaed4)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-08 22:07:55 +01:00
Ronald S. Bultje 9def2f200e error_resilience: initialize s->block_index[].
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 6193ff6854)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-08 22:07:55 +01:00
Ronald S. Bultje 7b676935ee svq3: protect against negative quantizers.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 11b940a1a8)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-08 22:07:55 +01:00
Justin Ruggles 4a15240a27 mov: set channel layout for AC-3 streams based on the 'dac3' atom info
fixes Bug 225
(cherry picked from commit 3798205a77)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-08 22:07:54 +01:00
Janne Grunau a47b96bdd3 rv34: handle size changes during frame multithreading
Factors all context dynamic memory handling to its own functions.
Fixes bug 220.
(cherry picked from commit 2bd730010d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-08 22:07:54 +01:00
Alex Converse 48ac765efe rv10/20: Fix slice overflow with checked bitstream reader.
(cherry picked from commit 9243ec4a50)
2012-03-06 15:31:23 -08:00
Michael Niedermayer 522645e38f h263dec: Disallow width/height changing with frame threads.
Fixes CVE-2011-3937

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 71db86d53b)

Conflicts:

	libavcodec/h263dec.c

Signed-off-by: Alex Converse <alex.converse@gmail.com>
2012-03-06 15:28:01 -08:00
Alex Converse e891ee4bf6 adpcm: Clip step_index values read from the bitstream at the beginning of each frame.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit bbeb29133b)
2012-03-06 15:28:01 -08:00
Alex Converse ef673211e7 tiff: Make the TIFF_LONG and TIFF_SHORT types unsigned.
TIFF v6.0 (unimplemented) adds signed equivalents.
(cherry picked from commit e32548d133)
2012-03-06 15:28:01 -08:00
Alex Converse eaeaeb265f dpcm: ignore extra unpaired bytes in stereo streams.
Fixes: CVE-2011-3951

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit ce7aee9b73)
2012-03-06 15:28:01 -08:00
Alex Converse db315c796d svq3: Prevent illegal reads while parsing extradata.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit 9e1db721c4)
2012-03-06 15:28:01 -08:00
Michael Niedermayer e3743869e9 ac3dec: Move center and surround mix level tables to the parser.
That way all mix levels as exported by avpriv_ac3_parse_header()
will have the same meaning.

Previously the 3-bit center mix level for E-AC-3 was used to index in a
4-entry table, leading to out-of-array reads.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit e6d9fa66f1)
2012-03-06 15:28:01 -08:00
Mans Rullgard 627f4621f5 ac3: Do not read past the end of ff_ac3_band_start_tab.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit 034b03e7a0)
2012-03-06 15:28:01 -08:00
Dale Curtis feed0c6b6a mpegaudiodec: Prevent premature clipping of mp3 input buffer.
Instead of clipping extrasize based on EXTRABYTES, clip based on the
amount of buffer actually left. Without this fix, there are warbles
and other distortions in the test case below.

http://kevincennis.com/mix/assets/sounds/1901_voxfx.mp3
(cherry picked from commit b716542691)

Signed-off-by: Alex Converse <alex.converse@gmail.com>
2012-03-06 15:28:00 -08:00
Alex Converse d0e53ecff7 mp3dec: Fix a heap-buffer-overflow
In some cases, what is left to read from ptr is smaller than EXTRABYTES.

Based on a patch by Thierry Foucu <tfoucu@gmail.com>.

Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit f372ce119b)
2012-03-06 15:28:00 -08:00
Alex Converse 1ca84aa162 mpeg12: Pad framerate tab to 16 entries.
There are many places where we read an unchecked 4-bit index into it.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit dfa37fe8a3)
2012-03-06 15:28:00 -08:00
Michael Niedermayer d5f2382d03 kgv1dec: Increase offsets array size so it is large enough.
Fixes CVE-2011-3945

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 807a045ab7)

Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit a02e8df973)
2012-03-06 15:28:00 -08:00
Alex Converse 416849f2e0 kmvc: Check palsize.
Fixes: CVE-2011-3952

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Based on fix by Michael Niedermayer
(cherry picked from commit 386741f887)
2012-03-06 15:28:00 -08:00
Martin Storsjö ca7e97bdcf g722: Fix the QMF scaling
This fixes clipping if the encoder input used the full 16 bit
input range (samples with a magnitude below 16383 worked fine).
The filtered subband samples should be 15 bit maximum, while
the code earlier produced them scaled to 16 bit.

This makes the decoder output have double the magnitude
compared to before.

The spec reference samples doesn't test the QMF at all, which
was why this part slipped past initially.

(cherry picked from commit b087ce2bee)

Signed-off-by: Martin Storsjö <martin@martin.st>
2012-03-06 15:45:30 +02:00
Justin Ruggles 4ae138cb12 ac3dsp: do not use pshufb in ac3_extract_exponents_ssse3()
We need to do unsigned saturation in order to cover the corner case when the
absolute coefficient value is 16777215 (the maximum value).

Fixes Bug #216
(cherry picked from commit d483bb58c3)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-06 13:55:35 +01:00
Fabian Greffrath 003f7e3dd0 Fix format string vulnerability detected by -Wformat-security.
Signed-off-by: Diego Biurrun <diego@biurrun.de>
(cherry picked from commit c9dbac36ad)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-05 18:01:37 +01:00
Ronald S. Bultje 85eb76a23f h264: fix mmxext chroma deblock to use correct TC values.
(cherry picked from commit b0c4f04338)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-05 18:00:43 +01:00
Ronald S. Bultje 5186984ee9 h264: change underread for 10bit QPEL to overread.
This prevents us from reading before the start of the buffer, and thus
prevents crashes resulting from this behaviour. Fixes bug 237.
(cherry picked from commit 291c9b6285)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-05 18:00:31 +01:00
Ronald S. Bultje b5331b979b cscd: use negative error values to indicate decode_init() failures.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 8a9faf33f2)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-05 14:48:35 +01:00
Vitor Sessak 11f3173e1b amrnbdec: check frame size before decoding.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit 882abda5a2)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-05 14:48:35 +01:00
Ronald S. Bultje cd17195d1c h264: prevent overreads in intra PCM decoding.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit d1604b3de9)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-05 14:48:35 +01:00
Justin Ruggles 1128b10247 wmaenc: fix m/s stereo encoding for the first frame
We need to set ms_stereo in encode_init() in order to avoid incorrectly
encoding the first frame as non-m/s while flagging it as m/s. Fixes an
uncomfortable pop in the left channel at the start of playback.

CC:libav-stable@libav.org
(cherry picked from commit 51ddf35c90)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-04 21:26:29 +01:00