Commit 3225daac authored by alastor0325's avatar alastor0325 Committed by Pier Angelo Vendrame
Browse files

Bug 2021788 - Guard against signed long overflow in... 

Bug 2021788 - Guard against signed long overflow in WMFVideoMFTManager::CreateBasicVideoFrame(). r=media-playback-reviewers,jolin

Guard CreateBasicVideoFrame() against invalid strides by rejecting
non-positive values early, widening the y_size and v_size multiplications
to int64_t, validating the results fit in uint32_t before narrowing, and
casting stride to int64_t before computing halfStride to avoid overflow
when stride == INT32_MAX.

IMF2DBuffer::Lock2D can return a negative stride for bottom-up images.
Multiplying a negative or large LONG stride by the frame height using
signed 32-bit arithmetic is undefined behaviour and produces a huge
uint32_t offset that is then used to index into the locked IMF buffer,
causing an OOB read.

Differential Revision: https://phabricator.services.mozilla.com/D288533
parent 627d8f56

dom/media/platforms/wmf/WMFVideoMFTManager.cpp

+16 −3
Original line number Diff line number Diff line
@@ -21,6 +21,7 @@ 
#include "gfx2DGlue.h"

#include "gfxWindowsPlatform.h"

#include "mozilla/AbstractThread.h"

#include "mozilla/CheckedInt.h"

#include "mozilla/ClearOnShutdown.h"

#include "mozilla/Logging.h"

#include "mozilla/SchedulerGroup.h"
@@ -571,6 +572,10 @@ WMFVideoMFTManager::CreateBasicVideoFrame(IMFSample* aSample, 
    NS_ENSURE_TRUE(SUCCEEDED(hr), hr);

    stride = mVideoStride;

  }

  if (stride <= 0) {

    LOG("CreateBasicVideoFrame: invalid stride %ld", stride);

    return E_FAIL;

  }



  const GUID& subType = mDecoder->GetOutputMediaSubType();

  MOZ_DIAGNOSTIC_ASSERT(subType == MFVideoFormat_YV12 ||
@@ -599,9 +604,17 @@ WMFVideoMFTManager::CreateBasicVideoFrame(IMFSample* aSample, 


  MOZ_DIAGNOSTIC_ASSERT(mSoftwareImageSize.height % 16 == 0,

                        "decoded height must be 16 bytes aligned");

  const uint32_t y_size = stride * mSoftwareImageSize.height;

  const uint32_t v_size = stride * mSoftwareImageSize.height / 4;

  const uint32_t halfStride = (stride + 1) / 2;

  mozilla::CheckedInt<uint32_t> y_size_checked =

      mozilla::CheckedInt<uint32_t>(static_cast<uint32_t>(stride)) *

      mSoftwareImageSize.height;

  if (!y_size_checked.isValid()) {

    LOG("CreateBasicVideoFrame: plane size overflow");

    return E_FAIL;

  }

  const uint32_t y_size = y_size_checked.value();

  const uint32_t v_size = y_size / 4;

  const uint32_t halfStride =

      static_cast<uint32_t>((static_cast<int64_t>(stride) + 1) / 2);

  const uint32_t halfHeight = (videoHeight + 1) / 2;

  const uint32_t halfWidth = (videoWidth + 1) / 2;