MediaTek chipsets are cheating in benchmarks — inflates PCMark scores by up to 75%

MediaTek chipsets have proven pretty impressive recently, but it seems like the way they achieved those scores may not have been entirely honest. Many smartphone makers have done it, got caught, and then changed their ways, but this time it’s not really the smartphone brands’ problem, but instead, MediaTek’s.

The issue comes about with MediaTek adding a whitelist of sorts for their benchmarks, allowing them to deliver better performance in benchmarks. MediaTek’s whitelist offers what appears to be a “Sports Mode”, which appears to be a performance mode of sorts. This mode is not accessible to end users, and is entirely in MediaTek’s control. It is part of MediaTek’s Board Support Package (BSP), something similar to a driver package for the chipsets. The whitelist itself is a file named power_whitelist_cfg.xml, and it is stowed away in the vendor/etc folder on most devices.

Reno3Pro MediaTek P95 whitelist

In Anandtech‘s testing, they used an anonymized version of PCMark to test the OPPO Reno3 Pro with a Helio P95, and there was a difference of 30% in the overall score, with up to a 75% difference in the sub-scores, between the whitelisted version of PCMark and the unlisted one. Aside from PCMark, the MediaTek’s whitelist also features a number of other apps and titles too, as NasiLemakTech also discovered in a recently announced 90 Hz MediaTek Helio G90T device. It’s also spotted in various other MediaTek devices too, with chipsets as far back as the OPPO F9 Pro with its Helio P60 and the much more recent realme C3 also featuring the whitelist file.

AnandTech did reach out to MediaTek and this was their response:

MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets. We work closely with global device makers when it comes to testing and benchmarking devices powered by our chipsets, but ultimately brands have the flexibility to configure their own devices as they see fit. Many companies design devices to run on the highest possible performance levels when benchmarking tests are running in order to show the full capabilities of the chipset. This reveals what the upper end of performance capabilities are on any given chipset.

Of course, in real world scenarios there are a multitude of factors that will determine how chipsets perform. MediaTek’s chipsets are designed to optimize power and performance to provide the best user experience possible while maximizing battery life. If someone is running a compute-intensive program like a demanding game, the chipset will intelligently adapt to computing patterns to deliver sustained performance. This means that a user will see different levels of performance from different apps as the chipset dynamically manages the CPU, GPU and memory resources according to the power and performance that is required for a great user experience. Additionally, some brands have different types of modes turned on in different regions so device performance can vary based on regional market requirements.

We believe that showcasing the full capabilities of a chipset in benchmarking tests is in line with the practices of other companies and gives consumers an accurate picture of device performance.

So MediaTek appears to be pretty defensive about their actions, and seems to believe that this is the way they want to portray their chipsets to the public eye. Parts of their statement is true, but pushing the chipset to perform above its nominal performance levels does seem a bit shady. In a way, you can almost see it as optimizations to help the chipset deliver better performance in the whitelisted apps, although it does mean that it essentially defeats the purpose of benchmarks, which is to let us compare the chipsets and devices to one another.

Boosting performance like this also results in an inconsistent experience, as some apps will run great thanks to the whitelist, but maybe a newer game title or benchmark will reveal some horrible performance. It’s not exactly “intelligently adapting” when you are referencing a whitelist, but to be honest I do see it from MediaTek’s point of view as it should deliver a better experience for end users, at least if they are using the whitelisted apps.

What about you?

Pokdepinion: Well, I was hoping for more out of MediaTek, but it seems like they can’t exactly compete on equal footing in terms of raw performance and had to use these shady techniques to get by.