For some people benchmarks don’t matter … and for others they are ALL that matter. For hardware companies, benchmarks provide an internal performance metric and external bragging rights. But new evidence shows that Samsung tweaked their Galaxy Note 3 to perform better on benchmarks than in ‘normal usage’. This isn’t the first time they’ve done this, either!
According to an article at ArsTechnica, Samsung tweaked their CPU performance when a known benchmark app was run, so it would give better than expected results. This was noticed when the Galaxy Note 3 outperformed the LG G2 — despite the two sharing nearly identical hardware. Samsung got caught doing this with the Galaxy S4 and even claimed it benefitted other apps like games, despite evidence that wasn’t true.
What do you think? Should companies be gaming benchmarks?
As an Amazon Associate, we earn from qualifying purchases. If you are shopping on Amazon anyway, buying from our links gives Gear Diary a small commission.
I have loved technology for as long as I can remember - and have been a computer gamer since the PDP-10! Mobile Technology has played a major role in my life - I have used an electronic companion since the HP95LX more than 20 years ago, and have been a 'Laptop First' person since my Compaq LTE Lite 3/20 and Powerbook 170 back in 1991! As an avid gamer and gadget-junkie I was constantly asked for my opinions on new technology, which led to writing small blurbs ... and eventually becoming a reviewer many years ago. My family is my biggest priority in life, and they alternate between loving and tolerating my gaming and gadget hobbies ... but ultimately benefits from the addition of technology to our lives!
Sadly it’s nothing new since…well dawn of time. I remember many PC benchmarks being cheated for graphics cards back in the day when I actually cared about it.
Benchmarks smenchmarks. As long as it does what I want then I don’t care what some stupid number says….and I am a geek. 😛
I understand what you mean, Joel, but it really bugs me that these games are played. The average consumer doesn’t know that benchmarks are so gamed … all they see is that the device they are contemplating is (or isn’t) the “best”. =/
The average consumer doesn’t even know what a benchmark is or means. The average consumer doesn’t have an idea on what a 64 bit processor in the iPhone 5s truly means either. They just ask: Is this the new iPhone 5s or the new Galaxy Note 3 and then they buy it.
The only ones who truly care about these are ones who are smart enough to research and find out if they are being gamed and many of them don’t care either. Oh and the guys who wring hands if their device isn’t the top one on the benchmark list.
Does it suck for those who do know/care? Probably. However, like I said, they are smart enough to know: never blindly trust a benchmark.
I don’t believe that the average consumer is that ignorant, nor do I think that they are immune to buzzwords like “benchmark”. I am sure that when most people hear a word like that — one that is supposed to MEAN something, they can’t help but look and see where their device might fall into the graph.
When “benchmarks” are used as part of an advertising campaign or when they are created by a third-party in an otherwise impartial review that potential buyers might read, to suppose that consumers wouldn’t be interested in how their device compares is a bit arrogant and condescending, don’t you think?
Possibly….but many don’t know where to even look and rely more on hearing what their friend says. My wife doesn’t even look at reviews online or anything else. She knows what she wants and could care less if it’s the top on a benchmark. She doesn’t even look at the advertising or listen to me! She was up just a month or so ago and she ran to the store to get a new device even though she could eek her old phone on long enough to get the 5s. So that means she bought a 5. Does she care that she doesn’t have a 5s? Nope. In fact, if she could get a decent Blackberry back when she got the 4 she last had she would have done that. As long as Apple continues to make iPhones that’s what she will get.
There are and always will be some that care about benchmarks. There will always be companies that try to game them. Luckily we have sites like Ars to expose this.
In the end what does it matter if your phone gets 100 points higher geekbench if it still instantly opens your e-mail when you tap the icon?
I disagree. Smartphones are making consumers far, far more savvy about their devices. I know people who have no clue what the various acronyms and abbreviations mean, but they know that 4G is faster and to wait in the fall before buying an iPhone
In many ways a little bit of knowledge is dangerous here-think about how often the megapixel debate crops up. People hear a high megapixel number and assume it means better camera, and no amount of geek-talk is going to change their mind. So when you lead with a false or exaggerated benchmark, it’s going to stick, and no amount of “well, technically…” is going to dislodge that initial impression. Just try explaining to someone that HSPA+ isn’t really 4G vs LTE…doesn’t matter. People are trained to hear 4G and think fast, hear megapixel and think “good camera” and hear benchmarks and think “fast computer”.
Sent from my iPad
Yeah I agree with that. They know that higher numbers mean better in many cases and that’s what they look for. Nevermind that in some cases you are talking about nanoseconds. It doesn’t matter to them. It has the higher “MBS” or whatever spec…thats what they want.
Oh and NONE of what I said excuses Samsung for the gaming. AT ALL. It seems, to me, that maybe it’s time for the benchmarks to look for that kind of trickery.
Perhaps the solution is for journalists to stop reporting using standardized benchmarks. I understand the desire to be able to measure things objectively, but maybe the best way to measure the speed of a phone is just to subjectively report how fast it seems, just as most journalists subjectively report on ease of use, camera quality, “hand feel”, call sound quality, etc.
And, yes, I have always thought it wrong when companies game benchmarks. Again, maybe the answer is for reviewers to refuse to recommend any device by a manufacturer who manipulated their devices to artificially score higher on benchmarks. I can’t think of any other way to penalize samsung than that.
I could not agree with you more. The numbers are always interesting but they don’t tell the full story. Not by a longshot. For example, Apple has less ran in most of their devices than their competitors and yet their devices run faster in most every regard. One would think more RAM means faster and better but it’s much more complicated than that. In addition, one has to wonder what the bit of speed difference makes for most people in the process.
Having said that, so long as there are benchmarks being used when describing devices – and I get it since people like to try to have some objective descriptors rather than merely subjective – there is a grave danger in dismissing the “average consumer” as not being savvy enough or interested enough as the initial comment thread seemed to suggest.
My other thought on this – graphics cards live and die by those numbers along with pixel throughput and so on … and anyone shopping for a card has an idea of what they are looking at. But like you say the synthetic numbers were easily gamed, and therefore we started seeing metrics of real-world performance in games.
Speed in a smartphone is a different story – and looking at the chart shows that the Galaxy Note 3 is a performance *beast*. So why continue to cheat with phone after phone? So they can claim ‘fastest smartphone ever’ in advertising? I don’t know, but it once again tweaks my cynical side that has always seen Samsung as having a culture of dishonesty and manipulation. The day after Apple announces 64-bit they say their next phone is 64-bit … which they were designing before Apple. The day after Apple posts massive gold iPhone demand … they have gold phones out there and talk about how they were doing it first . And so on …
As for whether benchmarks-as-marketing work … just think back to the ‘MHz wars’ in the 90s … it worked and it sold things. And suddenly people were claiming doubled speeds due to clock-doubling and so on – and it sold computers …
That is a tough one … because relative speeds become at least as tricky as benchmarks. Because then you have a completely context-sensitive set of criteria, and also depend on certain apps usage, pre-installed items, and so on. I get what you are saying – but think about ‘hand feel’ … if that was something other than meaningless, then why do we have so many phone designs? 🙂
They do work. Maybe it’s time to truly educate people about benchmarks. The people that truly need to know….marketing folks..have a hard time believing the people who DO know. They want numbers that they can point at and say hey ours is higher….when it really doesn’t matter.
The funniest thing is, like you said, the Note 3 is a beast. It’s probably got the highest specs of anything I’ve seen in a while. So why cheat? It’s like a Slashdot first post troll.
Yeah I was probably assuming that some don’t know that much about this stuff. There are some that do. Even people that are not as tech savvy…and there in lies the problem. The ones who recognize the better numbers buy a device based on that and then they find out that Samsung has cheated it. Some probably even want to return the phone. This is based on not understanding benchmarks and what they are: just an arbitrary number that can be gamed. So what to do? Find a benchmark that can’t be gamed to use as an industry standard. A difficult task for sure. One I think that may work might be Linpack or Lapack which are both used to measure super computer performance. Both are relatively open source, but under the BSD license instead of the GPL. You can still inspect the source and possibly fix the code whenever a manufacturer tries to game the system. They kind of did this with Geekpack. The downside is Linpack and Lapack both just look at raw power and doesn’t evaluate things that probably matter more like GPU performance. Ram speed…etc. There’s no real answer to actually measure which phone is “better”.
For me, the subjective is even more important than the objective numbers anyway. It doesn’t matter if it performs better on the benchmarks if the interface is laggy. It doesn’t matter if the benchmark is better if the phone is uncomfortable to hold. It also doesn’t matter if the benchmark is low, but I am happy with it…which is all that REALLY matters.
One podcast on TWIT.tv I like is Before you Buy. Oh and of course Geardiary. Both have never steered me wrong.
Samsung out-Sony’d Sony in the LCD TV wars and won. They are trying to do the same thing to Apple in the smartphone wars, but have found Apple to be a far more formidable opponent.
I don’t know all of the details, but you would think that what Samsung is doing would be the norm. The processors on these phones are more powerful than my 6 yr old laptop, and they have a lot of power saving features. If I run a benchmark app while the cpu has a couple of cores clocked down or off, I don’t want the benchmark to show me the state it is in right now. I want it to show me the theoretical maximum I can expect.
Maybe there needs to be some standardized way in which this is all done, and there certainly needs to be more transparency.
The issue is that I want the benchmarks to show me how the phone will work as I use it. Samsung (and others) are looking for the presence of benchmarks and changing how the phone runs in a way that would damage the phone if it ran that way all of the time. That’s the issue.
The sad thing is that the Samsung Note 3 would benchmark higher than anything if they did nothing to game the benchmarks. Silly.
tl;dr: Not only Samsung, but Asus, HTC and LG game the benchmarks, too. Motorola, the Nexus phones, and the NVidia Shield do not.
Sad. Key quote: “The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 – 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell.”
Do you have a link to that information? Because, everything I’ve read indicates that Samsung’s software isn’t over clocking the processor but just making it run at 2.3Ghz which is what the processor is advertised at. Just curious if I missed something.
“1) On the Exynos 5410, Samsung was detecting the presence of certain benchmarks and raising thermal limits (and thus max GPU frequency) in order to gain an edge on those benchmarks, and”
“2) On both Snapdragon 600 and Exynos 5410 SGS4 platforms, Samsung was detecting the presence of certain benchmarks and automatically driving CPU voltage/frequency to their highest state right away. Also on Snapdragon platforms, all cores are plugged in immediately upon benchmark detect.”
So, I was mistaken about the Note 3; it may not be as bad as the GS4 – at least thermal limits are not raised – but, in use, the phone will never max the cores and peg the CPU and GPU for very long in real-world use.
Sadly it’s nothing new since…well dawn of time. I remember many PC benchmarks being cheated for graphics cards back in the day when I actually cared about it.
Benchmarks smenchmarks. As long as it does what I want then I don’t care what some stupid number says….and I am a geek. 😛
I understand what you mean, Joel, but it really bugs me that these games are played. The average consumer doesn’t know that benchmarks are so gamed … all they see is that the device they are contemplating is (or isn’t) the “best”. =/
The average consumer doesn’t even know what a benchmark is or means. The average consumer doesn’t have an idea on what a 64 bit processor in the iPhone 5s truly means either. They just ask: Is this the new iPhone 5s or the new Galaxy Note 3 and then they buy it.
The only ones who truly care about these are ones who are smart enough to research and find out if they are being gamed and many of them don’t care either. Oh and the guys who wring hands if their device isn’t the top one on the benchmark list.
Does it suck for those who do know/care? Probably. However, like I said, they are smart enough to know: never blindly trust a benchmark.
I don’t believe that the average consumer is that ignorant, nor do I think that they are immune to buzzwords like “benchmark”. I am sure that when most people hear a word like that — one that is supposed to MEAN something, they can’t help but look and see where their device might fall into the graph.
When “benchmarks” are used as part of an advertising campaign or when they are created by a third-party in an otherwise impartial review that potential buyers might read, to suppose that consumers wouldn’t be interested in how their device compares is a bit arrogant and condescending, don’t you think?
Possibly….but many don’t know where to even look and rely more on hearing what their friend says. My wife doesn’t even look at reviews online or anything else. She knows what she wants and could care less if it’s the top on a benchmark. She doesn’t even look at the advertising or listen to me! She was up just a month or so ago and she ran to the store to get a new device even though she could eek her old phone on long enough to get the 5s. So that means she bought a 5. Does she care that she doesn’t have a 5s? Nope. In fact, if she could get a decent Blackberry back when she got the 4 she last had she would have done that. As long as Apple continues to make iPhones that’s what she will get.
There are and always will be some that care about benchmarks. There will always be companies that try to game them. Luckily we have sites like Ars to expose this.
In the end what does it matter if your phone gets 100 points higher geekbench if it still instantly opens your e-mail when you tap the icon?
I disagree. Smartphones are making consumers far, far more savvy about their devices. I know people who have no clue what the various acronyms and abbreviations mean, but they know that 4G is faster and to wait in the fall before buying an iPhone
In many ways a little bit of knowledge is dangerous here-think about how often the megapixel debate crops up. People hear a high megapixel number and assume it means better camera, and no amount of geek-talk is going to change their mind. So when you lead with a false or exaggerated benchmark, it’s going to stick, and no amount of “well, technically…” is going to dislodge that initial impression. Just try explaining to someone that HSPA+ isn’t really 4G vs LTE…doesn’t matter. People are trained to hear 4G and think fast, hear megapixel and think “good camera” and hear benchmarks and think “fast computer”.
Sent from my iPad
Yeah I agree with that. They know that higher numbers mean better in many cases and that’s what they look for. Nevermind that in some cases you are talking about nanoseconds. It doesn’t matter to them. It has the higher “MBS” or whatever spec…thats what they want.
Oh and NONE of what I said excuses Samsung for the gaming. AT ALL. It seems, to me, that maybe it’s time for the benchmarks to look for that kind of trickery.
Perhaps the solution is for journalists to stop reporting using standardized benchmarks. I understand the desire to be able to measure things objectively, but maybe the best way to measure the speed of a phone is just to subjectively report how fast it seems, just as most journalists subjectively report on ease of use, camera quality, “hand feel”, call sound quality, etc.
And, yes, I have always thought it wrong when companies game benchmarks. Again, maybe the answer is for reviewers to refuse to recommend any device by a manufacturer who manipulated their devices to artificially score higher on benchmarks. I can’t think of any other way to penalize samsung than that.
Doug,
I could not agree with you more. The numbers are always interesting but they don’t tell the full story. Not by a longshot. For example, Apple has less ran in most of their devices than their competitors and yet their devices run faster in most every regard. One would think more RAM means faster and better but it’s much more complicated than that. In addition, one has to wonder what the bit of speed difference makes for most people in the process.
Having said that, so long as there are benchmarks being used when describing devices – and I get it since people like to try to have some objective descriptors rather than merely subjective – there is a grave danger in dismissing the “average consumer” as not being savvy enough or interested enough as the initial comment thread seemed to suggest.
My other thought on this – graphics cards live and die by those numbers along with pixel throughput and so on … and anyone shopping for a card has an idea of what they are looking at. But like you say the synthetic numbers were easily gamed, and therefore we started seeing metrics of real-world performance in games.
Speed in a smartphone is a different story – and looking at the chart shows that the Galaxy Note 3 is a performance *beast*. So why continue to cheat with phone after phone? So they can claim ‘fastest smartphone ever’ in advertising? I don’t know, but it once again tweaks my cynical side that has always seen Samsung as having a culture of dishonesty and manipulation. The day after Apple announces 64-bit they say their next phone is 64-bit … which they were designing before Apple. The day after Apple posts massive gold iPhone demand … they have gold phones out there and talk about how they were doing it first . And so on …
As for whether benchmarks-as-marketing work … just think back to the ‘MHz wars’ in the 90s … it worked and it sold things. And suddenly people were claiming doubled speeds due to clock-doubling and so on – and it sold computers …
That is a tough one … because relative speeds become at least as tricky as benchmarks. Because then you have a completely context-sensitive set of criteria, and also depend on certain apps usage, pre-installed items, and so on. I get what you are saying – but think about ‘hand feel’ … if that was something other than meaningless, then why do we have so many phone designs? 🙂
They do work. Maybe it’s time to truly educate people about benchmarks. The people that truly need to know….marketing folks..have a hard time believing the people who DO know. They want numbers that they can point at and say hey ours is higher….when it really doesn’t matter.
The funniest thing is, like you said, the Note 3 is a beast. It’s probably got the highest specs of anything I’ve seen in a while. So why cheat? It’s like a Slashdot first post troll.
Yeah I was probably assuming that some don’t know that much about this stuff. There are some that do. Even people that are not as tech savvy…and there in lies the problem. The ones who recognize the better numbers buy a device based on that and then they find out that Samsung has cheated it. Some probably even want to return the phone. This is based on not understanding benchmarks and what they are: just an arbitrary number that can be gamed. So what to do? Find a benchmark that can’t be gamed to use as an industry standard. A difficult task for sure. One I think that may work might be Linpack or Lapack which are both used to measure super computer performance. Both are relatively open source, but under the BSD license instead of the GPL. You can still inspect the source and possibly fix the code whenever a manufacturer tries to game the system. They kind of did this with Geekpack. The downside is Linpack and Lapack both just look at raw power and doesn’t evaluate things that probably matter more like GPU performance. Ram speed…etc. There’s no real answer to actually measure which phone is “better”.
For me, the subjective is even more important than the objective numbers anyway. It doesn’t matter if it performs better on the benchmarks if the interface is laggy. It doesn’t matter if the benchmark is better if the phone is uncomfortable to hold. It also doesn’t matter if the benchmark is low, but I am happy with it…which is all that REALLY matters.
One podcast on TWIT.tv I like is Before you Buy. Oh and of course Geardiary. Both have never steered me wrong.
Yup! Some really like small screens. Some, like myself, like the big ones! 🙂 To each their own!
Samsung out-Sony’d Sony in the LCD TV wars and won. They are trying to do the same thing to Apple in the smartphone wars, but have found Apple to be a far more formidable opponent.
I don’t know all of the details, but you would think that what Samsung is doing would be the norm. The processors on these phones are more powerful than my 6 yr old laptop, and they have a lot of power saving features. If I run a benchmark app while the cpu has a couple of cores clocked down or off, I don’t want the benchmark to show me the state it is in right now. I want it to show me the theoretical maximum I can expect.
Maybe there needs to be some standardized way in which this is all done, and there certainly needs to be more transparency.
The issue is that I want the benchmarks to show me how the phone will work as I use it. Samsung (and others) are looking for the presence of benchmarks and changing how the phone runs in a way that would damage the phone if it ran that way all of the time. That’s the issue.
The sad thing is that the Samsung Note 3 would benchmark higher than anything if they did nothing to game the benchmarks. Silly.
Anandtech created a table (called “I Can’t Believe I Have to Make This Table”) which identified which recent model Android phones are gaming which benchmarks. See the table here: http://www.anandtech.com/show/7384/state-of-cheating-in-android-benchmarks
tl;dr: Not only Samsung, but Asus, HTC and LG game the benchmarks, too. Motorola, the Nexus phones, and the NVidia Shield do not.
Sad. Key quote: “The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 – 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell.”
Do you have a link to that information? Because, everything I’ve read indicates that Samsung’s software isn’t over clocking the processor but just making it run at 2.3Ghz which is what the processor is advertised at. Just curious if I missed something.
Yes. Anandtech (I believe) discovered the cheating in the Samsung Galaxy S4. Here’s an article about benchmark cheating: http://anandtech.com/show/7384/state-of-cheating-in-android-benchmarks
Here is a quote. Number 1 is about the GS4.
“1) On the Exynos 5410, Samsung was detecting the presence of certain benchmarks and raising thermal limits (and thus max GPU frequency) in order to gain an edge on those benchmarks, and”
“2) On both Snapdragon 600 and Exynos 5410 SGS4 platforms, Samsung was detecting the presence of certain benchmarks and automatically driving CPU voltage/frequency to their highest state right away. Also on Snapdragon platforms, all cores are plugged in immediately upon benchmark detect.”
So, I was mistaken about the Note 3; it may not be as bad as the GS4 – at least thermal limits are not raised – but, in use, the phone will never max the cores and peg the CPU and GPU for very long in real-world use.