ActionScript vs. JavaScript benchmark over time
(using Google V8 Benchmark)



The performance of ActionScript3 was quite impressive when Flash Player 9 came out in March 2007. Adobe promised and delivered us at least a 10x speed increase over AS1/AS2.

At that time JavaScript performances in the browsers were abysmal. But while few things changed on the AS3 front since that time, the browsers performance dramatically improved. At one point I started to think that JavaScript performance was better than ActionScript in most browsers.

In order to prove my point I’ve decided to port the Google V8 Benchmark (maybe not the most unbiased choice but the results are eloquent IMHO). To be fair I’ve tried to change as little as possible except strongly typing every variable/method whenever possible (what any developer would do with AS3). The port is released under the same license as the original V8 benchmark and available below.

Before I dig in deeper, here are my conclusions for anyone not interested in most details:

  • AS3 compilers have not brought any measurable performance improvement since AS3 is out. Is there room for improvement? I think so (and Alchemy quite demonstrates this).

  • RegExp performance in Flash Player is terrible, IE6-like terrible! I strongly suspect E4X performance to be also very bad but I don’t have any benchmark to confirm.

  • The overall AS3 execution engine performance roughly doubled from FP9 to FP10.3. FP10.1 brought most of these improvements.

  • Even without strong typing, every browser has now reached a level of performance at least comparable to Flash Player:
    • Chrome 1.0+ (01/2008)
    • Safari 3.1+ (03/2008)
    • Opera 10.5+ (03/2010)
    • Firefox 4.0+ (03/2011)
    • IE 9.0+ (03/2011)
  • Some browsers have found another performance boost and are now leaving Flash Player performance lagging behind:
    • Chrome 10.0+ (03/2011). Check this blog post explaining the perf boost.
    • Opera 11.50 (Swordfish) Beta
    • Firefox 5.0 Beta

Thibault Imbert talked a bit about future improvements to the VM in this blog post and while the improvements don’t look useless, a raw performance boost would be better (and there’s room to do so). Come on Adobe!

A Google V8 Benchmark port to ActionScript3

Porting JavaScript to AS3 is not always straightforward. Some parts of the benchmark are object oriented and are mostly AS1 to AS3 ports, but some use ugly approaches. There are hacks everywhere you have to emulate with static vars and other tricks.

My main port of the benchmark makes use of strong types, including Vector.<T>. Replacing Arrays with Vectors introduced some bugs and some loops had to be reversed because of the Vector.<T> behavior.

I took good precautions to change the algorithms as little as possible.

Two derivative code branches have then been produced:

  • one branch where Vectors have been replaced with generic Arrays
  • the last branch has been stripped of all typing

Download

Notes about EarleyBoyer

EarleyBoyer is a test where 95% of the code is generated by a translator tool named Scheme2Js. The result is a beast of over 4500 lines of code, and most of it is never executed. Only the executed code has actually been ported so the resulting file is only 1700 lines long.

There’s a lot of recursion in this test and old compilers (Flex 3.X, CS3 & CS4) produces something different with this code, resulting in a stack overflow of something.

The test protocol

Each test has been repeated 5 times (except very old browsers like IE6 which got only 3 or 4). The maximum score of each individual score was then extracted and combined into a global score using the same geometric mean as Google’s.

All tests have been executed on a VirtualBox VM under W7 x64. The host machine has a Intel Core i5 760.

ActionScript benchmarks

The following players were tested:

  • 9.0.115 (release / stand alone)
  • 9.0.280 (release / stand alone)
  • 10.0.2.54 (release / stand alone)
  • 10.1.52.14 (release / stand alone)
  • 10.2.159.1 (release / stand alone)
  • 10.3.181.14 (release / stand alone)
  • 11.0.1.3 beta (release / ActiveX)

The following compilers were used:

  • Flash Pro CS3
  • Flash Pro CS4
  • Flash Pro CS5
  • Flex SDK 3.0.0.477
  • Flex SDK 3.1.0.2710
  • Flex SDK 3.2.0.3958
  • Flex SDK 3.3.0.4852
  • Flex SDK 3.4.0.9271
  • Flex SDK 3.5.0.12683
  • Flex SDK 3.6.0.16995
  • Flex SDK 4.0.0.14159
  • Flex SDK 4.1.0.16076
  • Flex SDK 4.5.0.20967

I also tried to optimize a Flex 4.5 compiled swf using Apparat TDSI and tested it with the other swf.

A batch then executed each combination of code branch X player X compiler X 5 times, resulting in 1145 logged test results!

You can download:

JavaScript benchmarks

The majority of old versions were obtained on http://www.oldapps.com/

The following browsers ran successfully the Google V8 Benchmark v6 (only modded to log the results):

  • IE6/7/8/9/10(preview)
  • Firefox 2.0/3.0/3.5/3.6/4.0/5.0(beta)
  • Opera 7.0/8.0/9.0/9.6/10.0/10.5/11.0/11.1/11.5(beta)
  • Safari 3.1/4.0/5.0
  • Chrome 1/2/3/4/5/6/7/8/9/10/11/12(beta)

You can download:

Flash Player performance details


Compilers

There’s less than 1% difference between the best result of each swf produced by the different compilers (not counting EarleyBoyer results that error’d with older compilers), well within the margin of error. So the AS3 compiler appears to have never changed in terms of performance.

TDSI also made no measurable performance improvement.

Flash Player

Here are the scores by Flash Player version:

  • there’s almost as much performance gains in the AS3 ports with language features (typing) as there are with AVM2 improvements over time
  • the perf advantage with typing seem to add a fixed point boost to the score (and not a % of the no-typing branch). I wonder why?
  • the performance stagnates with 10.3 and 11.0 (not really fair since it’s a beta)

Here this same graphic with DeltaBlue only:

It’s interesting to see how much performance FP10.1 gained on this benchmark on untyped code on this many small-objects-benchmark. Was this their focus on FP10.1? Should it have been knowing that any performance limited code will be typed?

DeltaBlue: One-way constraint solver, originally written in Smalltalk by John Maloney and Mario Wolczko. The DeltaBlue benchmark is written in an object-oriented style with a multi-level class hierarchy. As such it measures how fast the JavaScript engine is at running well-structured applications with many objects and small functions.

Follow up: Adobe’s slides from Flash Camp Brasil

These slides are quite interesting. It looks like the VM team is also hard at work and everything is not oriented towards new features. I really hope the work on the JTing and the GC provides a huge performance boost. Thanks for communicating on this guys!


Flash-Camp-Brasil.small_.jpg

(click to open the PDF)


Comments (16)

zwetanJune 15th, 2011 at 15:31

FYI you have already a lot of performance test for ActionScript / Tamarin

see here
https://developer.mozilla.org/En/Tamarin/Tamarin_Acceptance_Testing

and here
http://hg.mozilla.org/tamarin-redux/file/2529571bfdfa/test/performance

fuJune 15th, 2011 at 15:49

@zwetan: thanks for these links, I didn’t know them

So most of the port to AS3 had already been made and I’ve lost quite a few hours coding this :/

All work is not lost anyway since these versions will not easily compile to .swf (or would they?) and I don’t know the relations between tamarin and Flash Player versions (or even if Adobe commits the newer ones). So I can’t rely on tamarin to do my benchmarks.

The whole performance comparison in time is what drove this work.

ShawnJune 16th, 2011 at 05:12

Thanks for shining a big fat spotlight on this issue, hopefully Adobe will be shamed into taking some major steps.

Alec McEachranJune 16th, 2011 at 13:39

Thanks for this, it’s really interesting. @Shawn: I’m not holding my breath!

TemposirJune 16th, 2011 at 13:43

Hm, maybe need try http://www.buraks.com/azoth/ and Alchemy Memory Classes?

ConstantineJune 17th, 2011 at 16:13

Impressive.

Though two little questions,
1) – why did Opera gained such huge boost in 11.50 ? If you compare 11.50 with 11.10 and 11.0 – there is not much difference. So, i can only suspect – you have not tested those versions.

2) – V8 tests calculation performance, i.e. “no rendering”. And i’m pretty sure that Flash is much faster than current browsers in that regard. For example – Firefox 4 is not able to handle even simple interactive games or animations, while it clearly beats flash in v8.

zedia.netJune 17th, 2011 at 16:44

Hey, this is all good and all, but I feel like these are not the areas where I have cpu bottlenecks when I build experiences and applications using Flash. The visual is what usually puts limits to what I can do. I think a benchmark like this one is better : http://www.craftymind.com/guimark3/

fuJune 20th, 2011 at 11:35

@constantine: I actually did test Opera 11.0 and 11.1. Check browsers_v8_log.csv linked above. I think the JS engine is not correlated to the major version number for Opera. The last speed boost was at a minor release (10.6). I don’t think the virtual machines can’t really improve at each version of a browser with a 3 month release cycle. You need to push major changes to see bigs leaps in performance. So the engine making big progresses from time to time makes perfect sense.

@zedia.net: I agree that the main bottleneck currently is the rendering engine, especially in 3D. But we know Molehill allows 60fps with nearly everything you can push to it. So the next bottleneck is the AVM2. Currently there’s also lots of specialized code (3D engines, AR engines, sound/image processing…etc) that is not visual and need to use tricks (like memory instructions with Apparat) to have correct performance. What bogs me right now is that it would make perfect sense to use Javascript as we use Alchemy (as a speed booster) and that’s a shame.

TangentJune 21st, 2011 at 05:03

Guys, don’t forget that bottle neck in most GUI’s are not script execution, but translating the vectors and DOM’s into pixels on the screen. Flash has done a lot of homework in the later end, so even though it lacks on the code execution, the drawings speed well compensates the rest.

After all, a good GUI should have a good balance of processing on the client side and server side.

Jeff MuizelaarOctober 27th, 2011 at 20:32

Can you post the SWFs so that others can run the same tests?

[...] Dart. Kasper mentioned the speed increase in Javascript execution in modern browsers (source: http://iq12.com/blog/as3-benchmark/) and Javascript popularity in general with projects like V8, Node.js, (based on V8) and [...]

ElliotJuly 25th, 2012 at 13:17

Now try the same thing with visual elements.
But not just bitmaps. Also do one with vector artwork.
To be fair here, but it would also need to be the same dimensions. I would try stepped view ports.

You might want to try different transforms too. I would try one that just moves x and y. Then another that adjusts scale. Try another with opacity. And another with color transforms.

Flash may have not gained much performance in the way of code execution. But it’s got rendering speed in spades!

IvanSeptember 9th, 2012 at 12:04

Hi, I have made a Plasma effect with JavaScript and AS3 (flash version on the link in top right corner). Results:

JavaScript: 60 FPS while using 10% of CPU.
AS3: 25 FPS while using 100% of CPU.

fuSeptember 11th, 2012 at 00:16

@Ivan: with a few basic things any as3 developer should do (type the variables!), framerate is WAY up
http://wonderfl.net/c/z1PM

ggNovember 25th, 2012 at 22:38

“with a few basic things any as3 developer should do (type the variables!)”

Why is it, then, that the JS does not require it?

fuDecember 19th, 2012 at 12:14

They’re not the same language! The design team of AS3 decided for a typed language. It doesn’t require everything to be typed (which is good IMHO). But as it’s a central language feature this is a basic good practice for any semi-pro developer to type variables.

Because Javascript is untyped, recent engines do an optimization technique called “type inference”. This allows them to guess the types of the variables while running.

Leave a comment

Your comment