ActionScript vs. JavaScript benchmark over time
(using Google V8 Benchmark)
The performance of ActionScript3 was quite impressive when Flash Player 9 came out in March 2007. Adobe promised and delivered us at least a 10x speed increase over AS1/AS2.
At that time JavaScript performances in the browsers were abysmal. But while few things changed on the AS3 front since that time, the browsers performance dramatically improved. At one point I started to think that JavaScript performance was better than ActionScript in most browsers.
In order to prove my point I’ve decided to port the Google V8 Benchmark (maybe not the most unbiased choice but the results are eloquent IMHO). To be fair I’ve tried to change as little as possible except strongly typing every variable/method whenever possible (what any developer would do with AS3). The port is released under the same license as the original V8 benchmark and available below.
Before I dig in deeper, here are my conclusions for anyone not interested in most details:
- AS3 compilers have not brought any measurable performance improvement since AS3 is out. Is there room for improvement? I think so (and Alchemy quite demonstrates this).
- RegExp performance in Flash Player is terrible, IE6-like terrible! I strongly suspect E4X performance to be also very bad but I don’t have any benchmark to confirm.
- The overall AS3 execution engine performance roughly doubled from FP9 to FP10.3. FP10.1 brought most of these improvements.
- Even without strong typing, every browser has now reached a level of performance at least comparable to Flash Player:
- Chrome 1.0+ (01/2008)
- Safari 3.1+ (03/2008)
- Opera 10.5+ (03/2010)
- Firefox 4.0+ (03/2011)
- IE 9.0+ (03/2011)
- Some browsers have found another performance boost and are now leaving Flash Player performance lagging behind:
- Chrome 10.0+ (03/2011). Check this blog post explaining the perf boost.
- Opera 11.50 (Swordfish) Beta
- Firefox 5.0 Beta
Thibault Imbert talked a bit about future improvements to the VM in this blog post and while the improvements don’t look useless, a raw performance boost would be better (and there’s room to do so). Come on Adobe!
A Google V8 Benchmark port to ActionScript3
Porting JavaScript to AS3 is not always straightforward. Some parts of the benchmark are object oriented and are mostly AS1 to AS3 ports, but some use ugly approaches. There are hacks everywhere you have to emulate with static vars and other tricks.
My main port of the benchmark makes use of strong types, including Vector.<T>. Replacing Arrays with Vectors introduced some bugs and some loops had to be reversed because of the Vector.<T> behavior.
I took good precautions to change the algorithms as little as possible.
Two derivative code branches have then been produced:
- one branch where Vectors have been replaced with generic Arrays
- the last branch has been stripped of all typing
Download
Notes about EarleyBoyer
EarleyBoyer is a test where 95% of the code is generated by a translator tool named Scheme2Js. The result is a beast of over 4500 lines of code, and most of it is never executed. Only the executed code has actually been ported so the resulting file is only 1700 lines long.
There’s a lot of recursion in this test and old compilers (Flex 3.X, CS3 & CS4) produces something different with this code, resulting in a stack overflow of something.
The test protocol
Each test has been repeated 5 times (except very old browsers like IE6 which got only 3 or 4). The maximum score of each individual score was then extracted and combined into a global score using the same geometric mean as Google’s.
All tests have been executed on a VirtualBox VM under W7 x64. The host machine has a Intel Core i5 760.
ActionScript benchmarks
The following players were tested:
- 9.0.115 (release / stand alone)
- 9.0.280 (release / stand alone)
- 10.0.2.54 (release / stand alone)
- 10.1.52.14 (release / stand alone)
- 10.2.159.1 (release / stand alone)
- 10.3.181.14 (release / stand alone)
- 11.0.1.3 beta (release / ActiveX)
The following compilers were used:
- Flash Pro CS3
- Flash Pro CS4
- Flash Pro CS5
- Flex SDK 3.0.0.477
- Flex SDK 3.1.0.2710
- Flex SDK 3.2.0.3958
- Flex SDK 3.3.0.4852
- Flex SDK 3.4.0.9271
- Flex SDK 3.5.0.12683
- Flex SDK 3.6.0.16995
- Flex SDK 4.0.0.14159
- Flex SDK 4.1.0.16076
- Flex SDK 4.5.0.20967
I also tried to optimize a Flex 4.5 compiled swf using Apparat TDSI and tested it with the other swf.
A batch then executed each combination of code branch X player X compiler X 5 times, resulting in 1145 logged test results!
You can download:
- the full log: http://iq12.com/files/v8/fp_v8_log.txt
- a csv of the processed results: http://iq12.com/files/v8/fp_v8_log.csv
JavaScript benchmarks
The majority of old versions were obtained on http://www.oldapps.com/
The following browsers ran successfully the Google V8 Benchmark v6 (only modded to log the results):
- IE6/7/8/9/10(preview)
- Firefox 2.0/3.0/3.5/3.6/4.0/5.0(beta)
- Opera 7.0/8.0/9.0/9.6/10.0/10.5/11.0/11.1/11.5(beta)
- Safari 3.1/4.0/5.0
- Chrome 1/2/3/4/5/6/7/8/9/10/11/12(beta)
You can download:
- the full log: http://iq12.com/files/v8/browsers_v8_log.txt
- a csv of the processed results: http://iq12.com/files/v8/browsers_v8_log.csv
Flash Player performance details
Compilers
There’s less than 1% difference between the best result of each swf produced by the different compilers (not counting EarleyBoyer results that error’d with older compilers), well within the margin of error. So the AS3 compiler appears to have never changed in terms of performance.
TDSI also made no measurable performance improvement.
Flash Player
Here are the scores by Flash Player version:
- there’s almost as much performance gains in the AS3 ports with language features (typing) as there are with AVM2 improvements over time
- the perf advantage with typing seem to add a fixed point boost to the score (and not a % of the no-typing branch). I wonder why?
- the performance stagnates with 10.3 and 11.0 (not really fair since it’s a beta)
Here this same graphic with DeltaBlue only:
It’s interesting to see how much performance FP10.1 gained on this benchmark on untyped code on this many small-objects-benchmark. Was this their focus on FP10.1? Should it have been knowing that any performance limited code will be typed?
DeltaBlue: One-way constraint solver, originally written in Smalltalk by John Maloney and Mario Wolczko. The DeltaBlue benchmark is written in an object-oriented style with a multi-level class hierarchy. As such it measures how fast the JavaScript engine is at running well-structured applications with many objects and small functions.
Follow up: Adobe’s slides from Flash Camp Brasil
These slides are quite interesting. It looks like the VM team is also hard at work and everything is not oriented towards new features. I really hope the work on the JTing and the GC provides a huge performance boost. Thanks for communicating on this guys!
(click to open the PDF)
