A scientific performance comparison: Flex/Flash vs. JavaFX vs. Silverlight vs. JavaScript

I finally finished my diploma thesis I mentioned before about performance comparisons between Flex/Flash, JavaFX, Silverlight and various JavaScript engines.

Why this analysis?

During an internship at IBM Germany back in 2009, I had to develop a Visualizer based on Flex that heavily relied on its charting library API. Even on strong machines, it was not possible to create more than 20 charts on one screen at the same time. If tried, the application terminated with a timeout exception after 60 seconds because it simply took the rendering engine to long to draw all the charts at once. These experiences lead to thoughts about questions why the Flash Player sometimes performs so slowly and if other technologies like JavaFX or Silverlight could do any better. While looking for answers, I encountered two benchmarks. One is Alexey Gavrilov’s Bubblemark test which moves around bitmaps on the screen capturing the current fps. The other one is Sean Christmann’s GUIMark, which simulates a common website layout and lets it scale up and down. While Gavrilov’s attempt is rather simple, Christmann’s benchmark is a bit more complex including aspects like transparency and overlapping layers. Both tests include technologies like Flash/Flex, JavaFX, Silverlight and Javascript. All these attempts have one thing in common though: They represent only one big benchmark instead of cutting down the issue into multiple aspects. This leads to the problem that one cannot clearly see what the reason is why solution A is faster or slower than B.

For example: Moving around bitmaps, as shown in Gavrilov’s Bubblemark benchmark, may sound simple but heavily relies on multiple aspects of a RIA runtime: First, to display images, a graphic-buffer needs to be filled with the bitmap data. Then it needs to be drawn to a canvas-like component and finally shown on the screen. To move around the images, mathematical calculations are required to let the balls bounce from the walls. Furthermore, some kind of data structure like (dynamic) lists or arrays must be used in order store each ball-object in. While running the test, one never knows what was the cause for performance decreases. Was it the »physics engine«, the image processing calls, the array/list operations or something else?

This lead to the idea of developing a series of tests to drill down to the core of performance issues, which leads to two benefits: One is that developers who already know their requirements for their applications can choose the RIA technology that fits best for their needs, based on the result of these test series. The other one is that RIA manufacturers can optimize their virtual machines and browser plug-ins based on the conclusions of this thesis.

The tests

Run the tests, download the source and view the results here.

Feel free to download everything and play around with it. Most of the sources are released under the MIT license. Some others use GPL or BSD so make sure to check the license agreement in the header sections of each project/file but in general you don’t really have to worry about them since they’re all open source licenses. Just watch out for the copyleft agreement in GPL.

Flash Player 10.1 performance explosion

I am currently writing my diploma thesis at the University of Ulm on performance issues regarding Rich Internet Application technologies like Adobe Flash/Flex, JavaFX, Silverlight and various JavaScript solutions.
(Update: It’s done! :-) )

I started back in December when I first noticed that Adobe’s Flash Player seriously has some performance issues. It always was by far the slowest of all technologies.
Today, I retried some of my self-written benchmarks using the new Flash Player 10.1 RC4 and I was absolutely blown away. The new version is so fast, it’s absolutely incredible.
I am not done yet with my thesis until mid-July, so I won’t publish to much about it here but I thought it might be interesting to show just one benchmark-result here.

I don’t want to go to much into detail regarding the test implementations since these might change til July, so if you want to know more about the exact details on my benchmark, you gotta wait til I am done with my thesis. So long, think of this as a “preview” ;-) since some things might still change. The final test/benchmark will be revealed in 2 months when it’s 100% finished. I just didn’t want to post anything that’s not done yet. As already said, this is just a little teaser for the final benchmark.

Test setup
All tests were run on a Macbook Pro with an Intel Core 2 Duo at 2.53 GHz and 8 GB of RAM.
Tests, which require a plugin were running using Safari.
Additionally, I also ran some JavaScript-based tests on Firefox, Google Chrome and Safari.
The code base for all tests is basically the same, except for differences regarding Syntax, of course. This makes all test results comparable.

Results (Click to enlarge):

All values are in milliseconds [ms] => Less is better

In numbers:

  • Flash Player 10.1: 528 ms
  • JavaScript (Safari 4.0.5): 1015 ms
  • JavaScript (Google Chrome 5.0.375.29: 1039 ms
  • Silverlight 4.0.50401.0: 1300 ms
  • JavaFX 1.3 (JRE 1.6.0_17): 1392 ms
  • Firefox 3.6.3: 1449 ms
  • JavaFX 1.2 (JRE 1.6.0_17): 1635 ms
  • Flash Player 10.0: 3201 ms

Adobe, what the hell did you do???

Note: Please be aware, that I am basically testing FP 10.1 vs FP 10.0 here. This has nothing to do with the latest RC4 version of FP 10.1 since I haven’t done any tests with older release candidates yet.

Update: Added test results for JavaFX 1.3 and fixed a mistake regarding the test results for Firefox.

Update II: According to a blog entry of Tinic Uro, an engineer at Adobe Systems, the reason why Flash Player 10.1 works so well on Mac OS is Apple’s Core Animation Framework (Thanks to Matthew for the link!)
What I really liked about this blog post is the following:

“You might have noticed that Core Animation is a Cocoa API. Yes, Flash Player 10.1 is a true Cocoa app now (with a Carbon fallback to support Firefox and Opera which are not Cocoa yet).”

Flash Player is a true Cocoa application now? Nice.

Tagxedo: Crazy tagcloud generator

To be honest, I don’t really like Silverlight apps, but Tagxedo is truly great.

Just enter a website’s url (or just plain words) of your choice and it will generate a cloud for you, based on its content, as shown in the image below.

Variable parameters include font size, color, alignment and shape.

The app itself feels a bit buggy and the export-to-png didn’t work quite properly for me, so I had to take a screenshot instead, but still, the idea itself it brilliant.

Tagxedo Tagcloud

Forward- and back-button support in Silverlight

I noticed in the screencast below, that Silverlight seems to offer a great implicit way of supporting browser’s forward- and back-buttons. The basic idea is to modify the current url by adding or removing tags without forcing a page refresh.

Example: In the video below, a basic Twitter client is being developed using Silverlight. The main part is wrapped into a navigation component which automatically stacks sub-components, similar to the ViewStack component in Adobe Flex. The difference is now, if the displayed component changes, also the current url changes. For example, if the user sees the “home” view and clicks on “search”, the url, displayed in the browser, changes from http://www.example.com/application#/home to http://www.example.com/application#/search.

If the user now clicks the back-button, the Silverlight application automatically switches back to the home view.

This truly is a great (and so simple!) way of supporting browser history inside of plugin-based RIA technologies. I wished, there would be something like this for Flex.

Flash + Chrome OS = ?

I am a passionate RIA developer, especially if it comes up to using Adobe’s Flex SDK. It’s so easy to use and one can built great web applications in no time.
The best thing I always liked (and still do) is that everything comes “from one hand”. Which means, that the whole API is made by Adobe and most common use-cases are already build-in.
Ever tried to create a div box with rounded corners and shadows just with html and css? Good luck. Using the Flash Builder, it’s just two clicks and you’re done. In my opinion, this makes Flex a development environment which is very consistent and easy to handle, unlike pure JavaScript-solutions that often rely on multiple 3rd-party frameworks.

Since I started coding RIA applications, I was always wondering what would the future look like? Would Flash-based apps conquer the web or will JavaScript (with a little help from Google) be the next step for the evolution of the www?
To be honest, I haven’t really found an answer yet.

As good as Flex can be used to efficiently write rich internet applications, so bad is its reputation on “real” programmers like C/C++, Java developers and even Webengineers that completely focus on pure JavaScript solutions not only cause of Flash’s serious security issues. Adobe seems to have both hands busy with fixing security holes while only a minority of the internet users runs the updaters frequently enough to be safe from a wide range of potential attacks on the flash plugin.

However, the one big advantage of Flash, which would be fast and easy video playback, seems to be melting down since W3C’s announcement that html5 will support real movie embedding and playback. I can already foresee how Youtube will be completely flash-free one day.

So, what if it is not necessary anymore to use Flash to embed a little video snipped on your favourite website? What is the reason for web developers to decide for the Flex framework instead of trying a pure JavaScript solution?

Yesterday, something funny came to my mind. I was thinking about Java and how great it failed to reach the desktop while becoming the standard technology if it comes up to web backends.
I find it funny, if you compare Java and Flash. Both rely on an own virtual machine that makes applications written for these framworks always look and behave the same, no matter which operating system you use.
You could also compare Javascript with C/C++ pretty much the same way. If you want to code applications in C, you have to keep many platform-specific things in mind. It’s pretty much the same with JavaScript. If you finished developing your brand new killer website, noone’s gonna say that it will work well in the Internet Explorer 6, only cause you tested it in Firefox 3.5. Even other “w3c-compatible” browsers like Safari, Opera or Chrome might give you a headache as a webengineer. To avoid these problems, most people rely on frameworks like Dojo, GWT and/or jQuery but I’d lie if I said that these would guarantee a clean and fluent development of web applications without any problems. On top of this, one needs to learn how to handle these frameworks, which is very often a pain in the a**.

Since the release of the sourcecode of Google’s Chrome OS, I can finally see how web applications could start conquering over classic desktop applications and I’m really exited about it.

Just imagine, you’d had to write an app for Chrome OS, like let’s say an online graphic editor like Photoshop, but on the web. Why’d you choose Ajax over Flash? Remember that using the Flex framework will speed up the development process a lot since so many components already work out of the box like menu-bars, for instance.

I hope Flash will do better here than Java did in the desktop world and become the cross-platform development enviroment for Chrome OS but without the drawbacks of Java like a slow loading time and low application performance.