I am biased (I work for Keynote Systems) but having done performance management for 10 years, using a synthetic transaction solves many of the variability/sample issues of other approaches.

Performance management is best done in a controlled manner and if you can guarantee that your sample is always the same (ie. same hardware, software, locations, network etc) than detecting even slight changes (that you can influence) is possible.

The weakness with this approach is that you can’t see every user and that some transactions cannot be done synthetically (think of applying for a credit card using unique SSN#). In those cases a instrumentation of some sort (or a network sniffer – though with CDN’s and other hybrid hosting I am not sure this approach is going to be valid for much longer) is the best approach.

We have been working lately on combing our metrics from the browser with the Dynatrace Ajax Edition tool with great success with customers. Now when there is a client/browser delay we can determine exactly what JS or Web Service call is causing the problem. The combination is very powerful. Adding a back-end view of application health is the logical next step. Doing this would provide the browser  network  application view which is critical.

Ben Rushlo
Director – Keynote Systems

This is the most comprehensive post regarding “end-user experience monitoring” (as you call it) yet, to my knowledge. Awesome!

Also, I agree with all points in this article. You even managed to point me to some things I didn’t know of yet:

the network-sniffing style performance monitors such as Coradiant
Speed Tracer Logging API

I’ve only got a minor addition to make. Jiffy is a system that’s similar to Episodes, but predates Episodes by some time. It was developed by whitepages.com for their specific performance monitoring goals. One of the goals was to be able to monitor every single page load. So Jiffy did just that. It was even able to analyze the log results in near real-time. The most important component was how the log was kept: it simply used the Apache web server’s logging capabilities. They wrote a script to parse and process those log files, in Perl if I remember correctly. They didn’t even have to apply fancy tricks like sending the logging to multiple servers. And most importantly: they generate millions of page views per month.
That strongly suggests that true real-time performance monitoring of actual end-users, and every actual end-user (!), is perfectly doable.

I do have a general request though: together with the people active in this area of performance, I’d like to define a definitive set of terms. The difference in naming strongly decreases general understanding, discoverability of articles and perhaps not least importantly, marketability I.e.:
- Steve Souders calls this “front-end performance”
- I called it “page loading performance” because it’s more accurate than “front-end performance”. It also distinguishes more clearly among page loading versus page rendering. The monitoring I called “page loading performance monitoring” as the general term and “real-world page loading performance monitoring” when the monitoring runs from a real-world perspective, i.e., in the end-user’s browser.
- You call “end-user experience monitoring” what I called “real-world page loading performance monitoring”.

I think the entirety of all web site performance optimizations should be named “Web Performance Optimization” (WPO) — as Steve Souders already called it.
An initial proposal toward a full-fledged taxonomy would then be:

Web Performance Optimization

page loading performance (CSS/JS + loading of resources; page loading therefore always consists of both client and server; putting client-side performance separately makes no sense since loading of resources is always necessary)

synthetic transactions
organic/real-world transactions (i.e. monitoring these would equal what you called “end-user experience monitoring)

server-side performance (whatever generates output + HTTP server, possibly behind a reverse proxy and whatnot)

synthetic transactions
organic/real-world transactions

This is most likely missing a lot of refinement, but roughly, it should be correct.