Week 14 – Building Your Own Amazon CloudWatch Monitor in 5 Steps
Amazon EC2 offers the CloudWatch service to monitor cloud instances as well as load balancers. While this service comes at some cost (0,015$/hour/instance) it offers useful infrastructure metrics about the performance of your EC2 infrastructure. While there are commercial and free tools out there which provide this service, you might not want to invest in them or add another tool to your monitoring infrastructure. This post will provide step-by-step guidance on how to extend your monitoring solution to retrieve cloud metrics. The code sample is based on the free and open-source dynaTrace plugin for agent-less cloud monitoring. Some parts however have been simplified or omitted in tutorial. The major parts that are missing in this sample are dynamic discovery of EC2 instances and an algorithm which is a bit more reliable and accurate in retrieving monitoring data.
Step 1 – Basic Infrastructure
So let’s get started by setting up our basic infrastructure. First we need to download the Java Library for Amazon CloudWatch . Alternatively you can create your own Web Service stubs or simply use the REST interface. For simplicity we rely on the ready-to-use library provided by Amazon. The we create a Java class for our cloud monitor which implements the basic functionality we need. For brevity I will omit any imports needed – in Eclipse CTRL – SHIFT – O will do the job
public class CloudWatchMonitor { private static class MeasureSet{ public Calendar timestamp; public HashMap<String, Double> measures = new HashMap<String, Double>(); @Override public int compareTo(MeasureSet compare) { return (int) (timestamp.getTimeInMillis() - compare.timestamp.getTimeInMillis()); } public void setMeasure(String measureName, double value) { measures.put(measureName, value); } public Set<String> getMeasureNames() { return measures.keySet(); } public double getMeasure(String measureName) { return measures.get(measureName); } } private String instanceId; private AmazonCloudWatchClient cloudWatchClient; public static void main(String... args) throws Exception { CloudWatchMonitor monitor = new CloudWatchMonitor("<instanceName>", Credentials.accessKeyId, Credentials.secretAccessKey); for (;;) { MeasureSet measureSet = monitor.retrieveMeasureSet(measureNames); if (measureSet != null) { printMeasureSet(measureSet); } Thread.sleep(60000); } } public CloudWatchMonitor(String instanceId, String accessKeyId, String secretAccessKey) { cloudWatchClient = new AmazonCloudWatchClient(accessKeyId, secretAccessKey); this.instanceId = instanceId; } }
So what have we done? We defined the CloudWatchMonitor which will contain all our logic. The main method simply queries every minute for new measures and prints them. We have chosen an interval of one minute as CloudWatch provides accuracy to one-minute intervals. Additionally, we defined the inner MeasureSet class which represent a set of measures collected for a given timestamp. We have used a HashMap to make the implementation more generic. The same is true for the retrieveMeasureSet method which takes the measures it retrieves as an input. Finally we defined the constructor of our monitor to create an instance of an AmazonCloudWatchClient – this is supplied by the Amazon library we use - and store the instanceID of the EC2 instance to monitor. The accessKeyID and secretAccessKey are the credentials provided for your Amazon EC2 account.
Step 2 – Retrieve Monitoring Information
Now we have to implement the retrieveMeasureSet method which is the core of our implementation. As there are quite a number of things we have to do, I will split the implementation of this method into several parts. We start by creating a GetMetricStatisticsRequest object which contains all information which data we are going to request. First we set the namespace of the metrics which in our case is AWS/EC2 (in case we want to retrieve load balancer metrics it would be AWS/ELB). Next we define which statistical values we want to retrieve. Then we define the period of the monitoring data. In our case this is one minute. If you want aggregated data you can specify any multiple of 60. Then we define which measure aggregates we want to retrieve. CloudWatch offers average, minimum and maximum values. As our aggregation will only contain one data point all of them will be the same. Therefore we only retrieve the average.
public MeasureSet retrieveMeasureSet(ArrayList<String> measureNames) throws AmazonCloudWatchException, ParseException { GetMetricStatisticsRequest getMetricRequest = new GetMetricStatisticsRequest(); getMetricRequest.setNamespace("AWS/EC2"); getMetricRequest.setPeriod(60); ArrayList<String> stats = new ArrayList<String>(); stats.add("Average"); getMetricRequest.setStatistics(stats); ArrayList<Dimension> dimensions = new ArrayList<Dimension>(); dimensions.add(new Dimension("InstanceId", instanceId)); getMetricRequest.setDimensions(dimensions);
Next we have to define the time frame for which we want to retrieve monitoring data. This code looks a bit complex simply because we have to do some number formatting here. CloudWatch expects the time in a special format and all date values in ISO 8601 format which use UTC and looks like this 2010-04-22T19:12:59Z. Therefore, we have to get the current UTC time and format the date strings in the proper format. We take the current time as the end time and the start time is 10 minutes back in the past. Why are we doing this? The reason is that CloudWatch data is written asynchronously and the latest metrics we will get will be a couple of minutes in the past. If we set the start time to one minute in the past we would not get any metrics.
String dateFormatString = "%1$tY-%1$tm-%1$tdT%1tH:%1$tM:%1$tSZ"; GregorianCalendar calendar = new GregorianCalendar(TimeZone.getTimeZone("UTC")); calendar.add(GregorianCalendar.SECOND, -1 * calendar.get(GregorianCalendar.SECOND)); getMetricRequest.setEndTime(String.format(dateFormatString, calendar)); calendar.add(GregorianCalendar.MINUTE, -10); getMetricRequest.setStartTime(String.format(dateFormatString, calendar));
Additionally we have to add the following code to the constructor to calculate our UTC offset and define the timezone member field.
TimeZone zone = TimeZone.getDefault(); timeOffset = zone.getOffset(new Date().getTime()) / (1000 * 3600);
The next thing we have to do now is retrieve the actual metrics. As we will get more than one measurement we have to store them to later on select the latest measurement. The inconvenient part here is that the CloudWatch API does not allow us to retrieve more than one timer at once. Therefore we have to make a request for each metric we want to retrieve. Additionally you will notice some possibly cryptic date parsing and calculation. What we do here is parse the date string we get back from Amazon and create a calendar object. The tricky part is that we will have to add (or subtract) the offset of our current timezone to UTC. The formatter is defined as private DateFormat formatter = new SimpleDateFormat(“yyyy-MM-dd’T'HH:mm:SS’Z'”);
HashMap<Long, MeasureSet> measureSets = new HashMap<Long, MeasureSet>(); for (String measureName : measureNames) { getMetricRequest.setMeasureName(measureName); GetMetricStatisticsResponse metricStatistics = cloudWatchClient.getMetricStatistics(getMetricRequest); if (metricStatistics.isSetGetMetricStatisticsResult()) { List<Datapoint> datapoints = metricStatistics.getGetMetricStatisticsResult().getDatapoints(); for (Datapoint point : datapoints) { Calendar cal = new GregorianCalendar(); cal.setTime(formatter.parse(point.getTimestamp())); cal.add(GregorianCalendar.HOUR, timeOffset); MeasureSet measureSet = measureSets.get(cal.getTimeInMillis()); if (measureSet == null) { measureSet = new MeasureSet(); measureSet.timestamp = cal; measureSets.put(cal.getTimeInMillis(), measureSet); } measureSet.setMeasure(measureName, point.getAverage()); }
The last part is to retrieve the latest available measurements and return them. Therefore we will simply sort the measurements and return the latest one.
ArrayList<MeasureSet> sortedMeasureSets = new ArrayList<MeasureSet>(measureSets.values()); if (sortedMeasureSets.size() == 0) { return null; } else { Collections.sort(sortedMeasureSets); return sortedMeasureSets.get(sortedMeasureSets.size() - 1); }
In order to make sorting work we have to make the MeasuresSet implement comparable
private static class MeasureSet implements Comparable<MeasureSet> { @Override public int compareTo(MeasureSet compare) { return (int) (timestamp.getTimeInMillis() - compare.timestamp.getTimeInMillis()); } // other code omitted }
Step 3 – Printing the Results
Last we have to print the results to the console. This code here is pretty straightforward and shown below.
public static void printMeasureSet(MeasureSet measureSet) { System.out.println(String.format("%1$tY-%1$tm-%1$td %1tH:%1$tM:%1$tS", measureSet.timestamp)); for (String measureName : measureSet.getMeasureNames()) { System.out.println(measureName + ": " + measureSet.getMeasure(measureName)); } }
Step 4 – Defining the Metrics to Retrieve
Our code is now nearly complete the only thing we have to do is define which metrics we want to retrieve. We can either pass them as command-line parameters or explicitly specify them. CloudWatch supports the following parameters for EC2 instances:
- CPUUtilization
- NetworkIn
- NetworkOut
- DiskReadBytes
- DiskWriteBytes
- DiskReadOperations
Step 5 – Visualizing and Storing the Data
As you most likely do not want to look at the data on your console, the final step is to visualize and store the data. How to implement this depends on the monitoring infrastructure you are using- Below you can see a sample of how this data looks in dynaTrace.
Conclusion
Building your own CloudWatch monitoring is pretty easy. The metrics provided enable an initial understanding how your EC infrastructure is behaving. These metrics are also input to the Amazon EC2 Auto Scaling infrastructure.
If you want to read more articles like this visit dynaTrace 2010 Application Performance Almanac




Hi Alois,
Could you please send the working code of the above post. Since i am trying to work with Elastic cloud watch, I am not able to do it, Your code helped me lot, but it could not worked with me, Please send the working copy of the code. The issues i am facing are :
1. Inside main method, you are using variable measureNames , which is not defined?
2. I am always getting , datapoint is null or size 0 at this line : for (Datapoint point : datapoints);
3. I am from india, so might be start time and end time also not functioning well for me.
Please resolve the above issues and help me, its very urgent.
Karunjay Anand
Technology Lead,
Artoo India
Hello Karunjay,
Ad 1: Have a look at Step 4. Here I describe the potential measure names you can use for your query. Just add the measures you want to retrieve to the list.
Ad 2: It seems you are passing in no measureNames (see above). This leads to no datapoints being returned. This should solve the problem. I omitted some data integrity checks in my code for brevity reasons.
Ad 3: The timezone matching code above actually should resolve your problem
I will try to put the code online. However it will take my some time to figure out the best way to do this.
Hi Alois,
Thanks for the reply.
After changing the MeasureNames, still not working.
Code here :
private static ArrayList measureNames = new ArrayList();
measureNames.add(“CPUUtilization”);
measureNames.add(“NetworkIn”);
measureNames.add(“NetworkOut”);
measureNames.add(“DiskReadBytes”);
measureNames.add(“DiskWriteBytes”);
measureNames.add(“DiskReadOperations”);
and for loop is taking one by one MeasureNames, but still the response is coming like this :
DiskReadOperations
4ea30428-7013-11df-bd33-c726ca094f5a
Please provide your input,
And how should i change my start time and end time in context of india , if i stary my instance at 6 PM at india?
Waiting for your valuable input,
Thanks
Karunjay Anand
Technology Lead
Artoo India
Hi Alois,,
I am waiting for your reply, whenever you feel free, please resolve my issue,
Thanks
Karunjay,
the code for changing the start time is listed above. Did you change the start time? For your convenience you should simple collect the results for the last 24 hours or so. Did you also use the right instanceID?
Thanks for the cloudwatch code. We use several cloud servers and this will definitely come in handy. I have been skeptical of choosing from the free sources out there and I had purchased a disaster recovery software just in case something were to go wrong. I am glad that I stumbled across your code. Now I can use this instead of the free services out there.
I can not thank you enough for the code. I work for the IT department & we have been given the task of finding a free cloud server to base our project off of, but there seems to be one too many draw-backs when using any of those free services. Our small Rancho Santa Fe company is going to implement your cloudwatch code (very slightly modified) for our upcoming multi-agent project. Much thanks from Dave in San Diego.
Hi! People are really well aware of global warming issue. The are somethings which can be implemented in our society to get rid of global warming. First of all we should grow trees and plantation which is a source of air. Secondly we should make ourself and our surrounding neat and clean. Thirdly we cannot misuse of water because water is life. The main thing is that the water level under the earth is decreasing. We can overcome this problem through that points. This is really a great site. Thanks. Wilma @ no fax payday loans
We were looking to move our sites to Amazon’s virtual servers so that we can take advantage of their cloud computing for some high-traffic content we’ll be serving, but I had no idea where to start… This is absolutely perfect. Thanks times a million.
Hello Karunjay,
Ad 1: Have a look at Step 4. Here I describe the potential measure names you can use for your query. Just add the measures you want to retrieve to the list.
Ad 2: It seems you are passing in no measureNames (see above). This leads to no datapoints being returned. This should solve the problem. I omitted some data integrity checks in my code for brevity reasons.
Ad 3: The timezone matching code above actually should resolve your problem
I will try to put the code online. However it will take my some time to figure out the best way to do this.
gehoorbescherming
exchange server 2010 training
microsoft training
This is helpful since we are moving our product acai berry detox soon to Amazon.com. Glad I stumbled upon this site. Thanks.
Thank you very much for a nice and cool article of amazon cloudwatch. It is an alternative post by you. That is so cute. This is very nice post! I will bookmark this blog.
the code for changing the start time is listed above. Did you change the start time? For your convenience you should simple collect the results for the last 24 hours or so. Did you also use the right instanceID?
The major parts that are missing in this sample are dynamic discovery of EC2 instances and an algorithm which is a bit more reliable and accurate in retrieving monitoring data.tHANKS
thanks for that explanation! i use it from now on! i had to google some stuff i don’t understand yet, but it works now
The information about building CloudWatch monitor is very good.it offers useful infrastructure metrics about the performance of your EC2 infrastructure.It’s a great post.Keep such type sharing continue.spanx
This say is existent appreciatable and thinking evolve. Mostly the vistors unit simple set of assembling. In myopic it is so awing. I one it really whatsoever. Thanks for the run.
It is an alternative post by you. That is so cute. This is very nice post! I will bookmark this blog.currency converter
This is a good post. This post gives truly quality information. I’m definitely going to look into it. Really very useful tips are provided here. thank you so much. Keep up the good works
Your blog shows how Techie you are? I appreciate you fabulous effort on Amazon Cloud monitor. Thanx a lot:)
Hi all. Please visit this website. There are many informations and many comments. it is useful
Alternatively you can create your own Web Service stubs or simply use the REST interface. For simplicity we rely on the ready-to-use library provided by Amazon. The we create a Java class for our cloud monitor which implements the basic functionality we need.Thanks
I can only agree. Really a great contribution. Unfortunately there are too few good blogs
best regards
Peter
Your observations are right on, I agree with most of what you say.Thanks so much for your comment. Thanks
Very good work on such topic. When I started to read article I was thinking about the way you present such great topic. By the way could you please post more article on such topic? I really appreciate it internetmarketingwithmark
I have been looking for a rare blog because I am tired of accessing almost the same topic discussed in a website. This blog is actually hitting what I want to expect. I am very glad that you are now providing the information where I am hunting for many days of surfing the net. Thanks for this post. You really hits the target!Gifts For Men
I am definitely tired of struggling to find relevant and intelligent commentary on this subject. Everyone nowadays seem to go to extremes to either drive home their viewpoint or suggest that everybody else in the globe is wrong and now you provide such valuable resource for free. Anyhow, I am blown away by your concept here, I will definitely subscribe to your RSS feed! hoover steamvac
I appreciate this one of a kind concept of yours very well organized.I want visit this to get more updates.
Spaced articles
Excellent work Alois..you’ve touched a real pain for many comapnies moving to the cloud..
I wonder if you’ve done (or dynatrace allow) the opposite; collect internal metrics (eg request processing time or throughput) from inside your Java application or server (say Jboss)and send it cloud watch? I’m interested in such scenario. it would be great if there is any insight?
We were looking to move our sites to Amazon’s virtual servers so that we can take advantage of their cloud computing for some high-traffic content we’ll be serving, but I had no idea where to start… This is absolutely perfect. Thanks times a million.
Veux simplement dire que votre site est étonnante. La clarté dans votre contenu est tout simplement spectaculaire, et je peux supposer que vous êtes un expert sur ce sujet. Eh bien avec votre permission me permettre de récupérer votre flux RSS à tenir à jour avec post à venir. Merci mille fois et s’il vous plaît suivre le travail gratifiant.
the code for changing the start time is listed above. Did you change the start time? For your convenience you should simple collect the results for the last 24 hours or so. Did you also use the right instanc
We were looking to move our sites to Amazon’s virtual servers so that we can take advantage of their cloud computing for some high-traffic content we’ll be serving, but I had no idea where to start… This is absolutely perfect. Thanks times a million.
The maximum number of datapoints that the Amazon CloudWatch service will return in a single GetMetricStatistics request is 1,440. If a request is made that would generate more datapoints than this amount, Amazon CloudWatch will return an error.
he code for changing the start time is listed above. Did you change the start time? For your convenience you should simple collect the results for the last 24 hours or so. Did you also use the right instanc
I abruptly stumbled on this great page while I was finding a metal louvre. I’m sure this web page invaluable. I wish to see new material as I browse the next time.
. I am sorry but I did not like this blog entry but I look forward to the next one.
I really enjoy reading comments and messages here from lot of bloggers thanks for your time guys.
Here post is really important and interesting.I love it very much because it has many valuable side.Thanks you very much for shearing this information.
I really like the Blog content, I like the information. This is what we are looking for. Thank for this and keep update the information .
Wow! This page is undoubtedly superb. Let me say thank you to my cousin, he’s working at a Brisbane Builder Firm and this man showed me your blog.
I am fond about businessmen like Andrew Reynolds, but I
spotted your blog just now. I am motivated of being
successful someday. Well, I’ve read about your blog and I
loved it. More power to you!
There is practically nothing which I adore more than unearthing unknown information. Finding out unpretentious yet noteworthy facts assist me improve as an individual. The composition on bushnell golf that I was skimming may seem to be paltry yet it presumes its own function. Either way, I enjoy putting reviews here and there. This is the reason why I have come by your impressive blog web page. I can frankly state that it has undoubtedly been fun.
There’s nothing at all which I adore more than finding out advanced facts. Finding out scanty yet important factors support me to grow as an individual. The write-up on christmas hampers that I was reading may seem nondescript yet it reckons its own objective. In any event, I adore posting comments everywhere. This is the reason why I have stopped by your exceptional blog website. I can truthfully say that it has certainly been pleasurable.
There is pretty much nothing which I appreciate more than unearthing unknown information and facts. Uncovering scanty yet vital information assist me develop as an individual. The column on bce pool table that I was skimming may seem insignificant yet it presumes its own purpose. Anyway, I love putting comments here and there. This is the why I have stopped by your amazing blog website. I can actually say that it has absolutely been pleasurable.
During my days off, I find comfort in going over write-ups on anything at all that I feel like reading. Just yesterday, I was reviewing a story on curtains online. It actually supported me gain more knowledge on the matter. At any rate, I even like posting opinions and sharing my notion so I chose to put a comment here. Thanks for letting me share my opinion and I hope to read features from this blog site soon enough.
I really like to go over commentaries on the internet because I actually dig up tons from them. The latest article which I was going through was on chocolate gifts. I always aspire to give assortment to my reading range. A few days ago, I was reading on the importance of online blog sites. This is the actual reason why I made a choice to sample placing opinions. Your online site is among the best web sites that I have definitely chanced upon so far. Many Thanks for letting me put a feedback on this website.
During my extra time, I obtain solace in studying write-ups on anything at all that I feel like reading. Yesterday, I was studying an article on flying lanterns. It genuinely allowed me obtain more understanding on the topic. At any rate, I also like setting remarks and sharing my thought so I made a decision to place a comment here. Thanks for letting me share my impression and I hope to discover write-ups from this blog page soon enough.
During my extra time, I obtain solace in studying write-ups on anything at all that I feel like reading. Yesterday, I was studying an article on flying lanterns. It genuinely allowed me obtain more understanding on the topic. At any rate, I also like setting remarks and sharing my thought so I made a decision to place a comment here. Thanks for letting me share my impression and I hope to discover write-ups from this blog page soon enough.
Our stretch limousine represents class, elegance and luxury. Whether you’re getting married, having a night out with the girls or any occasion. Our Stretch limousine is your best choice for. Our new car introductory offer is a free bottle of champagne with every booking plus you’ll get a fantastic hire rate. The longer you hire the cheaper it gets per hour.
I am very enjoyed for this blog. Its an informative topic. It help me very much to solve some problems. Its opportunity are so fantastic and working style so speedy. I think it may be help all of you. Thanks a lot for enjoying this beauty blog with me. I am appreciating it very much! Looking forward to another great blog. Good luck to the author! all the best!
adult toys
When I first launched my online business, I desired it to succeed however I have never dreamed it to prosper like this. The Business Plan Writer that I obtained seriously had a ton to do with my accomplishment. In any case, I wish that the many other businessmen and people right here comprehend the importance of finding an excellent one. Thanks for enabling me put up an opinion here. Good Luck!
I was browsing for some reports online about property to let reading when my guitar tutor instructed me to consider this web page. I right away realized why she was energized to share it to me.
Nothing at all is more fulfilling in this existence than gradual learning. That is why I continually research more information on the internet. Being a virtual writer, I usually have to study varied products. Right now, I’m examining external timber cladding. But nevertheless, I found this page so I decided to post a comment. I stumbled onto this page long before although I forgot about it. Right now, I am absolutely going to book mark it for future reference.
I was really looking for websites that provides an office design and it’s a good thing that I found your website. I didn’t get what I was initially looking for but I obtained a lot more than what I wanted. Your website is good! Thanks a lot for sharing it.
I was just conversing with my employer on the phone regarding online recruitment software and browsing several things in the laptop when I discovered your internet site. Fantastic! I think that you have a great site.
There is nothing more favorable within this life than sustained education. This is exactly why I persistently explore more facts on cyberspace. Being a writer, I usually have to research unique circumstances. Currently, I’m exploring behavioural profiling. Interestingly, I came upon this website and so I chose to place a simple message. I stumbled upon this blog long before however I forgot it. This time, I am surely gonna store it for future reference.
I am pleased with your good work. You share very helpful information.
As a computer expert I’m sure that one can find a lot of Free Competitions on the internet. Bingo websites tend to be one of those. Thanks for letting me drop by your blog site. It’s really nice here.
I have small knowledge about you topics but i know how to install such kind of project. I do agree with luka4get. He gave one of the approach. @ NH Weddings
It’s my day off from the office and I decided that I’d stay indoors for a change. I was just searching through a few internet sites that interest me and I came upon this site. The Wife Swapping site that I also found was absolutely enjoyable. I had a blast reviewing their blog posts and comments. Anyhow, I reckon that this website here is just remarkable. Many Thanks for letting me hang around on this webpage. I will definitely look forward to coming back when I have an additional time off at home.
Browsing the World Wide Web tends to be among my most preferred past-times. Since I furthermore like reading, I continually have an excellent time going through online posts on the internet. Some commentaries that I select to study have to do with SEO options and google places. Electronic marketing is 1 topic that I would like to excel in. Anyway, Thanks a lot for allowing me to publish this comment here. I totally relish it.
In these days I’ll try to read about Socially Responsible Products because I like to know about them and how they help me in my social life but fortunately I find these article which is also very Informative for me.
Geez! It was really a good idea that you created this site. I’m over viewing about Andrew Reynolds from the other web site, that’s the reason why I have my eyes in your site now. I’ll be reading more of your site!
Very well written article indeed, it really helps me with my programming skills, thank you so much for sharing such information with us The green energy company, i hope we will see more from author in the future.
Just after talking with a good friend online concerning web copywriter brisbane, I unintentionally came upon your page and I think your web posts in your web site is noteworthy. Carry on!
I am a programmer and i really thank those you post related to my field of study thanks a lot.
Thank you very much for this article. It´s what i was looking for.
Regards!!!
Wow! that was absolutely a fabulous, Your blog provided us with valuable information to utilize.
I like your post and all you share with us is up to date and quite informative. I hope you will spend much time commenting more to make us happier.
I got lot of information from this post. Seriously excellent entry to study on.! I am actually intrigued with this post. Searching forward for more information.