Develop for the Cloud, in the Cloud.

As a developer I’m lazy. I like to have all my favorite tools when I’m creating software, and these include windowing systems and IDE’s, in particular Eclipse. So when I go to develop software for the AWS cloud I find myself doing most of my work on my desktop machine (be it MacOs/Linux or Windows), and faking the Cloud bit using multiple tomcat/jetty servers. There always comes a point in a project where you can’t fake it any more. You have to get into the Cloud and deploy your software.  As much as the tools have matured for this in recent years, this last step is still very much “throw it over the wall and hope it works”.  As I rebuild my infrastructure, post Netflix, the first thing I wanted to do was get a remote Linux/Ubuntu desktop up in the Cloud so that I could develop for the Cloud, in the Cloud.  This turns out to be a not-so-easy thing to bootstrap, (unless I’ve really missed something obvious). My first thought was: Amazon Marketplace Workspaces. But these only support Windows at present, not what I want. Next I spent a whole day scavenging the internet for a workable Ubuntu desktop AMI, then how to build one, then how to build one with NoMachine 3.5 built in (since the NoMachine server/client combination outperforms Vnc by orders of magnitudes, and is SSH secure). Finally, after much grinding noise, I was able to stand up a basic AMI in the cloud, which I am making publicly available for anyone to use. It is preconfigured with NoMachine 3.5, but with no users so you will have to login and do the following:

$ sudo adduser ec2usr
$ sudo adduser ec2usr admin
You can login using the ssh client built into the AWS console with the keypair that you should have generated and used when you launched the image.
Remember to create this user and group assignment before trying to login using the NX password protocol (or some user/password of your own).
The AMI is ami-cdff8cfd and it is located in us-west-2. Ubuntu 12.04, runs on m1.small and larger (t1.micro does not have enough memory). Includes Eclipse.   Important: when logging in using NX client choose ssh/port22 as the protocol, but use the NX login to avoid having to generate public/private keypairs (until you have bootstrapped the system). Free for use, no support, etc.etc.etc. I do think there is a need for something like this in the AWS marketplace.

Notes: The main resource I drew on for this was the following excellent blog post (among others): plus I had to dig up old binaries for the nx 3.5 (manager,node,server) here:

Cloudetics — Studying The Cloud

Today I’m kicking off an open-source project to build/rebuild a real-time analytics system for cloud applications based on the work I created while at Netflix.  This system will support advanced analytics (think R) on real-time data (think 1 second sampling) and produce answers in real-time (think sub-30 seconds).

The goal is to tie together the latest technologies in streaming-data and basic analysis (Apache Spark) with powerful data analytics tools (R/OpenCPU) to create an infrastructure that can be used to monitor, diagnose, and control high-volume cloud systems.

The master repository is on GitHub: (it’s currently empty but this will change fast).

My overall goal is to build an open-source community around this effort to help drive cloud systems operations away from the focus on monitoring, toward a focus on analysis and decision making based on data.

– Simon Tuffs