SpamapS.org – Full Frontal Nerdity

One of the really cool things about using the cloud, and especially juju, is that it instantly enables things that often times take a lot of thought to even try out in traditional environments. While I was developing some little PHP apps “back in the day”, I knew eventually they’d need to go to more than one server, but testing them for that meant, well, finding and configuring multiple servers. Even with VMs, I had to go allocate one and configure it. Oops, I’m out of time, throw it on one server, pray, move to next task.

This left a very serious question in my mind.. “When the time comes, will my app actually scale?”

Have I forgotten some huge piece to make sure it is stateless, or will it scale horizontally the way I intended it to? Things have changed though, and now we have the ability to start virtual machines via an API on several providers, and actually *test* whether our app will scale.

This brings us to our story. Recently, Nick Barcet created a juju charm for Limesurvey. This is a really cool little app that lets users create rich, multi faceted surveys and invite the internet to vote on things, answer questions, etc. etc. This is your standard “LAMP” application, and it seems written in a way that will allow it to scale out.

However, when Nick submitted the charm for the official juju charms collection, I wanted to see if it actually would scale the way I knew LAMP apps should. So, I fired up juju on ec2, threw in some haproxy, and related it to my limesurvey service, and then started adding units. This is incredibly simple with juju:

juju deploy --repository charms local:mysql

juju deploy --repository charms local:limesurvey

juju deploy --repository charms local:haproxy

juju add-relation mysql limesurvey

juju add-relation limesurvey haproxy

juju add-unit limesurvey

juju expose haproxy

Lo and behold, it didn’t scale. There were a few issues with the default recommendations of limesurvey that Nick had encoded into the charm. These were simple things, like assuming that the local hostname would be the hostname people use to access the site.

Once that was solved, there were some other scaling problems immediately revealed. First on the ticket was that Limesurvey, by default, uses MyISAM for its storage engine in MySQL. This is a huge mistake, and I can’t imagine why *anybody* would use MyISAM in a modern application. MyISAM uses a “whole table” locking scheme for both reads and writes, so whenever anything writes to any part of the table, all reads and writes must wait for that to finish. InnoDB, available since MySQL 4.0, and the default storage engine for MySQL 5.5 and later, doesn’t suffer from this problem as it implements an MVCC model and row-level locks to allow concurrent reads and writes.

The MyISAM locks caused request timeouts when I pointed siege at the load balancer, because too many requests were stacking up waiting for updates to complete before even reading from the table. This is especially critical on something like the session storage that limesurvey does in the database, as it effectively meant that only one user can do anything at a time with the database.

Scalability testing in 10 minutes or less, with a server investment of about $1US. Who knew it could be this easy? Granted, I stopped at three app server nodes, and we didn’t even get to scaling out the database (something limesurvey doesn’t really have native support for). But these are things that are already solved, and that have been encoded in charms already. Now we just have to suggest small app changes to allow users to take advantage of all those well know best practices sitting in charms.

(check the bug comments for the results, I’d be interested if somebody wants to repeat the test).

So, in a situation where one needs to deploy now, and scale later, I think juju will prove quite useful. It should be on anybody’s radar who wants to get off the ground quickly.

Just a quick note.. I was watching Artur Bergman’s rant about Full Stack awareness at Velocity Europe with glee, until I saw how we, the Ubuntu devs, had drawn his ire for breaking something so simple, ‘restart’.

I’ve heard this before, and I realized that its true! Restart should ignore that the service is stopped, and just start it, otherwise just restart it as I requested.

So, first off, instead of:

/etc/init.d/apache2 restart

or (for upstart controlled services)

restart mysql

get in the habit of doing:

service apache2 restart

This command will do the right thing more often than not (including clearing out the environment for sysvinit jobs, which can help solve the “why does the service only work when I restart it manually”). And as of 12.04, it will start doing the right thing with restart too.

Its not perfect, and there are still bugs in upstart, but the version of ‘sysvinit’ that I just uploaded to Precise Pangolin (the future 12.04 / LTS release of Ubuntu) at least gives scripters and sysadmins a chance at uniformity by making upstart jobs and sysvinit scripts do the same thing with restart. I don’t think we can backport this all the way to 10.04, so sorry for that. However, please, users, keep the rants coming.. we need them!

A few weeks ago I gave a live demo during Canonical CEO Jane Silber’s keynote at the Essex OpenStack Conference, which was held in Boston October 4-7 (See my previous post for details of the conference and summit). The demo was meant to showcase our new favorite cloud technology at Canonical, juju. In order to do this, we deployed hadoop on top of our private OpenStack cloud (also deployed earlier in the week via juju and Ubuntu Orchestra) and fed it a “real” workload (a big giant chunk of data to sort) in less than 5 minutes.

I’ve had a few requests to explain how it works, so, here is a step by step on how to repeat said demo.

First, you need to setup juju to be able to talk to your cloud. The simplest way to do this is to sign up for an AWS account on Amazon, and get EC2 credentials (a secret key and a key ID is needed).

If you install juju in Ubuntu 11.10, or from the daily build PPA in any other release, you’ll get a skeleton environments.yaml just by running ‘juju’.

Once this is done, edit ~/.juju/environments.yaml to add your access-key: and secret-key:. Optionally, you can set them in AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the environment.

Now, you need the “magic” bit that turns juju status changes into commands for the “gource” source code visualization tool. Its available here:

http://bazaar.launchpad.net/~clint-fewbar/juju/gource-output/view/head:/misc/status2gource.py

(wgettable here)

You’ll also need to install the ‘gource’ visualization tool. I only tried this on Ubuntu 11.10, but it is available on other releases as well.

Make sure your desired target environment is either the only one in .juju/environments.yaml, or set to be the default with ‘default: xxxx’ at the root of the file. You need ‘juju status’ to return something meaningful (after bootstrap) for status2gource.py to work.

Now, in its own terminal, run this, note that cof_orange_hex.png is part of the official Ubuntu logo packs, but I forget where I got that. You may omit that commandline argument if you like, and a generic “person” image will be used.

python -u status2gource.py | gource --highlight-dirs \
--file-idle-time 1000000 \
--log-format custom \
--default-user-image cof_orange_hex.png \
--user-friction 0.5 \
-

This will not show anything until juju bootstrap is done and ‘juju status’ shows the machine 0 running. If you already have services deployed, it should build the tree rapidly.

So next if you haven’t done it already

juju bootstrap

Once your instance starts up, you should see a gource window pop up and the first two bits, the bootstrap node and the machine 0 node, will be added.

Once this is done, you can just deploy/add-relation/etc. to your heart’s content.

To setup a local repo of charms, we did this:

mkdir charms
bzr init-repo charms/oneiric
cd charms/oneiric
bzr branch lp:~mark-mims/+junk/charm-hadoop-master hadoop-master
bzr branch lp:~mark-mims/+junk/charm-hadoop-slave hadoop-slave
bzr branch lp:~mark-mims/+junk/charm-ganglia ganglia

Those particular charms were specifically made for the demo, but most of the changes have been folded back in to the main “charm collection”, so you can probabl change lp:~mark-mims/+junk/charm- to lp:charm/.

You will also need a file in your current directory called ‘config.yaml’ with this content:

namenode: job_size: 100000 job_maps: 10 job_reduces: 10 job_data_dir: in_one job_output_dir: out_one 

These numbers heavily control how the job runs with 1 or 100 hadoop instances. If you want to spend a couple of bucks in Amazon, and fire up 20 nodes, then raise job_maps to 100 and job_reduces to 100. Also job_size to 10000000. Otherwise its over very fast!

We started the demo after bootstrap was already done, so the next step is to deploy Hadoop/HDFS and ganglia to keep an eye on the nodes as they came up.

juju deploy --repository . --config config.yaml hadoop-master namenode
juju deploy --repository . hadoop-slave datacluster
juju deploy --repository . ganglia jobmonitor
juju add-relation namenode datacluster
juju add-relation datacluster jobmonitor
juju expose jobmonitor

This should get you a tree in gource showing all of the machines, services, and relations that are setup.

You can scale out hadoop next with this command. Here I only create 4, but it could be 100.. depending on how fast you need your data map/reduced.

for i in 1 2 3 4 ; do juju add-unit datacluster ; done

Finally, to start the teragen/terasort:

juju ssh namenode/0

$ sudo su -u hdfs sh /usr/lib/hadoop/terasort.sh

You may also want to note the hostname of the machine assigned to the jobmonitor node so you can bring it up in a browser. You will be able to see it in ‘juju status’.

Its worth noting that we had a fail rate of about 1 in 20 tries while practicing the demo because of this bug:

https://bugs.launchpad.net/juju/+bug/872378

This causes the “juju expose jobmonitor” to fail, which means you may not be able to reach the ganglia instance. You can fix this by stopping/starting the provisioning agent on the bootstrap node. That is easier said than done, but can be scripted. Its fixed in juju’s trunk, so if you are using the daily build, not the distro version, you shouldn’t see that issue.

So once you’re done, you’ll probably want to get rid of all these nodes you’ve created. Juju has a tool that strips everything down that it has brought up, which can be dangerous if you have data on the nodes, so be careful!


juju destroy-environment


It does not have a ‘–force’ or ‘-y’, by design. Make sure to keep the gource running when you do this. Say ‘y’, and then enjoy the show at the end.

I’d be interested to hear from anybody who is brave enough to try this how their experience is!

According to NASA, 70% of the earth is covered by clouds. Apparently, at least 70% of our computing needs can be covered by clouds as well. That seems to be the shared belief by the rather large crowd that gathered in Boston last week for the Essex edition of the OpenStack Design Summit and subsequent OpenStack Conference.

The amount of energy and corporate investment in OpenStack is staggering when one considers that it didn’t exist 2 years ago, and didn’t really do much more than spawn VM’s and store objects until this month with the Diablo release, which added some more capabilities, but from my point of view, mostly just refined those abilities and set the stage for the future.

Attending as a member of the Ubuntu Server team and a Canonical employee was quite a gratifying experience. Ubuntu Server has been the platform of choice for OpenStack’s development, and that has definitely led to a lot of people running OpenStack on Ubuntu Server. Its always nice to hear that your work is part of something greater.

On the surface, one might be concerned at a lack of vision in the OpenStack project. With so many competing interests, it may appear that it has no clear vision and is just growing toward the latest source of funding or food, much like an amoeba swallowing up its next meal. But the leadership of the project seems to understand that there is still a much greater mission here, that without intense focus the project will expend enormous energy and accomplish little more than falling a little less behind established players in the marketplace.

Its a bit vindicating for one of my more intense current interests, Juju, that others who are close to this discussion, like OpenStackers, are thinking along the same lines. In talking with Puppet and Chef guys and with people who are using the cloud, its clear to me that my hunch is right; chef and puppet are not really the same thing as Juju. The new project from Cisco, Donabe, seems to be thinking exactly like Juju, wanting to encapsulate and describe each service in what they call “Network Containers”. Also I’m told the desires of the Neutronium PaaS project are pretty similar as well.

Ultimately we don’t think that the current limitations of known PaaS stacks are always worth the effort to integrate with them. We do want to have a lot of the same capabilities without having to duplicate all the effort to set them up. We want to be able to make use of well understood technologies without having to understand every detail of their deployment and configuration. If I want to make use of MySQL or memcached, I should understand how they work, but I shouldn’t have to duplicate the effort that others have had to put in to make them work.

Chef and Puppet have made some inroads into this by making such things highly repeatable and getting them all into source control. However, its my belief that their implementations both limit the network effect that they can have to build up a full set of sharable services. Juju, I think, will really be a boost to those who have spent a lot on solid config management, as that config management will be easy to chop up into Juju charms, and then that will open up all the other existing charms for immediate use in such a shop.

Getting back to how this relates to OpenStack, it was also quite exhilarating to do a live keynote demo of Juju in all of its alpha glory. To raise the tightrope a little higher, it was driving OpenStack Diablo, which some might call beta-quality. We also got rid of the safety nets entirely, and had it running on top of Ubuntu 11.10 (pre-release). We had a few kinks through the week, but the awesome team I had around me was able to iron them all out and made both our CEO, Jane Silber, and me look very good up there. That includes my fellow server team members, the OpenStack developers, Canonical IS pro’s, the Juju dev team, and my main collaborator in the whole thing, Jorge Castro.

I hope to attend the next ODS, to see how much closer OpenStack is to completing its mission in 6 months. What is that mission currently? Quite simple really.. the mission is, figure out the mission.

I took a little trip down to San Diego yesterday to see what these CloudCamp events are all about. There are so many, and they’re all over, I figure its a good chance to take a look at what might be the “Common man’s” view of the cloud. I spend so much time talking to people at a really deep level about what the cloud is, why we like it, why we hate it, etc. This “un-conference” was more about bringing a lot of that information, distilled for business owners and professionals who need to learn more about “this cloud thing”.

The lightning talks were quite basic. The most interesting one was given by a former lawyer who now runs IT for a medium sized law firm. Private cloud saves him money because he can now make a direct charge back to a client when they are taking up storage and computing space. This also allows him to keep his infrastructure more managable because they tend to give up resources more readily when there is a direct chargeback as opposed to just general service fees that try to cover this.

There was a breakout session about SQL vs. NoSQL. I joined and was shocked at how dominant the Microsoft representative was. She certainly tried to convince us “this isn’t about SQL Azure, its about SQL vs. NoSQL” but it was pretty much about all the things that suck more than SQL Azure, and about not mentioning anything that might compete directly with it. I brought up things like Drizzle, Cassandra, HDFS, Xeround, MongoDB, and MogileFS. These were all swiftly moved past, and not written on the white board. Her focus was on how SimpleDB differs from Amazon RDS, and how Microsoft Azure has its own key/value/column store for their cloud. The room was overpowered into silence for the most part.. there were about 20 developers and IT manager types in the room and they had no idea how this was going to help them take advantage of IaaS or PaaS clouds. I felt the session was interesting, but ultimately, completely pwned by the Microsoft rep. She ended by showing off 3D effects in their Silverlight based management tool. Anybody impressed deserves what they get, quite honestly.

One good thing that did come out of that session was the ensuing discussion for it where I ended up talking with a gentleman from a local San Diego startup that was just acquired. This is a startup of 3 people that is 100% in Amazon EC2 on Ubuntu with PHP and MySQL. They have their services spread accross 3 regions and were not affected at all by the recent outtages in us-east-1. Their feeling on the SQL Azure folks is that its for people who have money to burn. For him, he spends $3000 a month and it is entirely on EC2 instances and S3/EBS storage. The audience was stunned that it was so cheap, and that it was so easy to scale up and down as they add/remove clients. He echoed something that the MS guys said too.. that because their app was architected this way from the beginning, it was extremely cost effective, and wouldn’t even really save much money if they leased or owned servers instead of leasing instances, since they can calculate the costs and pass them directly on to the clients with this model, and their commitment is zero.

Later on I proposed a breakout session on how repeatable is your infrastructure (basically, infrastructure as code). There was almost no interest, as this was a very business oriented un-conference. The few people who attended were just using AMI’s to do everything. When something breaks, they fix it with parallel-ssh. For the one person who was using Windows in the cloud, he had no SSH, so fixing any system problems meant re-deploying his new AMI over and over.

Overall I thought it was interesting to see where the non-webops world is with knowledge of the cloud. I think the work we’re doing with Ensemble is really going to help people to deploy open source applications into private and public clouds so they don’t need 3D enabled silverlight interfaces to manage a simple database or a bug tracking system for their internal developers.