SpamapS.org – Full Frontal Nerdity

Clint Byrum's Personal Stuff

Have you measured something lately?

cred
I think we all come upon situations where we can do something the simple way, maybe at the cost of efficiency, or the most efficient way, but it won’t be clear and easy to repeat or maintain long term. In software this is especially true because of the flexible nature of what we’re doing, but it happens in all walks of life.

You can take the freeway, and that is usually the fastest way between points. But anybody who has live in Los Angeles knows that sometimes the freeway is more like the parkway, as in, a parking lot full of cars going < 5mph. Just 100 yards to either side of this molasses like river of metal and rubber, there are surface streets whose average speeds are in the 20-30 mph range, even including stop signs and lights.

In the past, we would just take the freeway anyway. Who knows what lies on those streets? Pick the wrong one in LA and you’ll get a nice up close view of the spot where Reginald Denny was met by an angry mob back in the 90′s. Pick the right one and you will probably get to your destination a few minutes earlier and with a lot less feeling of helplessness and stress.

However, today we have some measurements available to us, and can make informed decisions. These days, before I go to drive anywhere, I pop up google maps. I instantly have some actual, reasonably accurate numbers, for where the parking lots and auto-bahn like areas of the freeways are. If I haven’t been on one of the streets, I drop my favorite little traffic helper on the spot, as I call him “street view man”, and get an idea for who I might encounter whilst enjoying the adventure of touring a new neighborhood.

When writing software, do you have similar measurements available to you? Why not? Is it too hard? Not valuable enough? How much is your time worth? How much is the program’s execution time costing you, or your clients?

Measure something today. Its fun, and graph porn is the best way to brag.

May 31, 2011 at 7:03 pm Comments (0)

Puppet Camp: Learn More About Open Source Data Center Automation | Puppet Labs

I’ll be attending Puppet Camp in San Francisco tomorrow and Friday. Come say hi if you’ll be there too!

Puppet Labs

Puppet Camp: Learn More About Open Source Data Center Automation | Puppet Labs.


October 6, 2010 at 4:51 pm Comments (0)

I learned ruby last week

Time to give myself a little pat on the back.

Last week I sat down to work for a whole working day on “whatever I wanted to”, as part of the Canonical Server Team’s pilot “Fedex Day” program. Mathias Gug and I both looked at this idea from Dan Pink’s book “Drive” and thought it made sense to try it out.

Management approved, and we set about on a day of “work on one thing, make it go, and then show it off the following week”.

I was originally going to work on improving the search capabilities of the MoinMoin wiki software that we use at Canonical. But it turns out, somebody already did that by adding Xapian support, and so we really just need to backport that to whatever version of Ubuntu canonical’s servers run on.

So, I decided to tackle another issue that has been nagging at me.

I love perl. I’ve used it for years, and I have always found that the ease with which one can get software from perl’s central repository, CPAN, was a huge differentiator from other languages.

So much so, that PHP, Python, and Ruby have in many ways gone even further than CPAN with their respective tools, PEAR, pypi, and rubygems.

Well one thing Debian developers love is perl too. So for a long time now, its been relatively simple, even braindead simple, to create a debian source package from a CPAN module.

Simply download the tarball, unpack, and ‘dh-make-perl’, that gets you 90% of the way there. All that is left is renaming a few dependencies, making sure the copyrights are done, and verifying that the module doesn’t do anything nasty.

How cool is that?

Python has a few tools like this too, like sdist, so they’re covered.

But Ruby had no such thing for rubygems. There are definitely tools to create debs from ruby source distributions, but often times projects don’t actually distribute source any other way than a .gem file or code repository. To make matters worse, there is a bit of confusion between some users of rubygems on Debian and Ubuntu about how it should work. Many people feel that gems is a poor way to distribute software, or that it lacks certain features. Others would just like to use it like any other developer tool.

Well I happen to think that many ruby users would probably grow to love .debs if they covered a large portion of rubygems software.

So, DebiGem is born! I created this project in about 8 working hours, with zero Ruby knowledge beforehand. I’m sure that shows in my style of Ruby. With some help from perusing the rubygems code base and about 80 Ruby tutorial pages on the net, I think I’ve created something that actually works.

If you want to try it, go ahead and fire it up, simply install the debs available in my PPA on Launchpad, and then download a .gem, and turn it into a source package (apologies for all the debug output, we’re talking about v0.0.2 here!).

clint@ubuntu:~/testgem$ wget http://rubygems.org/downloads/a2ws-0.1.9.gem
--2010-09-07 15:58:46-- http://rubygems.org/downloads/a2ws-0.1.9.gem
Resolving rubygems.org... 72.4.120.124
Connecting to rubygems.org|72.4.120.124|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: /gems/a2ws-0.1.9.gem [following]
--2010-09-07 15:58:46-- http://rubygems.org/gems/a2ws-0.1.9.gem
Reusing existing connection to rubygems.org:80.
HTTP request sent, awaiting response... 302 Found
Location: http://production.cf.rubygems.org/gems/a2ws-0.1.9.gem [following]
--2010-09-07 15:58:46-- http://production.cf.rubygems.org/gems/a2ws-0.1.9.gem
Resolving production.cf.rubygems.org... 216.137.37.222, 216.137.37.239, 216.137.37.11, ...
Connecting to production.cf.rubygems.org|216.137.37.222|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7168 (7.0K) []
Saving to: `a2ws-0.1.9.gem'

100%[================================================================================================================================================================>] 7,168 --.-K/s in 0.07s

2010-09-07 15:58:46 (94.9 KB/s) - `a2ws-0.1.9.gem' saved [7168/7168]

clint@ubuntu:~/testgem$ dh-make-gem a2ws-0.1.9.gem
Extracting gem file a2ws-0.1.9.gem
tar -xvf a2ws-0.1.9.gem data.tar.gz
data.tar.gz
tar: data.tar.gz: implausibly old time stamp 1969-12-31 16:00:00
tar -C liba2ws-ruby-0.1.9 -zxf data.tar.gz
tar: .document: implausibly old time stamp 1969-12-31 16:00:00
tar: .gitignore: implausibly old time stamp 1969-12-31 16:00:00
tar: LICENSE: implausibly old time stamp 1969-12-31 16:00:00
tar: README.rdoc: implausibly old time stamp 1969-12-31 16:00:00
tar: Rakefile: implausibly old time stamp 1969-12-31 16:00:00
tar: VERSION.yml: implausibly old time stamp 1969-12-31 16:00:00
tar: a2ws.gemspec: implausibly old time stamp 1969-12-31 16:00:00
tar: lib/a2ws.rb: implausibly old time stamp 1969-12-31 16:00:00
tar: lib/a2ws/base.rb: implausibly old time stamp 1969-12-31 16:00:00
tar: lib/a2ws/image.rb: implausibly old time stamp 1969-12-31 16:00:00
tar: lib/a2ws/image_search.rb: implausibly old time stamp 1969-12-31 16:00:00
tar: lib/a2ws/item.rb: implausibly old time stamp 1969-12-31 16:00:00
tar: lib/a2ws/item_search.rb: implausibly old time stamp 1969-12-31 16:00:00
tar: lib/a2ws/methodize.rb: implausibly old time stamp 1969-12-31 16:00:00
tar: spec/a2ws_spec.rb: implausibly old time stamp 1969-12-31 16:00:00
tar: spec/spec_helper.rb: implausibly old time stamp 1969-12-31 16:00:00
tar -zcf liba2ws-ruby_0.1.9.orig.tar.gz --exclude=liba2ws-ruby-0.1.9/debian liba2ws-ruby-0.1.9
changing working dir to liba2ws-ruby-0.1.9
DEBUG: a2ws-0.1.9.gem.gemspec
WARNING: no description specified
WARNING: no rubyforge_project specified
DEBUG: req = libhttparty-ruby (>= 0.4.3)
DEBUG: type=runtime
DEBUG: req = libactivesupport-ruby (>= 2.2.2)
DEBUG: type=runtime
Successfully built Debian source package from gemspec
Name: a2ws
Version: 0.1.9
PackageName: liba2ws-ruby
clint@ubuntu:~/testgem$ cat liba2ws-ruby-0.1.9/debian/control
Source: liba2ws-ruby
Section: ruby
Priority: optional
Maintainer: foo
Build-Depends: ruby, debhelper (>= 7), dh-rubygems,
Standards-Version: 3.9.1

Package: liba2ws-ruby
Section: ruby
Architecture: all
Depends: ruby, ${shlibs:Depends}, libhttparty-ruby (>= 0.4.3), libactivesupport-ruby (>= 2.2.2)
Description: Wrapper for Amazon Associates Web Service (A2WS).

clint@ubuntu:~/testgem$ cat liba2ws-ruby-0.1.9/debian/rules
#!/usr/bin/make -f
# This file was generated by 'gem debsrc'
#
%:
dh $@

override_dh_auto_build::
dh_rubygems build

override_dh_install::
dh_rubygems install

override_dh_clean::
dh_rubygems clean
dh_clean
clint@ubuntu:~/testgem$ ls -l liba2ws-ruby-0.1.9/debian
total 20
-rw-r--r-- 1 clint clint 151 2010-09-07 15:58 changelog
-rw-r--r-- 1 clint clint 2 2010-09-07 15:58 compat
-rw-r--r-- 1 clint clint 410 2010-09-07 15:58 control
-rw-r--r-- 1 clint clint 2078 2010-09-07 15:58 liba2ws-ruby.gemspec
-rw-r--r-- 1 clint clint 213 2010-09-07 15:58 rules

It also includes the dh_rubygems tool and dh-rubygems package you see mentioned above.

Given this tool, its relatively easy to create .deb files and focus on the real work of creating packages, which is reviewing licensing and code, rather than turning them into the actual .deb. Sure you still have to update the maintainer field, and will probably want to check those dependencies, but for the most part, this works and gets the gem into an installable state.

I do hope someone finds this useful, it was great fun implementing it.


September 7, 2010 at 11:09 pm Comments (0)

Multiple identities in one account with Apple Mail.app : Jonathan.inspect

Multiple identities in one account with Apple Mail.app : Jonathan.inspect.

Ok, I’m feeling a little silly that I never re-googled this. Apparently Mail.app can very easily do multiple email accounts.. though its completely non-obvious.

Who knew that sometimes even Apple requires you to RTFM.


May 7, 2010 at 6:07 pm Comments (0)

Why hasn’t OpenID, or something else, taken over yet?

I just happened upon a site that mentioned bubbl.us as a way to brainstorm. Cool tool. I played with it and decided I wanted to keep the data I had put in it to play with later, but was annoyed that I had to create yet another user id+email+password combination on yet another site that I probably won’t visit again for a long while. Plus, say I want to add it onto my facebook wall. Facebook might be able to extract the images, but they might now. How lame is that?

My current solution for the login problem is less than ideal. I use the java program Password Safe to save my accounts+passwords, which it generates randomly. The pass phrase for my password safe is pretty complex, and I change it on about an annual basis. The program re-locks the safe after 5 minutes of inactivity, so this is reasonably safe against casual compromise. Of course, keyboard shoulder surfing and a subsequent theft of my machine (or temporary control) could render it useless, but I’m willing to accept those risks and do what I can to maintain control of the laptop. If somebody steals my laptop, unless they can crack the encryption quickly, I feel pretty good that I’ll have enough time to restore from backup, change all the passwords, and set a new combination.

However, this is basically as good as our current “status quo” of online fractured identity can get. And I still don’t have anything to bring all of my online presence together.

I recall with fond memories watching Dick Hardt’s amazing Identity 2.0 presentation from the audience at OSCON 2005. I came away thinking to myself “oh good, somebody is on it.” I put it out of my mind as a systems administrator with a lot of things to think about on the backend, and no real concern for the frontend.

Fast forward 5 years, and I see that we’re not much better off now. Dick Hardt’s company Sxip produced Sxipper, which is pretty cool, but still puts it on the users to safeguard and manage their data. Oh and really, I never heard about it until I went looking for Sxip again, and I don’t know anybody using it, I think its just a cool curiosity, not a solution.

This really is an issue that affects people, but they may not know it. Look at the trouble this guy went through to make google accounts useful for people with multiple email addresses. As we start sharing and sending and moving data, our identities clearly can’t be defined as email addresses anymore. I have 3 that I use a lot, and a couple of others that just refuse to die for whatever reason. Changing them means trying to find every site on which I’ve used them. UGH.

OpenID was, and still is, a promising direction. There are some definite security pitfalls in the way its been done in the past, but I think they’ve solved most of them. It doesn’t really satisfy Dick’s Photo ID requirement where the issuer doesn’t get to know what you’re using it for. Still I love when I sign up for a site and I can use my OpenID login. I use my launchpad.net account for this, mostly because it was the first site that had a very clear “this is your open ID url” link.

FOAF-SSL or “WebID” also seems interesting as a way to promote social credibility and utilize existing technologies rather than try to invent the whole thing. Even twitter seems to have rudimentary support. But its still a long way off from being in control of our identity. Given the meager number of relying parties, I’d say it may not ever get there, which is too bad.

So now I’m just confused. How and when are we going to get this done? When can I say “this is me, here’s some proof that this is me, now lets get something done.”?

Social networks sort of try to do this with the social proof of many friends. But at issue there is how closed off those social relationships are. Facebook wants me *on Facebook*. They don’t want to enable me to also use myspace or my Ning community seamlessly.

Until we as users know why we’d want that, and somebody is able to provide it, I guess I’m just stuck with my password safe.


April 22, 2010 at 9:54 pm Comments (0)

SSH brute force protection – Its almost always already written

Every time I get my logwatch report and see the 20 – 40 daily brute force attempts on it, I cringe. I’ve locked it down to a point, but ultimately I prefer convenience on some level. Limiting any one IP to 2 ssh connections every 5 minutes has annoyed me as many times as it has probably saved me. Preventing root from logging in is nice too.

Ultimately though, I wanted a way to fight back against the brute forcers.. to get a step ahead of them. From seeing the success of projects like SpamHAUS and Project HoneyPot, I know that massive group collaboration works. Of course I started thinking how I’d write it in my head. Every time… for months.

Well, once I let go of my egotistical desire to write it, I found this great project, DenyHosts, which does the same thing for the brute force scanners. I just installed it, and already it has added a few IPs to hosts.deny. Go download it, run it, and stop the annoying scanners!


August 23, 2009 at 4:49 pm Comments (0)

Your code must suck

While attending OSCON 2009 w/ my faithful sidekick fluffy, we constantly kept finding instances of a common theme. The leading companies and projects seem to share one attribute that might shock you.

They all have at least *some* crappy code. At some point, all of them have set aside their principles and thrown in a hack to get things working. This is reinforced by those projects that have their dignity, but no market share. FreeBSD users are famous for saying that Linux is coded by 10,000 monkeys. FreeBSD is an awesome project, that has powered some huge websites. However, the primary Free OS is Linux. Even further along that line is Windows, which is pretty much a hack on a hack on a hack, but somehow, everybody ends up running it.

This isn’t to say that all of the code in popular projects sucks. Just that some of it does. I’m still waiting for the example of an organization that has produced pure, beautiful code with no compromises, and then gone on to garner a large market share and/or massive profits.

The site TheDailyWTF exists primarily because of this fact. I hit that site at least twice a week to have a good laugh. Many times it causes me to reminisce about some of the things I saw early in my career. Just as often, I’m reminded of something more recent. The trend doesn’t seem to stop, despite advances in computing and human understanding, it goes back decades. I imagine Ogg, the first guy who designed a wheel, snarked about how Thag’s wheels weren’t perfectly round. But ultimately, Thag was able to produce wheels that weren’t perfectly round, but rolled pretty well. He probably got them out in half the time, and ended up trading more wheels for Mammoth pelts than Ogg by a factor of five. No doubt Thag was able to attract more mates with his Mammoth Pelt fortune, so maybe its just in our nature.

Really though, this flies in the face of code purity, which we all want. Code sucking == profit? Hacks == market share? This doesn’t sit well with those of us who pride ourselves on brace placement discipline, and knowing at least 5 design patterns without looking them up in a book. But there it is, that pile of dung you knocked out at 3am the day before release to QA… 3 years ago. Still powering the site despite being closer to Alpaca bile than beautiful code.

This doesn’t mean projects fail without hacks. What it means though, is that projects that obsess over doing things “the right way” tend to languish, and rarely achieve success on a massive scale. For some that is ok, they’re happy to have produced something great that a few people like and that works right for them. In fact, this is largely the (healthy) attitude I see from the PostgreSQL project.

The PostgreSQL developers and users tend to feel strongly that their database is far superior to the likes of say, MySQL. They’ll tell you that they have always had full ACID compliance, that their bug counts are low, and performance continues to rise with every release.

I know a lot of people are successfully running PostgreSQL, but really, by contrast, seems like everybody’s running MySQL. MySQL is not bad code either. It just has hacks. Ok, having dug into it a bit now, it has a lot of hacks. But, why is MySQL the leader, and PostgreSQL the follower.

I think the answer is right there in that last sentence. As Cesar Milan will tell you, “choo gotta be da pack leader”. PostgreSQL probably would have continued on as a fine, but obscure, database engine had MySQL not revolutionized data storage in the same way Apache revolutionized web serving. MySQL has managed to carve out a huge market with Free software, while PostgreSQL’s market is only now beginning to grow. Really PostgreSQL has refused to follow in MySQL’s footsteps for a long time, and because of that, they’ve avoided many of the pitfalls MySQL has fallen in to as their scope creeps larger and larger like an amoeba slowly devouring the edges of the enterprise market that used to seem so far from its original targets.

However, even the Postgres guys know that hacks may be necessary. As of May, 2008 they have given in and will produce a general purpose master/slave replication system. The message to the “pgsql-hackers” list has an air of reluctance to it..

Users who might consider
PostgreSQL are choosing other database systems because our existing
replication options are too complex to install and use for simple cases.
In practice, simple asynchronous single-master-multiple-slave
replication covers a respectable fraction of use cases, so we have
concluded that we should allow such a feature to be included in the core
project.

Its like they’re finally saying “ok we want more users, so we’ll include this thing that goes against our principles.” Personally I think this is great, as PostgreSQL is a nice RDBMS, and to be able to use it for small-medium scaleout just like MySQL is really quite exciting.

So, the moral of the story is, if you want your project to be successful, throw in some crap code. Otherwise your developers will be up on their high horses too long, and not down in the trenches getting things done.


July 25, 2009 at 9:19 pm Comments (0)