A Trade Show booth with PF and OpenBSD

A few months after I started at Terracotta I attended my first JavaOne conference. Not as an attendee, but as an exhibitor. The boss came and asked me to build up some infrastructure to run a booth. Over the years, the setup of the booth and some of the software and equipment has changed, but the primary design principles have not.

• Allow all machines in the booth to share a single Internet connection
• Make it simple to setup and use
• Allow employees to check their email, etc. from the booth
• Allow the sales engineers to explore potential client websites
• Do not allow demo stations to be used by conference attendees to check their email or hog the demo station trying to show us their website
• Make it secure so that we don't have any demo "surprises"
• Make sure all the demo stations are consistent

I turned to one of my favorite operating systems to solve the problem, OpenBSD. Here is what the network looks like as of 2009.

We get our Internet connection from Priority Networks and every year it is rock solid, they are super easy to work with, and when you need help, they actually know what you're talking about!
As you can see, each daemon on the machine serves a purpose to running the overall network. Each daemon (other than PF) is only assigned to the internal interface.

named

We run a private domain inside the booth (javaone.tc) and also need standard resolving for internal clients

dhcp

Demo machines are given static IPs, all other clients are assigned to a different part of the subnet, more on this later

puppetmasterd

Now that machines have gotten faster and we have less graphical demos, we can run all Unix demo stations. Puppet makes sure all the machines are 100% consistent and makes it much easier to setup machines initially or substitute in a new station in case of some kind of problem

PF

This is where all the magic happens, why you can type www.yahoo.com and wind up at Terracotta.org

httpd

This was more important before puppet and when we still had Windows, but Apache is still a great way to serve up files to any network

ntpd

We're a Java clustering company and it's very important to have synchronized clocks in a cluster, then again, isn't it always?

As you can see above, we have a private domain inside the booth. It's just a simple /24 divided in two. Machines in the lower half of the subnet are assigned static IPs by MAC address, this is for the demo stations only. Machines in the top half of the subnet (129-254) are assigned IPs dynamically and this range is for any employee who brings their laptop to the booth and wants to login to check email, fix a bug, etc. PF treats the two IP ranges differently.

Here is the firewall ruleset:
ext_if="bge0" int_if="dc0" DEMOSTATIONS="192.168.100.0/25" EMPLOYEES="192.168.100.128/25" set skip on lo # allow demo stations to access Terracotta and a few other websites we rely upon table <TCOK> { 64.95.112.224/27, www.google-analytics.com, now.eloqua.com, secure.eloqua.com, download.terracotta.org } scrub all nat-anchor "ftp-proxy/*" rdr-anchor "ftp-proxy/*" nat on $ext_if from $int_if:network -> ($ext_if:0) rdr pass on $int_if proto tcp from $int_if:network to port 21 -> 127.0.0.1 port 8021 rdr pass on $int_if proto tcp from $DEMOSTATIONS to ! <TCOK> port 80 -> 64.95.112.233 port 80 rdr pass on $int_if proto tcp from $DEMOSTATIONS to ! <TCOK> port 443 -> 64.95.112.233 port 80 anchor "ftp-proxy/*" block log all pass quick on $int_if no state antispoof quick for { lo $ext_if } # fw inbound - for remote admin when Priority Networks allows this pass in quick on $ext_if proto tcp to ($ext_if) port ssh # fw outbound pass out quick on $ext_if proto tcp from ($ext_if) to any modulate state flags S/SA pass out quick on $ext_if proto udp from ($ext_if) to any keep state # int outbound pass in quick proto tcp from $DEMOSTATIONS to any port { 22 25 80 443 8081 } modulate state flags S/SA pass in quick proto udp from $DEMOSTATIONS to any port { 53 } keep state pass in quick proto tcp from $EMPLOYEES to any modulate state flags S/SA pass in quick proto udp from $EMPLOYEES to any keep state

The only problem with this ruleset is that the name resolution for domains that are hardcoded in the ruleset (e.g. www.google-analytics.com) can only really happen after the OS has booted. Otherwise, the boot sequence stalls on name resolution. The workaround for this is to disable PF in /etc/rc.conf.local and enable it with

pfctl -e -f /etc/pf.conf

in /etc/rc.local. That is really the only necessary workaround.

As you can see, it's actually a REALLY, REALLY permissive ruleset. Much more permissive than we allow in the office. Because there is rarely a Terracotta sysadmin on the show floor during the conference, and because there are tons of open access points which our employees would use if we locked them down too much anyway, we feel this is a pretty acceptable level of risk for the few days of the show. We could certainly lock down the ports employees could access, restrict to their MAC addresses, or even put in authpf for them to authenticate, but that would mean maintaining a password file outside the corporate office, or duplicating the LDAP server, or setting up an IPSEC tunnel, all of which are excessive for a few days of conference.

That's really all there is to it (other than some GENERATE statements in the zone files). Free, functional, easy, and secure by OpenBSD. Posted by Dave Mangot in Applications at 20090623 Comments[3]

EC2 Variability: The numbers revealed

Measuring EC2 system performance

I've been spending a lot of time at Terracotta working on cloud deployments of Terracotta and the cloud in general. People have been asking me what the difference is running apps on the cloud, and specifically EC2. There are a number of differences (m1.small is a uni-processor machine!) but the number one answer is "Variability". You just cannot rely on getting a consistent level of performance in the cloud. At least not that I've been able to observe.

I decided to put EC2 to the test and examine three different areas that were easy to measure: disk I/O, latency, and bandwidth

Disk I/O

• Environment: EC2 m1.small
• File size: 10 MB (mmap() files)
• Mount point: /mnt
• Testing software: iozone
• Duration: almost 24 hours

As you can see the numbers can vary a good deal. This is on an otherwise completely quiescent virtual machine and with a 10 MB filesize, the tests themselves took almost no time to run. Most of the numbers actually look remarkably consistent with the exception of Random Reads. Those numbers are all over the place which you might expect from "random" but this looks to be a bit much. The numbers are actually pretty respectable and compare to about a 7200 RPM SATA drive. Certainly not the kind of machine you would use for performance benchmarks, but if you threw enough instances at a clustering problem, you could certainly get the job done.

Latency

• Environment: EC2 m1.small
• Datacenter: us-east-1b
• Testing software: smokeping
• Duration: about 20 hours

Here, where networking is involved between instances, things start to get a little bit more varied. The median RTT is 0.3506 ms which is about 3 times more latency than you would get on a typical gigabit ethernet network. You can see the numbers hover there for the most part but there is a tremendous amount of variability around that number. Smokeping shows outliers about 2 ms but I have seen numbers as high as 65 ms or worse in ad hoc tests. I don't know what happened at 4 a.m. on this graph but I'm glad I wasn't running a production application at the time. If you look closely, you can also see a few instances of packet loss which is something we don't usually experience on a production network. Again, this is on an otherwise quiescent machine. For comparison's sake, here is the smokeping graph between Terracotta's San Francisco and India office which is actually carrying a fair bit of traffic. This is a LAN to WAN comparison so the numbers are not going to look as exaggerated because they are running on a different scale, but in the EC2 instance, we can see more than 5 times the variability in latency, which we don't see on the WAN segment (or ever on any of my lab switches for that matter).

Bandwidth

• Environment: EC2 m1.small
• Datacenter: us-east-1b
• Testing software: nttcp
• Duration: about 24 hours

In this graph, we can see that the gigabit connection between EC2 instances is hardly gigabit at all. I would say the numbers may trend upwards of 600 Mbps on average but they fluctuate pretty wildly between real gigabit to barely faster than a 100 Mbps connection. This was run on an otherwise quiescent machine. In a real "production" environment we would expect much more consistency, especially if trying to run performance numbers.

Conclusions

It is pretty safe to say that there won't be any vendors publishing performance numbers of what they are able to achieve with their software running on the cloud unless they had no other choice. You can easily get much more consistent, faster numbers running on dedicated hardware. In our tests with Terracotta, we've seen that you just have to throw that much more hardware at the problem to get the same kinds of numbers. It stands to reason however as a uni-processor m1.small instance is just not going to be as powerful as our quad-core Xeons. In throwing more instances at the problem you start to introduce more of the network into the equation, which as you can see, is a rather variable quantity. Thankfully, the amount of latency introduced even in this case is no big deal for the Terracotta server, so I've been having a pretty good time running bigger and bigger clusters in the cloud.

Posted by Dave Mangot in Applications at 20090513 Comments[4]

Linksys WET54G, a consumer product?

I recently bought a consumer electronic device that I wanted to hook up to the Internet. This device came with a hard wired Ethernet port, but of course, I had no Ethernet cable where I needed to hook it up and I also had no desire to run one.

There are various devices on the market that can turn a wired Ethernet jack into a wireless one and the one I choose was the Linksys WET54G.

I chose this device for a few reasons:

1. All my network devices at home are already Linksys
2. Seemed small and priced no higher than any similar devices
3. I could get it at my local computer store for the same price as ordering it online

Like most technical folk, I did a lot of reading before purchasing the device. Most of the reviews on Amazon were extremely negative, but I feel like I'm pretty good at sorting through the reviews of the inexperienced vs. the reviews of the knowledgeable. Big mistake!

I got the device and learned that because it is version 3.1 of the product, there are no firmware updates available on the Cisco/Linksys website. It is already a newer revision than anything that is even listed. All that was needed was to plug it in and configure it according to the instructions. Here's what happened.

1. Take the bridge (the WET54G is technically a wireless to Ethernet bridge) and plug it into my Linksys mini-hub. Run the Linksys provided utility on a PC (yeah, I still have an ancient XP box kicking around). Bridge not detected. Power cycle the hub and bridge a few times. Nothing.
2. Notice that it says the PC and the bridge must be plugged into the same hub. Ohhhh, that must be it. Wire the PC to the hub with a cable. Bridge not detected. Power cycle the hub and bridge a few times, still nothing.
3. Ok, I figure, how bad could it be to just use the web interface? I look at the docs and supposedly the bridge will autoconfigure at 192.168.100.121 or some weird address like that. Fine, I reconfigure the NIC on the XP machine and soon am on the web interface for the bridge
4. A few minutes later, the bridge is all configured with ip, netmask, gateway etc. I should be good to go. At this point, my PC says "Duplicate IP detected on the network". Hmmm.
5. I unplug the PC from the hub and reconfigure everything back to normal (i.e. wireless). The PC is still complaining about duplicate IP and I can no longer ping my default route. Something is fishy.
6. I plug my media device into the bridge as had been the plan all along, and the device instantly recognizes the network and says it needs a firmware update. Success! I tell it to get the update and it just hangs there, forever.
7. My wife ways that her PC is saying duplicate device detected as well and she can't get on the Internet. Huh?
8. Fire up the Mac and I get on the firewall. /var/log/messages tells me that another device on the network is advertising itself as the default route's IP. I check the MAC address and sure enough, it's the bridge!
9. I get on the web interface for the bridge and change its default route to a bogus address on the network. Why would my bridge need to get out on the Internet anyway? Instantly, all the devices in the house start working correctly.
10. I configure a static IP address on the media device and it is able to access the Internet without problems. I update its firmware through a USB key anyway.

So now the network is running fine and I haven't had any of the other issues people had described in their reviews. But the question remains: How is this a consumer product? I've designed LANs and WANs for multiple companies. I've configured networks on machine all the way on the other side of the world. I was stumped for a good 20 minutes as to why my network was behaving like it was drunk. What would your average gadgethead have done aside from sit on the phone with Cisco tech support for hours? Would they have figured it out? Crazy.

Update: Yesterday my bridge "lost it's mind" and I will have to reconfigure it from scratch. What a piece of junk. Posted by Dave Mangot in Hardware at 20090330

Choosing Zimbra as told to ex-Taosers@groups.yahoo

We're running Zimbra in production and have been for almost a year now. We're running Network Edition (paid) with Mobile.

Eh. It's ok.

Of course our philosophy is to let anyone run anything they want. Zimbra would seem to support this philosophy but I sometimes wonder if it tries to do too much.

Messaging is pretty rock solid. Other than some stupid bugs like showing 20 messages in your Drafts folder and then when you click on it, there is nothing there, it works fine. I've never heard about any lost messages or anything like that.

Calendar is wonky however. You can't set variable reminders for meetings. That means, either get a 5 (or 10, or 15) minute warning for every meeting you have or nothing. So if you have a sales meeting across town that you want to have a 30 minute reminder for, and a budget meeting down the hall you want a 5 minute reminder for, can't do it. It's one of the most voted on bugs in bugzilla and has been for a while, but they say they'll fix it maybe in 5.5. Weird.

Plus, the calendar is just buggy. Meetings don't show up all the time. If you change a meeting time, free/busy gets screwed up. I've had a user whose iSync just stopped putting on new meetings twice. All kinds of headaches.

Of course we support, Blackberrys (via NotifyLink Inc.), Treos, iPhones, Outloook connector, iSync connector. Basically, every bell and whistle you can ring or blow in Zimbra so sometimes it's user error, but sometimes it's Zimbra.

Despite this, I was planning on running Zimbra for mangot.com mail because my wife and I won't have a fit if we miss a meeting. I installed 5.0 and they've changed from Tomcat to Jetty, which is nice, but you still can't bind Zimbra to a single IP without hacking up all the conf files and putting some changes in LDAP. In fact, I had to change my hostname(1) during install temporarily just to get it on the right track. Then I went around hacking up the files to keep it from trying to hog the entire machine.

Of course, Zimbra will tell you that you need to dedicate the whole machine to ZCS. You would think the fact that it uses so many open source components means you could hack on it, but in reality most of your changes will be lost each time you upgrade and you will need to re-apply them (we do exactly that for Mailman integration). Until they get more of the config into the LDAP server, that's just what you're going to have to live with. I guess the fact that is OSS means that's at least an option.

On the plus side, it does have some nice Gee Whiz factor to it and we are talking about setting up a Zimbra server at a remote office that shares the same config as our current one and it's supposed to be very easy, but as with everything else, the devil is probably in the details. Like I said, I'm still planning on running it for my personal stuff, so I don't hate it as much as I might have come across. I don't know if I'd want to run it at my business again, except for the fact that there really is no one else who comes close.

(except Exchange, but we're not an MS-only shop, and please don't give me any of that it runs IMAP garbage, because we all know that's not a real solution)

Cheers, -Dave Posted by Dave Mangot in Applications at 20080207

Information Security Magazine Chuckle

This made me laugh. I get a "renew your subscription" notification from Information Security Magazine even though I've only received one issue. Fair enough, it's free. I fill out their form and click submit...and I get an (buffer?) overflow in Visual Basic on some Microsoft server.

Yeah, I'm really going to trust what these guys have to say now!

(yes, I need a new blog template, maybe after I upgrade)

Posted by Dave Mangot in General at 20080207

A SysAdmin's impressions of MacOS Leopard

I've had the chance to use Leopard for a few weeks now on my primary work machine, a 12" G4 Powerbook. The results have been mixed. On the whole, Leopard is very good, as good as Tiger (which is excellent) but there have been a few problems being an early adopter.

The aim in this post is to cover just a few of the experiences I've had, good and bad.

The bad

• 1st boot is a BSOD!

  Ok, not a real BSOD in the Windows sense but the kernel panic'd and the machine would not boot. This is a known issue and Apple has a fix. Still, it is mildly disconcerting to boot your new OS and get a big blue screen of nothing. We have a number of users on different Mac hardware and it's been very hit or miss as to who has been affected. The solution is easy, but I'd rather not see it at all.

• The X11 server is buggy I'm not the only one who has noticed this: Boredzo.org

  I've had two issues with the X11 server.

  1. The first is my dueling X11 dock icons. I need an X11 server to display remote X11 apps like the Terracotta admin console. In Leopard I have two icons in the dock, one that the OS thinks is in some weird state because it offers me to "Force Quit" the application. The other icon seems normal. "Force Quitting" the app has no effect incidentally, it remains. Lovely.
  2. While trying to use Wireshark it wouldn't start. A couple of searches and the problem turned up as a bug with X11.app discussed on the Wireshark Mailing Lists. The fix is simple, I don't care about having millions of colors in Wireshark, I'm just happy when it doesn't get my machine r00ted. Still, it worked without incident in Tiger.
• The new firewall configuration.

  This has been discussed ad naseum (trust me) on the focus-apple Securityfocus list. In 10.5.0, Apple had a setting where you could tell the OS to "Block all incoming connections". Sounds great, who doesn't like default deny? The problem was, that setting didn't block all incoming connections. Not even close. Anything that ran as root allowed incoming connections, plus anything Apple deemed essential like Rendezvous. The wording was updated for 10.5.1 to be a bit more accurate.

  

• X-Lite is straight up busted

  We rely on Asterisk and X-lite at work so we don't have to buy all the engineers desk phones as they rarely spend too much time on the phone. Plus the USB headset makes it convenient to talk on the phone while typing. The guys over at Counterpath have basically told everyone who is using X-lite to stick it . We have bought a number of copies of Eyebeam because we've had a good experience with X-lite. I thought that was their model. I guess times are hard. The best part is they say emphatically that it is a bug with Leopard, yet my SJphone and Xmeeting work perfectly fine. Hmm.

• General bugginess and insanity In no particular order
  • My machine can't eject firewire disks, it can mount them, but you can't check out. A firewire roach motel?
  • I can put my machine to sleep and once in a while, it just will not come back, reboot.
  • Since upgrading to Leopard, my machine will say I have over an hour of battery life left and then shut down without warning. Checking the battery shows it is dead. I know I probably just need to zap the PRAM, but I'm usually too lazy, or too busy to do that.
  • Apple filesharing gets totally confused. I mounted a drive using AFP and then tried to unmount it. The system showed the filesystem as mounted in the Filer, but it wasn't listed under /Volumes. I had to reboot to make it go away, yuk.

The good

ssh-agent integration with iTerm works!

One big annoyance with Tiger was starting up an ssh-agent(1) and it was only recognized for sessions I started in the default terminal. None of my bookmarks worked. I haven't upgraded iTerm but I have upgraded to Leopard. All of a sudden, my bookmarks recognize my ssh-agent. Sweet! iTerm is one of my favorite Mac programs, by the way. Great terminal, great support for tabs. A sysadmin essential!

The Cisco AnyConnect client work fine.

Ok, someone else I know who upgraded to Leopard had trouble until he went to the latest version. Mine worked fine however, which is pretty remarkable considering it's a new kernel. I'm not sure whether to give kudos to Cisco or Apple, but in either case, I was pleasantly surprised.

Otool is ldd!

Ok, I don't think this is a Leopard thing, but on every other operating system I care about, to find out what libraries are used by dynamically linked binaries, you use ldd. Not so on the Mac. I did discover that you can do the same thing with otool.

I wish I knew this when I was trying to find out if my psql Postgres client had SSL support built in! (it does)
dmangot-laptop:~ $ otool -L /sw/bin/psql /sw/bin/psql: /sw/lib/libpq.5.0.dylib (compatibility version 5.0.0, current version 5.0.0) /usr/lib/libpam.1.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/lib/libssl.0.9.7.dylib (compatibility version 0.9.7, current version 0.9.7) /usr/lib/libcrypto.0.9.7.dylib (compatibility version 0.9.7, current version 0.9.7) /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos (compatibility version 5.0.0, current version 5.0.0) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3) /sw/lib/libreadline.5.dylib (compatibility version 5.0.0, current version 5.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.1.6) /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

Conclusion

That's it. I know that it sounds like the OS has a number of problems, but on the whole it is pretty stable and doesn't do very many things that make you say, "What?!?", unlike Windows (is wireless networking really that hard?). Like I mentioned earlier, that's the price you pay for being an early adopter. I'm sure there are a ton of people out there waiting for Vista SP1 before even trying to install that beast. With Leopard, there were very few gotchas for a dot "Oh" release, but the best part was, the upgrade hardly slowed me down. Posted by Dave Mangot in General at 20071219 Comments[2]

Worlds collide: RMI vs. Linux localhost

A few times in my career as a Unix sysadmin working with Java applications I've into situations that use RMI remotely, and each time there have been problems.   The two most recent that come to mind are jstatd and the Terracotta admin console.

Because I now work at Terracotta, I've had a chance to actually do some testing to find out where the problem arises. We've seen this come up numerous times on our Forums, so much so that my buddy Gary who explained much of the what is going on "behind the scenes" has changed our default behavior in recent versions of the software.

When clients connect to the RMI server, the server returns the address of the JMX server to query to get our statistics. On clients where hostname(1) is set to localhost, that is what gets returned. Of course, when the client is told to connect to localhost by the remote server, it fails because the JMX server is really remote.  Turns out this is a pretty common problem in the Java world. You can find many examples of users complaining about it in Sun Java Forums.

When it works

To get the Terracotta server to use the RMI stub, we have to put it into authentication mode. Details on how to do this can be found in the Terracotta Configuration Guide

To actually see what is going wrong, first we'll look at a situation where it works correctly.
First we take a tcpdump of the client connecting to the server where the server actually has its hostname mapped correctly to its externally accessible IP address. We feed it into Wireshark (security holes be damned!) and do a Follow TCP Stream.

Here we can see that the RMI server has returned the IP address of the machine on which it runs.

When it fails

Next we change the hostname to localhost. We then do a Follow TCP Stream and see this:

The RMI server has returned 127.0.0.1 (localhost) to our remote client, and we get connection failed on the client.

Why are the hostnames of these machines being set to localhost?

I think what is happening is many of our users are testing out Terracotta on their desktops, or machines that have been built from the install CDs. When this happens, Linux is assuming that those machines will receive an IP address via DHCP. If that's the case, they don't want to hard code an IP into /etc/hosts. Therefore the only entry in /etc/hosts winds up being localhost and the hostname is set to the same. We see this kind of behavior to varying degrees even with our automated installs. Our kickstarts and autoyasts tend to put the correct name, hostname, and IP address information into /etc/hosts presumably pulling the address from DHCP. If there is no DHCP entry for a host, autoyast will put an entry with 127.0.0.2 in /etc/hosts, weird.

Posted by Dave Mangot in Applications at 20071102 Comments[1]

Hello World Finally got the blogging software up and running. I'll be posting mostly about systems administration. There seems to be a dearth of people out there writing about SysAdmins and the tasks/problems/solutions they deal with every day. Hopefully I can find others who are doing the same. Maybe we'll fill the void left by SysAdmin Magazine going the way of the dodo. Posted by Dave Mangot in General at 20071019