Archive
Finding the hostname in node.js
This is a simple thing, but something I tend to forget and have to go find again. So here it is for later…
To get the OS hostname use the os module.
var hostname = require(“os”).hostname;
to just get the first piece of the name:
var hostname = require(‘os’).hostname().split(‘.’).shift();
Of course, only do it that way if you don’t otherwise need the os module. Otherwise require the os module and set it to a variable so you can reuse it without require each time.
To get the hostname for the current request, look in
request.headers.host
Am I done with Ubuntu?
I made the mistake of hitting the Upgrade button on Ubuntu updates manager on my main development box the other day when it asked me if I really was going to go another day without Oneiric, and within a fairly short time had an unbootable Ubuntu box. Usually Ubuntu upgrades are fairly smooth, this one was bad. For a little bit of context, I have been using Linux for a long time. I started with RedHat, wandered through Fedora when it appeared, Gentoo, Suse, OpenSuse, CentOS, ClearOS, Debian, and Ubuntu, but for the past while I’ve been using Ubuntu. For years I used and advocated KDE until version 4, at which point (about the same time I switched to Ubuntu) I moved to Gnome. I have been using XFCE for a few months, and I’m basically done with both KDE and Gnome for now. So far I haven’t lasted for more than a few minutes on Unity before I get completely disgusted and change to something else. Somebody told me this week I’m just one of those old grumpy Linux guys.
I didn’t spend a lot of time figuring out what went wrong with the Ubuntu upgrade. Instead, I downloaded a few updated versions and put them on a thumb drive, and tried out some variations on the setup I’ve been using for a while. I installed Mint 11, Mint’s Debian XFCE version, and Xubuntu. As a side note, why aren’t there any really good tools to make bootable live Linux installs on USB for Linux? Most of the directions on the web say to do it on Windows. Blech. I ended up using unetbootin, which works.
My brief reaction to each of the three installs? Mint 11 has all the advantages of Ubuntu, except that its currently a version behind and has mintier branding. The main reason I’d use it is if I wanted to stick with Gnome, which is a possibility if it weren’t for the fact that I’m really liking XFCE. So, Mint 11 isn’t in my immediate future.
The Debian version of Mint is somewhat enticing. I like the idea of rolling updates. I’m not a huge fan of straight Debian, for no reason other than I have a kneejerk ideological reaction to software that is too ideological. Software should be practical. Debian is from a planet I’ve only ever visited for short periods. Yes, I know Ubuntu is Debian based, but it is suitably commercialized. Odd position for a Linux fan to take, isn’t it? I am fairly sure I’m not alone. All that being said, I could see myself using and liking this distro, certainly over the vanilla Mint 11 Gnome version.
Xubuntu works reasonably well. The new Ubuntu Software Center stinks. What happened to options and the ability to configure stuff? It’s pretty, but gutted. That’s basically my reaction to the direction Ubuntu is going generally. Ah, for the good old days when all the configurations were in bash and lisp files.
My first step on all three installations, after changing them so the focus follows the mouse properly, was to try to compile CouchDb 1.1. It failed on all three. There seems to be a mismatch between compiler versions and what CouchDb’s configure is expecting. I haven’t taken the time yet to figure out what the problem is. At this point I mostly just need to get on with my coding. The CouchDb binary package available on these distros is out of date. For my purpose on this dev box, it doesn’t matter enough to spend time on it. However, I will need to sort this issue out at some point. By contrast, node.js compiled easily on all three.
For now, I’ll probably use Xubuntu. When I have more time on my hands, it is likely I’ll wander off into a search for a different Distro, and move out of the Ubuntu family again. I’ll need to do something with my laptop (the machine I actually work on), which is a light weight Acer currently running Ubuntu 11.04 with XFCE. I’m open to suggestions, but I guess I’m not in much of a hurry. None of the recent installs on my dev box were exciting enough to make me want to spend more time on it. And for someone who’s spent way too many hours over the past fifteen years or so distro hopping just for fun, that’s too bad.
Launching a whole new thing
For the past 15 years I’ve worked in non-profits (missions, humanitarian relief organizations) to enable them to accomplish their ministry objectives through strategic application of technology. Early on, that consisted largely of systems, with administration and system design as the emphasis. Later on it shifted almost entirely to development and application architecture, but still with a strain of making sure services are always available for their intended use.
Now I’m entering a new chapter of my career, with the launch of NodePing, a partnership with my good friend Shawn Parrish. For me (this is my blog after all), that has meant something of a return to activities I haven’t focused on in quite a while: business plans and contracts. Happily though, most of my time in pulling this together has been in code, working with Node.js, Redis, CouchDb, and jQuery. That part has been a lot of fun, and there’s still a lot more to do.
The new service brings a lot of the things I have worked on through the course of my career together. I actually have used my education, a change from my normal work. It also leans on all the years I’ve been responsible for system administration departments, and ensuring services are available. And of course, it has included lots of system and software architecture and coding, by far the funnest parts.
Some posts that might otherwise appear on this blog about Javascript and systems for hosting Node.js applications will likely appear on the NodePing blog.
Huge amounts of work to do from here. I’m excited.
Moving CouchDb database files between servers
There are several methods available for copying data between CouchDb servers. The most obvious is replication, which CouchDb does extremely well and it’s built in. If that option is viable it is probably the way to go. A while back I posted about a method I have used to use bulk document handling in Couch to copy data. That process works well too, and I continue to use that from time to time for some data.
Recently I had a situation in which I needed to set up several development and testing servers with the same initial state for the data. These were not on the same networks, and replication wasn’t convenient. I was moving a good bit of data across several databases, so the bulk document approach wasn’t attractive either. So I resorted to just copying the data files between servers. CouchDb’s design makes this easy to do.
The steps I take are probably overly cautious, but here’s what I do:
- Stop the couchdb service on the source host
- tar.gz the data files. On my Ubuntu servers this is typically in /var/lib/couchdb (sometimes in a subdirectory based on the Couch version). If you aren’t sure where these files are, you can find the path in your CouchDb config files, or often by doing a
ps -A wto see the full command that started CouchDb. Make sure you get the subdirectories that start with . when you archive the files. - Restart the couchdb service on the source host.
- scp the tar.gz file to the destination host and unpack them in a temporary location there.
- chown the files to the user and group that owns the files already in the database directory on the destination. This is likely couchdb:couchdb. This is important, as messing up the file permissions is the only way I’ve managed to mess up this process so far.
- Stop CouchDb on the destination host.
- cp the files into the destination directory. Again on my hosts this has been /var/lib/couchdb.
- Double check the file permissions in their new home.
- Restart CouchDb on the destination host.
You may or may not have to stop the CouchDb services, but it seems like a good idea to me to decrease the chances of inconsistent files.
I have done this a number of times now with no problems, other than when I manage to mess up the file permissions.
Running out of disk space for CouchDB
I added some new views to CouchDb on a development server the other day. The views included three or four emits and the full document in the view, and the disk space used exploded to several times the size of the actual data stored. Everything came to a screeching halt as the server had no space to write logs, no space to build views, no space to store the data we were pushing in, and no space for the compacts that were trying to run. It was one of those moments when you are really happy that the host you just messed up is a development box. This episode raised several lessons for me, some of which were new in the specific context, some of which were good reminders.
First, CouchDb needs lots of disk space. If you are working with views, lots of disk space can disappear very fast, as Couch copies the data files for compacting. This is not a bad thing, its what allows Couch to keep running and performing while it is doing these operations, but it is something to plan for when sizing resources. Running out of disk space is a bad thing. In my case, I had estimated the size of the data and allowed some room for compacts and views, but not nearly enough.
Second, this episode started up quite a debate within my team about designing views. I tend toward making views overly inclusive. My friend tends toward making the views as small as possible, and then throwing in an include_docs if you need more in the results. This is a trade-off, and in the end you have to consider it with every view you write. Small views save lots of disk space, and speed up writes and simple queries where you don’t need much in the results. Using include_docs is fairly cheap if you are getting a handful of docs. If you’re looking for results from hundreds or thousands of records, include_docs loses its appeal, because the server has to fetch each of those documents. Its a balancing act. If you argue for larger views for faster queries as the side on which you should err, I’d suggest making sure you have plenty of disk space. Turns out it is a little embarrassing when you make that argument and then badly under estimate how much disk space you really need, even on a development box.
Third, I got to learn how to clean up from dead compact jobs. When the server is out of disk space when it tries to compact a database, it can leave behind compact files that prevent compacts from working even after the space problem is solved. In my case on Ubuntu, these are in /var/lib/couchdb. If there are 0 size .compact files in your database directory you just need to delete them and restart the compact. I also noticed that there were some non-zero size .compact files in databases that were not actually compacting, and I removed these as well. Everything went back to humming after that.
CouchDb is a great tool. It does have some peculiarities it is good to keep in mind. Life was simple when we had basically no real choices. “Welcome to LAMP! Would you like MySQL or Postres with that?” Now we have all kinds of options on the menu, and we actually have to think about finding the right tool and understanding its strengths, weaknesses, and quirks.
David's Tech Blog 