David’s Tech Blog

When teams are distributed, it is important to find ways to build constant communication and gain the benefits that might otherwise be gained by working together in one room.  This can be challenging.  As we move towards more mobile and nomadic work styles (see this interesting set of articles from the Economist), and as development and other teams are scattered around the world, this becomes increasingly important.

The primary purposes for having people sit together for development or other work include promoting the team mentality and relationships between team members, constant communication that promotes frequent but but typically short interaction, and the free flow and transparency of information.  All of these can be obtained with remote workers using the right tools, and with the right mental leap and cultural shifts.

One of the key elements is Presence.  When you share space with someone, you can see if they are sitting at their desk, and you can see from their body language if they are intently focused on something.  We need a way to convey this in situations in which people are not physically together.   The status and mood statements provided by most Instant Messaging systems can do this fairly well, if they are reliable and are used actively.  A good IM presence indicator should tell if a person is available, online but busy, online but should not be disturbed, or offline.  If the person can add text to this, using a mood line, or in some systems just as custom text on the status, this can add additional information for others.  A key element for all of this to work, though, is that the presence has to be reliable.  Skype’s presence indicators, for example, aren’t reliable.  You don’t really know from a status indicator in your contact list if that’s really the status of the person, or if it just hasn’t updated.  The same result happens when people don’t update their status (a failing I have, actually).  Effective remote team work takes the extra effort to use the tools to communicate in ways that would be inherent in physical proximity.

Another element lost when people are not physically together is the constant give and take, the chatter of people working together.  This constant communication is an element of status, telling people what you are working on, how it is going, and whether or not you should be bothered at the moment.  Another piece is a constant awareness of what is going on in the team.  Periodic meetings, even daily, don’t really get this.  The constant chatter of co-workers in close proximity is also important for team building.  Most of this can be obtained using micro-blogging services.  This makes the chatter easy, quick, non-obtrusive, but available to everybody in real time even if it isn’t aimed at them.  Services like Twitter or Jaiku can do this, but a service that is aimed at teams, like Yammer, is better.

I was skeptical of micro-blogging as an effective tool, until I saw it in action.  I invited a hand full of coworkers to Yammer, thinking they would say “Cool, but why?”  Instead, within a very short time the few people I had invited turned into around 50 people, many of them enthusiastically using the service.  Within a short time, I learned a couple important things that were going on in the work of people I don’t talk to very much (I’m not in the same city as they are).  I was amazed at how quickly the tool was picked up, and at how well it simulated hearing things in the hallway or other situations in which in an open office environment you would hear conversations between other coworkers.

Of course, not all conversations should happen around the water cooler or in the hallway.  Sometimes you need more indepth conversations, or you need privacy.  IM is still the way to go here.  It is important that IM have a way to archive and search old conversations, that it be really easy to use, that it allow mutliple people in the same conversation, and that it be a part of your working culture.

I should note that not all of the conversations that will happen on micro-blogging or IM services are going to be directly work-related.  In fact, you want to have a healthy ratio of personal give and take, joking, community building kinds of activity going on with these.  That’s the sort of thing that goes on when a real team is together, and it is important to get the same type of community buzz going on remotely.

It would be helpful if these tools would converge.  There are likely other ways to accomplish these goals.  IRC and other IM type systems that let you post comments in a “room” that is available to everybody might be part of it.  But you need to be able to subscribe to and follow multiple tags, people, and areas at the same time without switching contexts.  It has to come naturally.  The tools I mentioned are headed in that direction.

It would also be good to seamlessly tie voice and video to these mediums of communication.  Sometimes text isn’t working, and its best to pick up the headset and go to a voice chat.  Skype does that well.  If Google would make Google Talk cross platform maybe that service would work for this as well.  I am not a big fan of video (you don’t need or want to know what I’m wearing while I work), but for some people that would probably work best.  Sometimes face to face meetings are required, and its best to hop on a plane (if necessary) and go sit down for a face to face talk.  But for the vast majority of everyday work, there isn’t any reason that distributed teams can’t get the same results as teams sitting together.

I ran Ubuntu for about a month.  It is a very nice distribution.  Everything worked very smoothly.  I had no real problems with it.  I’m not going to use it any more.  There is nothing at all wrong with it.  On the other hand, there aren’t any real advantages over OpenSuse, so I’m going back to what’s more familiar.  If any Windows users ask me what Linux flavor they should try, now that I have actually used it I am comfortable recommending Ubuntu as a good choice.

OpenSuse 11 is excellent.  They have continued to develop a very good distribution, and here again everything pretty much just works.   The installation is very smooth.  The Network Manager for KDE is improved, as is the update manager.

One difference that threw me for a few minutes was configuring Xinerama across two screens on my Dell D620.  In Ubuntu, the key is to install the nVidia drivers, and use the nVidia tool to configure X.  On OpenSuse, you need the nVidea drivers, but you have to use SaX2 (accessible through YaST) to configure X.  Once you do this, its very smooth and easy to get working.  Just to be clear, the display works fine without the proprietary drivers, but to get the best use out of multiple screens you need the drivers from nVidia.

Once or twice a week I go into the office for meetings.  Its not unusual for me to unplug the monitor to carry my laptop to a conference room, and plug it into a projector without changing the configuration at all.  OpenSuse handles this without problems.  Ubuntu just wouldn’t do it satisfactorily at all.  I didn’t take the time to figure out how to get it to work, although I’d guess there’s a way to get the results I wanted.  It didn’t happen “out-of-the-box” like it does with OpenSuse.

I initially ran KDE4 under OpenSuse 11.  It looks very nice, but it is not ready for real use yet, at least not for me.  It is still lacking most of the options that make me really like KDE.  Panel configuration is simplistic and just not ready yet.  Fortunately, OpenSuse has packaged things so you can easily install KDE3 beside KDE4 and choose which to run.  Installing KDE3 was as easy as loading the package manager and picking KDE3 from the package patterns list.  Now I can check back on KDE4’s progress as it gets better.  KDE3 is just so good, they are going to have to do a lot more work on KDE4 to bring it up to being close to as good.  Until then, I’ll choose flexibility and customizability over the extra coolness of KDE4. KDE4 has a lot of promise.  It is going to be beautiful when its ready.

A few direct comparisons between OpenSuse running KDE and Ubuntu running Gnome (not attempting to differentiate which are distro and which are desktop environment features).  Ubuntu’s printer and network configuration is easier.  I like Ubuntu’s package management tools better than YaST, but both work quite well and are fairly easy to use.   The odd difference between the Ubuntu package manager on the normal menu vs the one on the system configuration menu is strange, but both worked for me fine.  I like the ease of YaST’s patterns, which I mentioned made it easy to install KDE3.  Its been quite a while since either of these distros has failed to resolve a dependency for me without having to go hunting.  Overall, I like YaST over the configuration tools in Ubuntu.  It is nicely organized and consistent, and they keep adding more stuff to it.  The real reason for my preference might be familiarity.  YaST has been improving steadily, but has been consistent in its look and feel for several versions.  The one thing I really don’t like to use YaST for is to configure Apache.

The short answer to a comparison between Ubuntu and OpenSuse is that there is no clear winner.  They are both excellent.  They both install and run without a lot of headaches.  They both perform well on my hardware, and they both found and configured all of my hardware without difficulty (including various USB devices I plug and unplug at whim).  The few differences are minor, and for the most part (other than the issue with unplugging the monitor and plugging it into a different display, which Ubuntu didn’t like), it comes down to a matter of taste more than it does functionality.

Both of them beat Windows soundly, on features, customization, and ease of installation and use.  Linux just keeps getting better, and the gap is widening.  Now if we could just get commercial software to distribute end user software that is cross platform…  but that’s a post for a different day.

My undergraduate degree was in history.  As credentials to build a profession on, its a pretty useless major, unless you want to teach history.  But one thing it gave me was a perspective on the flow of time in the human story.  Everybody always thinks that now is the ultimate stage of development.  Modern is wonderful.  Woops, I mean post-modern is better.  Oh, no, I mean post-post-modern is everything we ever needed.  Wherever we are now, if we’re inclined by temperament to be content we have reached the pinnacle.  If we’re inclined to optimism, now is messed up but surely truth and enlightenment are a stage or two down the line.  If we’re skeptics then whatever we did yesterday was definitely wrong, we’re not that much better today, and tomorrow is doubtful.

When we call something “post” something else, generally its a reaction–a rejection of whatever was in the something we are now post.  It is the plight of youth to react to whatever the generation before thought was great.  It is the plight of the young, bright professional to think that the voice of experience has stagnated.

Not really being the voice of youth any more, I have come to understand that my views on how to do things aren’t the pinnacle of perfection any more than the problems built into previous generations of methods were perfect.  The methods of tomorrow aren’t going to be perfect either.  While technology develops, people basically have the human capabilities that they have had for the past several thousand years, and unless we hit a technology singularity (or maybe even then, just faster) or the world ends we are about the same as the people a couple of thousand years from now.  Human nature, human creativity, human relationships, all work about the same over time.  And these are what our lives are made of, regardless of whatever the current hype.

Given that work is done by humans, and humans tend to be about the same as they have always been, why does it seem with each generation that the new methods for getting stuff done are so much better than what has gone before?  Is this just illusion?  Is Agile really better than TQM and the “tired” methodologies that went before?  Is post-Agile an improvement?  Total Quality has a lot of obviously valid points in retrospect.  The Theory of Constraints is mostly just common sense applied to processes.  To a great extent, the heart of Agile (as exposed by the Agile Manifesto) is just a reminder about the human core of all processes.  Here’s a guarantee:  our children will think Agile is nonsense, and whatever is the new fad is incomparably more effective.  Our grandchildren will be post-whatever our children are.  And they’ll be right, but at the same time they will be wrong.

I’m convinced that methodology itself mixed with human nature plus time is the main problem with methodologies.  Agile isn’t any different, as successful methodologies go.  It reminds us of the important stuff at the heart of processes:  people getting things done for people.  It reminds us that it is the people who are important, and that trust matters.  This is good stuff.  Important stuff.  Then, being creatures of habit, we formalize the rules of our methodology.  We try to make it reproducable.  We write books so that other people can do it too.  We tell them with confidence that if they do it like we did it, they will succeed.  If they don’t succeed that means they didn’t follow the method.  Bull.

What is successful about Agile isn’t the methodology.  Its the philosophy that processes are about people doing things for people.  Its the reminder that we should aim for the goal.  When the philosophy succumbs to the habits, the truth underlying its success dies.  The method becomes king rather than what made it successful in the first place.

Here is what makes Agile successful:

• People matter more than tools and processes.
• Software should work, and it should accomplish the goals for which it was created, as quickly and effectively as possible.
• It must accomplish the real goal, not my conceptualization or explanation of the goal.

Methods that enforce these concepts are successful, at least until the method takes over and the concepts are buried.  Processes stagnate, because humans are creatures of habit.  Habit isn’t bad.  It helps us reproduce good behavior.  But it can also kill us when we stop examining.  New processes, if they are any good, re-remind us of the things that matter.  They strip off the stagnation and bring us back to the things we were reminded of when the old processes were new.

Post-Waterfall is a very good thing.  Post-Agile is fine.  Post-Post-Agile will probably be fine too.  Maybe it will be a reaction against the fuzziness and a re-injection of more formality.  If so, it will be because people got too lazy with post-Agile development and devolved to ineffective chaos.  We’ll cycle around and figure out that whatever went before was all wrong.

At the base of it, its still about applying technology to make people more effective in reaching real goals.  Its the people and the goals that matter.  Pick a methodology that enforces these critical concepts, and do it until it becomes habit, then shake vigorously and repeat.  Do that until that cycle becomes habit, then do something else.  But always remember that its the people and the goals that matter, not the technology, and not the processes themselves.

I’ve been using Suse (and OpenSuse) for years, ever since moving off of RedHat when they started Fedora.  Other than a couple of Gentoo detours, RPM based distributions of Linux have been where I have lived since RedHat 6 point something.

I have been having some problems with my laptop (running OpenSuse 10.2) and it got to the point that I thought it might be easier to get a clean install than figure out what was causing my various problems.  That sometimes happens when you get a year and a half of messing with stuff accumulated.  You can’t easily roll back what you can’t remember you did.

So I’ve been waiting with anticipation for OpenSuse 11.  But I got tired of waiting, and this weekend I decided to try Ubuntu, which I’ve played with but haven’t actually used.  This is quite a change for me.  I’ve worked with Debian servers from time to time, but not by choice.  I have also been using KDE for a very long time (starting with KDE 2).  For the first time in recent memory, I’m moving to Gnome.  I might end up changing back to OpenSuse and KDE when it is released, but this will give me a month or two of spending the ten hours a day I spend on my laptop finding out what the other side is really like.

Ubuntu installed very nicely.  The installation process doesn’t seem as flexible as OpenSuse.  I missed OpenSuse’s easy package group selection.  But overall, I was fairly impressed.  Within just a few minutes I had a shiny new Ubuntu desktop.  Of course, it didn’t use my external monitor, and the display resolution it started with stunk.  But a quick install of the binary drivers (not installed by default because they aren’t “free” in the sense that they don’t come with a open source license), and a little configuration tweaking, and I had my dual monitor Xinerama setup working nicely.

I am not a fan of sudo, at least on boxes on which I am the only user, but I think I can get used to using it.  I’d rather open a terminal and su to root.  But its not a big deal, I’ll adapt.  On boxes where there are multiple people who might touch system wide settings, sudo is very nice.  That just doesn’t apply to my laptop.

Wireless, sound, and all of the rest of my hardware just worked.  Adding packages was very easy, and everything worked very nicely.  While there are some things I like about smart over synaptic, I’ve used synaptic before and it has come quite a distance since I last worked with it.

One thing that really impressed me was adding a printer.  The system-config-printer tool found my printer on the network (I print directly to the IP address using CUPS) and got it set up and configured almost without intervention, and it was about the easiest printer setup I’ve ever seen.   Way easier than it would have been in Yast (at least on OpenSuse 10.2), and of course way easier than Windows does anything.

I prefer administration tools that put everything in one place for you, rather than having to hunt for the right tool to tweak settings.  So I like the approach of the KDE preferences tool and Yast better than the “What was that thing called again” search for the right tool.  Some of that might be familiarity, but Ubuntu/Gnome could do a better job of streamlining the preferences process.

Overall, I am impressed.  The system install, adding software, getting things configured, and getting started using the new setup was very quick and easy.  I think just about anybody could do it.  Having recently struggled through trying to get Windows 2003 servers working properly, Linux definitely beats Windows, and I would not have any trouble at all recommending Ubuntu to anybody.

Some of my easy conversion might be because I’m not exactly new to Linux, and I knew going into it what some of the differences were going to be.   There are also a couple of non-distro related factors that make migration easier than it used to be.  I have a handy external USB hard drive.  So backing up my home directory so I could repartition my laptop was simply plugging the drive in and dragging the home directory over.  The hard part was waiting until it finished before I started messing with it.  When I got Ubuntu installed, I just dragged my stuff back to my new home folder.  Also, so much of my work is on Google, or in an svn repository, or other places out there somewhere in the cloud that saving the actual stuff on my hard drive is much less critical than it has ever been before.  My laptop is much more of an interface for stuff out there somewhere than it is actually used for storage.

If I change back, or if I find out things that make a difference, I’ll post them here.  I’m not sure how much I’ll like the differences in editors, terminal, and so on.  Only time will tell.  At the moment, it looks like my experiment might turn out to be a little boring, so I’ll stop talking about it for now.

I was delighted to see the announcement that Zend Framework is partnering with Dojo.  Life will be easier.  I like Zend Framework.  I like Dojo (although I’ve used jQuery on some recent projects).  What could be better?  Time from idea to working solution will be shorter.  As a result, customers and users will say “Wow, how’d you do that?” and get back to working more efficiently and in ways they couldn’t before, quicker.  That’s good stuff.

When I saw the information on Google App Engine, it occurred to me that we are witnesses to a paradigm shift. It isn’t just about Web Services and Services Oriented approaches to them. It isn’t just about Facebook and social apps. Its a different way to look at how we build and provide services in general. Its a different way to look at data and integrating it with services. Its a true shift, and we’re just at the beginning.

Since we are just at the beginning of this, its hard to see what it will really look like. We have the start of it in Google App Engine (and some of Google’s other services), and Amazons Web Services, and in IBM’s AlphaWorks QEDWiki. But it isn’t just about hosted services, or about mashups, or about APIs. In fact, I’m not even sure exactly what it is, or how to think about it. But get ready. Its going to be different from what we’ve ever done before.

If you are in the business of building business applications and your software does not run on the web, you are already two steps behind. If you are in the business of building or hosting web sites, applications, or services, get ready to change your conceptual and business models. It might not be fully realized this year, but it is coming.

In web applications it is fairly common to want to edit multiple rows of the same kind of information, formatted into a table, kind of like you can in a spreadsheet. These forms are generally for editing existing information pulled from a database, which is usually already in an array format. Most of the examples I can think of at the moment need explanations of the context to make much sense, so I’ll pick an example that isn’t as likely to be useful but doesn’t need explanation.

Suppose I have a table in my database that lists contacts with title, first name, last name, and so on. When I fetch the data from the database, I’m likely to have an array that looks something like this:

I want to get this into a form structured as a table, that submits the whole thing as one form. A row in the table should look like this:

So that the posted results like this:

I want the array to look like this because its much easier to handle the results than if we used the actual field name or id as the variable name.

I couldn’t find anything in the Zend Framework documentation, lists, or various tutorials that explained how to do this very clearly, so I thought I’d post at least one way to accomplish this in Zend Framework. This is based on 1.5. I’m going to start by showing the whole thing together. In practice it might not appear this way in the context of your controller class, but hopefully this is clear enough to understand how to apply it. After I show the whole thing I’ll walk through it and explain the pieces.

As you can see, this approach uses subforms, which allows us to automatically build the array in the form, as well as having a convenient structure for applying decorators. In order to get the array right, we use two layers of subforms (subForm for $contacts, and rowForm to create each row of the table). I couldn’t figure out a good way to not do that and still get the array I wanted in the html. setElementsBelongTo doesn’t quite do the trick.

This just sets up the form, tells the form it should post and have an id of ‘contactForm’. You might note that I didn’t set an action for the form. Zend_Form does set the action, but leaves the value blank. This conveniently posts the form back to our action (supposing all the browsers handle blank actions as specified in the relevant RFCs). The $subForm line just creates our first subForm for use later.

For each row in our data we create a subForm, which we populate with an element for each field in our data, setting the value for each to the value from our data. We decorate this subForm in the loop, wrapping each element in a td tag. Note that we don’t need to worry about the name of the element. We just set it to our data field name, and the framework’s subForm handling deals with the rest. Of course in practice your fields might not all be simple text fields, so the actual code would get more complicated but the process is the same. I also should mention that in practice you almost certainly want to check that $array is actually an array before you do this, or you’ll be looking at lots of warnings (at least) in your logs.

Now that we have a rowForm for each data row added to our subForm, we can decorate each of these with a tr tag, and assign the whole thing back to the parent form. This is where we tell Zend_Form that the parent should be ‘contacts’, so that gets in our result array. Then we decorate the contacts subForm with the tbody tag to get it nicely fit into the table.

We need to decorate the whole form with a table tag. This likely needs a class and id, but we leave it plain in this example.

Chances are we want a submit button, so here we add it to the bottom, and decorate it with its own row in the table.

The order of a surprising amount of this is arbitrary. You could set the table tag toward the top, for example, if it flows better for you. There are, of course, also completely different ways to do this. Several of the examples in the manual extend Zend_Form rather than do it the way that I have here. To me, this seems more clear.

Recent work changes have led me to think more intensely about development methodologies, project management for small teams, and similar issues, and I’ve been reading more about Agile development. I am much more interested in the philosophical underpinnings, the values of Agile, the overall approach of Agile development, than I am in the specific practices that make up a particular “official” methodology such as XP or Scrum. So I was interested to see Brian Marick’s post on his Exploration Through Example blog entitled “What is Agile? — beats me, but I know it when I see it“.

To me at my current stage of thinking, the most important components of Agile development include:

• short release cycles with working software to the customer as often as practical
• safety, trust and openness in the development team
• frequent (almost constant) and informal interaction between members of the team
• open communication between the user and the developers
• transparency and visibility of information
• focus on solving the business issues
• adaptability
• emphasis on the practical over the ideal, but always pushing for incrementally better
• an attitude that fosters creative solutions, dedication to solving real problems, and ownership of the problems, process and solutions

I think that the Agile manifesto wraps a lot of this up. It embraces the following values:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Reponding to change over following a plan

Agile processes focus on adapting to real needs, rather than trying to predict them and then sticking to the predictions regardless of how accurate they turn out to be. They focus on meeting the needs of people over the needs of the development process itself.

There is a lot in the various methodologies that is good, but to me the philosophy for development espoused by these ideas is what its about. Methodologies need to adapt to the situation, just as other components of planning and process must.

Most of the information I have come across in the “formal” Agile methodologies completely overlook or dismiss distributed teams. This is just not well connected to the real world. Instead of dismissing those of us who live in distributed teams (and the numbers are growing), we need ideas about how to reach the goals of Agile development in the reality we live in. That mostly means finding ways to work closely together, to have transparency, code review, collaborative approaches to working and coding, and high levels of interaction and trust in teams that might be physically together only rarely.

For me, the biggest component to success over the long term in development projects is in the relationships–in trust and open communication–between the members of the development team, between the developers and users and customers, and between the development team and management. If those are missing, things can quickly fall apart.  Building and maintaining these in distributed teams, and when the team is not local to the customer, is challenging.  The real world many of us live and work in day to day, the team, the boss, and the customer might be literally an ocean away.  In this world the values of Agile development that I outlined above hold just as true.  Its just the charts scribbled on paper, pair programming on one computer, and in person stand up meetings that do not.