Archive
Learning and community life cycles
My first round of experience learning a web development environment was in the 90’s, when I needed new ways to deliver financial reports to department managers and built web reporting using ASP pages on Windows NT servers. It was live. It was on demand. You didn’t need to run a special client program to get your reports. It was one of my first experiences of providing people with a tool that they hadn’t realized they needed, but within a few weeks everybody was convinced it was essential to getting their work done.
I jumped into the online communities (basically email back then) and asked and answered questions in various groups. I quickly realized that PHP fit what I was trying to do better, and I moved to the PHP community. I learned a lot from those groups. My personality is to listen more than I talk, both online and off, but in those days I was excited about what I learned and wanted to help others learn it too.
It didn’t take me long to notice that people ask pretty much the same questions over and over. Many groups online are used largely by people who are learning a new technology. That’s a critical function, and its a good way to distribute the learning. Like many people, I got tired of answering the same questions, and started thinking maybe people should at least make an attempt to look at the group’s archives.
Since I first started, we’ve moved through forums, blog posts with arguments in the comments, and then to sites like stackoverflow.com and microblogging. They are all useful, and all of these formats for distributing information and learning still continue to some extent even as they are replaced by newer models.
Its fun to go through the life cycle with each new technology. You learn something new, hit snags, and go looking for where the community is. Its even more fun when the community is new and the best practices are still being fleshed out. The beginnings of the CouchDb and Node.js communities were more recent examples of that for me. I didn’t catch either of those waves right at the beginning, but I was fairly early. True to my personality, I tend to listen more than talk, but I enjoy the discussions of sorting out the best way to do things.
The great thing about the early days is that the questions aren’t old for most anybody. Not that long ago the package manager for Node wasn’t settled, and there were discussions about the best way to handle that need. Now npm is the standard, and the question is fairly well settled. People still raise the question of whether it should work differently, but most of the questions have been answered. At least for the moment, npm’s place in the Node.js world is fairly settled. In this case, it happened fairly quickly.
Each generation of developers, of course, reinvents it all. Before long you start to see the same discussions popping up in every community. I’m amazed by people who continue to patiently answer the same question again and again. I don’t have the patience for it. I am much more tolerant of helping people in one on one, mentoring new developers in my team, especially when the new guy is really learning.
In Node, people get caught by async programming, and look for ways to make it work differently. Maybe at some point people will come up with a new paradigm, but Node is async and uses callbacks. If you don’t like that maybe you’re on the wrong platform. In Couchdb people get caught on how to construct views, and whether you can make views dependent on external conditions like the contents of other documents. Someone explains again why it doesn’t work like that. Some discussions never die, like when people find new ways to do things insecurely or rediscover familiar patterns. I’ve recently seen some discussions about dependency injection in PHP that seem rather worn to me.
The communities as a whole go through the same cycles too. People from the Erlang community might notice that many of the discussions and debates in the Node.js community are struggling with issues Erlang had to solve during its earlier stages. Languages evolve and have to struggle with “new” issues that are familiar to other platforms, like when PHP adds new pieces of the OOP puzzle and new debates about typing and inheritence in Java erupt in new contexts.
The world in general owes a great deal to all of the people who work through these cycles, and especially the role filled by people who seem to tirelessly answer the newbie questions. We’re all starting somewhere, and usually we find ourselves learning new things fairly often as the paradigms change. I’ve been through a few generations of PHP and MySQL, but I’m newer to CouchDb and Redis. The cycles repeat, but not exactly. Every once in a while something truly new emerges and things progress. That’s a side effect of the same challenges being faced again and again. Every once in a while, someone asking the same question the billionth time does come up with a novel answer. Then we wonder how we ever got by with the answers we all accepted before. Other times people give a name to something people have been doing for a while. Things move on. Technologies and people come and go. The tools we have now are worlds beyond what we had before. What’s next? Who knows, but there will be people learning it, and asking questions, and someone will be tirelessly answering the new questions for the billionth time. To the people who ask the same questions until the answers finally do change, and to all those people who answer the same things again and again: Thanks!
Google Apps Scripts
I was recently asked to review Google Apps Scripts as a potential platform for some simple business apps, tying spreadsheet data to Google sites. The organization already uses Google Apps, and it seems like a short leap to using Google Apps scripting. My conclusion from the review was to recommend looking for another platform for building apps, and integrate with other information in Google Apps in other ways. It might be that Google Apps Scripts are just too young, and the feature set will grow to fulfill its promise. At the moment, its just not there yet.
The first hurdle is that building apps in Google Apps Scripts is clunky, and doesn’t really feel like working with javascript. Javascript is flexible and easy to work with, if at times hard to wrap your head around its quirks. Working with Google App Scripts felt like going back a decade. A large part of that is because the Script platform is built around GWT. You don’t really work with HTML, CSS or the DOM. You can’t generate client side javascript or any kind of markup directly. Everything has to pass through a series of overlaid panels with grids and widget objects that you put field objects, labels, and text into. This works, and in the end you can make decent looking web pages out of it, but its very cumbersome.
There is no CSS that you can work with in Google Apps Scripts. In my original version of this post I had my workaround here, but in the interest of letting people find that without having to read through the rest of this, I pulled that out into its own post. The approach let me apply the whole set of styles to all “tableheaders” across the whole app, and if I wanted to change the styles I could change it one place and have it applied everywhere. That’s not particularly wonderful, but it sure saves a lot of setStyleAttribute calls. Without this approach, applying the four styles in my appStyles example above to the table headers would have taken four uses of setStyleAttribute for each column in each table that appeared in the app. This approach could be improved on significantly, but it solved my immediate headache.
What killed my ability to use Google Apps Scripts, though, was the lack of searches or queries for data in spreadsheets. When I started digging into this I assumed that Google would provide search capabilities, sort of like they do in the Spreadsheet data API. As far as I can tell this capability is completely missing, so far, from Google Apps Scripts. There’s a feature request for it, but it isn’t there yet.
This means that if you want to find something in a spreadsheet, you have to load the whole sheet (or range of cells) and iterate through the whole thing looking for the bit you’re actually needing. This works, but it is slow. It would be fine if you had a few dozen rows, but it doesn’t take much data to be getting into a thousand or few thousand records, and this isn’t an acceptable way to look up data. I wasn’t expecting this to be an approach suited for hundreds of thousands of records or anything, but from my quick tests it’s too slow for 1000 rows to be practical.
On top of that, it isn’t really asynchronous as we expect things to be in a modern platform. If I was building the same type of thing with jQuery or something, and parts of it took more time than I really wanted to keep users waiting (which these days isn’t long), I’d slap out the basic HTML and fill in all the pieces as they became available. That would hide the slower data on a panel or tab or whatever that isn’t displayed on start up, allowing a delayed load. You can’t really do that with Google Apps Scripts, since you can’t write custom client side code beyond the relatively few event driven callbacks they provide. From what I can tell, you pretty much have to load all of your data at the start, including data you aren’t going to use until people start clicking. That loses you those seconds to get your data in order that might otherwise be available before the user looks behind the curtain. There isn’t any way to swap out content, other than simple things like text values. You can’t manipulate the DOM. That’s no way to build a web application.
Most of the problems I had using Google Apps Scripts were related to how young it is. The documentation is incomplete at best. If you want to know what most of the classes do, you have to dig into the GWT documentation. Most of Google’s code documentation is decent (I recently had occasion to work through their OpenId documentation as well, but that’s a different post for a different day), and I expect that before too long they’ll fix this documentation as well.
But if this platform is really going to be useful Google is going to have to take significant steps to embrace javascript much more than they are here. Obviously they can. They have other things based on javascript that are awesome. Unfortunately, this isn’t one of them. Yet.
Clean Models
The flip side of what I said in my last post about Models in frameworks is that models shouldn’t know anything about the controller or view. The data that the model is providing might eventually end up in JSON, or XML, or in a web page or HTML snippet, or in a CSV file. None of that should matter to the model.
Generally models should not return error messages or “no data” messages that are intended for the eventual user, unless they are generic enough to be meaningful regardless of context. It should always be obvious for the controller whether the call was successful, but the controller and view should handle how to display or send any messages. Typically errors and other similar messages should be communicated through a central error or messaging handler. The messages formatting and even content will vary quite a bit depending on how the results will eventually be delivered.
The data might be destined for a javascript heavy table, or for a CSV or Excel file, or a PDF file, or for a clean display on a light HTML page, or to pre-fill a form on a user’s return visit. All of these will have very different formatting requirements, both for the data and for any messages that might get displayed to the user. The deliveries that download files might actually never download anything if there’s no data, and might display messages in some other way.
Not building clean models, either by requiring that the controller know too much about the model, or by embedding information about the eventual delivery of the information in the model, is a recipe for spending time rewriting later, or for ending up with lots of duplicate code.
Models in frameworks
One of my biggest frustrations with most MVC frameworks is that their approach to models are generally broken. Models should hide the data from the controllers. A controller shouldn’t need to know anything about the data, including what kind of storage is used. Most frameworks don’t use this approach.
Suppose, for example, that I have a model for Users. The model should provide the ability to add a User, remove a User, and update a User. We’ll also probably need some methods for listing and other ways of manipulating the user information. However, the model should be written in such a way that if we start with the user information in a database, and later move it to a remote LDAP server, or some SOAP or REST service, or a text file it won’t matter. All of the controllers should be able to continue to call the same model methods that they were calling before, in the same way, and it shouldn’t matter to them where the data is coming from. Knowing where the data is stored and how to get it and work with it is the model’s job.
This means that the model should not extend some database class. The framework’s model structure should not assume that all models use databases. In general, most discussions of MVC frameworks try to isolate the actual database work from the controllers, but they still require the controller to know too much about the data repository. The effect is that you could probably move from a MySQL database to SQLite, but if you stop using a database entirely you are going to have some major work to do on your controllers that use that model. To me, a major measure of success for any MVC approach is how much your controllers have to know about the data. Less is better.
Zend Framework tries to accomplish this, but they basically do it by not giving you a start at any models at all. Even if it was a very abstract class it would be nice to have something to extend. At least that would give people a place to start that’s better than extending one of the DB classes, which just about all of the tutorials and articles on ZF models recommend doing. ZF’s approach to just leave it entirely out of the framework means most instructions tell people to do it wrong.
I’m not a purist. I’m a pragmatist. I don’t care a great deal about patterns for their own sake, in any sort of Platonic sense. I just want to be able to be lazy when I find out later that indeed change is normal and I need to adapt an application to shifting requirements. Because as predictable as the weather in Phoenix, just about the time that an inventory application is just about through a couple of iterations and really getting useful, they are going to want to integrate it with some remote POS system, and oh yeah can it get its user information from some place in the cloud rather than the local accounts we wanted to use last week. I’d much rather say “Sure, that’ll be ready day after tomorrow” than say “DOH!” That requires the controller to be ignorant about the data repository.
Post-Anything
My undergraduate degree was in history. As credentials to build a profession on, its a pretty useless major, unless you want to teach history. But one thing it gave me was a perspective on the flow of time in the human story. Everybody always thinks that now is the ultimate stage of development. Modern is wonderful. Woops, I mean post-modern is better. Oh, no, I mean post-post-modern is everything we ever needed. Wherever we are now, if we’re inclined by temperament to be content we have reached the pinnacle. If we’re inclined to optimism, now is messed up but surely truth and enlightenment are a stage or two down the line. If we’re skeptics then whatever we did yesterday was definitely wrong, we’re not that much better today, and tomorrow is doubtful.
When we call something “post” something else, generally its a reaction–a rejection of whatever was in the something we are now post. It is the plight of youth to react to whatever the generation before thought was great. It is the plight of the young, bright professional to think that the voice of experience has stagnated.
Not really being the voice of youth any more, I have come to understand that my views on how to do things aren’t the pinnacle of perfection any more than the problems built into previous generations of methods were perfect. The methods of tomorrow aren’t going to be perfect either. While technology develops, people basically have the human capabilities that they have had for the past several thousand years, and unless we hit a technology singularity (or maybe even then, just faster) or the world ends we are about the same as the people a couple of thousand years from now. Human nature, human creativity, human relationships, all work about the same over time. And these are what our lives are made of, regardless of whatever the current hype.
Given that work is done by humans, and humans tend to be about the same as they have always been, why does it seem with each generation that the new methods for getting stuff done are so much better than what has gone before? Is this just illusion? Is Agile really better than TQM and the “tired” methodologies that went before? Is post-Agile an improvement? Total Quality has a lot of obviously valid points in retrospect. The Theory of Constraints is mostly just common sense applied to processes. To a great extent, the heart of Agile (as exposed by the Agile Manifesto) is just a reminder about the human core of all processes. Here’s a guarantee: our children will think Agile is nonsense, and whatever is the new fad is incomparably more effective. Our grandchildren will be post-whatever our children are. And they’ll be right, but at the same time they will be wrong.
I’m convinced that methodology itself mixed with human nature plus time is the main problem with methodologies. Agile isn’t any different, as successful methodologies go. It reminds us of the important stuff at the heart of processes: people getting things done for people. It reminds us that it is the people who are important, and that trust matters. This is good stuff. Important stuff. Then, being creatures of habit, we formalize the rules of our methodology. We try to make it reproducable. We write books so that other people can do it too. We tell them with confidence that if they do it like we did it, they will succeed. If they don’t succeed that means they didn’t follow the method. Bull.
What is successful about Agile isn’t the methodology. Its the philosophy that processes are about people doing things for people. Its the reminder that we should aim for the goal. When the philosophy succumbs to the habits, the truth underlying its success dies. The method becomes king rather than what made it successful in the first place.
Here is what makes Agile successful:
- People matter more than tools and processes.
- Software should work, and it should accomplish the goals for which it was created, as quickly and effectively as possible.
- It must accomplish the real goal, not my conceptualization or explanation of the goal.
Methods that enforce these concepts are successful, at least until the method takes over and the concepts are buried. Processes stagnate, because humans are creatures of habit. Habit isn’t bad. It helps us reproduce good behavior. But it can also kill us when we stop examining. New processes, if they are any good, re-remind us of the things that matter. They strip off the stagnation and bring us back to the things we were reminded of when the old processes were new.
Post-Waterfall is a very good thing. Post-Agile is fine. Post-Post-Agile will probably be fine too. Maybe it will be a reaction against the fuzziness and a re-injection of more formality. If so, it will be because people got too lazy with post-Agile development and devolved to ineffective chaos. We’ll cycle around and figure out that whatever went before was all wrong.
At the base of it, its still about applying technology to make people more effective in reaching real goals. Its the people and the goals that matter. Pick a methodology that enforces these critical concepts, and do it until it becomes habit, then shake vigorously and repeat. Do that until that cycle becomes habit, then do something else. But always remember that its the people and the goals that matter, not the technology, and not the processes themselves.
David's Tech Blog 