Jun 182013
 

I recently finished reading Clayton Christensen's book The Innovator's Dilemma (TID), which talks about technological innovation and how companies can effectively manage it.  TID was originally written in 1997, and I read a (slightly) updated version published in 2011.  I found this book fascinating, especially the way that Christensen broke down innovation into two distinct categories: sustaining innovation and disruptive innovation.  Each of these categories requires different strategies to manage productively, which are supported by historic studies of innovation in many different industries such as hard drives, excavators, steel, computers, motorcycles, and more.

Sustaining innovation is technology and process updates that give a company's existing customers more of what they want.  This is in contrast to disruptive innovation, which initially has no known customers, unknown applications, and capabilities that existing customers do not value.  But once the market for the disruptive innovation is found and grows, it will eventually take over the company's existing customers, often driving the company out of business.  TID points out that it is very odd that companies so often have problems managing disruptive innovation: sustaining innovation can be incredibly complex and expensive, but leading companies will almost always spend the resources necessary to see it through and bring the sought-after innovation to market.  Disruptive innovation is the opposite--it is often cheap and quite simple compared to what an industry-leading company already produce, but it is usually of "inferior" quality to existing customers.  Since existing customers do not want the disruptive technology, the market must be found for it before money can be made.  Large, successful companies historically do not perform this search--Christensen believes that is because established companies see the search for an unknown market as risky when they have existing customers they can continue to please through their established processes.

I found the ideas presented in TID particularly fascinating in terms of the higher education industry because of my work with the Birdseye College Cost Comparison site and because of the time I spent down in the trenches as a university professor.  The university business looks, to me, very similar to the large, established firms discussed in TID that, over and over again, got driven from their markets by smaller upstarts peddling disruptive technology.  The similarities in brief:  universities are established, they offer exceptionally complicated products in terms of accredited degree programs in many disciplines, and they are completely reliant on high-end customers feeding them fat margins.  Almost all of their development caters to these high-end customers.  To illustrate reliance on high margins, consider that many private, non-profit universities derive a large portion of their net budget, often a majority, from room and board charged to students for ever-more palatial "dorms."  Really, these dorms are more similar to luxury apartments than the two-to-a-tiny-room block houses of old.  Universities are essentially getting a significant portion of revenue from an expensive, high-margin add-on to their main product, at a time when news services regularly report that cost is a great concern.

After reading TID, I do not know what disruptive innovation will rise.  In fact, TID makes the point that this cannot be known.  But I can speculate based on what I know of the market and what cheap, readily-available technology currently exists.  Video courses might be one option:  with the expansion of low-cost fast Internet connections and the affordability of streaming and receiving video, entire video lecture series and accompanying material can be created and distributed.  The rise of Wikipedia offers another option:  a vast quantity of small, discrete, digital, educational elements (videos, interactive programs, web pages, homework assignments, etc.) created by a diverse number of experts that can be mixed and matched to create educational packages tailored to meet knowledge or skill goals for individual customers.  Or it could be something else entirely.  Whatever product arises, it will initially be something that existing customers (students going to university right now) neither want nor need.  These products will also not have many features the current higher-education market demands.  Perhaps no one in the new market is concerned about homework, tests, football games, lectures, grades, personalized feedback, dorms, accreditation, degrees, certification, etc.  The lack of some or all of these features will make the new product look "inferior" when compared to the established market, but the new product will have features that appeal to a different, initially unknowable set of people--the new market.

I have no idea what this low-cost, higher-education market is or what products to serve it will look like.  TID makes a compelling case that markets for disruptive innovation cannot be known before they arrive.  That said, I do not think that companies currently providing free college courses over the Internet such as Corsera or Udacity have figured out who the customers in this new market are or what they want.  At least not yet.  Both companies seem too locked into the current university model, expending a great deal of effort trying to emulate features of the university system, trying to become equivalent to a university.  I am not sure this is serving them well.

But whatever the new higher-education market turns out to be, it will be largely ignored by existing universities when it arrives because its margins are too low and the products too simple.  Years later, seemingly overnight, the new market's products will be sufficient for most then-current university students, and there will be a massive shift away from the current university model to the new model.

We live in interesting times.  Wherever this evolution eventually takes us, it is an exciting time for higher education.  I cannot wait to see what develops!

Apr 052013
 

I recently finished a cover-to-cover reading of Frederick Brooks' classic book on large software project management, The Mythical Man-Month (TMMM).  Originally published in 1975, TMMM details Brooks' management philosophy, which he developed as manager of the IBM OS/360 operating system from 1964-1965.  I read the 20th anniversary edition (published 1995), in which two additional chapters were added that update the philosophy put forth in the original edition from the beginning of the timesharing era up to the PC era of computing.

My impression of TMMM is very positive overall; it is easy to see why this is still a required text on software project management, even nearly 50 years after Brooks finished his stint managing the OS/360 project.  There are many interesting anachronisms in TMMM that would necessitate a history lesson in computing for today's college-age audience, e.g., software development in pencil and paper (since machine time was rare and expensive), separate punch-card readers from computers, development primarily in machine code, a preoccupation with memory size of programs, a view that structured programming might be a good idea that likely goes too far, etc.  Clearly, a very different world technologically than that which we work in now.  But despite all the differences in computing technology from 1965 to today, the fundamental problems in large software projects and the structural solutions to combat them that Brooks presents have not changed all that much.  It is rather fascinating--when I was teaching, there were always a few students who went into conniptions if an assignment required them to use a terminal interface instead of a graphical IDE, a 10 year change in development habit historically.  But current techniques for building large software systems remain fundamentally quite similar over 50 years of computing history.

In TMMM, Brooks posits, quite rightly in my opinion, that conceptual complexity is the primary hindrance to quick and quality software development.  In addition, conceptual complexity is a problem that can never be truly solved.  Conceptual complexity here means the number of logical components in a software system coupled with the the amount of communication required between these components. Because of the complex interrelationships between the components, a particular component could interact with hundreds or thousands of other components, requiring a huge amount of communication overhead between the people building these components.  This communication eats up a lot of time for the people involved, which in turn requires more people to get the project done in a timely manner, which in turn adds more necessary communication...and so forth.  The cycle is recursive.  The curse of complexity is that at some point, when more people are added to a project, the amount of time the project takes actually gets longer, instead of shorter.  This is why big software is almost always late and over budget.

Brooks presents a number of management techniques in TMMM to tame the overhead brought on by this complexity.  Many of these are standard industry practice today for large projects:  Brooks recommends having a rigidly hierarchical top-down design and development process, a chief architect at whom the buck stops for design decisions, a separation of implementation from design, teams dedicated to testing and finding bugs, a team dedicated to keeping documentation up-to-date, keeping track of the project through document maintenance, starting the design with a user manual/model, source control, and regression testing.  Brooks also presented some ideas that are not really used today, including his small "surgical team" concept, an insistence on the Waterfall design method (which he renounced in the 1995 retrospective chapter), and fairly hands-off upper management (which might exist somewhere?).

While reading TMMM, it seemed to me that Brooks had an idea of future developments that actually did eventually show up.  This includes documentation contained within the source code itself (Javadoc and the like), Git-style version control, a Wiki-like system for managing project documents, and in some sense Agile development (at least for individual components).  While reading through the original chapters, I noticed a few places where Brooks hinted at object-oriented-like abstraction, but in the 1995 retrospective, he claimed to have not seen that coming.  Brooks consistently reads as being ahead of his own time throughout.

As Brooks points out in TMMM, large software projects are the most complex things that humanity has ever built, and this complexity makes them very difficult to build on time and on budget.  Further, we have collectively already solved most of the easy problems in tackling the software project overhead, which means that most of the time spent on large software projects now is simply inherent to the problem being solved.  But despite this, TMMM ended on an optimistic note:  we may not ever solve the tractability problems in building large software, but we continue to build large software, producing ever more wondrous marvels of human ingenuity.