Knight Drupal Initiative

I'm getting ready for the Future of News and Civic Media Conference, and as part of the preparation we have been putting together a research/development site as part of our work for our KDI project. We are still evaluating the different options that will make it into the initial versions of our platforms.

For this stage of the research, we chose to focus in on the events of the Iranian election -- first, I was woefully underinformed about the events of this election, and given the noise (or, some would say the lack of it) about the event, this seemed like a good opportunity to set up a tool that would provide an overview of the event, with a cross-section of primary source material (largely from YouTube and Twitter) and content from more polished sources (both blogs and traditional/mainstream media). In a few ways, this provides a real use case: if an organization doing grassroots organizing wants to find out about and publicize events occurring in several places on the same day, this type of aggregation from multiple sources allows real-time data collection from disparate sources. People can continue to use the tools they already use to discuss their work, and the main site can collect information from these sources and present/recontextualize it from a central location.

With minimal effort, we were able to put together a rough set of tools that allows people to get a perspective on the events going on in Iran. We built the site using tools freely available within the Drupal community. The bulk of the heavy lifting is done using FeedAPI and friends, and the folks over at Development Seed deserve huge kudos for unleashing these tools into the world. We also used the Views module to split out the content of the feeds. Obviously, the resulting site is a proof of concept, more of a pre-alpha prototype than anything else, but the site is useful as a research tool. We'll also continue developing on the site after publishing this post, so the site will undergo changes over the next few days as we modify things/tinker.

On our testing site, the Twitter traffic provides a pretty scattered overview, but taken in aggregate it allows one to scan raw data over time and get a sense of the ebb and flow of events on the ground. Initially, this feed was pretty free from spammers, but lately some opportunists have taken to using the #iranelection hashtag to get a broader audience for their content.

The YouTube videos provide another means of getting a sense of what is happening. As with any piece of media, the source of the content and bias of the author need to be taken into account. Also, our means of collecting these videos is certain to miss some content, as we are just aggregating the feed for the search term "Iran election protest".

The content coming from MSM outlets and blogs is admittedly arbitrary -- we tended to favor sources that had a clean tag for either Iran or the Middle East; the dearth of effective tagging on information coming from both traditional news outlets and some group blogs is discussed in more detail later in this post.

The important piece of this from our perspective: this tool can be easily built and focused on just about any topic. When it is set in motion, the site will gather and import information that can be used, analysed, recontextualized, or otherwise modified. This can be a tool used for any topic discussed on the web:

• grassroots organizing/community media -- use various data streams to collect information in real time that can be analyzed/collected/synthesized over time
• lesson creation -- aggregate writing about a specific topic, then choose the imported resources that align with your learning goals. Edit these assets as needed, or add in information that is missing
• farmer's markets -- farmers/sellers/market organizers use a microblogging platform to describe what they will be selling, and where; this information can be aggregated and geotagged, allowing an accurate breakdown of what is for sale at local markets.

As we built this out, we encountered some surprises. A short list includes:

The Wall Street Journal uses feedburner as their for their RSS tracking. However, this is exposed in their feeds (or at least in their World News RSS feed), and the original URL of their article points to http://feedproxy.google.com, as opposed to a location within WSJ.com. Additionally, the only tag for all content coming out of this feed is "Free". At the risk of stating the obvious, tagging all posts in your outgoing RSS feed as "free" is worse than useless. I have a hard time believing that they don't have the resources to do this well, which makes me wonder why it is allowed to be so sloppy.

NOTE: The following paragraph was edited because, well, it is completely wrong. The Huffington Post nails syndication. The links on the syndication page point out to RSS feeds of several dozen tags. In short, it rocks.END NOTE

The Huffington Post, which makes extensive use of tags to categorize posts, only offers 4 RSS feeds (Full feed, Latest news, The Blog, Featured posts) of content. Even though you can browse posts by tag on their site, you can't actually aggregate by these same tags. Given how easy it is to expose the content of any tag via an RSS feed, I can only conclude that the choice to not support feeds based on tags is tied to their business strategy. Given how little of the Huffington Post homepage is actually original content, it's surprising to see them reducing the number of ways people can interact with their site.

As a final note on this, it's not uncommon to see other more popular group blogs/major news outlets doing similar things. Talking Points Memo eschews aggregation by tags, and none of the major papers do much beyond feeds that summarize articles appearing in their standard sections. Within their RSS feeds, most major papers do little in the way of tagging content. Additionally, most papers/blogs include little more than a brief teaser within their feed. Given that most of these sites devote a fair amount of screen real estate to advertising (and some, like the NY Times, even embed ads in their feeds), their desire to bring eyeballs back to their sites is understandable.

However, an advertising-driven paradigm seems unlikely to work, and it seems especially shortsighted given that excessive reliance on advertising money is frequently cited as a contributing factor in the decline of newspapers. The new media economy seems unlikely to be a link economy; micropayments, paywalls, and/or "better" targeted ads feel equally fruitless. A remix-with-attribution economy feels more likely, with the looming caveat that no one has really figured out how to make that work in a way that makes all the people in the supply chain happy. But the necessary first step is to move away from the notion that the finished work is the starting point or ending point for profiting from that work; that places too high a value on the role of content, and how people interact with information. Content -- the article -- is the middle point of the process, and on the web "content" can be understood as one point in an ongoing chain of synthesis/recontextualization.

Working with aggregation -- arguably the simplest means of republishing and recontextualizing content -- gives an incomplete yet suggestive view into two elements: how an organization understands the web, and how they view the role of content. From what I have seen, both mainstream news outlets and popular blogs do a poor job of making the most of their content. If I had to reduce this down to a single reason, I would say that there is a perceived need to control how people consume content, and that this is tied to the need to pass eyeballs over ads.

However, tethering the distribution mechanism of online content to a strategy designed to generate more pageviews (ie, News as SEO) seems destined to fail, as the gimmickry of SEO doesn't mix well with unbiased reporting.

And I would love to end this post with the next great idea on how to support working writers within this model, but hey, it's late and I need to pack. But I'm looking to forward to learning more about other approaches over the next few days. I'll write up any ideas as they come, and for those who want to follow along, dip into the feed.

Part of our work for the Knight Drupal Initiative includes work to lower the barriers to getting started using Drupal sites to power community-driven media. As we work toward this end, we have multiple tools at our disposal, including:

• Better documentation, for both site administrators and people using the site
• Easier installs, to allow people to enable more complex functionality with less work
• Easier and more flexible theming

At the risk of stating the obvious, the work we do for the Knight Drupal Initiative will also be immediately transferrable to all of Drupal development.

Getting Started: Journal

The Journal module allows us to keep a running record of changes made on a Drupal site. Currently, it allows you to take notes as you build out your site. It's a great tool to track the specific steps people take as they work within a site.

We will build out two additions to Journal:

1. Tag journal entries, which would allow us to create categories of journal entries. For example, while configuring the WYSIWYG API to work with Filefield and Filefield Insert, we could tag these entries as "Text Editor" -- later, this would allow us to see exactly how the editor was configured.

2. Export journal entries by tag or by date range, in csv, ordered list, or unordered list format. This would allow us to generate documentation on how sites are built. This documentation can be used on drupal.org, for site administrators as sites are delivered, or to create end user documentation.

By way of example, let's say we have built a site that is designed allow teachers to create online portfolios as part of professional development. Prior to delivering the site, we could walk through the steps of creating a portfolio. At each step, we would comment on the process of creating the portfolio using the Journal module. We would tag each entry as "portfolio creation." Then, once all the steps have been completed, we would export the steps as an ordered list, and voila, we have our end user documentation.

Moreover, this process also gives us the instructions and order we need to develop functional tests using Simpletest.

With these additions to the Journal module, we will have the base package for creating more comprehensive documentation about all aspects of building and using a site. However, one of the challenges of documenting a site build is getting the documentation into the hands of end users, or people not directly involved in the build. Often, comprehensive documentation can be overwhelming as it floods someone with information.

Inject Me!

The Advanced Help Injection module builds off the Advanced Help module, and allows us to draft documentation using Drupal's core book module. Then, the documentation can be inserted on specific pages within a site, so people only see the documentation they need when they are at a place where they need it. So, the documentation generated using the Journal module can be presented to people exactly when it makes the most sense.

And finally: Advanced Help Injection allows the documentation to be exported as a module. So, all the documentation can be pushed to code, and delivered with the site. This brings us closer to the goal of having a site that is self-documenting, for both site admins and end users.

Moreover, the process of creating these tools can actually be documented using the tools themselves. While it feels like a blend of eating our own dogfood and chasing our own tail, it works.

What Are We Documenting, Anyways?

In this post so far, we have looked at simplifying the task of creating documentation, and using existing tools to get that documentation to people in a more efficient way. However, documentation assumes featuresets, as we can't have documentation about what to do without something to do.

This is where we cue the Features module. Take a minute to check out the latest post illustrating the power of Features over on the Development Seed blog.

As the video and post shows, the Features module allows us to create bundles of featuresets that can be moved from site to site. We envision a system where two things happen:

• Featuresets are paired with documentation, so that any set of features can be accompanied by inline documentation; and
• The process of creating new sets of features is fully documented using the Journal module, to empower more people to build and share features.

Making It Look Pretty

Theming and design are both enormous categories, so I'll be brief. In parallel with the work described here, we are working with Joon Park to build out a base theme (currently named Annex, but it might get changed to Mango Smoothie for the official release) that simplifies the process of creating a unique look and feel for your site. For end users, it supports some simple eye candy out of the box; as one example, it has multiple block regions, including some that support accordion and tab effects by default. For themers and designers, the process of inheriting features from the base theme to subthemes has been overhauled and simplified, allowing people more control and flexibility in building sub themes. More to come on this; ideally, we will have a release at some point in June.

Earlier this spring, the Knight Foundation let us know that our proposal to the Knight Drupal Initiative was accepted.

We are equal parts honored and excited, and work is underway.

Our project targets community media, and seeks to lower the barrier to entry for communities looking to collaborate with other like-minded groups via the web. One of the uses for our work will be within journalism, but other uses include collaborative creation of open courseware.

To be clear: this project will simplify the process of creating, distributing, and using open courseware. We want schools to spend less money on textbooks, and our work -- which will be freely available to all -- will support that goal.

As the project progresses, we will update this space with tutorials, status updates, etc. This page collects our Knight Drupal posts, or you can also subscribe via the feed.

For a good portion of 2008, I have been writing a book on using Drupal in Education. It has been a pretty incredible process, filled with rewards and challenges I didn’t envision at the outset.

Among the challenges: I began writing the book when Drupal 6 core was still in active development, and the contributed modules featured in the book did not yet exist in their D6 versions. As a consequence, I ended up writing two books to create one; the first version using Drupal 5 to help frame the scope of the book, and the second, final, version updated to reflect the improvements and changed processes in Drupal 6.

Among the rewards: a chance to see Drupal through fresh eyes. I’ve been working with Drupal for nearly four years now; writing a book targeted for people new to Drupal, and/or with a limited technical background, provided me the opportunity to slow down and examine procedures we had come to take for granted – things like adding a new content type, or adding a view. CCK and Views are critical to building a site within Drupal; we haven’t rolled a site out in the last couple years without these modules. The process of documenting their use helped me see the barriers that new users face when trying to learn these modules for the first time.

And while we are on the subject of Views, one of the other rewards of writing the book was being able to focus on the improvements between Views 1 and Views 2. The conversations and the development of Views 2 have been ongoing for over a year, and the work and effort has resulted in a tool that is more powerful while being easier to use. The ease of use of Views 2 in Drupal 6 shifts how we can develop, as Views 2 eliminates even more problems that used to require custom development.

The other realization I had throughout the course of writing the book centers around how we approach training in general, and Drupal-based training in particular. In discussions of training and usability, one main challenge revolves around identifying your audience: who are you training? What are their skillsets? What do they need to know to work effectively?

Most Drupal sites have at least three primary types of users: people who read content in the site; people who create content in the site; and people who maintain the structure of the site. There can (and usually are) overlaps between these roles, and some larger sites also have additional roles: for example, people who only add video content, or administrators who only edit/moderate content. And this is where things start to get interesting from both a training perspective and a book-writing perspective. Administrative tasks -- things like creating a new content type, building a navigational structure, configuring user profiles, configuring groups, etc -- are mostly strategies designed to meet needs. These strategies, once built into a site, provide a structure that people can use to do their work. The better these strategies have been executed, the easier it is to work within a site, and the more usable the site is for all stakeholders.

Which is all a long way of saying: site admins need to learn how to solve problems with Drupal. Other types of users shouldn't have to care. They are coming to the site to do work, and they shouldn't need to be bothered with *how* the site runs. From a training perspective, this results in multiple trainings around a single site

And with that said, the more we can simplify managing Drupal for site admins, the better. On more complex sites, we are already creating custom interfaces to make site administration easier, or less "drupal-ly."

Really, I'm still digesting the lessons (I think/hope) I have learned regarding Drupal, training, and usability. I'm going to be optimistic and assume that these thoughts will become more coherent, and if/when they do I'll share them here.

In the meantime, now that the heavy lifting involved in getting the book out is behind me, I'm looking forward to devoting more attention to other projects. In the upcoming weeks, we'll be doing some (much needed/long overdue) work with DrupalEd, and doing some more work with RSS Import (along these lines, but with an eye toward making this happen). We have some code that we have developed on some ongoing projects we need to release out, including some Drupal 6 code that can be used to create an amazingly flexible and simple online portfolio application. We're also still in the pipeline for the Knight-Drupal Initiative; as progress occurs I'll update this space.

We formally submitted our Knight Drupal Initiative application last week. We're now officially in the pipeline; when we hear anything from the Knight Foundation we'll post details here.

Until then, this paragraph from our application sums up what we're trying to do:

We would like our project to help catalyze profound disruptions across several pre-existing markets. We would like to shift the definition of journalism; and we would like for this shift to diminish the reach and influence of corporate media outlets. We want citizen and community media to become more accessible, and barriers impeding prolonged collaboration between local, regional, national, and international groups to disappear. We would like to see schools and training organizations reduce their reliance on textbooks, as a secondary use of the tools created by our proposal is developing and distributing open courseware. On a more fundamental level, we want to see web publishing tools relegated to the level of the desktop text editor: something that we use daily without thinking or worrying about the details.