Blogs

Some thoughts after reading this piece in Wired (although this actual blog post could have been written anytime in the last few years).

Let's imagine that the US Government announced that they had started a web site. On this site, you needed to enter your personal information, including an address, and various interests. Once this was done, you could tell the government – via the web site – all about your day to day activities: what you read, where you were going, what movies you like, etc. Then, you could identify your friends, and upload pictures and video of these friends.

This is a small subset of what Facebook users do every day, by choice. Facebook is probably the single largest opt-in surveillance program ever seen. If any government ever tried to build a site like this – even with an ostensibly worthwhile goal, like mapping public services to people based on interest, geographic location, and perceived need – the outcry would be deafening.

Facebook's "services" – and I'm thinking specifically of Facebook Connect – extend that surveillance to what people do on sites outside of Facebook. However, Facebook's internal search – powered by their deal with Microsoft – will provide an enormous amount of raw data about what individual people want. Given that these searches will be conducted by people logged in to Facebook, the search strings used can be mapped to specific individuals. As we have seen before, even a little bit of information about search strings can lead to some awkward revelations.

When people get a glimpse of how much Facebook knows about them, they generally freak out. Yet, the freak outs subside, and people keep plugging away, adding more data into the system.

Okay, time to go. Need to update my status:

Adjusted my tinfoil hat. It had tilted precariously back, exposing most of my frontal lobe.

I'm getting ready for the Future of News and Civic Media Conference, and as part of the preparation we have been putting together a research/development site as part of our work for our KDI project. We are still evaluating the different options that will make it into the initial versions of our platforms.

For this stage of the research, we chose to focus in on the events of the Iranian election -- first, I was woefully underinformed about the events of this election, and given the noise (or, some would say the lack of it) about the event, this seemed like a good opportunity to set up a tool that would provide an overview of the event, with a cross-section of primary source material (largely from YouTube and Twitter) and content from more polished sources (both blogs and traditional/mainstream media). In a few ways, this provides a real use case: if an organization doing grassroots organizing wants to find out about and publicize events occurring in several places on the same day, this type of aggregation from multiple sources allows real-time data collection from disparate sources. People can continue to use the tools they already use to discuss their work, and the main site can collect information from these sources and present/recontextualize it from a central location.

With minimal effort, we were able to put together a rough set of tools that allows people to get a perspective on the events going on in Iran. We built the site using tools freely available within the Drupal community. The bulk of the heavy lifting is done using FeedAPI and friends, and the folks over at Development Seed deserve huge kudos for unleashing these tools into the world. We also used the Views module to split out the content of the feeds. Obviously, the resulting site is a proof of concept, more of a pre-alpha prototype than anything else, but the site is useful as a research tool. We'll also continue developing on the site after publishing this post, so the site will undergo changes over the next few days as we modify things/tinker.

On our testing site, the Twitter traffic provides a pretty scattered overview, but taken in aggregate it allows one to scan raw data over time and get a sense of the ebb and flow of events on the ground. Initially, this feed was pretty free from spammers, but lately some opportunists have taken to using the #iranelection hashtag to get a broader audience for their content.

The YouTube videos provide another means of getting a sense of what is happening. As with any piece of media, the source of the content and bias of the author need to be taken into account. Also, our means of collecting these videos is certain to miss some content, as we are just aggregating the feed for the search term "Iran election protest".

The content coming from MSM outlets and blogs is admittedly arbitrary -- we tended to favor sources that had a clean tag for either Iran or the Middle East; the dearth of effective tagging on information coming from both traditional news outlets and some group blogs is discussed in more detail later in this post.

The important piece of this from our perspective: this tool can be easily built and focused on just about any topic. When it is set in motion, the site will gather and import information that can be used, analysed, recontextualized, or otherwise modified. This can be a tool used for any topic discussed on the web:

• grassroots organizing/community media -- use various data streams to collect information in real time that can be analyzed/collected/synthesized over time
• lesson creation -- aggregate writing about a specific topic, then choose the imported resources that align with your learning goals. Edit these assets as needed, or add in information that is missing
• farmer's markets -- farmers/sellers/market organizers use a microblogging platform to describe what they will be selling, and where; this information can be aggregated and geotagged, allowing an accurate breakdown of what is for sale at local markets.

As we built this out, we encountered some surprises. A short list includes:

The Wall Street Journal uses feedburner as their for their RSS tracking. However, this is exposed in their feeds (or at least in their World News RSS feed), and the original URL of their article points to http://feedproxy.google.com, as opposed to a location within WSJ.com. Additionally, the only tag for all content coming out of this feed is "Free". At the risk of stating the obvious, tagging all posts in your outgoing RSS feed as "free" is worse than useless. I have a hard time believing that they don't have the resources to do this well, which makes me wonder why it is allowed to be so sloppy.

NOTE: The following paragraph was edited because, well, it is completely wrong. The Huffington Post nails syndication. The links on the syndication page point out to RSS feeds of several dozen tags. In short, it rocks.END NOTE

The Huffington Post, which makes extensive use of tags to categorize posts, only offers 4 RSS feeds (Full feed, Latest news, The Blog, Featured posts) of content. Even though you can browse posts by tag on their site, you can't actually aggregate by these same tags. Given how easy it is to expose the content of any tag via an RSS feed, I can only conclude that the choice to not support feeds based on tags is tied to their business strategy. Given how little of the Huffington Post homepage is actually original content, it's surprising to see them reducing the number of ways people can interact with their site.

As a final note on this, it's not uncommon to see other more popular group blogs/major news outlets doing similar things. Talking Points Memo eschews aggregation by tags, and none of the major papers do much beyond feeds that summarize articles appearing in their standard sections. Within their RSS feeds, most major papers do little in the way of tagging content. Additionally, most papers/blogs include little more than a brief teaser within their feed. Given that most of these sites devote a fair amount of screen real estate to advertising (and some, like the NY Times, even embed ads in their feeds), their desire to bring eyeballs back to their sites is understandable.

However, an advertising-driven paradigm seems unlikely to work, and it seems especially shortsighted given that excessive reliance on advertising money is frequently cited as a contributing factor in the decline of newspapers. The new media economy seems unlikely to be a link economy; micropayments, paywalls, and/or "better" targeted ads feel equally fruitless. A remix-with-attribution economy feels more likely, with the looming caveat that no one has really figured out how to make that work in a way that makes all the people in the supply chain happy. But the necessary first step is to move away from the notion that the finished work is the starting point or ending point for profiting from that work; that places too high a value on the role of content, and how people interact with information. Content -- the article -- is the middle point of the process, and on the web "content" can be understood as one point in an ongoing chain of synthesis/recontextualization.

Working with aggregation -- arguably the simplest means of republishing and recontextualizing content -- gives an incomplete yet suggestive view into two elements: how an organization understands the web, and how they view the role of content. From what I have seen, both mainstream news outlets and popular blogs do a poor job of making the most of their content. If I had to reduce this down to a single reason, I would say that there is a perceived need to control how people consume content, and that this is tied to the need to pass eyeballs over ads.

However, tethering the distribution mechanism of online content to a strategy designed to generate more pageviews (ie, News as SEO) seems destined to fail, as the gimmickry of SEO doesn't mix well with unbiased reporting.

And I would love to end this post with the next great idea on how to support working writers within this model, but hey, it's late and I need to pack. But I'm looking to forward to learning more about other approaches over the next few days. I'll write up any ideas as they come, and for those who want to follow along, dip into the feed.

Part of our work for the Knight Drupal Initiative includes work to lower the barriers to getting started using Drupal sites to power community-driven media. As we work toward this end, we have multiple tools at our disposal, including:

• Better documentation, for both site administrators and people using the site
• Easier installs, to allow people to enable more complex functionality with less work
• Easier and more flexible theming

At the risk of stating the obvious, the work we do for the Knight Drupal Initiative will also be immediately transferrable to all of Drupal development.

Getting Started: Journal

The Journal module allows us to keep a running record of changes made on a Drupal site. Currently, it allows you to take notes as you build out your site. It's a great tool to track the specific steps people take as they work within a site.

We will build out two additions to Journal:

1. Tag journal entries, which would allow us to create categories of journal entries. For example, while configuring the WYSIWYG API to work with Filefield and Filefield Insert, we could tag these entries as "Text Editor" -- later, this would allow us to see exactly how the editor was configured.

2. Export journal entries by tag or by date range, in csv, ordered list, or unordered list format. This would allow us to generate documentation on how sites are built. This documentation can be used on drupal.org, for site administrators as sites are delivered, or to create end user documentation.

By way of example, let's say we have built a site that is designed allow teachers to create online portfolios as part of professional development. Prior to delivering the site, we could walk through the steps of creating a portfolio. At each step, we would comment on the process of creating the portfolio using the Journal module. We would tag each entry as "portfolio creation." Then, once all the steps have been completed, we would export the steps as an ordered list, and voila, we have our end user documentation.

Moreover, this process also gives us the instructions and order we need to develop functional tests using Simpletest.

With these additions to the Journal module, we will have the base package for creating more comprehensive documentation about all aspects of building and using a site. However, one of the challenges of documenting a site build is getting the documentation into the hands of end users, or people not directly involved in the build. Often, comprehensive documentation can be overwhelming as it floods someone with information.

Inject Me!

The Advanced Help Injection module builds off the Advanced Help module, and allows us to draft documentation using Drupal's core book module. Then, the documentation can be inserted on specific pages within a site, so people only see the documentation they need when they are at a place where they need it. So, the documentation generated using the Journal module can be presented to people exactly when it makes the most sense.

And finally: Advanced Help Injection allows the documentation to be exported as a module. So, all the documentation can be pushed to code, and delivered with the site. This brings us closer to the goal of having a site that is self-documenting, for both site admins and end users.

Moreover, the process of creating these tools can actually be documented using the tools themselves. While it feels like a blend of eating our own dogfood and chasing our own tail, it works.

What Are We Documenting, Anyways?

In this post so far, we have looked at simplifying the task of creating documentation, and using existing tools to get that documentation to people in a more efficient way. However, documentation assumes featuresets, as we can't have documentation about what to do without something to do.

This is where we cue the Features module. Take a minute to check out the latest post illustrating the power of Features over on the Development Seed blog.

As the video and post shows, the Features module allows us to create bundles of featuresets that can be moved from site to site. We envision a system where two things happen:

• Featuresets are paired with documentation, so that any set of features can be accompanied by inline documentation; and
• The process of creating new sets of features is fully documented using the Journal module, to empower more people to build and share features.

Making It Look Pretty

Theming and design are both enormous categories, so I'll be brief. In parallel with the work described here, we are working with Joon Park to build out a base theme (currently named Annex, but it might get changed to Mango Smoothie for the official release) that simplifies the process of creating a unique look and feel for your site. For end users, it supports some simple eye candy out of the box; as one example, it has multiple block regions, including some that support accordion and tab effects by default. For themers and designers, the process of inheriting features from the base theme to subthemes has been overhauled and simplified, allowing people more control and flexibility in building sub themes. More to come on this; ideally, we will have a release at some point in June.

Wolfram Alpha was released recently, and by all accounts it's a great tool. Groklaw breaks down the terms of service of Wolfram Alpha, and it's a good read for those interested in both search and journalism. The short version: WA claims copyright over search results, and requires attribution. Failure to attribute WA as a source could possibly be "academic plagiarism or a violation of copyright law." The full Terms of Service are worth a read; the above quotation is pulled from the Attribution and Licensing section.

This reminds me of the efforts by some news outlets to control how people link to content. Wolfram Alpha claims to be "an authoritative source of information," and that the process of "gather(ing), compar(ing), contrast(ing), and confirm(ing) data from multiple external sources" gives it copyright over its responses to questions." (from WA Terms of Service -- Attribution and Licensing).

Given that some WA results will likely use data pulled from news outlets and other copyrighted sources, it will be interesting to see where this leads.

Earlier this spring, the Knight Foundation let us know that our proposal to the Knight Drupal Initiative was accepted.

We are equal parts honored and excited, and work is underway.

Our project targets community media, and seeks to lower the barrier to entry for communities looking to collaborate with other like-minded groups via the web. One of the uses for our work will be within journalism, but other uses include collaborative creation of open courseware.

To be clear: this project will simplify the process of creating, distributing, and using open courseware. We want schools to spend less money on textbooks, and our work -- which will be freely available to all -- will support that goal.

As the project progresses, we will update this space with tutorials, status updates, etc. This page collects our Knight Drupal posts, or you can also subscribe via the feed.