Performance Measurement for the Public Community Newspaper

Introduction

Journalists perform a vital role in democratic societies by providing citizens with the information needed to effectively participate in decision-making. According to Entman, five criteria define traditional journalism: accuracy; balance; checks on pure profit maximization; democratic accountability; and editorial separation. "Traditional news organizations attempt to hold government to account by maintaining a high level of surveillance of public officials and promoting monitorial citizenship. Highest priority (page-one placement, most prestigious-skilled journalists assigned) is therefore accorded to major policy issues and decisions..."

In addition to these traditional roles, a reform movement that started in the 1980s known as "public journalism" sought to improve trust in journalism as an institution by identifying a positive subjective role for newspapers to play within a community. Such advocates as Buzz Merritt and Jay Rosen suggested that journalists should play an active role in democratic self-governance by "becoming the experts on public life in the community" and by acting as "a fair-minded participant in a community that works." Many of the early advocates for public journalism were inspired by the writings and activities of pollster Daniel Yankelovich, who founded the group Public Agenda. A variety of pilot public journalism projects engaged citizens in defining a prioritized agenda of issues or in detailed discussions about specific issues. These projects included surveys, questionnaires, citizen discussion groups, and focus groups.

Performance measurement offers a means of determining how well public journalism achieves its desired outcomes. Hatry defines performance measurement as the "measurement on a regular basis of the results (outcomes) and efficiency of services or programs." Hatry and others draw a clear distinction between the "outputs" directly tied to an organization's activities and the "outcomes" that are related to the organization's ultimate mission. Outcome-sequence charts "provide a visual depiction of what a program is expected to produce, and they allow staff to construct the anticipated time path of program activities from initial inputs through end outcomes" [Hatry, page 48]. The chart maps out the implicit assumptions of the program designers, showing the connections between activities, outputs, intermediate outcomes, and end outcomes. Hatry also recommends developing a mission statement as the first step in producing an outcome sequence chart. In articulating the organization's mission statement, Hatry suggests clearly differentiating between the outcomes and the means of achieving those outcomes. The mission statement should consist of a "to" part and a "by" part, he says, clearly separating the results from the methods used to attain it.

Measurable Indicators and Benchmarks

Hatry recommends that considerable care and precision go into the design of the indicators, which he defines as one or more specific numerical measures that indicate progress toward achieving an outcome [Hatry, page 55]. Because indicators get attention, the set of indicators should be relevant, important, understandable, feasible, and comprehensive: "The goal is to provide valid, useful data on which managers and other officials can rely--and to recognize explicitly the limitations of those indicators."

Figure 1. Outcome-sequence chart for the public community newspaper.

Benchmarks represents values against which the indicators can be compared. They can represent either specific numerical target values, ranges, or percentages. The benchmarks can represent values for other time periods, other organizations or work units, other geographical areas, or broad averages of performance across many time periods, work units, or geographical areas. For valid comparisons, data must frequently be normalized.

Outputs, Outcomes, Indicators and Benchmarks

This article seeks to develop a performance measurement system for the newspaper's activities that promote democratic self-governance within the broader context of its community. This performance measurement system does not seek to measure the routine operations of the newspaper such as advertising, reporting, publishing, and delivery, or reporting on topics other than public affairs and community organizations.

The performance measurement system seeks to answer four questions:

1. Is the newspaper reporting on all of the agencies and organizations within the community, and all of the issues of importance to the community?

2. For each individual issue of importance, is the newspaper representing all possible views on that issue within the community?

3. In the number of stories and the importance assigned to them by story placement and reporting resources, do the newspaper's relative priorities match the importance of the issue within the community?

4. Does the newspaper's act of providing comprehensive information about these agencies, organizations, and issues effectively foster citizen participation within the community's organizations and in democratic self-governance?

The mission statement for the public community newspaper is: "To be an active participant in a community that works by identifying the issues of importance to a given community and by providing citizens and decision-makers with the wide range of facts and opinions needed to help the community evaluate and decide public policy."

The remainder of this section describes the outputs, intermediate outcomes, and end outcomes for this core activity of the public community newspaper (as depicted in Figure 1), providing the information necessary to participate in the community and in democratic self-governance. It also describes the benchmarks that will be used to evaluate performance, and the indicators that will be used to measure progress toward the benchmarks.

Output (Box 2): Editors and reporters maintain lists of story assignments and issues important to the community based on public meeting agenda, discussions with community members, and public comments at meetings. These are also based on the list of jurisdictions/organizations within the readership area and their mission statements; the list of top issues identified by local officials and candidates for local offices during campaigns; and the list of officials/individuals within the community participating in local organizations.

Indicators: Total number of items in the list; number of new items added to the list during the reporting period.

Benchmarks: Comparison to own performance in past; congruence with survey results asking local residents to list issues of importance to the community. Goal is to cover 95 percent of the issues that are later identified as important to the community.

Output (Box 3): Published items relating to public issues of concern to the community as a whole: News stories, editorials, opinion columns, letters to the editor.

Indicators: Number of published items; word count; page placement of the story within printed version of newspaper; seniority of reporter assigned to story; percentage of public sessions attended and reported upon; number of distinct individuals identified as participating/speaking.

Benchmarks: Target percentages. Stories should be written for 100 percent of all candidates seeking office; 100 percent of all meetings where more than five local residents attend and speak; 100 percent of all special/emergency meetings (such as policy initiative informational meetings, wildfire community information meetings); 75 percent of regular school/fire/hospital/rural planning board meetings; 75 percent of local civil and criminal cases. Compare with own performance in past periods; compare with other newspapers in same readership area.

Intermediate outcome (Box 4): Range of facts and opinions representative of the community

Indicators: Number of discrete views for an issue (narrative); Number of articles/words for each discrete view; Percentage of articles for each discrete view; Fairness index: deviation from equal representation. There are two goals here: to ensure that the newspaper provided opportunities for all individuals/agencies to respond to questions/comments; that its published work represents all facts and opinions pertinent to the issue; and that if there are multiple possible categories of responses, that they are they represented equally by the published materials.

Benchmarks: Compare with own past performance; compare against idealized goal of equal representation. This outcome will be described in a narrative format.

Intermediate outcome (Box 5): Prioritization of issues within community.

Indicators: Total count of published items on an issue of importance; Weighted count 1 (considering story placement and reporter seniority; add .05 for page 1 placement, .04 for page 2, etc.; add .05 for most senior reporter, .04 for next, etc.); Weighted count 2 (considering number of unique individuals; multiply total count of published items by number of unique individuals represented); Ranking of issues by total count; Ranking by issues by weighted counts; Congruence of rankings to survey results.

The usual measure of a story's importance is its position in the print version of the newspaper. The top stories are on page 1, above the fold, and generally the next highest priority stories follow in the order assigned by the editor, on pages 2, 3, 4, etc. Story page placement is usually a one-to-one correspondence to the relative priority given to that story by the editor. Another way to weigh the relative importance of a story is by author: The most important stories usually get the most senior or skilled writers. Aside from the number of stories on a topic, the placement of the story and the reporter assigned to it offer clues to the relative importance of the story for the newspaper.

Benchmarks: Compare with past performance; compare with results from annual member survey for congruence with opinions of community. The goal is 80 percent congruence with the annual survey.

End outcome (Box 6): Newspaper is a fair-minded participant in a community that works.

Indicators: Count of persons in attendance at public meetings, broken out by agency. Count of opinions and letters published in the newspaper. Volunteerism within the community. Membership in organizations. Survey membership in organizations annually; for local organizations, seek membership figures annually, and conduct a survey to estimate participation in other organizations. Polyarchy index as defined by Dahl and developed by Vanhanen; ranking of polyarchy index by precinct and aggregate for readership area. The Vanhanen polyarchy index is equal to [(Competition index * Participation index) / 100] where the Competition index equals (100 - percentage of votes going to the winning candidate or party), with a maximum possible value of 70; and the Participation index equals the percentage, (Number of voters / Total population). The raw data for percentage of votes going to the winning candidate or party, total population, and number of voters are available for each precinct from local election officials.

Benchmarks: Compare attendance over time by agency. Compare submissions to the newspaper over time. Break out volunteerism by agency and track changes within each agency over time. Break out membership by organization and track membership over time. Compare volunteering and membership against state and national figures. Compare to local polyarchy index over time. Compare to other similar areas outside the readership area of the newspaper. Compare Mineral and Missoula counties. Compare ratings for individual political precincts. Compare changes in aggregate polyarchy index for the newspaper coverage area over time.

Reporting and Analyzing Performance Information

One important goal of the public newspaper movement is to instill public trust in journalism. Accordingly, all performance measures for the newspaper will be publicly reported and available to be independently verified.

Aggregated data for all published material within the newspaper as a whole will be used to compare the newspaper's actual performance against benchmarks and target values. This aggregate report will include a combination of tabular data and narrative data, based on the output/outcome that is being measured. For example, tabular data will be used for most counts, percentages, and calculated numeric values, while the analysis of the breadth of the newspaper's coverage--its completeness index--will rely on a narrative framework that establishes categories for the numerical data.

Data collection will be automated to reduce costs and ensure continuing performance measurement. All lists of story assignments and published stories are already currently available in databases and available for automated processing, such as automated subject classification. This approach would allow software to annotate the stories with metadata--information about the information within the stories. For example, the News Industry Text Format (NITF) defines tags that identify common elements of a news story, as well as allowing tags within the news story that allow identification of persons and organizations (for example, the NITF DTD 3.4 includes the elements "event", "function", "location", "money", "num", "org," and "person"), with sufficient flexibility to define custom categorization of the persons and organizations. Such tags could identify the name of a person or a government agency appearing in, or expressing a statement in, the story. Adopting an industry standard for metadata such as NITF makes it more likely to obtain low-cost software to insert and extract the metadata, reducing the startup implementation costs. Software systems using the current NITF standard will be evaluated before developing custom software. These tools will be used to automate insertion of metadata for stories developed over the past several years, from 2004 through 2007. For future stories, authors and editors also have the opportunity to manually insert metadata tags as the stories are posted.

In addition to the aggregated data for the newspaper, individual performance reports can also be generated for each writer or for each story. These will not be used for individual performance evaluation.

In the case of the Clark Fork Chronicle, nearly all of the data that must be collected for performance measurement is available in two distinct databases: the internal assignment database called "chronlog," and the newspaper's public website at http://www.clarkforkchronicle.com.

An Internet-based program to access the chronlog database appears in Figure 2:

Figure 2. Internal list of categories (issues) and story assignments.

As suggested in the figure above, each story or photo assignment can be associated with a particular category. These can be created so that they correspond to the "issues of importance to the community" that are being monitored.

The Chronicle's public website (an interface to its story database) appears as follows:

Figure 3: The Clark Fork Chronicle's public website. Performance data will be automatically collected by examining stories in the underlying database.

A list of sections appears at the left-hand side of the Chronicle website. The sections are named according to category. For the purposes of performance measurement, only those stories within the sections News, Features, Opinion, and Letters to the Editor will be used. The Features section typically includes regular columns and stories about local organizations.

In addition to using an Internet-based user interface, both the internal and public databases can be accessed directly, allowing the use of software that can query the database to generate performance information. A sample snippet of a program that counts the number of words written by a specified author appears in Appendix A. Appendix B demonstrates the use of generic metadata tagging within a story. With metadata tagging, most data collection can be completely automated. This will allow data to be generated for stories published in the last four years, allowing performance to be tracked over several years.

The following table describes each of the indicators, how data is to be collected, analyzed and reported:

Ensuring System Integrity

Once the prerequisites for a performance measurement system are in place-- including agreement on program goals and the availability of data processing support, analytic support, and management support--Hatry says full implementation is achieved only when the system "is taken for granted" and its ongoing data is regularly used to make program changes that improve quality and outcomes. He points out many possible obstacles along the way to routine collection of valid and reliable performance data. These include: a lack of training, which can reduce the effectiveness of personnel in reporting and collecting valid data; resistance from individuals who are reluctant to adopt the system; and skewed results provided by individuals with an incentive to fudge the numbers. Hatry strongly recommends independent third-party audits to help maintain the integrity of the performance measurement system.

The public newspaper seeks to mitigate these potential obstacles with four key strategies: (1) limiting use of the performance measures to aggregated performance data describing the newspaper's performance as a whole, (2) extensive use of automation to collect and analyze performance data, (3) opening the performance-measurement toolkit to the general public and (4) regular publication of performance goals and results.

1. Performance data will not be used as the basis for individual performance evaluations. This policy should remove any incentives to individuals to provide inaccurate reports by reducing any possible motivation for them to fudge the numbers.

2. Extensive use of automation in the data collection process will not only ensure greater consistency in the dataset, but will also limit the opportunities for individuals to fudge the numbers. Because the automated system relies on metadata tags inserted within the stories, it is still possible for individuals to insert inaccurate tags that would skew the results for any particular story. This possibility should be mitigated by strategy (3).

3. The toolkit will be made available to the public, allowing readers to conduct their own analysis using the same tools used by the newspaper. This will allow rapid identification of inaccurate tags in individual stories, and will help build public trust in the performance results. Critiques of individual stories and metadata tagging may also lead to improvements in the overall system.

4. The newspaper will periodically publish stories reporting on its own performance goals and results. This will also have the benefit of publicizing the toolkit in an ongoing fashion, making it available to a wider public. After the startup costs for development of automated tools, the costs for generating reports are negligible and the reports can be produced monthly, quarterly, or annually. The initial reports featuring intermediate incomes will be generated quarterly, while examining data for the previous 12 months. (Indicator data for most end outcomes are available only on an annual basis.) The reporting frequency can be changed as needed based on experience with the system.

Recommended Implementation

The overall approach to performance measurement in this paper offers a method to objectively generate numeric ratings of the newspaper's performance according to three of the five traditional criteria defined by Entman: balance, editorial separation, and democratic accountability. The end outcome of public journalism, enabling effective citizen participation in democratic self-governance, can also be objectively rated using data readily available from the broader community.

The approach described in this paper relies heavily on automation to reduce costs and to make the performance measurement transparent to the staff, reducing the need for training and mitigating resistance to system implementation. The primary development effort involves adoption of a news industry metadata standard, encompassing the following sequence of 10 tasks:

1. Define extensions to NITF tags that use the "class" attribute to customize the "person", "org", and "event" tags to allow identification of public officials, government agencies, community organizations, and public meetings. Identify an existing NITF tag or custom extension to identify issues of importance to the community.

2. Define extensions to NITF tags to be used in categorizing different points of view in regard to an issue, to be used in measuring the fairness of the newspaper's coverage. Define tags that identify discrete points of view within the spectrum of all potential views and tags that identify statements as expressions of one identified point of view.

3. Perform hand-tagging of a variety of selected stories and test those stories to ensure that NITF tagging does not affect the existing database and usability of the existing website.

4. Develop and test software to automatically insert NITF tags within the existing story database. Develop small functions for each individual tag to keep each function small, minimizing the potential for software errors and allowing rapid development.

5. Develop and test software to count the NITF tags within one or more stories, allowing the stories to be filtered by author, range of publication dates, issue of interest to the community, officials, agencies, and other entities identified by the NITF tags.

6. Develop and test software to process the counts of the NITF tags to aggregate these counts into numbers and percentages that represent the outputs and intermediate outcomes identified in the section, "Reporting and Analyzing Performance Information."

7. Use the existing archive of PDF format pages to automate the insertion of page numbers for all published stories, enabling weighting by page placement.

8. Modify the internal "chronlog" assignment database to support performance measurement: (a) explicitly add a field representing "issues of concern to the community" that corresponds to the issues tracked by performance measurement reporting; (b) add fields for entry dates, deadline dates, and removed dates to the database; (c) change the user interface for the assignment database to list only those items that are not removed, and (d) change the delete function to insert a "removed date" into the database.

9. Develop and test software to produce reports from the lists present in the chronlog internal assignment database.

10. Develop and test software to produce reports from the tagged stories.

The secondary effort involves developing software tools to assist a person with tagging where the tagging cannot be completely automated. This software will make it as easy as possible to manually tag statements representing one distinct point of view among a spectrum consisting of all possible points of view on an issue of interest to the community.

Appendix A: Example of software to automate data collection

The following code snippet demonstrates how many of the software routines will automate the data collection. This function, get_word_count, calculates the total number of stories and total number of words by the specified author.

function get_word_count($author, $startdate, $enddate)

{

... // connect to database

$myquery = "select title, date, introtext, bodytext from gl_stories where date>='$startdate' and date<='$enddate' and introtext like '%by $author%'";

$result = mysql_query($myquery);

if ($result) {

$nrows = mysql_num_rows($result);

echo "$author contributed $nrows stories\n";

$totalwords = 0;

for ($i = 0; $i < $nrows; $i++) {

$row = mysql_fetch_array($result);

$title = $row['title'];

$date = $row['date'];

$introtext = $row['introtext'];

$bodytext = $row['bodytext'];

$numbodytextwords = sizeof( explode( ' ', strip_tags( $bodytext )));

$numintrotextwords = sizeof( explode( ' ', strip_tags( $introtext )));

$numwords = $numintrotextwords + $numbodytextwords;

echo "$title, $date, $numwords \n";

$totalwords = $totalwords + $numwords;

} // end for

echo "$author, $nrows stories, $totalwords words";

if ($nrows > 0) echo ", average = ". ($totalwords/$nrows);

...

The output from this function for the author "John Q. Murray" between January 1 and June 30, 2007 is as follows:

John Q. Murray, 114 stories
Frenchtown seniors seek relief from school 15 percent tax hike, 2007-01-04 14:02:36, 1163 
The Frenchtown-Griz connection, 2007-01-04 14:15:56, 489 
Supreme Court to vote Friday on Lolo post-burn case, 2007-01-04 14:26:10, 842 
Rep. Hendrick starts his second session, 2007-01-04 14:44:58, 426 
Registered sexual offender has left Tami Drive, 2007-01-04 14:54:47, 187 
State House is blocking Yahoo email; contact Gordon by phone, 2007-01-05 10:48:33, 168 
Chief, taxpayers argue numbers, 2007-01-11 10:29:49, 1575 
Post-burn case denied: Lolo, county 'disappointed', 2007-01-11 10:39:59, 557 
Sen. Elliott: Tax relief will require compromise, 2007-01-11 10:42:02, 988 
4 of 7 Frenchtown school board positions up for election, 2007-01-11 10:45:02, 338
...
Weather helps crews catch Four Mile Ridge Fire, 2007-06-28 17:23:43, 426 

John Q. Murray, 114 stories, 98903 words, average = 867.5701754386

Appendix B: Metadata Example

The following example shows generic (non-NITF) tags embedded within the text of a story, demonstrating how they can facilitate automated data collection. The plain text story appears at http://www.clarkforkchronicle.com/article.php/20070711173651853.

The metadata tags identify that this is a story about a public meeting, record the attendance, and identify persons and officials quoted or paraphrased in the story. The tags also differentiate between persons present as participants or mentioned at the meeting.

<storyid>20070711173651853</storyid>

<meeting agencyid="17" type="regular" frequency="monthly" datetime="July 8, 2007 7:00 p.m." totalattendance="43" numberagency="12" numberpublic="29" numbermedia="2"/>

<title>Evaro residents seek to keep fire station open</title>

<byline>by <reporter id="4">Richard Werst</reporter></byline>

<quoteinparagraph>Evaro Fire Station was listed as an "information item" on the July <agency fullname="Frenchtown Rural Fire District">Frenchtown Fire Board</agency> agenda, but to many of those who attended the meeting the words meant "imminent closure."</quoteinparagraph>

Closing the station is just one of the possible options, <official agency="Frenchtown Rural Fire District" title="Trustee" firstname="Mitchell" lastname="Hicks">Trustee Mitchell Hicks</official> told those in attendance. Putting the subject on the agenda did not guarantee the station would be closed--it is just a way to get people involved and get people to the meeting to discuss the station.

<quoteinparagraph>You should tell the people living in the area that, <meetingparticipant><person firstname="unidentified1-20070711173651853" lastname="unidentified1-20070711173651853"></meetingparticipant>one concerned citizen said, “before we start reaching for our nitro pills.” </quoteinparagraph>

At issue with the Evaro station is the fact that no volunteers currently live in the area.
<meetingparticipant><person firstname="unidentified2-20070711173651853" lastname="unidentified2-20070711173651853">One person</person>, who identified himself as a typical resident of the area, told the board that he worked two jobs with a thirty mile commute, and was out of town for work about 50 days a year.

<quoteinparagraph>“I'm not really volunteer material,” <person firstname="unidentified2-20070711173651853" lastname="unidentified2-20070711173651853">he</person> said, adding that he was delighted to pay for fire protection – that he would like to take the time to thank the staff and volunteers, and that it was wrong to cut services in places where the population is not declining.

The satellite station in Evaro was established in 1997 and serves 99 homes, according to the 2003 census figures, <official agency="Frenchtown Rural Fire District" title="Fire Chief" firstname="Scott" lastname="Waldron">Fire Chief Scott Waldron</official> said. He said according to the same source, there are 142 homes in the Petty Creek area, counting Upper Petty Creek and the West Fork.

<official id="201">Waldron</official> said he figured ten percent could be added to both figures to offer an accurate count for 2007.

The station costs about $100 per month to maintain, <official id="201">Chief Waldron</official> said, adding that they need an engine there that could hold and carry about 500 more gallons.

<quoteinparagraph><official id="201">He</official> suggested that Frenchtown could establish a joint station with Arlee Fire. “They staff it and we help fund it,” <official id="201">he</official> said.</quoteinparagraph>

<meetingparticipant><person firstname="Don" lastname="Klepper">Don Klepper</person></meetingparticipant>, a long time resident of the area and a self-described advocate of the Evaro Fire Station, told the group that it was time to roll up their sleeves and get to work to see what could be done to save the station.

<quoteinparagraph>“Evaro is going to get taxed even though the station is closed,” <person id="202">he</person> said, adding that the residents of the area need to know what is going on before a decision is made.</quoteinparagraph>

<quoteinparagraph>“We need to participate,” <person id="202">he</person> said.</quoteinparagraph>

In other business, the <person lastname="Taapken">Taapken family</person>, which adopted Station 8 in Alberton by pulling weeds, cleaning windows, and tackling other projects, was named "Volunteer of the Month."

We focus on equipment and response, <official id="201">Fire Chief Scott Waldron</official> said, and it is difficult to get time for the station duties.

<official id="201">Chief Waldron</official> told the board that about 20 homes have participated in the fuel mitigation program, with between ten and twelve thank you notes and donations received by the department to date.

To illustrate the need for the service as a tool to prevent wildfire <official id="201">he</official> told the group about an area in Colorado that only allowed the residents to clear up to thirty feet around their homes.

<quoteinparagraph>“Those lots are cleared out now,” <official id="201">he</official> said referring to a fire that destroyed the majority of the homes in the area because they did not have adequate protection.</quoteinparagraph>

One of the department's engines has been sent to help fight the 160,000 acre wildfire in Colorado, according to <official id="201">Chief Waldron</official>. <issuestatement agencyid=17" issueid="4" positionid="2"><official id="201">He</official> said if such wildland revenue was removed, there would be a great strain on the department's budget.</issuestatement>

He also revisited the subject of impact fees and told the board that the Missoula County failed to take any action in regards to imposing the fees.

<agencydecision agencyid="17">By a unanimous vote the <agency id="17">Fire Board</agency> approved a set of minimum standards for fire fighters, EMTs and officers, and an annual reimbursement of $200 for all volunteers who complete the minimum standards by the end of the fiscal year.</agencydecision>

The $200 payment will be in addition to the current compensation that they receive for fire calls, travel expenses and training.

<quoteinparagraph><official id="201">Chief Waldron</official> also promised that he would "soon" provide complete payroll data and information about any payments to board members, as requested in a letter from Frenchtown resident <person firstname="Ray" lastname="Winn">Ray Winn</person> dated June 11.</quoteinparagraph>

<quoteinparagraph>That request letter stated, "We are requesting the payroll sheets and time sheets submitted for both paid and volunteer firefighters and contracted personnel services and also any monies paid to board members either for compensation or reimbursement. The records we wish to have transmitted are to be for the period January 1, 2003 and January 1, 2007."</quoteinparagraph>

<event type="meeting">There will be a special board meeting at <eventtime>6:00 PM</eventtime> on <eventdate>July 16, 2007</eventdate> in <eventlocation>Fire Station One</location></event>.

Appendix C: Lists of public agencies and organizations

In reporting on breakouts of news stories by government agency, the performance system for the public community newspaper generally sets higher performance targets for local government agencies and officials, focusing on county and town offices and officials. The exceptions are the local state representative and senator, and the two local Forest Service ranger districts of the Lolo National Forest. Candidates for state and federal offices are covered to the extent that their positions on issues reflect positions on issues of interest to the local community. Other state and federal offices and officials will be added to the performance measurement system only as they appear in news stories.

Comprehensive lists of government and community organizations to be included in the initial performance measurement appear at http://www.clarkforkchronicle.com /article.php/20070729154642773 and 20070729155254369, respectively.

References

Hatry, Harry. Performance Measurement: Getting Results. First Edition. 1999. Urban Institute Press.

Entman, Robert M. "The Nature and Sources of News." Overholser and Jamieson, editors, Foundations of American Democracy: The Press, Oxford University Press, 2005.

McCool, Daniel, editor. Public Policy, Theories, Models, and Concepts: An Anthology. Includes several articles discussing analyzing Robert Dahl on pluralism and polyarchy: McCool p. 29; William Kelso, p. 42; Jack Walker p. 83; McCool p. 97; Paul Sabatier p. 344; McCool p. 383.

Vanhanen, Tatu. A New Dataset for Measuring Democracy, 1810-1998. Journal of Peace Research, Vol. 37, No. 2. (Mar., 2000), pp. 251-265.