What makes a news article credible? Is it a trusted byline, a particular tone, the number of links, or the range of expert opinions presented?
To identify the credibility indicators in news media, Meedan and Hacks/Hackers set up a working group, currently in the process of developing a framework and mark-up a first round of test articles.
Initially called The Credibility Indicators Working Group and recently renamed as The Credibility Coalition, the initiative builds on existing frameworks such as Google’s way of labelling fact-checked articles in search results in order to build common definitions and a common data schema that all those who are working in this space can use to communicate.
“It’s an effort to develop a standard of third party mark-up of stories to inform platforms and AI initiatives about their credibility,” said An Xiao Mina, director of product at Meedan, who led workshops on credibility indicators at MozFest in London at the end of October.
“The coalition involves journalists, data scientists, librarians, designers and others because we believe that credibility has many different facets.”
Those already contributing to the working group come from MIT Computer Science and Artificial Intelligence Laboratory, Berkman Klein Center for Internet and Society, Global Voices, The New York Times, The Associated Press, Berkeley Institute for Data Science, Storyful, and Schema.org among others, as well as Meedan and Hacks/Hackers.
The project was born at MisinfoCon in February, and the team has hosted workshops in San Francisco at Northwestern University’s campus, and in New York at Columbia University.
The workshop at MozFest focused on identifying credibility indicators in articles based on analysing two stories, one deemed to be of high credibility and one low. Elements such as headlines, the tone and language used in the story, the sources, the byline, the number of inbound and outbound links and more were taken into account.
“There are different layers and we aggregate that data, and then it’s up to platforms and AI systems how they interpret them,” said Mina.
“Some will want to create a score, but what’s interesting to me is that we could use this as a process to educate the general public on what makes a credible article.
“There are many ways that we can use these indicators beyond just a score, beyond just a simple number.”
The project has four core aims: to develop the framework of indicators to test; refine the process by testing a wide variety of articles; evaluate the training data in partnership with potential users, and define the process for the group. klgirls
In June, The Credibility Coalition received support from The Knight Prototype Fund to focus on solving three key challenges. The team is working on identifying an accessible user experience for mark-up so that anyone can contribute; developing a framework and training data; and communicating the framework in such a way that all those involved can understand how the conclusions were reached.
A session at newsrewired on 7 March will introduce the Credibility Coalition and some pilot work: an initial set of indicators for article credibility defined collaboratively by a diverse coalition of experts and applied to a dataset of 40-50 articles on health and climate change. Find out more from Connie Moon Sehat, director, NewsFrames, Global Voices; collaborator, the Credibility Coalition.
