I created an AI tool to help investigative journalists find stories in audit reports. Here’s how I did it and why it matters

1. The project in a nutshell.

Can generative AI help make the lives of investigative journalists easier when it comes to handling complex documents? While ChatGPT does not have the capability to generate sufficiently detailed results for investigative processes, using Custom GPTs to set guardrails and parameters gives promising results that could significantly reduce research time and empower reporters to uncover important stories with ease. So this is what I wanted to explore. 

2. The problem I wanted to solve.

For the longest time, journalists in the Philippines have been relying on reports from the government’s Commission on Audit (COA) to help uncover potential graft in government agencies. The audit reports usually provide leads that reporters could explore further to see if there are anomalies involving how public funds are spent.

A recent example was the purchase by the country’s Department of Education of P2.4 billion ($134 million) worth of laptops that state auditors deemed to be “pricey” and “outdated” in 2021.

The information from the state audit report was uncovered by GMA News Online reporter Llanesca T. Panti. I was editor-in-chief of GMA News Online at the time. The story quickly went viral, and public outrage was swift. In a matter of months, a Senate panel recommended filing charges of graft and perjury against government officials involved in the procurement.

Panti was the first journalist to publish the overpriced laptop procurements, and while other news outlets later reported further on it, the issue might not have made it to public consciousness if not for her eagle eyes. The executive summary for the Department of Education 2021 audit report was 46 pages long, and the observation about laptops was buried in page 26. Only a couple of lines provided details about the laptop purchases.

The Center for Media Freedom and Responsibility, a media watchdog in the Philippines, notes that “COA’s many audit reports are agency-specific and may be difficult to read and understand.” It is usually up to reporters and editors from each newsroom to go through the stacks of pages and try to figure out what would be newsworthy from each report.

Because journalists have limited attention and newsrooms increasingly have even more limited resources, I investigated whether AI could help augment journalists’ capacity to go through audit reports looking for clues that could uncover anomalies.

3. The problems I encountered.

To start, I tried using ChatGPT’s GPT-4 model from OpenAI. A $20 monthly subscription to ChatGPT Plus allows you to upload a PDF document that it can scan and summarise.

Asking the model to summarise the document – the executive summary of a COA report from the Philippines’ National Economic and Development Agency – came back with a description of the contents of the document, which might be nice for a grade school term paper on the subject but is virtually useless for an investigative journalist.

Asking ChatGPT if there are red flags in the document provides a little more insight, but many of the points in the summaries are still too vague for a reporter looking for a story. For example, ChatGPT said the audit “found errors affecting asset, liability, and equity accounts. Such errors can indicate weaknesses in the accounting processes and systems, possibly leading to inaccurate financial reporting.”

I tried another prompt: “I’m an investigative reporter. Are there any newsworthy items from this report?” Again, it gave vague responses.

At this point, a reporter would probably realise that it might be easier to read the audit report themselves rather than trying to wrestle with ChatGPT to find the exact prompt that would uncover newsworthy information.

4. How I found a solution. 

In November, OpenAI announced that it was allowing users to create what it calls GPTs, custom agents on top of ChatGPT. I explored whether a custom GPT, helpfully named COA Beat Assistant, could help an investigative reporter deal better with audit reports.

After testing different configurations for the COA Beat Assistant, I came up with the following guidelines:

1. To try to find meaningful information in the Executive Summary of COA reports, I had to limit what I wanted the COA Beat Assistant to look for. Each document contains a section called “Summary of Significant Audit Observations and Recommendations”, and I wanted the GPT to generate a summary of each item there, which would serve as sort of a menu for the reporter.

2. I wanted the COA Beat Assistant to use only the information found in the corpus of the document, and not include anything outside that document such as from previous training data.

3. After summarising the Significant Audit Observations and Recommendations, I wanted the GPT to guide the user with prompts for the next steps.

4. After uploading the report, I wanted the COA Beat Assistant not to respond immediately. Instead, I wanted to ask the user if they wanted to summarise the Significant Audit Observations and Recommendations. The technical reason for this is that the token limit for ChatGPT counts both input and output for the same prompt. This means that if you upload a 46-page document for the GPT, it might not have enough space to generate a useful reply.

Given this, I ended up giving COA Beat Assistant the following instructions:

"Upon the upload of an audit report, the COA Beat Assistant will process the report, which contains a list of Significant Audit Observations. It will not yet summarise the report. Instead, it will ask the user if it wants a summary of all the observations in one sentence, followed by the Observation Number. After displaying the summary of observations, it will prompt the user with possible options. It will use only information from this document and not use any previous training data."

5. How I tested this tool.

I tried using COA Beat Assistant with the Executive Summary of the 2021 COA Report for the Department of Education, the same report that contains the item about the overpriced laptops. After uploading the document, it asked me if I wanted a summary of observations, to which I said yes.

It returned with a summary of the observations. It was helpful, albeit incomplete and riddled with formatting errors. For example, it didn’t include all the observation numbers in its summary:

But the information it generated would immediately be useful to any reporter looking for leads. At the very least, it included the issues related to overpricing and budgets – including for teachers’ laptops. Asking the COA Beat Assistant for Observation 4 led to the following insight:

I also asked it about Observation 5, flawed budget utilisation:

At each of these points, there should be enough leads for an investigative reporter to decide whether they should do a deep dive on the report for a potential story. The COA Beat Assistant had already fulfilled its mission. But of course, I already knew what I was looking for with the Department of Education 2021 audit report. I tried out the tool on reports about other government agencies that I had not read before:

• Department of Agriculture
• Department of Tourism
• Cebu Provincial Government
• Pampanga Provincial Government
• Quezon City Government

In each of these cases, the assistant successfully summarised the audit observations from COA and prompted the user about which items they wanted more detailed responses. To my eyes, this was already a very useful tool for doing an intensive task.

COA Beat Assistant performed better with smaller documents. Interestingly, for some of those documents, the bot completely ignored my instruction not to put out a summary and went ahead and replied with summaries. (The summaries were, at least, accurate.)

All told, I spent a total of about 16 hours working on customising COA Beat Assistant, trying different sets of instructions and prompts, to finally get it to behave close to the way I wanted it to behave. Given that there are dozens of audit reports published every year and each would ordinarily take at least an hour or maybe two to comb through, this kind of upfront investment could save a lot of time in the long run.

6. How the tool works in practice 

I wanted to test how useful COA Beat Assistant was for actual reporters working on audit reports. I gave my colleague Panti access to the tool to Panti. She was the beat reporter who originally discovered the overpriced laptop issue. So I wanted to see if the tool would be useful to her.

After a few hours of testing, it quickly became apparent how much it would make life easier for someone like her. When dealing with COA reports, Panti says, one of the biggest pain points was getting the PDF reader on her laptop to play nice with the documents, which always came with different formatting depending on the auditor producing the report. The summaries that COA Beat Assistant provides are useful at first glance, as they give an experienced journalist like her a good idea whether to spend more time on a particular agency or to move on to another report that might yield something more newsworthy.

Without additional guidance from me, Panti also discovered further uses for the COA Beat Assistant. Because she was familiar with the structure of the full COA report, she would take insights from bullet points generated by the COA Beat Assistant off the executive summary, and let that guide where she would go next. She would then feed other documents from the full report where further information might be, to gain further insights. In one instance, she tried feeding it with the expanded Observations documents. In another instance, she fed it with an Annex document containing notes from the auditors, where more important details might be buried.

Because of issues like these and her concerns about generative AI, Panti doesn’t see herself relying solely on the insights and summaries generated by the COA Beat Assistant when it comes time to finally write her story. Instead, she only uses them to find the sections that would be the logical starting point for her report. Then she would still do her deep dive on the document. She also wouldn’t trust a bot to help her write the story, preferring instead to do everything from scratch the old-fashioned way.

I introduced the tool in a workshop for journalists organised by the Philippine Center for Investigative Journalism (PCIJ). Sheila Coronel, the Director of the Toni Stabile Center for Investigative Journalism at Columbia J-School and the founding executive director of the PCIJ, tried the tool out on the audit reports for provinces in Mindanao in the southern Philippines, some of which are among the poorest areas in the country. Immediately, she noticed an item about the millions of pesos being spent by one area on Gender and Development and flagged it as a potential investigative story.

7. How the tool can be improved.

The tool isn’t without flaws. Aside from the formatting issues I had mentioned, Panti found shortcomings with how the COA Beat Assistant processed documents. In one instance, she noticed that the figure that the custom GPT returned didn’t seem quite right. It turned out that the bot was returning that figure from the first column of a financial report, rather than the total at the rightmost column.

Our Custom GPT tends to struggle with larger documents. Additionally, there are still restrictions on ChatGPT 4 that limit users from using it heavily; even paying users are limited to a certain number of queries over a given period.

Right now, the application of the tool is pretty narrow. At the PCIJ workshop, journalists asked if it could work with audit reports from multiple agencies from the same year or audit reports from the same agency over a number of years. 

Initial testing indicates that Custom GPTs are unable to handle multiple documents quite as well. Perhaps a pre-processing step (i.e. extracting text from PDFs, converting them into a single document) would be helpful and should be explored.

8. How this can help other newsrooms. 

COA Beat Assistant is a useful timesaver. Panti estimates that it could cut down on the time she spends on one report by as much as 80%. Because she also covers the House of Representatives beat for GMA News Online, she sees a tool like this being helpful for journalists who do a lot of multitasking. “I could be monitoring a committee hearing while feeding audit reports to the COA Beat Assistant, and then dive deep into them later if there’s something that might be newsworthy,” she told me.

The introduction of custom GPTs is a potential game-changer for newsrooms, as it allows news managers and developers to limit the scope of work for the bot: it doesn’t have to think critically, it just needs to find the right information and summarise it. This allows for the development of new GPT apps customised to do the job within the parameters and guardrails set by the newsroom. It’s also impressive how fast it is to develop and deploy a useful tool for a specific purpose. Previously, you had to do costly and intensive machine learning training to deploy a similar application, and even then you might not get as effective a tool.

Aside from audit reports, similar tools could be developed to help newsrooms and reporters deal with financial statements and annual reports from corporations; legal documents, including court rulings and case files; environmental impact assessment reports; public policy documents and legislative texts; procurement and contracting documents; and many other document types. Custom GPTs could be created for all sorts of corpora of documents to make life easier for investigative journalists. 

9. What’s next for the project.

There’s a low barrier given the $20 monthly fee to use ChatGPT Plus. But for some newsrooms in the Global South it can still be prohibitive to get all their reporters a ChatGPT Plus account, even at that low monthly rate. For document research, journalists could use free tools such as Perplexity.AI or Copilot on the Microsoft Edge browser. However, unlike Custom GPTs, these do not allow guardrails for better handlings of documents.

Equivalent free solutions could be explored by newsrooms in the Global South, including the use of open-source language models to help in document processing, although this could entail a lot more coding work.

An earlier version of this piece was originally published by Generative AI in the Newsroom and can be found in this link.