Master’s Thesis on Evidence-Based
Claim Assessment:
An NLP Approach Based on
Source Credibility
Master’s Thesis on Evidence-Based
Claim Assessment:
An NLP Approach Based on
Source Credibility
Jana Burns-Balog, MSc, studied Software Engineering in her Master’s program and wrote her master’s thesis titled “Evidence-Based Claim Assessment: A Source-Credibility-Based NLP Approach.” She was supervised by DI (FH) Stephan Leitner on behalf of the company and by FH-Prof. PD DI Dr. Michael Affenzeller on behalf of the university of applied sciences.
Jana: My master’s thesis is about the automated verification of claims, in other words, the question of how AI can help detect misinformation. Misinformation often spreads online faster than it can be manually checked. Existing fact-checking systems usually treat the search for evidence, the classification of that evidence, and the trustworthiness of sources as separate steps. They also rarely disclose how they arrive at their results.
That’s why I developed a prototype that combines these three steps into a single pipeline, specifically for German-language Austrian online news. When a claim is entered, the system first uses classical information retrieval methods (TF-IDF, BM25, and a cluster-based search) to find the most relevant articles from a news archive. An ensemble of language models then determines, for each article, whether it supports the claim, contradicts it, or remains neutral. In the final step, these assessments are combined into an overall verdict, with each article weighted according to the credibility of its source. This credibility is calculated using a heuristic scoring model that I developed as part of the thesis. It takes into account reputation, transparency, content quality, and external ratings.
It was important to me that the system wouldn’t just output “true” or “false”. It provides a traceable basis for decision-making: which articles were found, how each one was assessed, and how trustworthy the respective source is. This allows people to review the result and make corrections if needed. The prototype was evaluated using nearly 2,000 articles from two Austrian and five synthetic news sources. The search found almost exclusively relevant articles, and stance detection achieved around 79% accuracy.
Jana: In practice, fact-checking faces a scaling problem. Newsrooms and fact-checking organizations can only verify a fraction of the claims shared daily on social media. Many media users have neither the time nor the expertise to verify statements themselves. A tool that pre-sorts relevant evidence and presents it transparently can take a lot of work off people’s hands, for example for journalists, fact-checkers, or organizations that need to quickly assess information. For Austria, there’s the added challenge that most fact-checking systems and datasets are geared toward English-language media. They can’t easily be transferred, either linguistically or in terms of content.
That’s exactly where the biggest challenges came from. There was no ready-made dataset of Austrian news articles with matching claims. I therefore had to build the corpus myself: with a custom web scraper that had to deal with changing page structures, paywalls, and legal and ethical requirements, supplemented by synthetically generated articles and manually annotated data. Another difficulty was that credibility can’t be measured objectively. The scoring model therefore had to be traceable and justifiable, without claiming to know the “truth”. Stance detection is also more demanding than it sounds. Articles that merely describe a topic without taking a position are hard for language models to distinguish from texts that support or contradict a claim. And since there was no comparable system for this combination, I couldn’t simply measure the overall pipeline against an existing benchmark. I had to evaluate each component individually.
DI(FH) Stephan Leitner: Today, false reports and unsubstantiated claims spread across news portals and social media in no time at all. This is especially evident in the political arena: misleading statements, targeted disinformation campaigns, or figures taken out of context can shape public opinion and even influence elections. Fact-checking newsrooms try to verify such claims, but can barely keep up with the sheer volume of content. The same mechanisms increasingly affect the business world as well, especially since political debates about topics such as energy prices, tariffs, or new regulations often directly impact companies. A rumor about supply problems, product defects, or an alleged takeover can affect a brand’s reputation or stock price within hours. For PR and communications departments, it’s therefore essential to quickly identify which statements about their own company are circulating and how well substantiated they are.
The challenge lies in the sheer volume of information published every day, which can no longer be handled manually. An automated system must first find, among thousands of articles, those that actually relate to a specific claim. It must then determine whether a text supports the claim, contradicts it, or merely mentions it neutrally, which is difficult even for modern language models. On top of that, not every source is equally trustworthy, yet credibility can’t be measured directly. For practical use, it’s also crucial that the system explains its assessment in a traceable way. Only then can experts within the company review the result, trust it, and correct it if needed, rather than having to rely blindly on an opaque decision.
DI(FH) Stephan Leitner: In her master’s thesis, Ms. Burns-Balog demonstrated that she can independently structure a complex and highly topical problem area and turn it into a functioning overall system. Rather than limiting herself to a single aspect of automated fact-checking, she combined information retrieval, stance detection, and source credibility assessment into a modular pipeline. This required broad methodological knowledge, ranging from classical information retrieval techniques and clustering to the targeted use of large language models. Particularly noteworthy is how she translated a hard-to-grasp concept like source credibility into traceable, measurable criteria, and how well the resulting ranking aligns with the expected assessment of the Austrian media landscape.
Jana: I’ve been working at RISC Software GmbH since the start of my master’s program. As my thesis approached, it was clear to me that I wanted to work on a topic in Natural Language Processing, ideally one connected to a socially relevant problem. Stephan Leitner supported me in finding a topic, and that’s how the idea came about to work on automated fact-checking for Austrian news. On the company side, Stephan Leitner took care of the organizational supervision and supported me in completing the thesis alongside my work at the company. In terms of content, I worked very independently. That was challenging, since I had to make and justify every decision myself, from building the data corpus to the retrieval and stance detection components to the evaluation. At the same time, that’s exactly how I learned so much. I had the freedom to design the system the way I thought was right, and I learned a great deal as a result.
Jana: I’d like to continue working in NLP and data science, with a particular focus on how AI systems can make their decisions traceable. Transparency was already a central theme in my master’s thesis. The system was meant not just to deliver a verdict, but to disclose which evidence and sources it was based on, so people could review and correct it. In the process, I also saw how difficult it is to provide good explanations: they need to be understandable, accurately reflect the model’s actual decision-making process, and must not encourage blind trust.
That’s exactly what I’d like to build on in a doctoral thesis. I’m planning a PhD project on explainable AI in the legal field. Especially in law, where decisions have far-reaching consequences and must be justified, it’s not enough for a model to simply deliver a result. You need to be able to understand why it arrived at that result. Over the next few years, I’d like to further deepen this connection between language processing, traceability, and the responsible use of AI.
We’ve received your documents and will review your application carefully.
As soon as we have feedback for you or need any additional information, we’ll get in touch.
Thank you for your interest – we’re excited to learn more about you!
In the meantime, feel free to pass the time by discovering a bit more about us 🙂
