Today, a great deal of political discourse takes place on the Internet, and is thus subject to the logic governing online media. Messages contain images, sounds, and videos along with written text; algorithms tend to display the messages that elicit the most reactions and generate personalized advertising for users. From May to September 2020, the “Multimodal Rhetoric in Online Media Communications” research group at ZiF investigated how these processes can be tracked and analyzed. In a capstone symposium, titled “Institute on Multimodality: Minds, Media, Technology”, which will take place at ZiF from 28 August to 6 September, the researchers will compile their results and discuss them with international experts and young researchers. The schedule of events also includes a public podium discussion on 30 August.
“It is important to first understand how online discourse works, and for this we need appropriate analytical tools,” explains linguist Professor Dr Kay O’Halloran from the University of Liverpool, who co-chaired this research group together with linguist Professor Dr John Bateman (University of Bremen) and cognitive/computer scientist Professor Dr Mehul Bhatt (Örebro University).
“Previous approaches to such analyses have been predominantly language-centric and lack sound methods for teasing out the meanings that emerge from the interplay of images, videos and language,” explains John Bateman. “So, for most social media, they’re insufficient.”
© University of Liverpool/Örebro University/University of Bremen
This is why the researchers, together with colleagues from computer science, network theory, political science and communication studies, developed tools for the multimodal analysis of online communication, including the analysis of news articles and other media as well as the evaluation of reactions to such articles across various social media, blogs and other websites. Their research encompassed case-studies and news reports appearing on prominent news agencies.
“Our approach leverages modern AI methods to enable fine-grained understanding of small samples of media content, while also bridging together such analyses with traditional big data analyses, where nuances of specific context are typically lost”, explains Mehul Bhatt. “Our methodology is designed top-down by AI specialists working hand-in-hand with researchers in the social sciences and humanities, as well as practitioners and other stakeholders in journalism and media”, he explains.
The research group’s final symposium, the Institute on Multimodality 2022, will begin with an outreach event consisting of a summer school and doctoral colloquium for early-career researchers (28 August until 2 September). The summer school will feature methods and tools for analyzing multimodal interaction from the viewpoints of artificial intelligence, cognitive science, and linguistics. This will be followed by an outward-oriented workshop held by the research group, and will focus on articulating emerging perspectives (5 until 6 September).
A public discussion entitled “Multimodal Rhetoric and Media Communications“ will take place on 30 August at 5 pm. Speakers will include Professor Joanna Björklund, PhD (Umeå University, Sweden), Professor Dr Charles Forceville (University of Amsterdam, the Netherlands), Dr Inés Olza (University of Navarra, Spain) and Professor Barbara Tversky, PhD (Stanford University, USA).
“It is a key goal of ours to pass on the findings and methods we have developed to colleagues from different disciplines, as well as describe and define next steps” explain the conveners of the research group.
All symposium events will be held onsite at ZiF in English.
