Multi-Modal Fake News Detection via Bridging the Gap between Modals.

Peng Liu, Wenhua Qian, Dan Xu, Bingling Ren, Jinde Cao
Author Information
  1. Peng Liu: School of Information Science and Engineering Yunnan University, Kunming 650500, China.
  2. Wenhua Qian: School of Information Science and Engineering Yunnan University, Kunming 650500, China.
  3. Dan Xu: School of Information Science and Engineering Yunnan University, Kunming 650500, China.
  4. Bingling Ren: School of Information Science and Engineering Yunnan University, Kunming 650500, China.
  5. Jinde Cao: School of Mathematics, Southeast University, Nanjing 210096, China. ORCID

Abstract

Multi-modal fake news detection aims to identify fake information through text and corresponding images. The current methods purely combine images and text scenarios by a vanilla attention module but there exists a semantic gap between different scenarios. To address this issue, we introduce an image caption-based method to enhance the model's ability to capture semantic information from images. Formally, we integrate image description information into the text to bridge the semantic gap between text and images. Moreover, to optimize image utilization and enhance the semantic interaction between images and text, we combine global and object features from the images for the final representation. Finally, we leverage a transformer to fuse the above multi-modal content. We carried out extensive experiments on two publicly available datasets, and the results show that our proposed method significantly improves performance compared to other existing methods.

Keywords

References

  1. IEEE Trans Pattern Anal Mach Intell. 2017 Jun;39(6):1137-1149 [PMID: 27295650]
  2. Z Gesundh Wiss. 2021 Oct 09;:1-10 [PMID: 34660175]
  3. IEEE Trans Image Process. 2021;30:1180-1192 [PMID: 33306468]
  4. Proc Natl Acad Sci U S A. 2019 Apr 16;116(16):7662-7669 [PMID: 30642953]
  5. Neural Comput Appl. 2022;34(24):21503-21517 [PMID: 34054227]
  6. Neural Comput Appl. 2022;34(18):16019-16032 [PMID: 35529091]
  7. Neural Comput Appl. 2021;33(14):8597-8613 [PMID: 33424132]
  8. Science. 2018 Mar 9;359(6380):1146-1151 [PMID: 29590045]

Word Cloud

Created with Highcharts 10.0.0imagestextsemanticfakeinformationimagenewsdetectionmethodscombinescenariosgapcaption-basedmethodenhancetransformermulti-modalMulti-modalaimsidentifycorrespondingcurrentpurelyvanillaattentionmoduleexistsdifferentaddressissueintroducemodel'sabilitycaptureFormallyintegratedescriptionbridgeMoreoveroptimizeutilizationinteractionglobalobjectfeaturesfinalrepresentationFinallyleveragefusecontentcarriedextensiveexperimentstwopubliclyavailabledatasetsresultsshowproposedsignificantlyimprovesperformancecomparedexistingMulti-ModalFakeNewsDetectionviaBridgingGapModals

Similar Articles

Cited By

No available data.