BQ1D: Bringing it all together Umut Gl (LiI Postdoc) Marcel van - PowerPoint PPT Presentation

BQ1D: Bringing it all together Umut Güçlü (LiI Postdoc) Marcel van Gerven (BQ1D coordinator) Stefan Frank, Ivan Titov, Roel Willems

Research objective Provide an algorithmic account of language processing in humans by validating computational models using neurobehavioural data BQ1D aims to answer the following questions: - Can neural networks provide accurate models of neural information processing? - How are semantic representations encoded in the human brain? - Can we reconstruct (inner) speech from patterns of human brain activity

Objective: Validate neural models on naturalistic linguistic processing vision sound linguis*cs van Gerven (2016) J. Math. Psych; Güçlü et al. (2017). Neuroimage; Seeliger et al. (2017). Neuroimage. Rev.; Güçlü et al. (2017). J. Front. Comput. Neurosci

Desired outcome: a framework for model comparison • A computational framework for modeling and understanding neural processing of semantic information

Cognitive plausibility: models that maximize model evidence • Test hypotheses about language processing via model comparison

Advanced models: Memory networks New understanding about memory-related and predictive processes in language via novel neural network architectures Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwi ń ska, A., … Hassabis, D. (2016). Nature

Advanced models: Question answering systems Semantic structure embedded in neural network Q&A systems

Objective: Mapping of semantic representations Established correspondences between sensory representations of these models and those of the sensory cortices (J Neurosci, 2015; Neuroimage, 2015). Preferred layer Güçlü, U., & van Gerven, M. A. J. (2015). J. Neuroscience.

Objective: Mapping of semantic representations Analogous results established in auditory cortex (NIPS 2016). Güçlü, U., & van Gerven, M. A. J. (2016). NIPS.

Objective: Mapping of semantic representations Trained low-level feature models and high-level semantic models end-to-end using RNNs Güçlü, U., & van Gerven, M. A. J. (2017). Frontiers in Computational Neuroscience

Objective: Mapping of semantic representations Recent invasive work; neural correlates of speech and music processing Berezutskaya, J., Freudenburg, Z. V, Güçlü, U., Gerven, M. A. J. Van, & Ramsey, Nick, F. (2017). J Neurosci.

Automated mapping • Automated mapping of the functional organisation of semantic representations in the human brain Huth, A. G., Heer, W. A. De, Griffiths, T. L., Theunissen, F. E., & Galant, J. L. (2016). Nature.

Decoding (inner) speech Decoding semantic content from measurements of brain activity Simanova, I., Gerven, M. A. J. Van, Oostenveld, R., & Hagoort, P. (2014). Journal of Cognitive Neuroscience .

Decoding inner speech Inversion of neural networks provides accurate reconstructions of neural representations (NIPS 2017).

Digression: Predicting cognitive traits • Can we train neural networks to predict high-level cognitive traits? • Can we interrogate neural networks to understand the underlying principles? Güçlütürk, Y., Güçlü, U., van Gerven, M.A.J., van Lier, R., 2016. Deep Impression: Audiovisual Deep Residual Networks for Multimodal Apparent Personality Trait Recognition

Digression: Multimodal model Model trained on 6000 15-second- long video clips Can we predict personality traits? Does mul*modal fusion help? Güçlütürk, Y., Güçlü, U., van Gerven, M.A.J., van Lier, R., 2016. ECCV

Digression: Explainability Visualising how individual pixels are involved in trait predic*on Applicable to language models as well Güçlütürk, Y., Güçlü, U., van Gerven, M.A.J., van Lier, R., 2016. ECCV

Digression: Predicting cognitive traits Conscientiousness Agreeableness Extraversion Neuroticism Openness Güçlütürk, Y., Güçlü, U., van Gerven, M.A.J., van Lier, R., 2016. Deep Impression: Audiovisual Deep Residual Networks for Multimodal Apparent Personality Trait Recognition

Where’s the data? • Focus on within subject analysis • Fitting of complex models using hours of single subject data • Rigorous control of confounds via headcast, etc.

Concrete goals - Develop new language models (e.g., for summarization, generation, translation, etc.) that account for di ff erent aspects of language processing in the human brain. - Provide a computational framework that allows one to fit di ff erent neural networks (BQ1L/P/N/D) to neural recordings obtained under naturalistic language processing tasks. - New approaches to compare and interpret neural nets - New approaches for automated mapping of semantic representations - Next-generation neural decoding models

Challenges and opportunities - Exchanges with other BQ1 subprojects: - Rapid model testing and sharing, common codebase, bridge levels of analysis - Foster exchanges between theorists and experimentalists (other BQs) - Ensure sustainability and impact within and beyond the consortium: - Dissemination, grants, student projects - Where’s the data? Opportunities for data collection and dissemination…

THANK YOU