BQ1D: Bringing it all together Umut Gl (LiI Postdoc) Marcel van - - PowerPoint PPT Presentation

BQ1D: Bringing it all together

Umut Güçlü (LiI Postdoc) Marcel van Gerven (BQ1D coordinator) Stefan Frank, Ivan Titov, Roel Willems

Research objective

Provide an algorithmic account of language processing in humans by validating computational models using neurobehavioural data BQ1D aims to answer the following questions:

• Can neural networks provide accurate models of neural

information processing?

• How are semantic representations encoded in the human

brain?

• Can we reconstruct (inner) speech from patterns of human

brain activity

sound vision

Objective: Validate neural models on naturalistic linguistic processing

linguis*cs

van Gerven (2016) J. Math. Psych; Güçlü et al. (2017). Neuroimage; Seeliger et al. (2017). Neuroimage. Rev.; Güçlü et al. (2017). J. Front. Comput. Neurosci

Desired outcome: a framework for model comparison

• A computational framework for modeling and

understanding neural processing of semantic information

Cognitive plausibility: models that maximize model evidence

• Test hypotheses about language processing via model comparison

Advanced models: Memory networks

New understanding about memory-related and predictive processes in language via novel neural network architectures

Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., … Hassabis, D. (2016). Nature

Advanced models: Question answering systems

Semantic structure embedded in neural network Q&A systems

Objective: Mapping of semantic representations

Established correspondences between sensory representations of these models and those of the sensory cortices (J Neurosci, 2015; Neuroimage, 2015).

Preferred layer Güçlü, U., & van Gerven, M. A. J. (2015). J. Neuroscience.

Analogous results established in auditory cortex (NIPS 2016).

Güçlü, U., & van Gerven, M. A. J. (2016). NIPS.

Objective: Mapping of semantic representations

Trained low-level feature models and high-level semantic models end-to-end using RNNs

Güçlü, U., & van Gerven, M. A. J. (2017). Frontiers in Computational Neuroscience

Objective: Mapping of semantic representations

Recent invasive work; neural correlates of speech and music processing

Berezutskaya, J., Freudenburg, Z. V, Güçlü, U., Gerven, M. A. J. Van, & Ramsey, Nick, F. (2017). J Neurosci.

Objective: Mapping of semantic representations

Automated mapping

• Automated mapping of the functional organisation of

semantic representations in the human brain

Huth, A. G., Heer, W. A. De, Griffiths, T. L., Theunissen, F. E., & Galant, J. L. (2016). Nature.

Decoding (inner) speech

Decoding semantic content from measurements of brain activity

Simanova, I., Gerven, M. A. J. Van, Oostenveld, R., & Hagoort, P. (2014). Journal of Cognitive Neuroscience.

Decoding inner speech

Inversion of neural networks provides accurate reconstructions of neural representations (NIPS 2017).

Digression: Predicting cognitive traits

Güçlütürk, Y., Güçlü, U., van Gerven, M.A.J., van Lier, R., 2016. Deep Impression: Audiovisual Deep Residual Networks for Multimodal Apparent Personality Trait Recognition

• Can we train neural networks to predict high-level cognitive traits?
• Can we interrogate neural networks to understand the underlying principles?

Digression: Multimodal model

Güçlütürk, Y., Güçlü, U., van Gerven, M.A.J., van Lier, R., 2016. ECCV

Model trained on 6000 15-second- long video clips Can we predict personality traits? Does mul*modal fusion help?

Digression: Explainability

Güçlütürk, Y., Güçlü, U., van Gerven, M.A.J., van Lier, R., 2016. ECCV

Applicable to language models as well Visualising how individual pixels are involved in trait predic*on

Digression: Predicting cognitive traits

Güçlütürk, Y., Güçlü, U., van Gerven, M.A.J., van Lier, R., 2016. Deep Impression: Audiovisual Deep Residual Networks for Multimodal Apparent Personality Trait Recognition

Extraversion

Agreeableness Conscientiousness Neuroticism Openness

Where’s the data?

• Focus on within subject analysis
• Fitting of complex models using hours of single

subject data

• Rigorous control of confounds via headcast, etc.

Concrete goals

• Develop new language models (e.g., for summarization, generation,

translation, etc.) that account for different aspects of language processing in the human brain.

• Provide a computational framework that allows one to fit different neural

networks (BQ1L/P/N/D) to neural recordings obtained under naturalistic language processing tasks.

• New approaches to compare and interpret neural nets
• New approaches for automated mapping of semantic representations
• Next-generation neural decoding models

Challenges and opportunities

• Exchanges with other BQ1 subprojects:
• Rapid model testing and sharing, common codebase, bridge levels of

analysis

• Foster exchanges between theorists and experimentalists (other BQs)
• Ensure sustainability and impact within and beyond the consortium:
• Dissemination, grants, student projects
• Where’s the data? Opportunities for data collection and dissemination…

THANK YOU