Lecture

Psychology with Deep Learning

22 Dec 2022 • Richard Kuo

This introduction includes Sentiment Analysis, Emoton Detection, Speech Emotion Recognition, MBTI, Behavior Prediction, Social Simulation, Theory of Mind, Brain Model, Episodic Memory, Semantic Memory, The Emotion Machine.

Python libraries for Psychology researchers

PsychoPy
PsychoPy is also a Python application for creating Psychology experiments.

Blog: Best Python libraries for Psychology researchers

PsyUtils “The psyutils package is a collection of utility functions useful for generating visual stimuli and analysing the results of psychophysical experiments.
Psisignifit is a toolbox that allows you to fit psychometric functions. Further, hypotheses about psychometric data can be tested.
Pygaze is a Python library for eye-tracking data & experiments.
MNE is a library designed for processing electroencephalography (EEG) and magnetoencephalography (MEG) data.
Kabuki is a Python library for the effortless creation of hierarchical Bayesian models.
Scikit-learn is an excellent Python package if you want to learn how to do machine learning

Machine Learning in Psychometrics and Psychological Research

Paper: Review of Machine Learning Algorithms for Diagnosing Mental Illness

Bio-inspired Computer Vision

Towards a synergistic approach of artificial and biological vision

Book: Bio-inspired computer vision: Towards a synergistic approach of artificial and biological vision

Computational graph of a foveated spatial transformer network

Code: int-lab-book

Sentiment

Sentiment Analysis of Tweets

Blog: SENTIMENT ANALYSIS OF TWEETS WITH PYTHON, NLTK, WORD2VEC & SCIKIT-LEARN
Dataset: First GOP Debate Twitter Sentiment
Code: Sentiment-Analysis-NLTK-ML and LSTM
Kaggle: Sentiment NLTK
Kaggle: Sentiment LSTM

Depression Detection

Paper: Deep Learning for Depression Detection of Twitter Users
Paper: Machine Learning-based Approach for Depression Detection in Twitter Using Content and Activity Features
Paper: A comprehensive empirical analysis on cross-domain semantic enrichment for detection of depressive language
Paper: DepressionNet: A Novel Summarization Boosted Deep Framework for Depression Detection on Social Media
Code: Detect Depression In Twitter Posts

Emotion Detection

Facial Expression Recognition

Dataset: FER-2013

7 facial expression, 28709 training images, 7178 test images
labels = [“angry”, “disgusted”, “fearful”, “happy”, “neutral”, “sad”, “surprised”]

Kaggle: fer2013-cnn

Code: EmoPy

First Impression

Paper: Predicting First Impressions with Deep Learning
Code: mel-2445/Predicting-First-Impressions
Annotations : Age, Dominance, IQ, Trustworthiness

Large-Scale Facial Expression Recognition

Paper: Suppressing Uncertainties for Large-Scale Facial Expression Recognition
Dataset: Real-world Affective Faces Database (RAF-DB)

29672 number of real-world images,
a 7-dimensional expression distribution vector for each image,
two different subsets: single-label subset, including 7 classes of basic emotions; two-tab subset, including 12 classes of compound emotions,
5 accurate landmark locations, 37 automatic landmark locations, bounding box, race, age range and gender attributes annotations per image,
baseline classifier outputs for basic emotions and compound emotions.

Code: Self-Cure-Network

Emotion Recognition from Body Gestures

Paper: A Generalized Zero-Shot Framework for Emotion Recognition from Body Gestures

Speech Emotion Recognition (SER)

Datasets

Multimodal Sentiment Analysis

MISA

Paper: MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis

Transformer-based joint-encoding

Paper: A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis
Code: bdel/MOSEI_UMONS

Multimodal Transformer Encoder for two modalities with joint-encoding

Multimodal Fusion

Paper: Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis
Code: Multimodal Deep Learning

Bi-Bimodal Modality Fusion

Paper: Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis
Code: declare-lab/BBFN

K-EmoCon

Paper: K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations

CycleEmotionGAN++

Paper: Emotional Semantics-Preserved and Feature-Aligned CycleGAN for Visual Emotion Adaptation

Myers–Briggs Type Indicator (MBTI)

Myers–Briggs Type Indicator (MBTI) Assignment

An Overview of the Myers-Briggs Type Indicator

MBTI scales:

Extraversion (E) – Introversion (I) 外向型 vs 內向型
Sensing (S) – Intuition (N) 實感型 vs 直覺型
Thinking (T) – Feeling (F) 思考型 vs 情感型
Judging (J) – Perceiving (P) 判斷型 vs 感覺型

The MBTI Types:

ISTJ - The Inspector
ISTP - The Crafter
ISFJ - The Protector
ISFP - The Artist
INFJ - The Advocate
INFP - The Mediator
INTJ - The Architect
INTP - The Thinker
ESTP - The Persuader
ESTJ - The Director
ESFP - The Performer
ESFJ - The Caregiver
ENFP - The Champion
ENFJ - The Giver
ENTP - The Debater
ENTJ - The Commander

Predicting MBTI with RNN

Dataset: (MBTI) Myers-Briggs Personality Type Dataset
Kaggle: mbti-lstm

Personality Analyzer

Code: personality prediction from text
Using FB webscraper, based on RandomForestClassifier & TfidfVectorizer

Install MongoDB
Scrape friends info from your FB account (fb_webscraper.py)
Train model (model.py)
Make Prediction (predict.py)

Behavior Prediction

Understanding Consumer Behavior with RNN

Paper: Understanding Consumer Behavior with Recurrent Neural Networks

CPC-18

Paper: Predicting human decisions with behavioral theories and machine learning
Raw Data: All raw data for CPC18
Code: CPC-18 baseline models and source code

MLP + Cognitive Prior

Paper: Cognitive Model Priors for Predicting Human Decisions

Predicting human decision making in psychological tasks

Paper: Predicting human decision making in psychological tasks with recurrent neural networks
Code: HumanLSTM

System-level simulations (SLS) - Explore a given situation as a whole and how individuals and groups respond to the presence of certain variables.
System-level modeling (SLM) - Creates a more complex and sophisticated environment and aims to be able to make predictions about the behavior of any individual entity or thing within the simulation.
Agent-based social simulation(ABSS) - Models societies on intelligent agents and studies their behavior within the simulated environment for the application to real-world situations.
Agent-based modeling (ABM) - Involves independent agents with individual-specific behaviors interacting in networks.

Neural MMO

Blog: OpenAI 打造 Neural MMO 遊戲，觀察 AI 在複雜開放世界中表現
Paper: The Neural MMO Platform for Massively Multiagent Research
UserGuide

Paper: Learning to Communicate with Deep Multi-Agent Reinforcement Learning
Paper: Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
Paper: TarMAC: Targeted Multi-Agent Communication
Blog: About communication in Multi-Agent Reinforcement Learning

Code: Sequential Social Dilemma Games

Theory of Mind

Theory of Mind: refers to the mental capacity to understand other people and their behavior by ascribing mental states to them Paper: Mindreaders: the cognitive basis of theory of mind
Blog: How the Theory of Mind Helps Us Understand Others
One study found that children typically progress through five different theory of mind abilities in sequential, standard order.6
Tasks Listed From Easiest to Most Difficult:

The understanding that the reasons why people might want something (i.e. desires) may differ from one person to the next
The understanding that people can have different beliefs about the same thing or situation
The understanding that people may not comprehend or have the knowledge that something is true
The understanding that people can hold false beliefs about the world
The understanding that people can have hidden emotions, or that they may act one way while feeling another way

Machine Theory of Mind

Paper: Machine Theory of Mind
Code: VArdulov/ToMNet

APES

Paper: APES: a Python toolbox for simulating reinforcement learning environments
Github: Artificial Primate Environment Simulator (APES)
Code: APES-simple

Perspective Taking

Paper: Perspective Taking in Deep Reinforcement Learning Agents
Perspective taking is the ability to look at things from a perspective that differs from our own [15].
It could be defined as “the cognitive capacity to consider the world from another individual’s viewpoint”
(a) Overview of the simulation environment and visual encodings. The artificial monkey with green circle is the subordinate agent and the one with red circle is the dominant agent.
(b) Two examples of a subordinate agent goal-oriented behavior as driven by our neural network controller. In the top panel the agent should avoid the food as it is observed by the dominant. In the bottom panel the agent should acquire the food as it is not observed by the dominant. Code: APES_PT
Kaggle: rkuo2000/APES-PT

Visual Perspective Taking

Paper: Visual Perspective Taking for Opponent Behavior Modeling
Visual Perspective Taking (VPT) refers to the ability to estimate other agents’ viewpoint from its own observation.

ToMNet+

Paper: Using Machine Theory of Mind to Learn Agent Social Network Structures from Observed Interactive Behaviors with Targets
Code: ToMnet+ project

Visual Behavior Modelling

Paper: Visual behavior modelling for robotic theory of mind
Code: Visual Behavior Modelling

ToM for Humanoid Robot

Paper: Theory of Mind for a Humanoid Robot

The phenomenon of habituation has been described generally as decrement in response to repeated stimulation (Harris 1943)

CX-ToM

Paper: CX-ToM: Counterfactual Explanations with Theory-of-Mind for Enhancing Human Trust in Image Recognition Models

Brain Model

Brain-like DNN

Paper: Brain hierarchy score: Which deep neural networks are hierarchically brain-like?

DNNBrain

Paper: DNNBrain: A Unifying Toolbox for Mapping Deep Neural Networks and Brains
Code:BNUCNL/DNNBrain
DNN Model: AlexNet

GNN

Paper: Graph Neural Networks in Network Neuroscience

BrainGNN

Paper: BrainGNN: Interpretable Brain Graph Neural Network for fMRI Analysis

DMBN

Paper: Deep Representation Learning For Multimodal Brain Networks

BrainNNExplainer

Paper: BrainNNExplainer: An Interpretable Graph Neural Network Framework for Brain Network based Disease Analysis

Multi-GCN

Paper: Multiplex Graph Networks for Multimodal Brain Network Analysis

Learning

Self-supervised Learning

Paper: Self-supervised learning through the eyes of a child
Dataset: SAYCam: A large, longitudinal audiovisual dataset recorded from the infant’s perspective
Code: eminorhan/baby-vision
databrary

Generalized Schema Learning

Paper: Learning to perform role-filler binding with schematic knowledge
Code: cchen23/generalized_schema_learning
Coffee shop World
The “engine” takes a schema and generates a bunch of stories!
python run_engine.py poetry fight 2 2

Neural Model of Schemas and Memory

Paper: A Neural Model of Schemas and Memory Consolidation

Episodic Memory

Episodic Memory Reader

Paper: Episodic Memory Reader: Learning What to Remember for Question Answering from Streaming Data

Episodic Memory in Lifelong Language Learning

Paper: Episodic Memory in Lifelong Language Learning
Code: episodic-lifelong-learning

Replay Episodic Memory

Paper: DRILL: Dynamic Representations for Imbalanced Lifelong Learning

Semantic Memory

Paper: Semantic Memory: A Review of Methods, Models, and Current Challenges

Semantic Memory Representation
- Network-based Approaches
- Feature-based Approaches
- Distributional Approaches : Distributional Semantic Models (DSMs)
Semantic Memory Learning
- Error-free Learning-based DSMs
- Error-driven Learning-based DSMs: word2vec
Contextual and Retrieval-based Semantic Memory
- Ambiguity Resolution in Error-free Learning-based DSMs
- Ambiguity Resolution in Predictive DSMs: ELMo, BERT, GPT-3
Retrieval-based Models of Semantic Memory