Sven - this was really wonderful, and although it took me a few days to read it, I really enjoyed the post. Thank you for writing it. I attempted a PhD in psych about a decade ago, realised it wasn't working out, and left for data science/ML in industry. I've always had a soft corner for the behavioural sciences (in a broad sense, not thinking of specific academic disciplines).
Over the past year or two, I've found myself becoming more and more disillusioned by results in psychology. I had doubts even during the PhD attempt, but (ironically) quelled them with an argument from authority. I was always surprised by the dramatic conclusions psych studies came to despite tiny, non-representative samples drawn primarily from WEIRD populations such as US university students who happened to need credit for a psychology class they were enrolled in.
Cognitive psychology made grand claims about memory, learning, etc., in a very "non-indexical" manner, as you and Science Banana might put it. My own study involved using video game simulations to study how humans navigated, to see if the presence or absence of certain environmental cues influenced their behaviour (this feels like priming to me because we didn't alert them to changes in the environmental cues, but I'm not 100% sure). We were trying to study memory and learning, but I'm now puzzled by how simplistic the study design was relative to the conclusions we hoped to draw.
Not only this, it also felt like cognitive psychology entirely ignored culture and "life history" (for lack of a better term). I grew up (and now live) in India, but was in the US to study. The environments couldn't be more different. Even if the experimental conditions used in my study had shown non-null results, I doubt I could have generalised them to India. I recall trying to bring this up once or twice, but there wasn't much interest in listening from professors. I don't entirely blame them, they probably understand the US and have no idea of what India is like, so why bother?
But the challenge for me was that neither the experiment nor its potential conclusions would have worked outside the specific experimental (and cultural) setting. So how useful would the conclusions have been anyway? To return to the point about indexicality, it now feels like psych studies (non-survey ones, specifically) capture some indexical knowledge, but present it as a more generalised or global conclusion. These are then amplified in the media.
A separate point that I found utterly puzzling was that studies were conducted in a manner that bore little or no resemblance to the outside world. The participants would take part in the study in very "clinical" settings. Behaviour in the real world obviously takes place in much more messy circumstances and studies didn't capture this at all. Once again, the issue of generalisability comes in - are the conclusions valid outside the experiment room?
Obviously, I have some "home field" bias, but I wonder if fields like machine learning have something to add here - rather than having a specific model of behaviour and see if it pans out through experiments, maybe attempting to predict behaviour using models could teach us more. I haven't spent much time on this, but there are obvious issues - representativeness of the data, sampling methods, etc. But perhaps something worth thinking about.
Long comment, but I hope some of this resonates with you.
Sven - this was really wonderful, and although it took me a few days to read it, I really enjoyed the post. Thank you for writing it. I attempted a PhD in psych about a decade ago, realised it wasn't working out, and left for data science/ML in industry. I've always had a soft corner for the behavioural sciences (in a broad sense, not thinking of specific academic disciplines).
Over the past year or two, I've found myself becoming more and more disillusioned by results in psychology. I had doubts even during the PhD attempt, but (ironically) quelled them with an argument from authority. I was always surprised by the dramatic conclusions psych studies came to despite tiny, non-representative samples drawn primarily from WEIRD populations such as US university students who happened to need credit for a psychology class they were enrolled in.
Cognitive psychology made grand claims about memory, learning, etc., in a very "non-indexical" manner, as you and Science Banana might put it. My own study involved using video game simulations to study how humans navigated, to see if the presence or absence of certain environmental cues influenced their behaviour (this feels like priming to me because we didn't alert them to changes in the environmental cues, but I'm not 100% sure). We were trying to study memory and learning, but I'm now puzzled by how simplistic the study design was relative to the conclusions we hoped to draw.
Not only this, it also felt like cognitive psychology entirely ignored culture and "life history" (for lack of a better term). I grew up (and now live) in India, but was in the US to study. The environments couldn't be more different. Even if the experimental conditions used in my study had shown non-null results, I doubt I could have generalised them to India. I recall trying to bring this up once or twice, but there wasn't much interest in listening from professors. I don't entirely blame them, they probably understand the US and have no idea of what India is like, so why bother?
But the challenge for me was that neither the experiment nor its potential conclusions would have worked outside the specific experimental (and cultural) setting. So how useful would the conclusions have been anyway? To return to the point about indexicality, it now feels like psych studies (non-survey ones, specifically) capture some indexical knowledge, but present it as a more generalised or global conclusion. These are then amplified in the media.
A separate point that I found utterly puzzling was that studies were conducted in a manner that bore little or no resemblance to the outside world. The participants would take part in the study in very "clinical" settings. Behaviour in the real world obviously takes place in much more messy circumstances and studies didn't capture this at all. Once again, the issue of generalisability comes in - are the conclusions valid outside the experiment room?
Obviously, I have some "home field" bias, but I wonder if fields like machine learning have something to add here - rather than having a specific model of behaviour and see if it pans out through experiments, maybe attempting to predict behaviour using models could teach us more. I haven't spent much time on this, but there are obvious issues - representativeness of the data, sampling methods, etc. But perhaps something worth thinking about.
Long comment, but I hope some of this resonates with you.