Last changed 16 May 2014 ............... Length about 900 words (11,000 bytes).
(Document started on 16 May 2014.) This is a WWW document maintained by Steve Draper, installed at You may copy it. How to refer to it.

Web site logical path: [] [~steve] [resources] [main crs] [this page] [CT: a general mental ability?]

Some problems with common criticisms by psychology students

By Steve Draper,   Department of Psychology,   University of Glasgow.

More points

General 2: Keeping a grip on the purpose of early studies

Not knowing the cause / not having a theory, is not of primary interest in applied work i.e. work about benefits to people.

Unrepresentative or unequal gender balance

Be careful here and think what the logic is of the study. The following remarks apply not only to gender, but to other demographics e.g. age, family income, nationality.

You want representative balances e.g. of males and females, if you want to describe the population accurately. E.g. if you want to know how many anxious people there are in the population; or if I want to know what's wrong with my teaching, I don't want just the few angry ones, or the few fans, to respond but a representative sample.

You want equal balances e.g. of males and females, if you want to study gender differences as an important effect (biggest causal factor). In this department, the student gender ratio is about 1:5 male:female. So to study gender effects, you need an artificial and unrepresentative but equal balance.

Otherwise, if you suspect a gender difference but it is not your primary research question, then it may be better to ONLY use one gender and so get rid of a confounding factor; get a clear picture of causal relationships without it, then leave gender differences to further work.

Convenience and self-selected samples

A much bigger worry in many cases is studies using convenience samples.

What matters, is random selection from the population you wnat to generalise about.

Convenience samples: country and culture

Get in the habit of stating what country the research was done in. Psychologists like to pretend that psychology is about universal truths: but that isn't true of education (the system is different in each country) nor politics (ditto), nor even medical treatments (ditto); and in fact may not be true of a lot of Social (cultures differ).

Big numbers (2): The experiment needs more participants

If you say a study needs bigger numbers even though it reports a SigDiff then you, the critic, are saying you don't trust statistics. Numbers are not good in themselves: rather, a smaller p-value is good, in the sense that it implies a greater degree of certainty.

Big numbers: The experiment needs more participants

Someone criticised one paper on ACT treatment for porn addiction for having a sample of only 6, all males, in a restricted age range, from one city. "The sample was therefore not very diverse and has low generalizability." This is a cheap comment. Would you criticise a study of treatment for menopausal symptoms for not having male participants? children? women under 30? If you want to do an experiment to test a statement like "All humans everywhere at all times have an IQ between 90 and 110" then you need a sample that represents the population. If you are studying a possible new treatment for a new disease like porno addiction, then you start with a sample only of porno addicts. (If you want to study how prevalent the disease is, then you might want to sample the whole population, though you would have to sample many thousands to get 6 such people, so few funders would pay for that.)

That is why studies of disease and treatments would be silly to sample the whole population. However there is nevertheless a trap that goes with pre-selecting a "clinical population". The trap is, that for all diseases, certainly mental ones, there is a large spontaneous recovery rate. So just because someone was ill, got a treatment, and then recovered is actually no proof that the treatment made any difference unless you have a control group. Even if you do, many, many researchers make the wrong stats / calculations about recovery rate. They do this because they pre-select for being ill (meeting clinical criteria) at time 1; and then repeat the measure at time 2, but do NOT allow for how some will have recovered anyway (NOR for how some in any control group would have got ill during that time). Basically, it means you can't use pre- to post-test differences if you have pre-selected the sample. You might all like to read Senn (2009) which is a paper addressed to the non-expert about this, normally called in Stats "regression to the mean", because it is a standard real criticism to always consider for any clinical trial.

But given that many studies must use clinical populations, you have to raise a discussion about what this means for a syndrome which does not already in advance have a fully known cause. What counts as a case when you are trying to study whether there is a disease at all? For addictions not in DSM, this is the issue. Many of you touched on this, but probably to improve your CRs you would have to discuss this issue of selection explicitly. Who self-labels as an addict? who is labelled by others? what are the internal psychological mechanisms? are you studying treatments? Each of these is a different selection criterion for the study and none of them define the disease in a way that is the way you will in future, post hoc, want to define it.

If there is an agreed (and testable) single cause, then simple treatment trials make sense. If there is an agreed diagnosis, then you are treating "patients with a diagnosis of X" but must expect some variability in outcome because very likely they do not all have the same real underlying disease. If you are addressing those who self-label as sex addicts (say) then you are dealing with a distressed population, but can only say what effect the treatment has on the set of self-labelling addicts. In CRs of this kind, you probably need to comment on this to some extent. I note that, according to the DSM-5 list of criteria which Olivia gives for excoriation disorder, one is "the skin picking causes clinically sig. distress or impairment ... " Note that this criterion is NOT a causal one of the same kind as "tests positive for HIV", but rests entirely on the subjective reaction of the patient and those around her or him. Thus DSM is not a purely scientific, certainly not a purely biological, tool. Discussions of both DSM and of any treatments in relation to DSM, cannot therefore be discussed in those terms. Doctors sometimes call something a syndrome, when they have a cluster of diagnostic criteria but no idea about causation: as in "AIDS" (... Syndrome). Any paper that talks as if a disorder is a disease is usually assuming without any evidence whatever that there is a single disease, with a single underlying causal mechanism. In a CR I would always point out what they are doing, and that their approach is pseudo-scientific in that they talk as if the causation was of a known type, when the main question is whether or not that is true. In CRs on the topics you had, I would be tempted to use the language of "syndromes" to make this clear all the time, to paraphrase the author's terms for groups e.g. "had some effect on the group of self-labelling sex addicts" to reflect the truth, not the author's wishes.

A different set of high level general issues that perhaps should be addressed in CRs in such "early stage" research are touched on in David's CR and in Catriona's (but perhaps should be emphasised in all). That is the issue of what KIND of concept should be used in addiction.

1) Should addiction be a binary or spectrum kind of concept. Either "you are ill/well"; or "autistic spectrum" kind of approach. Binary is appropriate for (say) a disease caused by a specific bacterium. Obesity however ....

2) Social reward mechanisms. Many substances have a big social component: alcohol in bars for instance. Often a big step to shedding an addiction is to entirely shed all your friends who are fellow users; and conversely, it is the social component that may be a big part of the reward. Chemists don't want to admit this, but psychological rewards can come from social as well as substance causes.

3) Social inhibition mechanisms. Conversely, social mechanisms may help us limit our intake of some things, or some of our behaviours. Twelve step programmes are largely based on social mechanisms.

Any approach to addiction which says nothing about these issues seems unlikely either to account for addiction behaviour, or to account for some of the most successful treatment approaches. In discussing possible new addictions, this probably needs to be addressed full on.

So the general kind of issues here include: a) Even if a completely chemical theory is adequate for some addictions, how are we to address other kinds of distressed patients who share some but not all such characteristics? If you reserve the word "addiction" to a subset of these conditions, does that mean DSM needs to have a whole new category with a new word to deal with the rest? b) There are probably a whole chain of mechanisms affected by addiction. We may see disorders that involve some, but not all, those mechanisms. E.g. not only what makes you try it; how it is rewarding; how the behaviour / impulse is normally inhibited (or not)...

Your comment on an unrepresentative sample confuses the need for a clinical sample with a need to make generalisations about the whole human race in an early study of 6 people.

Web site logical path: [] [~steve] [resources] [this page]
[Top of this page]