I wrote a blogpost a while back on the White Swan hypothesis, where I challenge my research students to test the hypothesis that all swans are white, as in the UK and Europe, this is a generally accepted notion. Either students suggest finding as many white swans as possible to support their hypothesis, or look for a single black swan to disprove their hypothesis. As all good scientists know, the latter is ideal, but the former is so often the reality in the ‘publish or perish’ era of academia, and is cause for so much bad science. When I wrote the blog post, I promised to highlight some of the Black Swans that I’ve located, and in doing so, disproved my Professor’s hypotheses, made myself mildly unpopular and in some cases, managed to set the academic record straight.
In the link above (bad science) Amgen, a large Biotech company, attempted to replicate 53 papers in the area of bone oncology, and found the central hypothesis in 47 of them was not supported. Worrying for me, this is the area where I research. Explanations for so many cancer studies being impossible to replicate include direct research fraud, delusional scientists who are convinced they are right and fool themselves into believing their results and so are fooled by their own cognitive bias (white swans), and to some extent, poor quality reagents.
We were working on Osteoprotegerin, a protein that Amgen were interested in, partly because it was reported to bind and inactivate the cytotoxic ligand TRAIL. TRAIL was (and still is) being developed by Amgen as an anti-cancer therapy, and the concern was in the early 2000s that Osteoprotegerin (OPG) might stop their new drug from working if produced at biologically relevant levels by either the cancer cells themselves, or the supporting mesenchymal/connective tissue cells. So that’s what we worked on.
The plan was simple. Grow cancer cell lines in the lab, allow them to accumulate enough OPG so that they are ‘protected’ from TRAIL. Alternatively grow cancer-associated connective tissue cells in the lab, and see if they produced enough OPG to protect the cancer cells. The experiment worked. Both breast and prostate cancer cell lines secreted TRAIL, and in these experiments, the in vitro results supported the notion that OPG prevented TRAIL-induced cell death, and that the cancer cells could produce it themselves. This produced several papers and therefore, OPG production was the major concern for resisting TRAIL-based therapies.
Cancer cell lines grown in the lab are a poor proxy for cancer, but in many cases they are the best we have access to. The really important question was: What was happening in tumours within the patient? So we immunostained a series of breast and prostate cancers using a widely used and well published antibody to OPG and saw abundant staining in these cells, suggesting that they might produce abundant OPG in vivo, and therefore rendering them resistant to TRAIL. So far so good. Many other research groups also found high OPG staining in tumours and other tissues, and crucially, we all used the same antibody.
The antibody in question did indeed bind OPG. It was used as an antibody pair in an ELISA assay, whereby one OPG-binding antibody is attached to a plastic well of a microplate, which captures the protein (OPG), and the bound protein is then detected with a different antibody to OPG which is labelled, so signal is proportional to OPG in the sample. One of these antibodies was also widely used for immunostaining in tissues, so applying the antibody to a thin section of tissue on a microscope slide, so that it binds where OPG is present in the tissue section. This detection antibody can then be easily detected via a second antibody which is labelled to allow detection and visualisation.
We had been conducting some Western blots with the same antibody, so separating out the proteins based on mass, then detecting with the same antibody, rather than immunostaining of proteins in situ in cells or tissue, and found abnormal sized OPG in some breast and prostate cancer cell lines, and the notion was that this could be a novel variant and OPG is known to be alternately spliced, whereby the OPG gene could code from alternate exons (protein-coding) and introns (non-protein coding). We knew that the OPG mRNA sometimes included an intron that created a premature ‘stop’ codon, potentially accounting for the smaller observed size of OPG on the Western Blots. Many other groups had mentioned these mRNA and protein variants in conference presentations but hadn’t yet published their observations. We were awarded funding to complete this project to investigate this potential novel biomarker. That’s where it all went wrong (or right).
Our plan was to selectively knock-down the mRNA to the splice variant mRNA using RNA interference experiments (RNAi) experiments, whilst leaving the full-length coding mRNA intact, to test the hypothesis that the splice variant produce the abnormal sized protein variant. We showed that RNAi to selectively knock-down the OPG splice variants of OPG mRNA failed to affect this abnormal sized product. The hypothesis that OPG had a novel protein variant derived from a splice variant was dead. We then looked at the abnormal sized protein in more detail, to find out what it was. We performed 2-Dimensional gel electrophoresis, so separating proteins by charge, then by size, to give a better separation such that each protein was isolated into a spot on the gel. Then by transferring the proteins to a membrane, we could perform a 2D-Western blot using the anti-OPG antibody to localise the abnormal-sized protein, without other proteins of the same size contaminating the same spot. We then isolated protein from the region when the antibody bound. The protein was sent for peptide sequencing by Mass Spectrometry. That protein turned out to be a different protein, called carbonic Anhydrase-2. Carbonic anhydrase-2 (CA-II) is an existing tumour marker, known to be expressed in many cancers. We published the findings here in International Journal of Cancer. The notion that breast and prostate cancers expressed abundant OPG was questionable, although our widely used ELISA to detect OPG was valid.
We showed the staining pattern observed with the anti-OPG antibody matches CA-II staining in many tumours, but not all, and that some were indeed expressing OPG. The paper cited all the studies that had used the antibody and whose data might be affected, including one of our own studies.
Most groups working on this wouldn’t have known about the antibody problem, and wouldn’t have thought to even look for a problem, and some still might not know. We could have gone on for years collecting “evidence” that supported our hypothesis, probably publishing more papers whose conclusions might not stand up to retesting by alternate methods. Instead, we tested our hypothesis to destruction, on the basis that if we did all we could to disprove our hypothesis, and failed to do this, then maybe, just maybe, the hypothesis is correct. Our hypothesis that the abnormal-sized product was derived from splice variant of the OPG gene had been disproved. In doing so, we inadvertently highlighted that the immunostaining in a number of papers looked more like the CA-II pattern.
And what happened? Well, the antibody supplier initially wasn’t impressed, but the antibody data sheet now cites our paper and makes it clear for future researchers about the issue. The academic community surely would cite this paper as a beacon of good practice, letting everyone know that those early OPG-staining papers might be unreliable. Erm… no. The paper has been cited twice. TWICE! In a journal with an impact factor of >5, you would expect 5 citations per year. However the papers with the questionable OPG staining are still getting cited, and some of the earliest papers noting differential staining (and citing alternate explanations for this, see Fig. 1 here, and see Fig.1 here) between two different anti-OPG antibodies, including the problem antibody in our study, are still being cited (169 times to date for one of them). What is clear is that even if we identify a Black Swan, most people don’t want to accept that it exists. Instead, they insist it is a large, dark Goose.