chemotherapy how is success measuredIn this article, we discuss how chemotherapy is measured and how it often gives a false sense of success.  We will see that the seeds of this bias were sowed back in the early days of chemotherapy when doctor such as Sidney Farber tried to put a positive spin on what was otherwise questionable success.  This history and method to measure “success” are seldom discussed.“Response Rates”

When discussing the success of chemotherapy in helping cancer patients, oncologists typically discuss the “response rate.” The response rate is a measure of how much a tumor or tumor metastasis decreased in size or how much a tumor marker declined. It is easy to assume that a tumor response is equivalent to an increase in survival, but, unfortunately, it is not. In fact, tumor responses with chemotherapy for solid cancers often have no relationship whatsoever to an increase in survival. A tumor may temporarily shrink only to explode in growth a short time later. This is especially true for advanced and metastatic solid tumors.

In this example from the medical literature, five children with medulloblastoma (a type of brain tumor) “responded,” but as of 1979, three had already died.

“Five children with recurrent medulloblastoma were treated with Vincristine, BCNU, Methotrexate and Dexamethasone. All five patients responded to therapy. Two of the patients are alive …”
– Duffner PK, Cohen ME, Thomas PR, Sinks LF, Freeman AI. Combination chemotherapy in recurrent medulloblastoma. Cancer 1979 Jan; 43(1): 41-5.

“Evaluable Patients”

In fact, response rates, as inaccurate as they are in telling us anything about survival in solid cancers, can be inflated by excluding some patients who died. This is known as counting only “evaluable” patients. Patients not considered “evaluable” are often those who did not get the “benefit” of the entire treatment plan. This is an example from the medical literature.

“Twenty-two consecutive patients with recurrent malignant brain tumors after radiation therapy and systemic combination chemotherapy with BCNU and vincristine, four of whom were not evaluable due to early death, were treated with etoposide. Response was observed in three of 18 (17%) evaluable patients …”
– Tirelli U, D’Incalci M, Canetta R, Tumolo S, Franchin G, Veronesi A, Galligioni E, Trovo MG, Rossi C, Grigoletto E. Etoposide (VP-16-213) in malignant brain tumors: a phase II study. J Clin Oncol 1984 May; 2(5): 432-7.

In the example above, a response rate was calculated after removing certain patients who died from the calculation. This obviously inflates the response rate.

Observational Bias

In addition to being a poor metric in respect to the most important measure – survival, and being subject to statistical manipulation by simply excluding patients, response rates are also subject to observational bias. For example, the following example comes from an article published by Memorial Sloan-Kettering Cancer Institute. An “institutional review” documents a 33% response rate. However, when the same patients are seen by the “central review” (doctors less vested in the success of the protocol) the response rate drops to 18%. We now understand that a “response rate” may lie in the eyes of the beholder.

“One hundred and thirty children less than 21 years of age with newly-diagnosed high-grade astrocytoma were treated with ‘eight-drugs-in-one-day’ chemotherapy … Of 79 patients with evaluable post-operative residual tumor on CT or MRI scans 26 (33%) were determined on institutional evaluation to have had an objective response. However, central review of scans documented responses on only 14 (18%) … ”
– Finlay JL, Geyer JR, Turski PA, Yates AJ, Boyett JM, Allen JC, Packer RJ. Pre-irradiation chemotherapy in children with high-grade astrocytoma: tumor response to two cycles of the “8-drugs-in-1-day” regimen. A Childrens Cancer Group study, CCG-945. J Neurooncol 1994; 21(3):255-65.

Why Use This Metric?

If response rates can have little to do with survival and are subject to statistical manipulation and observational bias why are they used? One answer is because they are useful for research and publication purposes. Oncologists often want to publish papers for professional reasons. They need to report on the outcomes of their latest experiment, but if they had to wait for survival data it could take months or years until all the data was aggregated. In contrast, data on response rates can be collected quickly. Another answer is that their use was originally created in reporting the results of leukemia. In this blood cancer responses can often equate with survival. Sometimes, the more responses, the more remissions, the greater the survival. But there is a third answer why this measurement system is so widely used in solid cancers and continues in use more than 60 years after its inception. It is possible that “response rates” or “improvement rates” give oncologists the opportunity to take a more optimistic look at therapies that have limited success. Here, for example, is one of the first published uses of this metric.

A History Lesson

On March 11, 1951, Sidney Farber, MD, professor of pathology at Harvard Medical School at the Children’s Medical Center of Boston, and Scientific Director of the Children’s Cancer Research Foundation organized a conference on folic acid antagonists in the treatment of leukemia. Its proceedings were published in the medical journal Blood: The Journal of Hematology in January, 1952 (Proceedings of the Second Conference on Folic Acid Antagonists in the Treatment of Leukemia). Today, oncologists call this type of chemotherapy an antimetabolite. Antimetabolites can be thought of as wolves in sheep’s clothing. They are man-made molecules that are designed to resemble a substance that our cells need such as a vitamin or an amino acid. Once the antimetabolite enters the cell it creates damage because the cell cannot function with the counterfeit substance. The cell dies. This chemotherapy, like all traditional chemotherapy, is indiscriminate. It kills healthy cells along with cancer cells. Examples of antimetabolites currently in use include: methotrexate, fluorouracil or 5-FU, cytarabine, mercaptopurine or 6-MP, and thioguanine or 6-TG.

Farber’s antimetabolites were “folic acid antagonists” which meant that it was something that looked like folic acid to a cell. Folic acid is also known as vitamin B-9. Today, folic acid is suggested to be a key player in the prevention of cancer. Farber tested the antimetabolites in 238 children with various types of leukemia. The results? According to Farber, “the total improvement rate in this group was 54.6%.” He called this an “important improvement in unselected, consecutive children with acute leukemia.” He defined “improvement” as including complete remission, partial remission, simple clinical improvement and “so on.” The tables he published and the comments of his colleagues, however, are quite enlightening.

54.6% Improvement but Only 19% Living

Table 5 from page 110 of the Proceedings shows 54.6% improvement rate. But this is generated from combining 98 dead patients and only 32 living patients. (See the row titled “Total Improvement.”) Also, note that out of 238 patients, a total of 200 have passed away and only 38 are alive. But an improvement rate of 54.6% sounds significantly better than a survival rate of 19%. The next table is even more revealing.

100% improvement but All are Dead

On Table 8, Farber reports that he has 100% improvement with the drug Ninopterin. The only problem is that all those children are in the “dead” column. With the drug Dichloro-aminopterin, Farber depicts a 75% improvement but none of these children are living either. With Denopterin the improvement rate is 50% but all of these children are also dead. Apparently, “improvement” can have little to do with survival.

The thought process of Farber and his colleagues are suggested in some of the comments recorded in the Proceedings. For example, Farber is quoted as saying, “One of the first important questions we would like to ask and have answered today, if possible, is this: Why are these patients, as many as 45% or 50%, who do not respond to treatment with folic acid antagonists?” This is quite revealing in that Farber wonders out loud why half the patients don’t respond as opposed to perhaps a more defining question – what is the relationship between responses and improvement and survival and why have the overwhelming majority of children who responded have died?

This question becomes even more pointed when one considers that Farber also helped introduce the concept of the “evaluable patient” mentioned above. Farber states, “If we treat all patients for three weeks, I think that we can fairly evaluate the efficacy of the compound, which takes that long, on the average, before it can be regarded as effective. Therefore, if we disregard all of those patients who died in the first day or two or three after admission to the hospital, or after the onset of therapy, and include only those treated twenty-one days or more, we find that we have 190 children, with acute leukemia, treated with folic acid antagonists since June 1, 1947.” (Proceedings pages 109-110).

Fast Math

Table 6 of page 110 of the Proceedings reports 155 patients as “dead” and 35 patients as “living.” Also according to this table, 36.9% of the patients who were “dead” did not respond or improve from treatment, while 31.6% of patients who were “living” did not respond or improve from treatment. Therefore, of the living patients approximately 5% more patients responded to or improved from treatment. This again begs the question – what is the relationship between responses and improvement and survival and why have so many of those children that have responded also died?

The Proper Scientific Attitude

While Farber does not discuss this question or toxicity or quality of life of his patients as reported in these Proceedings, some of these issues were apparently brought forward by others in attendance.

One physician wondered why autopsies of the treated children revealed liver damage. Dr. E. Clarence Rice, Director of the Children’s Hospital of Washington D.C. stated, “I would be interested in hearing others say something about the findings at postmortem examination. When we first started using folic acid antagonist therapy, we saw four children who had rather marked scarring of the liver, similar to that of cirrhosis… We would like to know how you interpret this. Is this an effect of the drug, or how can one account for this?” (Page 114 of the Proceedings.)

Perhaps even more revealing is when one of Farber’s colleagues had a family member diagnosed with leukemia. The researcher, Dr. William Dameshek, Professor of Clinical Medicine at Tufts College Medical School did not suggest that his brother-in-law undergo chemotherapy. This is what Dameshek stated at the conference:

“Still, I must confess that I continue to be pessimistic about folic acid antagonist therapy despite what has been said thus far. This was brought forcibly to mind recently when a brother-in-law of mine developed acute leukemia and we were faced with a situation as to whether or not to give him folic acid antagonist, ACTH, or cortisone. He was so sick and so obviously near to death that we decided finally to leave him alone and give him simply antibiotics and not too many transfusions, and he went on his way and died, perhaps a little more comfortably than if he had been given folic acid antagonist therapy. As we go along in our therapeutic efforts, we come to the point in some cases where we hate to inflict the so-called toxic reactions of folic acid antagonists on some of our friends, neighbors and relatives in whom this unfortunate condition may develop. I realize that this is by no means the proper scientific attitude … ”

It is striking that the doctor ended his remarks that “this is by no means the proper scientific attitude … ” Concerned with his brother-in-law’s quality of life he did not feel the administration of toxic chemotherapy was appropriate. Are science and the humane care of patients at odds? If so, where should the accommodation be made?

It deserves emphasis that chemotherapy does significantly prolong survival for some patients with blood and lymph cancers. It also deserves emphasis that Sidney Farber, MD made many important and lasting contributions to the understanding of clinical treatments of cancers. There’s even a hospital called Dana-Farber named after him. But, it also important to understand that there is a long legacy of measuring “responses” and “improvements” and that these metrics, especially in advanced and metastatic solid cancers often have nothing to do with survival or quality of life. Once again, it is incumbent on the patient and the patient’s professional caregivers to obtain the information needed to make informed treatment decisions.

Share this post on: