[Ip-health] Size of trials by status (S or P) - Some 2004 FDA
and Parexcel data compared
Joseph DiMasi
joseph.dimasi@tufts.edu
Tue Aug 1 12:43:02 2006
Jamie,
At the risk of appearing to beat a dead horse, let me offer a few
responses (some minor, some more significant) to your posting. I would
have preferred that you had my previous response copied in your message,
so I have copied it below (along with your previous posting). I will
take some of your comments in the order that they were made.
> What Michael Palmedo reported were the number of patients in clinical
> trials referred to by the FDA in a new drug approval, in the
> "clinical trials" section of the approval letter.
I realize that the numbers he used were taken from the "Clinical
Studies" sections of the labels. This is what I had written below in my
prior posting. Let me correct you on terminology, since you have
repeatedly referred to approval letters in the long string of e-mails.
The information of concern comes from product labels, not approval
letters. Approval letters are literally letters. They are typically
one to two pages and basically just inform a sponsor that a product
application has been approved.
> These are not "CPTech numbers." They come from the FDA. The numbers
> are not "estimates," but rather the numbers reported by the FDA as
> the evidence the drug is safe and effective (assuming Michael reads
> the FDA letters correctly).
Who said anything about CPTech generating their own numbers? It is
abundantly clear from my earlier messages that we have been discussing
what is in FDA-approved product labels. The numbers pulled out of the
Clinical Studies section are also not the only evidence that the FDA
assesses to determine that the "drug is safe and effective." The FDA
evaluates everything it receives. In particular, its safety evaluation
is based on a much larger record than is obtained from the handful or
less of efficacy trials that it chooses to describe in the Clinical
Studies section. In addition, you cannot simply rely on an FDA
imprimatur here. I could count the number of pages in different drug
labels. These would be "official FDA numbers," but they would be
meaningless. You have to show that what you use and how you use it is
meaningful. Also, picking out n's from the Clinical Studies section and
representing or implying that they mean something in particular IS
something that you own, not the FDA, since the FDA has not presented and
interpreted these numbers (nor would they, I believe, accept your
interpretations).
> Relative size of trials: As you know (but did not really address
> below), Michael was asked to look at the "relative" size of trials
> between standard and priority drug approvals. We are interested in
> this issue for the following reason. We want to know if company R&D
> investments are directed at priority or non-priority medicines.
> Relatively smaller trials would suggest (all other things being
> equal) investments in clinical trials for priority products may be
> smaller than investments in trials for "standard" drugs.
I have already addressed issues related to the relative numbers of
subjects in priority and standard approvals in a number of e-mails in
this string. You have now added the qualifier "all other things being
equal." This makes the statement true if total trial sizes are as you
say, but the problem is that all other things are not, in general,
equal. Given that nearly all orphans get priority ratings and many
drugs with fast-track status get priority ratings, you might well expect
that pre-approval trial sizes are lower (at least for the indications at
issue). The fast-track drugs also come with post-approval commitments,
which will add to trial sizes (when and if they are done). However,
this says nothing about what was actually done in these trials,
infrastructure costs (which will move the numbers toward equalization),
work on other indications, and discovery costs and risks in novel
areas. Total costs are also going to depend on the therapeutic class
distributions and it is all confounded by the fact that the bulk of
development in classes is done more or less contemporaneously and often
it is the first drug to win this race that gets a priority rating, while
the losers tend to be given standard ratings at approval. You cannot
infer relative total costs from clinical trial sizes alone in the
aggregate, let alone for any particular drug.
> --------------Michael Palmedo quote----------
> "There are two parts of the FDA labels that typically include patients
> in clinical trials - the Clinical Trials section and the Adverse
> Reactions section. At one point I tried to incorporate the adverse
> reactions section, but found that this approach was problematic as
> well. For starters, it does not include control
> groups, so it too gives incomplete information. It often gives a
> higher number than the Clinical Trials section, but not always, and
> it sometimes gives a lower number. Sometime it gives estimates instead
> of numbers "Approximately 2,000" or "more than 1500." So if
> you take the highest number off the label for one drug, and then
> compare it to the highest number on other labels for other drugs, you
> are comparing:
> - full numbers of patients from a specific subset clinical trials
> - the number of patients in an unknown number of trials (maybe all
> the trials, maybe not) minus the control groups
> - vague estimates"
> --------------end quote-------------
Mike Palmedo is wrong that the Adverse Reactions sections never include
information on control groups. The example that I gave below for
Enablex proves that. Information was given on the numbers of subjects
in controlled and uncontrolled phase II and III trials. The other
example, Lyrica, shows that this is not always the case. This and what
Mike wrote just makes my point that you cannot consistently get total
clinical trial sizes from the FDA labels.
> Michael said he wanted to "keep the comparisons apple-to-apple" using
> what seemed like the most important data, the trials cited as the main
> basis for the FDA approval.
For there to be an appropriate and useful apples-to-apples comparison,
the n's in the Clinical Studies section have to consistently measure the
same thing and what they measure must be meaningful. The only thing
that is consistent here is that the n's that you pick out were all from
the Clinical Studies section. There is no requirement that the FDA
report n's for all "relevant" or "important" clinical trials in this
section. The intent seems to be to discuss trials that illustrate what
the efficacy effects are for the drug. You certainly cannot interpret
the handful or less of phase III trials discussed in the Clinical
Studies section as all that was "important" in the drug approval
process. Are phase I trials not important (and while individual studies
may be small, in aggregate they can be substantial [see the Enablex
example below])? Are phase II trials unimportant? Are phase I-III
trials for other indications not important? Simply counting the trial
sizes for those trials mentioned in the Clinical Studies section is a
deeply flawed measure of importance in either absolute or relative terms
(because of inconsistency in what is measured and variability in what is
left out that is also "important").
> You are correct, if Michael presents this data, he (we) should always
> mention that these data under-report the total number of patients,
> and be clear that the data is only that reported in the Clinical
> Trials section of the FDA approval. With this explanation and
> context, it is interesting to compare the size (apple to apple) of
> trials for priority and non-priority products.
I am glad that in the future you would always mention that these numbers
understate total clinical trial sizes and that they come from one
section of the labels. However, I would also hope that you refrain from
using these numbers in inappropriate apples-to-oranges comparisons with
other numbers. The original posting from Mike quite clearly made a
comparison between these Clinical Studies section n's and an estimate of
average total clinical trial sizes from our cost study.
Joe DiMasi
> Jamie,
>
> Now that I am back from being away for a while and have seen your new
> posting, I am going to do just one more posting on clinical trial
> sizes. A number of years ago I pointed out to you that the FDA approval
> labels cannot and should not be used as evidence of total clinical trial
> sizes. In your report from 2003 that I cited in one of my postings, you
> used these kinds of numbers for earlier years, although you did
> acknowledge in some manner that the numbers obtained from the FDA
> documents may not tell you everything about clinical trial sizes. The
> reason that I posted in the first place was that the original posting
> from Mike Palmedo on 2004 approvals tried to make an unfavorable
> comparison to a number in our R&D cost study using the same approach
> that you had used for the 2003 piece, but without even acknowledging
> this time that there might be something wrong with your numbers (not
> hat you should be using the numbers anyway).
>
> There is no question in my mind that what you did underestimates total
> clinical trial sizes, often by substantial margins. As you note, the
> 2004 PAREXEL numbers that I gave you are all larger than the numbers
> that Mike Palmedo reported in his posting for the same drugs. You
> wondered why. As far as I know, PAREXEL uses FDA documents as a
> starting point, but they realize that they are inadequate and they make
> some attempts to get additional information. Regardless of what PAREXEL
> does, why your numbers are much too low is no mystery to me. It is
> clear from examining a handful of cases some years ago and again for
> some of the 2004 approvals what you are doing. The FDA does not report
> on the sizes of all clinical trials on a drug in its approval labeling
> (nor has it been their intent to do so). What you have done is to
> simply go to the "Clinical Studies" sections of the labels and count up
> the trial sizes that the FDA might mention there. However, what is
> discussed in that section is just some of the efficacy trials that the
> reviewers considered pivotal for the approved indication. They do not
> discuss every clinical trial for even the approved indication, let alone
> other indications that were examined. They generally discuss a few
> phase III trials (not necessarily all of them). They might occasionally
> mention a phase II trial, and probably never discuss phase I trials. For
> this reason, what you get out of this section is clearly not everything
> that there is. You can even sometimes see this from information in
> other parts of the label. Let's take as examples, the two drugs that
> you mention below where there are large differences between what you and
> PAREXEL have (Lyrica and Enablex).
>
> You had 1,508 as the total number of subjects in trials for Lyrica,
> while PAREXEL had 9,100. 1,508 is the number that you get out of the
> Clinical Studies section. There are six efficacy trials mentioned
> there. However, if you go to another part of the label you will see a
> statement that in all controlled and uncontrolled premarketing trials
> "MORE (my emphasis) than 9000 patients have received pregabalin." There
> are two things to note about this statement. One is that it by
> definition gives you an underestimate of the number of subjects that
> received the drug. The other thing to note is that it only mentions the
> active ingredient. That is, it does not tell you how many subjects
> received a placebo or an active comparator (if there was one). Perhaps
> this is what PAREXEL used and 9,100 was just a typo, or perhaps they had
> some additional information. However, you can use other information in
> the label to go beyond even 9,100 (but still have an underestimate).
> The number of subjects in the placebo arms of five of the six efficacy
> trials mentioned in the Clinical Trials section are noted there. These
> total 425. So now, we are up to 9,425. This is six and a quarter times
> larger than the number that cptech reported. And still, we know that it
> must be an underestimate. We don't know the number of subjects that
> received a placebo in the sixth study mentioned in the Clinical Studies
> section, there probably were subjects in other trials that received
> placebo (or active comparator), and 9000 was acknowledged to be an
> underestimate of the number of subjects that received the drug that was
> approved.
>
> Finally, let's look at the case of Enablex. Cptech had the total number
> of subjects at 1,454, while PAREXEL had it at 8,830. Again, 1,454 is
> what you get out of the Clinical Studies section for four efficacy
> trials. However, elsewhere in the label you see the statement that the
> safety of Enablex was evaluated in "Phase II and III controlled clinical
> trials in a total of 8,830 patients." Of course, even 8,830 is an
> underestimate. It does not tell you anything about phase I uncontrolled
> trials. Once again, we can do better by examining other information in
> the label. We are told that 6,001 of the 8,830 patients received
> Enablex. We are also told that a total of 7,363 subjects were treated
> with Enablex from 3.75 mg to 75 mg once daily. So, at least 1,362
> subjects received the active drug outside of controlled phase II and III
> trials. So now we are up to at least 10,192 subjects, or seven times
> what cptech reported as coming from FDA documents.
>
> What all of this tells me is that approach that cptech uses
> systematically underrepresents the number of subjects in clinical
> trials, the numbers that cptech uses are, on average, substantially
> below those reported by PAREXEL, and that even the PAREXEL numbers at
> least sometimes underestimate clinical trial sizes. You should not
> continue to report the kinds of numbers on clinical trial sizes that you
> have reported.
>
> Joe DiMasi
>
> James Love wrote:
>
> The tables and the links work best from the Blog version. Jamie
>
> http://www.cptech.org/blogs/drugdevelopment/2006/07/size-of-trials-by-
> status-s-or-p-some.html
>
> Size of trials by status (S or P) - Some 2004 FDA and Parexcel data
> compared
> James Packard Love, Friday, July 14, 2006
>
> This note is a follow-up to discussions stimualted by Michael
> Palmedo's note on 2004 FDA NME drug approvals. In particular, it
> follows discussions on ip-health by Joe DiMasi and myself on a fairly
> narrow question -- are clinical trials trials larger for Standard (S)
> FDA NME drug approvals than for Priority (S) approvals?
>
> The following table reports the size of clinical trials for 5
> priority and 8 standard FDA NME drug approvals. The products are the
> union of those reported by Michael Palmedo for 2004 FDA approvals,
> and data from Parexel. These are the data that Joe DiMasi referred to
> in his June 14 post to ip-health.
>
> Drug Rating Size, FDA letter Size, Parexcel
> Clolar P 66 138
> Lyrica P 1,508 9,100
> Prialt P 1,434 1,634
> Sensipar P 1,146 2,000
> Tarceva P 1,837 6,000
> Apidra S 2,467 4,093
> Cymbalta S 1,850 6,100
> Enablex S 1,454 8,830
> Fosrenol S 2,357 2,697
> Ketek S 2,016 5,900
> Lunesta S 2,100 2,909
> Spiriva S 2,663 3,168
> VESIcare S 3,027 3,327
>
> Below are the mean and median , for the FDA and Parexcel data,
> reported by (P) and (S) drugs.
>
> Mean-FDA Median-FDA Mean-Parexecl Median-Parexcel
>
> Standard Approvals 2,242 2,229 4,628 3,710
> Priority Approvals 1,198 1,481 3,774 2,000
> Difference 1,044 854 795 1,710
> % larger 87% 55% 23% 86%
>
>
> Two quick points. First, Parexcel reports more patients for every
> trial. Second, the number of data points is pretty small (5 P and 8 S
> drugs), so one has be careful about drawing conclusions.
>
> Joe notes that when you look at means from the Parexel data, the
> trials for Standard approvals (S) are only 23 percent larger than for
> the Priority products. Joe notes that by comparison, when looking at
> the FDA data, the mean size of the trials for Standard approvals were
> 87 percent higher, suggesting a possible bias when looking at FDA data.
>
> However, I would add, that when looking at the MEDIANS of the
> Parexcel data (for the 13 products), the differences between the size
> of standard and priority drug trials are quite pronounced. For the
> Parexcel data, the median size of trials for the Standard drugs is 86
> percent larger than the size of the median trial for the priority
> drugs -- actually higher than the 55 percent difference (in medians)
> that Michael reported, looking at FDA data for the same drugs.
>
> Ultimately, this is too small a sample to say that much. We'll take a
> look at a larger sample, and report that. But before doing so, it is
> also interesting to look at the differences between the FDA data and
> the Parexecl data. Paraexcel always reported more patients in the
> trials than did Palemedo, looking at the FDA approval letters. In
> some cases, much more. Pfizer's Lyrica, for example, was reported by
> Palmedo as 1,508, and Parexcel as 9,100. Enablex, reported by Palmedo
> as 1,454, is reported by Parexecl is 8,830. In looking further at
> this issue, we will also look closer on these differences. One person
> suggested the initial FDA approvals may not report parallel trials in
> the works for other indications (Lyrica is now approved for 3
> indications, for example). Another comment is that some of the
> "trials" reported by Parexel may be of lesser scientific importance
> (possibly having value for marketing purposes), or may be un-reported
> by the FDA other reasons. People may speculate or offer some evidence
> on these points in the comments to this note.
>
> This issue has generated some debate with Joe DiMasi, because we have
> questioned his repeated finding (in 1991 and 2001/2003) that priority
> products are more costly than standard drugs, at least in terms of
> the important area of clinical trials. Our reviews of the data, on a
> couple of different occasions, have suggested that priority drugs
> consistently have smaller clinical trials than do standard approvals
> (findings borne out here again). If priority drugs have smaller
> trials and quicker approvals, they would seem to be less expensive,
> all other things being equal. Joe's comments have been informative
> and constructive, and we will revisit the issue, incorporating both a
> broader analysis of the Parexecl data, and a closer look at the
> differences between the FDA and Parexel data, as well as other
> evidence on this topic.
>
> Finally, we remind people that neither the Parexel nor the earlier
> (2001, 2001/2003) DiMasi et all data claim to present data for all
> drug approvals. Most importantly, DiMasi has said that "It should
> also be noted that our study was based on the R&D experiences of
> major traditional pharmaceutical firms," in contrast to "small
> biotech and niche pharmaceutical firms." This is not a criticism of
> the DiMasi studies, as any analysis is going to be limited in some
> way. It is rather a reminder that some of the estimates provided by
> DiMasi are based upon particular samples that may not be
> representative of other drug development efforts. Indeed, DiMasi's
> 2001/2003 paper, which is now so widely quoted, drew important
> conclusions about relative investments in priority and non-priority
> drugs from just 10 priority products and 14 standard products (DiMasi
> 2003 page 172). His estimates of out-of-pocket outlays were also more
> than twice as high as the previous PERI study involving 117 drug
> development projects (Project Management in Pharmaceutical Industry:
> A survey of Perceived Success Factors 1995-1996, PERI), raising some
> questions about the nature of the sample he studied. To deepen the
> understanding of these issues, people have to look at more data, and
> do some modeling of their own.
>
>
--
-----------------------------------------------
Joseph A. DiMasi, Ph.D.
Director of Economic Analysis
Tufts Center for the Study of Drug Development
Tufts University
192 South Street, Suite 550
Boston, MA 02111
tel: 617-636-2116; fax: 617-636-2425
URL: http://csdd.tufts.edu
-----------------------------------------------