Ryan's Blog on NMR Software

Bosutinib: A Case of What Could Have Been?

For many years I have been looking for a high profile public case that represents a good example of why an automated NMR structure verification system can help prevent bad things from happening. We all know these types of examples exist, but they don't generally become public knowledge. 

Yes, I've blogged about this topic many times over the  years. To name a few:

Back to Reality #1- LC/MS vs. NMR

The Realities of NMR in the New(ish) Discovery Environment

Are We There Yet? Automated Structure Verification (ASV)

The Black Swan of Compound Libraries

Recently, an article in C&E News was published. It can be read here:

Bosutinib Buyer Beware

In short, there appears to be a lot of bottles of bosutinib out there that actually contains the wrong isomer. 

Bosutunib is a tyrosine kinase inhibitor that Pfizer is hoping will be one of their next big drugs. Also known as SKI-606, it is currently in Phase III trials to treat chronic myeloid leukemia. 

Naturally, for such a high profile compound there are a fair amount of research groups who have been conducting their own studies on this compound. Problem is, there is a lot of the incorrect stuff floating around out there. The latest update on the PKC Pharmaceuticals, Inc. website have reported that up to 18 vendors have actively sold to one or more customers in the biochemical reagent marketplace, an incorrect bosutinib isomer.

Derek Lowe at In the Pipeline has also sounded off on the issue.

Naturally, the first thing that came to my mind when reading this story was, could an automated NMR software system have caught this mistake. From the C&E News article:

Levinson and Boxer put their publication on hold, ordered bosutinib from a different vendor, and did a battery of tests to determine which material was the genuine bosutinib. They soon figured out the original compound they had done all their research on turned out not to be bosutinib. “We had wasted a huge amount of time and money on the wrong isomer,” Boxer says.On the basis of multidimensional nuclear magnetic resonance (NMR) experiments, Boxer and Levinson believe that this isomer not only has a chlorine at the 3-position rather than the 2-position, but also that the chloro and methoxy groups that appear in the 4- and 5-positions, respectively, in bosutinib’s aniline moiety have been switched.The difference between the two molecules is subtle. Mass spectrometry and elemental analysis would be the same for both compounds. There are differences in the aromatic region of the ¹H NMR, but one wouldn’t necessarily pick up on them unless they compared the two compounds side by side. It’s only when you use ¹³C NMR that the symmetrical nature of the aniline group in the isomer becomes clear.

 

Here are the two molecules in question:

Much of the discussion has been that the average chemist would let the incorrect isomer through with just the LC-MS information and quick glance at the ¹H. Of course the ¹³C NMR would clearly point out the difference but this isn't standard practice for many bench chemists. 

But would one have even needed to acquire a ¹³C NMR spectrum for this?

Would a ¹H NMR have been enough? At the very worst, a HSQC experiment? The supporting documentation for this article suggests clearly yes to the latter. 

Phil Keyes from Lexicon Pharmaceuticals and I are working on this right now and should have some results showing how our automated NMR verification system would have handled this case. 

More details to follow...stay tuned!

 

 

Back to Reality #1- LC/MS vs. NMR

Reality #1:

LC/MS has become the primary analytical check for the medicinal chemist. I Love NMR, but it's the reality

So how did we get here? Clearly there a lot of reasons, but one of my hypotheses is based on the evolution of open access and the walk-up environment. No doubt that moving away from the traditional analytical service departments has been a great thing from a productivity standpoint.

LC-MS lends itself really well to the open access environment. There are good systems out there that can auto-process data well and provide a nice report that gives the chemist a simple answer. In most cases, the chemists can get the answer they are looking for with a quick glance at the report. In addition, for this kind of routine analysis, the interpretation of the results is very easy. 

Contrast that to NMR. How often are the printouts from the instrument good enough? Not nearly as often. More and more it is becoming common practice for the chemist to import their raw or processed data into the processing software on the instrument or in some offline processing software. Within these applications, they can get a better look at their data, zoom in on areas of interest, re-integrate the spectrum so it makes sense, perhaps even generate a multiplet report for a potential patent down the road.

Add up those tasks alone, and the chemists may be forced to spend 5-10 more minutes on a routine NMR than an LC-MS when you factor in interpretation as well! Unfortunately, the NMR instrument is not printing out an answer, it is printing out a spectrum that needs to be interpreted by the chemists which on a routine basis is a lot more difficult and complex task than checking an LC-MS for reaction completion. Furthermore, let's keep in mind that intrepreting analytical data is not the medicinal chemists job. They are evaluated, compensated, etc. for making molecules. The more molecules the better so the inherent risk here is obvious and I don't think it's fair to simply "blame the chemists"

Given the above, is there any wonder why LC-MS has become the primary analytical check?

The Realities of NMR in the New(ish) Discovery Environment

Reality #1:

LC/MS has become the primary analytical check for the medicinal chemist. I Love NMR, but it's the reality

Reality #2: 

More often than not, NMR spectra are merely glanced at:

• Sometimes interpretation is based on the presence or absence of one peak (did my reaction go to completion?)
• How often are NMR spectra fully assigned by a medicinal/synthetic chemist? Not very often, it's not their job. 
• Is all the information a ¹H NMR spectrum can provide being effectively used? Some chemists are extremely good at NMR, other couldn't be bothered with the intimate details, and I can't say I blame them. It's largely a training/education issue
• How often are 2-dimensional experiments acquired and interpreted? Of course this varies chemist by chemists, organization..but despite strong advances in hardware, acquisition, etc, I believe the safe answer is, "not enough"

Reality #3:

In many organizations, upwards of 95% of compounds are being registered by synthetic/medicinal chemists without having undergone any analytical review by an analytical specialist (NMR spectroscopist). 

***

How and why did we get here? I have some hypotheses, and will write about them in the coming days. 

Are We There Yet? Automated Structure Verification (ASV)

When I created this blog several years ago, one of the first links I added to my blog was to Stan's NMR Blog. Over the years I have had the pleasure to meet Stan who is a wonderful individual. His blog is very interesting, Stan is incredibly knowledgeable, and I really enjoy his writing style. 

Recently, Stan posted about a topic, those readers who follow my blog know I am very passionate about; Automated Structure Verification (ASV). The initial entry was very thought-provoking and while I think the numbers Stan posts regarding the number of spectra chemists interpret below are too high, I think he's put together an interesting argument nonetheless about the need and application of ASV in the scientific community:

The typical application area for such a product is drug discovery, meaning Pharmas. I was told that these Companies employ masses of chemists who have to interpret some 50 spectra a day, having only 8h/50 = 9:36 minutes per spectrum. Clearly, they don't wag their tails, nor wish to pursue more balls. They were reduced to the echelon of routine workers and they hope that the new software might give them a few minutes more time to maybe ponder again that strange spectral peak that looked like an interesting challenge. Illusion, of course, because their top CEO's hope that the software will make half of the chemists redundant, leaving the rest doing a spectrum every 4:48 minutes (guess who will prevail).

Stan's entry definitely triggered some interesting comments from the community as well, in addition to a follow up letter from John Hollerton, GSK.

My take is that we don't need to be waiting 3-5 years for this technology to become more relevant...it already is! It is already being used in the industry. However, the secret is finding and recognizing the right application for it within the industry, and that application will vary considerably from lab to lab. 

It seems the main discussion about this completely surrounds whether or not ASV can replace the NMR interpretation of the medicinal or synthetic chemist. Of course this is the ultimate goal, but it should not be considered the only goal. John eludes to some other applications in his letter and there are other uses as well. 

I will blog more later about my thoughts on the "Reality of the New(ish) Discovery Environment" as it pertains to NMR and more thoroughly justify why I feel the way I do about this technology. But for me, the bottom line is that this is not just a concept, it's not a dream that we need to wait be realized in the next 3-5 years. It is also not a new technology on the horizon. Our first publication on this topic was back in 2006 and we've continued to build on those approaches in 2007 and 2008. Work was done by Griffiths et. al in the late 90s and early 2000s. The techonology has only gotten better and better. We have been working diligently with industry partners for over the last decade on this concept. It is not new and it is not an unsolved area. 

Just to be clear, am I saying this technology is ready to completely replace the NMR interpretation responsibilities of the average medicinal/synthetic chemist? Definitely not. 

Can it be used to help chemist make better, faster, and more independent decisions? Yes. Can it be used in complimentary way to the chemists existing interpretation workflow? Yes. Can it used to be validate the quality of registration libraries? Yes. Can it be used to verify compounds purchased from external sources, whether this are assay materials or simple starting materials? Yes.  The list of applications go on for a while. 

I believe current ASV applications can improve all the above workflows. The key is identifying the best applications and practicing caution in how this technology is deployed and used by chemists. 

In conclusion, my advice is that if we continue to treat this technology as only applicable to the ultimate goal, perhaps it will take longer than 3-5 years, or perhaps NMR will become irrelevant before we get there. The goal of NMR scientists, hardware, and software vendors should be to work hard to continue to make NMR technology relevant in all industries. If we can do this, we all win.

The Black Swan of Compound Libraries

Recently I finished a great book that made me think about different things on a lot of different levels. The book is called The Black Swan and the author is Nassim Nicholas Taleb. 

In short, a black swan is defined as an event, positive or negative, that is deemed improbable yet causes massive consequences. 

From wikipedia:

What we call here a Black Swan (and capitalize it) is an event with the following three attributes. First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.

Anyone who follows this blog knows that I have a passion around the concept of automated structure verification by software methods and the different applications it can play within the pharmaceutical industry. 

Reading through this book, one of the things that popped in my mind was the black swan impact as it pertains to compound registration libraries in pharma.

I spend many hours in my role discussing the concepts of automated structure verification (by NMR of course) to spectroscopists, chemists, directors, VPs, Managers of compound management, etc. 

In many of these conversations, the risk of having an incorrect compound in the registration library is always a fruitful and interesting discussion. However, this conversation is almost always dominated by low probabilities. The low probability that the compound being registered is wrong in the first place. I agree, this is a low probability. After all, a trained chemist has synthesized said material and has used a variety of analytical methods to confirm it's identity. In some organizations, the compounds in this library will be validated further either before they are accepted in the store, or before they are sent off to assay. 

So the argument goes that incorrect compounds in the registration library are low probability. And furthermore, there are steps being taken in some organizations to proactively catch incorrect compounds (most often LC-MS). Finally, perhaps the lowest probability event of them all is the compound being identified as a hit during the assay. 

So in the end, what are the chances that a meaningful compound that is identified as a "hit" is actually an incorrect compound? Furthermore, in this instance what's the chances that this compound will make it too far downstream without it's false identity being exposed?

In short, it's a pretty low probability event, and because in most cases it will eventually be caught it's a stretch to really call this a massive consequence or impact. 

That's not the black swan I am talking about in this post. 

In my opinion, the Black Swan concept is more relevant to those incorrect compounds that lie in the registration library and never get cherry picked out. A compound that doesn't produce anything interesting from the assay. A compound that effectively hides in the back of the shelves in a compound management cold room, that has been documented as "tested" but never advanced any further because of poor assay results.

Of course this compound is not the Black Swan. However, it's possible that the compound that it was supposed to be is. The compound that the chemist thought they made, and the compound that was referenced in the inventory.   

While it's low probability, there are probably tens of thousands of misrepresented compounds that haven't truly been assayed over the last 20 years. Of these thousands could any of them turned into the highly coveted, blockbuster drug?

Sure, it's doutbful, but the infinitely small possibility exists. 

And further, this very idea contains one of the core components of a Black Swan; "nothing in the past can convincingly point to it's possibility"

I am not certain that there is an example of a blockbuster drug that was originally missed because of a mistake in synthesis, or a misrepresented registrant. My guess is "not really" And if it did, perhaps as Taleb suggests, it would have been rationalized by hindsight. 

A Case for more NMR, and maybe even Automated Structure Verification?

I've spent a lot of time over the past couple of years visiting both scientists and directors in Pharma and having discussions around the concept of automated structure verification, whether or not there is enough characterization being done on screens, libraries, purchased compounds, etc. Whether the possibility exists to do more characterizations and whether companies have the personnel to do this.

Derek's post entitled, the Infinitely Active Impurity doesn't necessarily capture all the feedback I have gotten, but it certainly does bring up some pretty solid arguments, and some of the comments are very interesting. Here are some excerpts from Derek's post:

Yesterday's post touched on something that all experienced drug discovery people have been through: the compound that works - until a new batch is made. Then it doesn't work so well. What to do? You have a fork in the road here: one route is labeled "Blame the Assay" and the other one is "Blame the Compound". Neither can be ruled out at first, but the second alternative is easier to check out, thanks to modern analytical chemistry. A clean (or at least identical) LC/MS, a good NMR, even (gasp!) elemental analysis - all these can reassure you that the compound itself hasn't changed.

But sometimes it has. In my experience, the biggest mistake is to not fully characterize the original batch, particularly if it's a purchased compound, or if it comes from the dusty recesses of the archive. You really, really want to do an analytical check on these things. Labels can be mistaken, purity can be overestimated, compounds can decompose.

And I've seen plenty of things that have fallen apart on storage, and several commercial compounds that were clean as could be, but whose identity had no relation to what was on their labels (or their invoices for payment, dang it all). Always check, and always do that first.

So why aren't these original batches fully characterized? Cost? Acquisition times? A human required to prep the sample? To interpret the results?

And some comments from Derek's loyal readership who share interesting stories, experiences, and practices:

 

 

 

In yet another program, we were doing an SAR study, and one particular compound didn't really fit the SAR the way we had predicted. Initially we took it in stride, and had all sorts of hand-waving arguments of why that compound might not fit the SAR. With time it became more obvious that something was rotten in Denmark - and a close look at the NMR spectrum revealed that in fact it wasn't the compound depicted, but a regioisomer.

Its probably less likely to occur these days I think due to modern tools and better upfront characterizations as suggested but I have heard of similar stories throughout my career and was involved in a several programs where the activity was from either from a minor by-product or a misidentified structure. The by-product was more puzzling at first but did lead to interesting series of analogs (before failing tox screen unfortunately). The misidentified compounds came from a wrong (mislabeled) regioisomer by supplier (so heed Derek's advice to pre-check purchased material as in this case was only obvious after a 2nd bottle gave a slightly different spectra) and another was an unanticipated chemical rearrangement during the synthesis (again only obvious after 2nd round gave less rearrangement). Although frustrating at the time these events were some of the most interesting times in the lab.

The other example came from a compound being contaminated in the Genevac. It's always a problem with shared equipment that many people don't think about. My own rule now on this is to always have LCMS and NMR (so many vendors only do LCMS) and if you get a good hit make a batch 2 on a bigger scale so you can really look at what's in there.

Anyhow, I've seen all of the above mentioned scenarios played out at some time or another. My sense is that the cost of a compound (i.e. how pure?, how sure?) should parallel the cost of the biology being done on it. You don't need EA, NMR, LC/MS, IR on each of 250,000 compounds going into a high throughput screen. And you better be mighty sure what you're putting into a dog tox study or into the clinic. If you get a high throughput hit, you need to confirm (more $) before you move forward very far (more $). There's nothing wrong with occasionally screening garbage if you don't act stupid after it hits.

My experience is that, at most companies, discovery research needs to spruce up its sloppy attitude to characterisation. (I dread to think what happens in academia, where we used to wait weeks for MS because the people were too busy doing *research* to be *bothered* with mundane structure confirmations). The approach that works in most cases has already been hinted at; synthesise batch 2, for preference by a different chemist, on a gram scale, and fully analyse (tlc and hplc) and characterise - Yes! Use C-nmr to check that all the carbons are there (!), and that their 1-bond and 3-bond couplings are logical. This is also an excellent nmr learning exercise for new BSc chemists.

The overall theme of this post, as well as the comments are that more diligent characterization is needed, even in high volumes.

This outlines one of the issues in this industry that we are trying to address through automated structure verification. But the argument for the system is of course that organizations do not have the personnel to diligently check the identity of the structure against analytical data from all different sources. An automated QC of these results can help address a good chunk of these concerns.

Of course one of the major concerns that comes up, is whether or not we can completley eliminate false positives. The answer of course right now is no. And of course it will likely always be no. Sometimes there are just really challenging problems.

Of course, nothing beats the diligent manual analysis by an analytical expert, but we all know that there is the time/money dilemma which is inevitable, and it's the reason why many organizations are unable to practice the due diligence that Derek and some of his readers are advocating.

Yes, Chemists Will Still Have To Look at their NMR Data

I've been away for the better part of the last few months, haven't been able to spend anytime blogging, and I apologize for those of you who have subscribed to the blog and anticipate hearing from me on a regular basis. 

During my travels I have had a lot of discussions about ACD/1D NMR Assistant our software tool that has been designed specifically in mind for synthetic and medicinal chemists. Based many of these discussions I feel like I should address this idea of structure verification algorithms in software.

I think the point is NOT to replace the chemists QC or review of their NMR analysis. I know that may be an attractive marketing message...but that's not the purpose.

A chemist runs an NMR to ensure that their reaction has come to completion, and to confirm that they indeed made what they thought they made. Of course they have other reasons for running NMR (quantitation for example) but this is generally the major reason.

I think there is widespread acknowledgment throughout the industry that the detail and emphasis put into this personal QC varies greatly from chemists to chemist. Some chemists are very diligent in their NMR analysis assigning their data and ensuring that every peak and multiplet makes sense. From my own personal experience as a graduate student and chemist in the industry, I will be honest that for every NMR I ran following a reaction, I didn't diligently assign every multiplet from every ¹H spectrum. If I was in the 3rd or 4th step of a reaction, I was simply looking primarily at 1 or 2 peaks to see if my substitution was successful. If that peak disappeared at 3.1 ppm or that peak appeared at 2.7 ppm...I was off to the next reaction.

Does that make me an irresponsible chemist? Perhaps?...But I keep hearing that this practice is pretty common in the industry. In fact, I've had some chemists admit that they don't look at the NMR at all. If the LC-MS looks good...it's time to move on.

I think that this is precisely  where a tool like 1D NMR Assistant fits in. No, it's not necessarily smarter than the average chemist, and it's not necessarily better at assigning data than a chemist. But it can be a valuable tool to complement the chemist's workflow and help them make better decisions.

Further more....it's not a black box that tells you whether something is right or wrong.

Here's an example output from the software:

Note the language:

The structure does not appear consistent with the spectrum.

It is not suggesting that the structure is wrong only that it "appears inconsistent"

This is a key point. It is flagging certain parts of this spectra that perhaps the chemist should spend an additional 10-20 seconds looking at to ensure that they indeed made what they thought they made.It does this because it takes a brute force approach towards the assignment of all protons in the chemical structure to multiplets in the experimental spectrum. If it can't find a matching multiplet, or the assignment doesn't make sense...it will tell you.

In some cases these flags may not be serious. Maybe it's a solvent peak, or a slight impurity, or the software didn't predict the chemical shift accurately. That's OK...but the purpose is to alert the chemist that this is one they might want to take a look at. One that might otherwise end up as another false positive in the organizations registration database. Every once in awhile the software will uncover something that might have gotten overlooked.

In addition to being a standard 1D NMR Processing package, it is this functionality that has made 1D NMR Assistant a popular choice early in it's first year of availability.

False Negatives and False Positives are Waiting...

Great post from Derek Lowe from In the Pipeline the other day talking about the dangers of not quality checking those fine-looking starting compounds for your project. Chemistry happens and yes, mistakes do too.

In fact, it appears that Derek has been on a kick as of late referring to personal QC.

I Can Has Ugly Molecules?

Oops.

I thought this would once again be a good opportunity to provide you with a link to a poster Sergey Golotvin presented at ENC 2008 entitled, "Validating the Quality of Large Collections of NMR Spectra Automatically".

Long story short, 15,000 1H NMR Spectra from the Aldrich collection were evaluated in complete automation and the software was able to confirm 88% of the collection as having chemical structures that were consistent with the respective spectra. In addition, 4% were flagged by the software as being inconsistent. A closer, manual look at those 4% revealed that there were indeed some truly wrong structures (or incorrect tautomers) in the collection.

This was evaluating the 1H NMR data only. Using additional 2D experiments, such as HSQC, will likely improve these results.

Just an example of a check an organization can build into their process for additional QC of their registration database for example.

Is it perfect? Absolutely not. There are perhaps a few more false positives in there that the software didn't catch, and of course the software provided some false negatives as well, annoying because presumable someone has to look over them manually only to realize that they were indeed the right structure all along. But at least this doesn't involve manually pining over 15,000 spectra!

We continue to run these datasets, and actually have a consortium consisting of several NMR experts in the industry we call ASCI (Automated Structure Confirmation Initiative) where we are testing and validating this technology in the real pharmaceutical world. Identifying the common areas where false negatives and false positives occur and trying to address them with algorithms.

Will we ever solve all the problems, especially in the world of novel chemistry? Of course not, and for that matter there are some existing problems that appear to be too hard to solve.

But that being said, what is the acceptable limit of false positives and false negatives for automated verification by software for the verification of  registered compounds in a library?

Interested in hearing your thoughts.

Automated Structure Verification by 1H NMR Only

I've blogged several times about the progress and applications of automated structure verification with the help of ACD/Labs software.

There are really two main approaches right now:

1. Combined verification which includes automatically verifying the correspondence between a proposed chemical structure and the 1D ¹H and 2D HSQC spectra. I've blogged about the publications and applications of this method previously.
2. ¹H only verification. Of course the first approach is preferable from an accuracy standpoint as the additional information gained from the HSQC spectrum increases the selectivity and specificity of the results.

However, I always get questions on how well we perform on ¹H NMR only because in some organizations and environments it is simply not feasible to always run an HSQC in tandem with a routine ¹H NMR analysis.

As mentioned in previous posts, we've already conducted a validation study on this approach and it was published in a 2006 article in MRC. We've continued to investigate and validate this approach and we recently presented our latest results using 1D ¹H NMR data only at ENC 2008.

The poster highlighted a study on the automatic evaluation of over 15,000 Aldrich compounds and spectra from Aldrich NMR Spectra Database.

The results of this study revealed that the software was able to confirm 88%of all spectra as consistent and flagged less than 5% as inconsistent.

One of the more interesting discoveries in this study was that it revealed some truly wrong structures in the Aldrich NMR database.

More information on these as well as shortcomings in prediction, processing, and analysis are provided in the poster that can be downloaded:

http://www.acdlabs.com/download/publ/2008/enc08_aldrich.pdf

What's New in Version 11- Improved 1D NMR Structure Verification Accuracy

Automated structure verification using ACD/Labs software is a method that compares the chemical shifts, intensities, and multiplicities of signals in experimental NMR spectra with those from a predicted NMR spectrum with a proposed structure.

Naturally, in order for this process to be effective the chemical shift prediction, multiplet characterization, and integration measurements must be accurate. This process is described in detail in a 2006 publication in MRC.

As mentioned in the previous post, in version 11 the automated multiplet analysis
algorithm in ACD/Labs software has been significantly improved.

The purpose of this post is to show you how much the improvements of the automated multiplet analysis algorithm in version 11 impacts the performance of automated verification of ¹H NMR spectra.

The following results compare the verification improvements in version 11 based on the multiplet analysis enhancements ONLY. In version 11 we made several other improvements in NMR prediction and analysis, and I will get to them in a future post. For now:

For this study, two different data sets each consisting of the ¹H NMR spectra of 30 samples and the correct chemical structures were automatically processed, analyzed, and evaluated by the software.

Test Set 1- A set of 30 spectra and their respective chemical structures with reasonably good signal to noise:

Test Set 2- A set of 30 spectra and their respective chemical structures with lower signal to noise:

As you can see clearly, the improvements in multiplet analysis in version 11 heavily impact the performance of automated NMR verification of ¹H NMR data in both datasets. Based on the dataset used for this study, the software (employing the version 11 multiplet analysis algorithm) was able to correctly confirm the consistency between the proposed chemical structure and the experimental spectrum ~60% of the time in both datasets.

For a real world application of this system, check out a previous post that described Anthony Macherone's workflow at ASDI (and a link to his presentation).

The next questions is, "If you use more data, how well does this system perform?"

A future post will describe a study that highlights the latest performance statistics of a "combined verification" approach that can automatically identify  correct and incorrect chemical structures based on their 1D ¹H and 2D HSQC NMR data. This system was described in a 2007 publication in MRC.

Stay Tuned.