Ryan's Blog on NMR Software

Back to Reality #1- LC/MS vs. NMR

Reality #1:

LC/MS has become the primary analytical check for the medicinal chemist. I Love NMR, but it's the reality

So how did we get here? Clearly there a lot of reasons, but one of my hypotheses is based on the evolution of open access and the walk-up environment. No doubt that moving away from the traditional analytical service departments has been a great thing from a productivity standpoint.

LC-MS lends itself really well to the open access environment. There are good systems out there that can auto-process data well and provide a nice report that gives the chemist a simple answer. In most cases, the chemists can get the answer they are looking for with a quick glance at the report. In addition, for this kind of routine analysis, the interpretation of the results is very easy. 

Contrast that to NMR. How often are the printouts from the instrument good enough? Not nearly as often. More and more it is becoming common practice for the chemist to import their raw or processed data into the processing software on the instrument or in some offline processing software. Within these applications, they can get a better look at their data, zoom in on areas of interest, re-integrate the spectrum so it makes sense, perhaps even generate a multiplet report for a potential patent down the road.

Add up those tasks alone, and the chemists may be forced to spend 5-10 more minutes on a routine NMR than an LC-MS when you factor in interpretation as well! Unfortunately, the NMR instrument is not printing out an answer, it is printing out a spectrum that needs to be interpreted by the chemists which on a routine basis is a lot more difficult and complex task than checking an LC-MS for reaction completion. Furthermore, let's keep in mind that intrepreting analytical data is not the medicinal chemists job. They are evaluated, compensated, etc. for making molecules. The more molecules the better so the inherent risk here is obvious and I don't think it's fair to simply "blame the chemists"

Given the above, is there any wonder why LC-MS has become the primary analytical check?

The Realities of NMR in the New(ish) Discovery Environment

Reality #1:

LC/MS has become the primary analytical check for the medicinal chemist. I Love NMR, but it's the reality

Reality #2: 

More often than not, NMR spectra are merely glanced at:

• Sometimes interpretation is based on the presence or absence of one peak (did my reaction go to completion?)
• How often are NMR spectra fully assigned by a medicinal/synthetic chemist? Not very often, it's not their job. 
• Is all the information a ¹H NMR spectrum can provide being effectively used? Some chemists are extremely good at NMR, other couldn't be bothered with the intimate details, and I can't say I blame them. It's largely a training/education issue
• How often are 2-dimensional experiments acquired and interpreted? Of course this varies chemist by chemists, organization..but despite strong advances in hardware, acquisition, etc, I believe the safe answer is, "not enough"

Reality #3:

In many organizations, upwards of 95% of compounds are being registered by synthetic/medicinal chemists without having undergone any analytical review by an analytical specialist (NMR spectroscopist). 

***

How and why did we get here? I have some hypotheses, and will write about them in the coming days. 

Are We There Yet? Automated Structure Verification (ASV)

When I created this blog several years ago, one of the first links I added to my blog was to Stan's NMR Blog. Over the years I have had the pleasure to meet Stan who is a wonderful individual. His blog is very interesting, Stan is incredibly knowledgeable, and I really enjoy his writing style. 

Recently, Stan posted about a topic, those readers who follow my blog know I am very passionate about; Automated Structure Verification (ASV). The initial entry was very thought-provoking and while I think the numbers Stan posts regarding the number of spectra chemists interpret below are too high, I think he's put together an interesting argument nonetheless about the need and application of ASV in the scientific community:

The typical application area for such a product is drug discovery, meaning Pharmas. I was told that these Companies employ masses of chemists who have to interpret some 50 spectra a day, having only 8h/50 = 9:36 minutes per spectrum. Clearly, they don't wag their tails, nor wish to pursue more balls. They were reduced to the echelon of routine workers and they hope that the new software might give them a few minutes more time to maybe ponder again that strange spectral peak that looked like an interesting challenge. Illusion, of course, because their top CEO's hope that the software will make half of the chemists redundant, leaving the rest doing a spectrum every 4:48 minutes (guess who will prevail).

Stan's entry definitely triggered some interesting comments from the community as well, in addition to a follow up letter from John Hollerton, GSK.

My take is that we don't need to be waiting 3-5 years for this technology to become more relevant...it already is! It is already being used in the industry. However, the secret is finding and recognizing the right application for it within the industry, and that application will vary considerably from lab to lab. 

It seems the main discussion about this completely surrounds whether or not ASV can replace the NMR interpretation of the medicinal or synthetic chemist. Of course this is the ultimate goal, but it should not be considered the only goal. John eludes to some other applications in his letter and there are other uses as well. 

I will blog more later about my thoughts on the "Reality of the New(ish) Discovery Environment" as it pertains to NMR and more thoroughly justify why I feel the way I do about this technology. But for me, the bottom line is that this is not just a concept, it's not a dream that we need to wait be realized in the next 3-5 years. It is also not a new technology on the horizon. Our first publication on this topic was back in 2006 and we've continued to build on those approaches in 2007 and 2008. Work was done by Griffiths et. al in the late 90s and early 2000s. The techonology has only gotten better and better. We have been working diligently with industry partners for over the last decade on this concept. It is not new and it is not an unsolved area. 

Just to be clear, am I saying this technology is ready to completely replace the NMR interpretation responsibilities of the average medicinal/synthetic chemist? Definitely not. 

Can it be used to help chemist make better, faster, and more independent decisions? Yes. Can it be used in complimentary way to the chemists existing interpretation workflow? Yes. Can it used to be validate the quality of registration libraries? Yes. Can it be used to verify compounds purchased from external sources, whether this are assay materials or simple starting materials? Yes.  The list of applications go on for a while. 

I believe current ASV applications can improve all the above workflows. The key is identifying the best applications and practicing caution in how this technology is deployed and used by chemists. 

In conclusion, my advice is that if we continue to treat this technology as only applicable to the ultimate goal, perhaps it will take longer than 3-5 years, or perhaps NMR will become irrelevant before we get there. The goal of NMR scientists, hardware, and software vendors should be to work hard to continue to make NMR technology relevant in all industries. If we can do this, we all win.

The Black Swan of Compound Libraries

Recently I finished a great book that made me think about different things on a lot of different levels. The book is called The Black Swan and the author is Nassim Nicholas Taleb. 

In short, a black swan is defined as an event, positive or negative, that is deemed improbable yet causes massive consequences. 

From wikipedia:

What we call here a Black Swan (and capitalize it) is an event with the following three attributes. First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.

Anyone who follows this blog knows that I have a passion around the concept of automated structure verification by software methods and the different applications it can play within the pharmaceutical industry. 

Reading through this book, one of the things that popped in my mind was the black swan impact as it pertains to compound registration libraries in pharma.

I spend many hours in my role discussing the concepts of automated structure verification (by NMR of course) to spectroscopists, chemists, directors, VPs, Managers of compound management, etc. 

In many of these conversations, the risk of having an incorrect compound in the registration library is always a fruitful and interesting discussion. However, this conversation is almost always dominated by low probabilities. The low probability that the compound being registered is wrong in the first place. I agree, this is a low probability. After all, a trained chemist has synthesized said material and has used a variety of analytical methods to confirm it's identity. In some organizations, the compounds in this library will be validated further either before they are accepted in the store, or before they are sent off to assay. 

So the argument goes that incorrect compounds in the registration library are low probability. And furthermore, there are steps being taken in some organizations to proactively catch incorrect compounds (most often LC-MS). Finally, perhaps the lowest probability event of them all is the compound being identified as a hit during the assay. 

So in the end, what are the chances that a meaningful compound that is identified as a "hit" is actually an incorrect compound? Furthermore, in this instance what's the chances that this compound will make it too far downstream without it's false identity being exposed?

In short, it's a pretty low probability event, and because in most cases it will eventually be caught it's a stretch to really call this a massive consequence or impact. 

That's not the black swan I am talking about in this post. 

In my opinion, the Black Swan concept is more relevant to those incorrect compounds that lie in the registration library and never get cherry picked out. A compound that doesn't produce anything interesting from the assay. A compound that effectively hides in the back of the shelves in a compound management cold room, that has been documented as "tested" but never advanced any further because of poor assay results.

Of course this compound is not the Black Swan. However, it's possible that the compound that it was supposed to be is. The compound that the chemist thought they made, and the compound that was referenced in the inventory.   

While it's low probability, there are probably tens of thousands of misrepresented compounds that haven't truly been assayed over the last 20 years. Of these thousands could any of them turned into the highly coveted, blockbuster drug?

Sure, it's doutbful, but the infinitely small possibility exists. 

And further, this very idea contains one of the core components of a Black Swan; "nothing in the past can convincingly point to it's possibility"

I am not certain that there is an example of a blockbuster drug that was originally missed because of a mistake in synthesis, or a misrepresented registrant. My guess is "not really" And if it did, perhaps as Taleb suggests, it would have been rationalized by hindsight. 

Most Accessed Article on Journal of Cheminformatics of All Time!

An article co-authored by several ACD/Labs scientists, the Russian Academy of Scientists, and ChemZoo, Inc. has been generating a lot of interest over the past year.

In fact, this article entitled, "Computer-assisted methods for molecular structure elucidation: realizing a spectroscopist's dream" is the most accessed article of all time, on the Journal of Cheminformatics website.

The news gets better. This article is FREE for your reading pleasure! As per the policy of the Journal:

"The Journal of Cheminformatics offers scientists the opportunity to publish their research rapidly in an open access medium that is freely available online to researchers worldwide. All manuscripts submitted to the journal are subject to rigorous peer review. Published material may include electronic supplementary material, such as animations (e.g. graphical chemical structures) or data (e.g. spectra) that can be downloaded for a reader's use.

The Journal of Cheminformatics aims to work closely with operators of open data repositories such as PubChem. Requests for collaboration are welcome; please contact the Editorial Office."

This motivation for this article was to mark the 40 year anniversary of published developments regarding CASE (computer assisted structure elucidation systems).

It a very interesting read that draws from many meaningful publications on CASE over the last 40 years and tracks history to outline some of the approaches that have been considered, implemented, and successful.

Some of the more significant developments such as fuzzy generation, used to deal with the ambiguity of 2D NMR experiments, as well as the determination of relative stereochemistry of large and rigid molecules with many stereocenters are discussed in this article.

I could spend more time talking about this article, but as I've already mentioned, it's FREE and can be accessed here:

http://www.jcheminf.com/content/1/1/3

Enjoy and feel free to leave any comments for the authors, and I will be sure to get them. 

How do Medicinal and Synthetic Chemists use NMR Software?- Part 6

This is a continuation of a series of posts that began here.

In the previous post of this series, I discussed how chemists can extract multiplet reports in journal format on the fly using the same procedure to re-cut integrals in the spectrum. Well in addition to having this nice reporting tool, the software can make valuable use out of this multiplicity information.

The software has the ability to assist the chemist with the interpretation of their spectrum. It does this by using the NMR prediction algorithms embedded in the software. With the use of these algorithms structure interpretation and verification can be employed.

Essentially when a chemical structure is attached to a spectrum, the software will predict a spectrum in the background and automatically compare the predicted spectrum to the experimental spectrum. To do this automatic evaluation, the software will compare three properties between the spectra:

1) Chemical Shift
2) Integration
3) Multiplicity information

As a result, by re-cutting your integrals in the software (and hence defining multiplet patterns and coupling constants) the software will use this information to help a chemist interpret their spectrum. Specifically, it will use this information to provide the chemist with immediate feedback as to what atoms in the chemical structure a multiplet is consistent with. See the movie below:

The Balancing Act of False Postives vs. False Negatives

A few months back, I referenced Derek Lowe's excellent blog, "In the Pipeline"

Some of the most interesting entries for me are:

Backtracking, Necessary, and Unnecessary

I Can Has Ugly Molecules?

Oops.

One of the take home messages for me, is when Derek says, "False Negatives and False Positives are waiting in your dataset, depend on it"

I think this is a point that is largely acknowledged in the industry. For example, I think it is pretty obvious to most organizations that there are a significant amount of incorrect structures in their registration database. Some will have a pretty good idea of how much, and others will have no idea.

I think the key point however, is to acknowledge it.

Now this brings me to the topic of the balancing act between false positives and false negatives. If I go back to the application of validation compound registrations with automated NMR verification. This application is one I have blogged about quite a bit over the last year and a half.

The key point to emphasize here, is that this application is NOT replacing the chemists analysis and QC

Chemists are still looking at their data and registering their compounds the normal way. But is an automated validation step necessary after the fact to ensure the quality of the registration database is accurate?

I asked the question from an earlier post, "what is the acceptable limit of false positives and false negatives for automated verification by software for the evaluation of  registered compounds in a library?"

I didn't get any comments on the blog, but I got a few email responses and I have discussed this with some industry people over the last few months. Not surprisingly, the results vary significantly.

Of course everyone would prefer a 0% False Positive Rate, and a 0% False Negative Rate. Yes, in a perfect world that would be great. But it's not going to happen. Not ever. So decisions need to be made about how important this this exercise is, and furthermore what is deemed acceptable?

If we acknowledge that there are incorrect structures in the registration database, then the question is how much, and furthermore, how many can be removed with the use of some automation.

I think the challenge that most organizations face today when evaluating such a system is trying to balance out to sides, false positives vs. false negatives. Why?

My understanding is that if you have a system that is false positive tolerant, well, you are sacrificing the overall quality of your organizations registration database. On the other side of the coin, if you have a system that is false negative tolerant, well, someone needs to look at the data manually to see if this is indeed the right or wrong structure. So there are two forces pulling here. One, is actually knowing that despite your efforts, you are still letting incorrect structures in your database. The other force is knowing that you need a real person's time to spend on manually evaluating a bunch of spectra.

Most spectroscopists I know, do not have a lot of time for this. So of course it would be preferable to have a system that is potentially more false positive tolerant. However, the whole point of implementing the system in the first place was to identify the false positives.

Tough decision...

But at the end of the day, I believe there are systems and applications out there that can help drastically improve the quality of your registration databases.

However, the balancing act between false positives and false negatives is an issue that you are most certainly going to have to juggle.

Yes, Chemists Will Still Have To Look at their NMR Data

I've been away for the better part of the last few months, haven't been able to spend anytime blogging, and I apologize for those of you who have subscribed to the blog and anticipate hearing from me on a regular basis. 

During my travels I have had a lot of discussions about ACD/1D NMR Assistant our software tool that has been designed specifically in mind for synthetic and medicinal chemists. Based many of these discussions I feel like I should address this idea of structure verification algorithms in software.

I think the point is NOT to replace the chemists QC or review of their NMR analysis. I know that may be an attractive marketing message...but that's not the purpose.

A chemist runs an NMR to ensure that their reaction has come to completion, and to confirm that they indeed made what they thought they made. Of course they have other reasons for running NMR (quantitation for example) but this is generally the major reason.

I think there is widespread acknowledgment throughout the industry that the detail and emphasis put into this personal QC varies greatly from chemists to chemist. Some chemists are very diligent in their NMR analysis assigning their data and ensuring that every peak and multiplet makes sense. From my own personal experience as a graduate student and chemist in the industry, I will be honest that for every NMR I ran following a reaction, I didn't diligently assign every multiplet from every ¹H spectrum. If I was in the 3rd or 4th step of a reaction, I was simply looking primarily at 1 or 2 peaks to see if my substitution was successful. If that peak disappeared at 3.1 ppm or that peak appeared at 2.7 ppm...I was off to the next reaction.

Does that make me an irresponsible chemist? Perhaps?...But I keep hearing that this practice is pretty common in the industry. In fact, I've had some chemists admit that they don't look at the NMR at all. If the LC-MS looks good...it's time to move on.

I think that this is precisely  where a tool like 1D NMR Assistant fits in. No, it's not necessarily smarter than the average chemist, and it's not necessarily better at assigning data than a chemist. But it can be a valuable tool to complement the chemist's workflow and help them make better decisions.

Further more....it's not a black box that tells you whether something is right or wrong.

Here's an example output from the software:

Note the language:

The structure does not appear consistent with the spectrum.

It is not suggesting that the structure is wrong only that it "appears inconsistent"

This is a key point. It is flagging certain parts of this spectra that perhaps the chemist should spend an additional 10-20 seconds looking at to ensure that they indeed made what they thought they made.It does this because it takes a brute force approach towards the assignment of all protons in the chemical structure to multiplets in the experimental spectrum. If it can't find a matching multiplet, or the assignment doesn't make sense...it will tell you.

In some cases these flags may not be serious. Maybe it's a solvent peak, or a slight impurity, or the software didn't predict the chemical shift accurately. That's OK...but the purpose is to alert the chemist that this is one they might want to take a look at. One that might otherwise end up as another false positive in the organizations registration database. Every once in awhile the software will uncover something that might have gotten overlooked.

In addition to being a standard 1D NMR Processing package, it is this functionality that has made 1D NMR Assistant a popular choice early in it's first year of availability.

False Negatives and False Positives are Waiting...

Great post from Derek Lowe from In the Pipeline the other day talking about the dangers of not quality checking those fine-looking starting compounds for your project. Chemistry happens and yes, mistakes do too.

In fact, it appears that Derek has been on a kick as of late referring to personal QC.

I Can Has Ugly Molecules?

Oops.

I thought this would once again be a good opportunity to provide you with a link to a poster Sergey Golotvin presented at ENC 2008 entitled, "Validating the Quality of Large Collections of NMR Spectra Automatically".

Long story short, 15,000 1H NMR Spectra from the Aldrich collection were evaluated in complete automation and the software was able to confirm 88% of the collection as having chemical structures that were consistent with the respective spectra. In addition, 4% were flagged by the software as being inconsistent. A closer, manual look at those 4% revealed that there were indeed some truly wrong structures (or incorrect tautomers) in the collection.

This was evaluating the 1H NMR data only. Using additional 2D experiments, such as HSQC, will likely improve these results.

Just an example of a check an organization can build into their process for additional QC of their registration database for example.

Is it perfect? Absolutely not. There are perhaps a few more false positives in there that the software didn't catch, and of course the software provided some false negatives as well, annoying because presumable someone has to look over them manually only to realize that they were indeed the right structure all along. But at least this doesn't involve manually pining over 15,000 spectra!

We continue to run these datasets, and actually have a consortium consisting of several NMR experts in the industry we call ASCI (Automated Structure Confirmation Initiative) where we are testing and validating this technology in the real pharmaceutical world. Identifying the common areas where false negatives and false positives occur and trying to address them with algorithms.

Will we ever solve all the problems, especially in the world of novel chemistry? Of course not, and for that matter there are some existing problems that appear to be too hard to solve.

But that being said, what is the acceptable limit of false positives and false negatives for automated verification by software for the verification of  registered compounds in a library?

Interested in hearing your thoughts.

Automated Structure Verification by 1H NMR Only

I've blogged several times about the progress and applications of automated structure verification with the help of ACD/Labs software.

There are really two main approaches right now:

1. Combined verification which includes automatically verifying the correspondence between a proposed chemical structure and the 1D ¹H and 2D HSQC spectra. I've blogged about the publications and applications of this method previously.
2. ¹H only verification. Of course the first approach is preferable from an accuracy standpoint as the additional information gained from the HSQC spectrum increases the selectivity and specificity of the results.

However, I always get questions on how well we perform on ¹H NMR only because in some organizations and environments it is simply not feasible to always run an HSQC in tandem with a routine ¹H NMR analysis.

As mentioned in previous posts, we've already conducted a validation study on this approach and it was published in a 2006 article in MRC. We've continued to investigate and validate this approach and we recently presented our latest results using 1D ¹H NMR data only at ENC 2008.

The poster highlighted a study on the automatic evaluation of over 15,000 Aldrich compounds and spectra from Aldrich NMR Spectra Database.

The results of this study revealed that the software was able to confirm 88%of all spectra as consistent and flagged less than 5% as inconsistent.

One of the more interesting discoveries in this study was that it revealed some truly wrong structures in the Aldrich NMR database.

More information on these as well as shortcomings in prediction, processing, and analysis are provided in the poster that can be downloaded:

http://www.acdlabs.com/download/publ/2008/enc08_aldrich.pdf