Ryan's Blog on NMR Software

How Accurate are Experimental Chemical Shifts?

Several months ago, I asked, "How Accurate Should NMR Predictions Be?"

Today, I ask how accurate and consistent are actual experimental chemical shifts?

In many ways, this post probably should have preceded the one I linked above because in reality, before a discussion about prediction accuracy can begin, the topic of experimental accuracy needs to be addressed.

The issue of experimental accuracy can be important from two perspectives. For example, accuracy is important for a chemical shift database that is used for producing the predictions (Hence ACD/Labs' Purgatory Database). In addition, it is also important in identifying the accuracy of a predicted chemical shift when comparing it to an experimental one. How can we determine where the inaccuracy occurred?

In the process of producing an experimental NMR spectra, there are many variables that can affect a chemical shift that are not always carefully controlled. They include, but are certainly not limited to:

• Concentration of the sample
• Temperature of the probe
• Equilibration time in the probe
• Solvent type
• Residual water content of the solvent
• pH of the sample (if aqueous media)
• Digitization of the spectrum
• Shimming and phasing inaccuracies
• Choice of reference standard

Of course many of these factors, can significantly affect the chemical shift of the peaks in the spectrum. How much they are affected is sometimes hard to measure, but as an example, we can consider the range of database entries in our database for the shift of the methyl group protons in toluene. All of the following have been published in peer-reviewed journals.

Of course the deviations in these shifts are primarily based on the fact that each chemical shift was recorded in a different solvent. The reason for adding 8 sources of toluene in our database is so we can attempt to take the solvent into account when solvent-specific prediction is performed.

But as mentioned, solvent is not the only variable that can affect how an experimental chemical shift is recorded.

Thoughts?

References:

1. Prog. Nucl. Magn. Reson. Spectrosc.,1996,v.28,p.161 (Toluene-d8)

2. Zh. Org. Khim.,v.12,p.275 (Tetrahydrofuran-d8)

3.  J. Org. Chem.,1997,v.62,p.7512 (Chloroform-d; 300 MHz; 24 C)

4.  J. Org. Chem.,1997,v.62,p.7512 (Acetone-d6; 300 MHz; 24 C)

5.  J. Org. Chem.,1997,v.62,p.7512 (Dimethylsulfoxide-d6; 300 MHz; 24 C)

6.  J. Org. Chem.,1997,v.62,p.7512 (Benzene-d6; 300 MHz; 24 C)

7.  J. Org. Chem.,1997,v.62,p.7512 (Acetonitrile-d3; 300 MHz; 24 C)

8.  J. Org. Chem.,1997,v.62,p.7512 (Methanol-d4; 300 MHz; 24 C)

 

Wolfgang Robien and Modgraph vs. ACD/Labs

Those of you who have been following this blog through it's first few month will no doubt have seen a few posts about the comparisons between ACD/Labs, Modgraph, CSEARCH, and the NMRShift DB. For those of you who have not followed this story but are intrigued, you have a lot of reading to do, starting here:

http://acdlabs.typepad.com/my_weblog/2007/05/nmrshiftdb_acdl.html

If you go to Modgraph's website for example, you will find the following web page:

http://www.modgraph.co.uk/product_nmr_shiftdb.htm

Here Modgraph presents results that show, most notably to this blogger:

Modgraph NMRPredict: 1.40 ppm overall average deviation

ACD/Labs CNMR Predictor: 1.59 ppm overall average deviation

These values correspond to the average deviation of predicted chemical shifts as compared to the experimental values (over 200,000) in the NMRShiftDB. Let me first state that the quoted average deviation for ACD/Labs CNMR Predictor is INCORRECT in this context. The number is correct in a different context. More on that in a moment.

In the meantime, there is one flaw in these numbers that I continue to object to, yet Modgraph continually chooses to pass over. It involves the following statement from their website:

For this evaluation the combined databases from CSEARCH and SPECINFO holding a total of 345,308 reference spectra were used. Based on this higher number of reference spectra a somewhat higher structural overlap between our databases and the NMRShiftDB-testdata has been detected. In order to compensate for this, we have recalculated our overall average deviation of 1.40 ppm using the lower structural overlap as detected by ACD. The value of 1.40 ppm corresponds to 92,927 known carbon environments and 121,209 unknown carbon environments - without this compensation our overall average deviation would be slightly better, but a comparison with ACD's results would be impossible.

Am I the only one who thinks that the bolded part is rather inappropriate and unscientific?

We have acknowledged the overlap between our database and the NMRShiftDB to be 57%. Modgraph declines to share what their overlap is and admits only that it is higher. So in order to compensate for this they have used our lower structural overlap? So what structures and shifts are they removing from their databases to compensate for this overlap? How do they decide which ones get removed?

Irregardless of how this study was conducted, it should be known that the average deviation reported on their website of 1.59 ppm is incorrect in this context as a data point of comparison with Modgraph's number. I have already addressed that number and written a post about it on this blog earlier.

The average deviation of ACD/Labs CNMR Predictor compared to the entire NMRShiftDB (with an overlap of 57% included) is actually 0.96 ppm.

Of course I will re-iterate here my thoughts that I shared in my original posting about this. These reported average deviations (0.96 ppm for ACD/Labs and 1.40 ppm for Modgraph)  are likely not the best measure of how well NMR prediction software will perform for an end user. Why? Because as clearly stated, these numbers are based on an experimental dataset (the NMRShiftDB) that has a 57% overlap with the databases in the prediction engines.

So unless you, the end-user can expect a very significant overlap between your own compounds and the NMR predictors (57%) you shouldn't expect this kind of accuracy. In reality, chances are you are working with novel chemistry and the majority of your compounds are completely novel and thus not represented in a prediction softwares database.

In my opinion, we have come up a best practice for NMR prediction validation. Not only do we provide an average deviation for the entire NMRShiftDB dataset (0.96 ppm and acknowledging a 57% overlap), but we also provide a a validation study on the completely novel chemical shifts. This study evaluated only the chemical shifts that are not present in our database (the other 43%).

Modgraph has yet to produce a study consistent with the second approach (that is, a study on completely novel chemical shift not represented in their database). In fairness to them, it can be argued that this is not a fair comparison either as we would likely be comparing two completely different data sets with completely different structures. Our overlap is not the same after all.

Nonetheless, ACD/Labs has provided the public with both numbers for their reference. The bottom line, is that Modgraph has chosen to publish some number that doesn't have a clear explanation behind it. It neither represents an evaluation of the entire dataset, nor one without any overlap. Their reported number is somewhere in between and they have declined to offer details. If they  produced an average deviation for  the entire dataset (including all overlap) or the results of a dataset excluding any overlap, it would most definitely be different than 1.40 ppm value they have quoted. All we hear from them is that if they didn't compensate for the difference in overlap between ACD/Labs and Modgraph, their results would be SLIGHTLY better. 

That's fine, and while I question their experimental method of removing chemical shifts to compensate for the difference between ACD/Labs and Modgraph's overlap, this is the study they have chosen to go public with. So in the end the final results are:

Modgraph NMRPredict: 1.40 ppm overall average deviation

ACD/Labs CNMR Predictor: 0.96 ppm overall average deviation

The Price of NMR Software

So I will be presenting at our annual seminar at SMASH in a few weeks, and I always enjoy doing so. One thing I have picked up on over the years when presenting in our seminars, in addition to other seminars, is that look of curiosity and waiting that appears on the audience's faces as we go from slide to slide. That look that can be translated to, "How much does this stuff cost?"

I am going to talk a little about a fairly touchy subject. Pricing.

It's no secret that ACD/Labs does not share it's pricing online while some of the other vendors of NMR software do. Sometimes we take criticism for that. I have also heard many times from many people that ACD/Labs software is really expensive.

I am not going to hide those conversations, they are out in the open right here and right now.  But the real question is, "what defines expensive?"

I think it is very subjective. I happen to think that $400 for a purse is very expensive. My wife may disagree. I think that $5000 for a guitar is not expensive. My wife definitely disagrees. It's subjective.

But we are talking about NMR software after all. I think when you think about pricing for software you have to look at many different variables.

So there are  a couple of questions I will try to answer to somewhat address the "elephant in the room":

1) Is our software expensive?

Some people have a pre-conceived idea that no matter how good a piece of software is, it shouldn't cost more than $500. Heck, some will argue that all software should be free. I disagree with this for a number of reasons, but let me start by outlining some of the things that have to be considered when determining the price of software. To name a few:

- The cost of developing the software
- The cost of maintaining the software. If you sell it for less than the cost of development and improvement, than it becomes very difficult to maintain the software, add new products and features, and fix bugs. If the software company goes out of business because they can't maintain and improve their software, everybody loses.
- The cost of supporting the software and the customer base. i.e. product management, development, technical support, sales, marketing, production, etc.
- The market. How big is it? Is there competition? Are they better than you? What can they afford?
- The costs the consumer encounters for deploying the software in their institution (i.e. installations, training, etc.)

This kind of thing is of course not unique to ACD/Labs. It is the similar to what other industries have to consider when selling books, education, or pharmaceutical drugs!

So answer the question Ryan, "is your software too expensive?" Well I would say no. But of course I would, I am an employee of ACD/Labs after all, I can't admit that it is expensive, can I? The truth is that I have spent a lot of time in my last 4 years speaking with customers and users of our software. I have heard excellent stories about how valuable our software is, how much time it has saved, and how much it has benefited their research, students, departments, etc. One of my favorite comments from a Structure Elucidator user was, "If the software can either dereplicate or elucidate even one of my unknowns, it has paid for itself!" In instances like these, I don't think the software is too expensive because the benefits these groups are receiving have far outweighed the cost it took for them to obtain the software.

Unfortunately, there are also cases where users just haven't found the software useful for their research, or haven't even gotten around to installing it yet. In those cases they might have considered the software not valuable and thus the upfront cost was not justified. It happens. We try to prevent it as much as possible, but unfortunately it is unavoidable in some cases. 

I think we have very good software. We work very hard on the development of good software products. We fix bugs and add new things year round and we release a MAJOR version of the software every year with new enhancements and features. With the help of NMR spectroscopists and scientists around the world, we will often identify market needs that allow us to further innovate and augment our core software through new software product offerings. We have been in existence for over 11 years and we continue to make good software and we continue to improve and innovate from version to version. We work hard to make our different modules integrate with each other so users can extend the capabilities of the software. Finally, we sell the software. We have many, many customers from all over the world. To name a few:

http://www.acdlabs.com/clients/
http://www.acdlabs.com/educators/acadclients.html

We've sold to these companies and institutions and we have survived more than 11 years in this industry selling and supporting our software in many of the above institutions and more. I don't think we would have come this far if our software was flat out "too expensive".

Finally, I think I have to acknowledge the market that we are in strictly from an NMR software perspective. There are several companies who primarily develop NMR software out there:  Acorn NUTS, Mestrelab, Modgraph, NMRTec, to just name a few. But I don't think any of these companies cover the scope of the NMR Spectroscopy applications like we do. This is not a criticism of them, they just have a different business and strategic focus for what they want to achieve with their software. We offer NMR processing, prediction, databasing, automated structure verification, computer assisted structure elucidation, high-throughput quantitation and verification, and enterprise solutions. Furthermore, we support a wide range of other analytical techniques so it is possible (with the correct modules in place) to view, process, and database NMR, MS, Chromatographic, UV-IR, (and more) data all in one place. In other words whether you are just starting with our simple NMR Processor, there is the opportunity to expand the solution in many different ways. I don't think this is possible with many other NMR software companies. Of course some people aren't interested in the complete solution. That's completely understandable and as a result you can just purchase what you need. For that I think our individual NMR processing, prediction, databasing (and other) packages are very competitively priced with other similar products in the market. Check out the quotes from some of these press releases, there is a common theme in all of them:

http://www.acdlabs.com/clients/pr_nmr_1206.html

"All of our chemists are now able to manipulate their NMR data at their convenience which has eliminated the lengthy line-ups we used to face in our walk-up lab. ACD/Labs was able to provide us with great tools that meet our needs and were competitively priced."

http://www.acdlabs.com/clients/pr_bristol0806.html

"ACD/Labs were also able to offer us a competitively priced and flexible site license that meets the range of access requirements of both the students and staff members at Bristol."

http://www.acdlabs.com/clients/pr_kalexsyn0506.html

"All of our chemists are now able to manipulate their NMR data at their convenience which has eliminated the lengthy line-ups we used to face in our walk-up lab. ACD/Labs was able to provide us with great tools that meet our needs and were competitively priced."

It of course becomes a bit more difficult to price some of the bigger products because there is simply nothing comparable to it in the market. And generally these bigger products are subject to huge amounts of development time (i.e. ACD/Structure Elucidator and ACD/2D NMR Expert).

Is ACD/Labs software expensive?

No. Some of it is, yes...but I think it is expensive for a reason. And that of course is subjective, if you receive the benefits that we claim from the software, then it is definitely not expensive, in my opinion.

The point I am trying to get at is that the value of the software is what you yourself get out of it. If you have a major problem with a crowded instrument room because all you have is one seat of the vendor's processing software available, then deploying offline processing for all of your chemists would probably have a significant amount of value to it.  Then again, if you don't have this problem but you view it as a luxury to have desktop processing for everyone, then perhaps you would think it is too expensive.

2) How much is our software?

Our company has a policy regarding not publishing our prices and it is not in my right to publish those here. That policy has been implemented for many reasons, and it does raise an interesting question.

Why wouldn't we share our prices publicly?

Well I think that the logical response most people would come to is, "because they are too expensive"

I don't believe that's the case.

In fact, years ago, we DID publish our pricing on the website, we even had a webstore for all of our products. Unfortunately, this  was not successful. In fact, we continued to sell software the way we do now, through people contacting our account managers and having a discussion about their wants, needs, and desires.

And this, leads to one of the problems we encountered by posting our pricing online. It's one of the drawbacks of having 100+ different products. Potential customers tend to get overwhelmed by the number of choices and they want to make sure that they don't pull the trigger on a product that they don't need. So they simply write an email or pick up the phone. Furthermore, the pricing of our products is not always static. There are often special discounts on certain products, and bundles depending on whether you are a new customer, current customer, an academic, the number of products or licenses you want to buy (do you want a copy for everyone, or would you like 15 people to share 5 copies), etc.

Therefore, as a company, we have resigned ourselves to the fact that we need to involve the sales person in all sales discussions. In doing so we can ensure that they can take the prospective customer through different options and ensure they get set up with the best solution for their needs at the best possible prices. I think that's a good thing. We pride ourselves on good customer interactions and customer service, and I hope that some of you have benefited from that already.

If you have made it to the end of this entry, I am grateful for your attention. I hope that I have at least somewhat addressed the elephant in the room, lab, or blogosphere.

Neural Network Algorithms vs. HOSE Code Algorithms

You may or may not be aware that version 10 of ACD/HNMR and CNMR Predictor now offer the option to generate predictions using a neural network algorithm.

For those of you who have followed ACD/Labs for years, you may observe this as an intriguing development.

Since the release of our NMR prediction packages many years ago, ACD/Labs has been strongly committed to a modified HOSE (Hierarchical Organization of Spherical Environments) code approach for NMR prediction.

The reasons for this commitment were obvious. Over the years, the developers have undertaken many internal validation studies and the only conclusions made were that HOSE code was clearly the better choice and thus warranted the most future development (one recent example).

So it was during the development of version 10 of ACD/Structure Elucidator that the inclusion of neural network algorithms was once again considered within ACD/Labs NMR software. In the beginning, the consideration of employing a neural network algorithm was based on an effort to try and enhance the speed of NMR predictions within the structure elucidation software. The process involves generating predictions on suggested structures and comparing them directly to the experimental data. Because neural network algorithms are known to perform 100s to 1000s of times faster than their HOSE code counterpart, research was conducted on this.

What was not expected, was a discovery made by the Structure Elucidator Project Leader, Kirill Blinov during his evaluation of the neural network algorithm performance.  It turns out that Kirill's neural network algorithms appeared to be outperforming our current (at the time) HOSE code implementation in version 9.0 in several areas.

As a result, the information gathered from this development work allowed us to improve and optimize our HOSE code algorithm for release in version 10 NMR predictors.

To make a long story short, what was the final result?

As of right now, the HOSE Code algorithm available in version 10 of ACD/Labs NMR predictors, still outperforms the neural network algorithm in all criteria except speed, hence it remains the default algorithm in the current version of ACD/HNMR and CNMR Predictors. This being said, the neural network algorithm is available in version 10 of the software for users to try out and evaluate on their own. It can be accessed through the Options menu and selecting Spectrum Plot...

Of course the ideal scenario is one where a user does not have to choose which algorithm they want to use. We are currently working on an intelligent hybrid approach that hopefully will be available in version 11.  One example of a combined approach has been attempted elsewhere, but we are looking more deeply than this for an alternative solution that will result in improved prediction.  Much work is also being done on atomic increment-based algorithms as well. 

Lastly, it is important for me to point out a couple of limitations of the neural network algorithm:

1. Neural net predictions are a black box. Therefore a "calculation protocol" which shows which experimental data was used in the prediction (available with HOSE code) is not available.
2. Prediction training, while possible, has not proven to outperform HOSE code training.

For more information on our internal validation results of these algorithms, see this presentation from ENC 2007

Final Note on the NMRShiftDB Debate- 0.96 ppm

Perhaps some readers will believe that I have spent too many posts on the NMRShiftDB debate. I would like to re-iterate that one of my main purposes for creating this blog was to keep the public informed on the world of NMR software. In this particular debate, it has been possible to inform users of NMR software around the world with more information on three different NMR software resources, The NMRShiftDB, ACD/CNMR Predictor, and Modgraph NMRPredict.

So with that I would like to make one final posting on the prediction validation of the NMRShiftDB. The previous presentation of results may lead one to believe that Modgraph has the edge in prediction accuracy. 

I have continually asked Modgraph to clear up the overlap issue that we have so heavily discussed here. Again, what is the amount of overlap of identical chemical structures between the NMRShiftDB and Modgraph NMRPredict database? Once that overlap is acknowledged and removed, what is the accuracy of this software package on completely novel chemical shifts? 

As a co-author of the original validation of prediction accuracy on the NMRShiftDB dataset, I firmly believe that we have created a best practice for prediction validation where we openly shared our structural overlap and generated a prediction error on completely novel chemical shifts. 

Our core belief is that information that is provided to the public should be shared with the best interest of quality science in mind. As a result when we publish prediction accuracy numbers, we strive to produce numbers that best reflect the performance of the prediction algorithms in a way that is relevant to an end user. In our original study we produced an error of 1.59 ppm for the entire NMRShiftDB. In order to produce a number that is a true reflection of the prediction algorithms, we always exclude equal codes from the database of structures when doing an accuracy assessment. In addition to this, we also clearly acknowledged that there was a significant overlap between our prediction database and the NMRShiftDB (57%). As a result we produced a number for a subset of the NMRShiftDB (43%) that excluded all of this overlap (1.74 ppm). This is what we believe was a true prediction validation on novel chemical shifts, and would be similar to what a user could expect to get for novel structures and is therefore a relevant number for users to know.

So if Modgraph is unwilling to share the important details of this study regarding overlap with the public, we can only respond by providing the public with a prediction accuracy value that can fairly be compared to theirs. 

As a result, we have produced a prediction accuracy result that DOES NOT exclude identical HOSE codes. It is simply the result that we have generated by downloading the SDF file of the NMRShiftDB and running it through version 10 of ACD/CNMR Predictor, the same version that is available to customers today. 

The result of this study is an average deviation of 0.96 ppm. 

That is an incredible result, no doubt. But understand that it is not the most relevant result in practice. We have simply provided this new number of 0.96 ppm in order to provide our users and followers a number that they can accurately compare to the number of 1.40 ppm provided by Robien using Modgraph’s NMRPredict. 

In our original validation document, we freely shared the news that 57% of the structures in NMRShiftDB were also found in the ACD/Labs CNMR prediction database. We believe that the most important result to an end-user would be the performance of the predictor on the remaining 43% of the structures that are not present in ACD/Labs CNMR prediction database, hence the value of 1.74 ppm. 

I should also add that the major reason we conduct these validation studies on independent datasets is to challenge our own technology. We are always very eager to test our predictors on new data because it provides us with an unbiased evaluation of the performance of our software. In addition, many times, these validations help us improve our software. As a matter of fact, during the evaluation of the NMRShiftDB we have identified areas within our software in which prediction can be improved. We have already made changes in preparation for version 11 and have achieved significantly better prediction accuracy numbers than the 0.96 ppm I have presented today. I will however not share these numbers to the public since these improvements will not be available until the release of Version 11 in the fall. 

For now: 

An average deviation of 0.96 ppm on the entire NMRShiftDB database (214,136 chemical shifts). 

An average deviation of 1.74 ppm on the subset of shifts in the NMRShiftDB database that are not represented in ACD/CNMR Predictor (92,927 chemical shifts). 

If the overlap between the two databases is not acknowledged, prediction accuracy results cannot be trusted. For this reason, I will not post any other prediction accuracy results on this blog, either from ACD/Labs or any other software company or research group, unless the overlap with the independent dataset (NMRShiftDB) is clearly acknowledged. 

However, in the end, I agree with Jeff Seymour from Modgraph, Consultants Ltd. when he says: 

“Of course customers are really interested in how accurately a prediction program can predict THEIR molecules - not a collection of external data such as NMRShiftDB.”

Go ahead and evaluate the software packages on your own.  Contact me if you would like to conduct an evaluation of ACD/CNMR Predictor. 

The Purgatory Database

So the NMRShiftDB and our comparison to Modgraph and CSEARCH has caused quite a stir.

Currently we are still awaiting Modgraph's response on the following two points:

1. What is the overlap between NMRShiftDB and Modgraph’s NMR prediction databases? Further, with several different database sources how much duplication of data exists across the Modgraph databases?
2. Once that overlap is removed from the dataset, what is the final deviation produced by NMRPredict?

Modgraph's website currently claims the following two statements:

"Modgraph NMRPredict shows itself to be the most accurate carbon 13 NMR predictor in an independent evaluation!"

"NMRPredict now contains the world's largest collection of NMR data - over 424,632 records in total!" (>345,000 are 13C records)

Let me tackle each statement one by one:

On accuracy:

Well I think we all know where we stand. Modgraph claims a greater accuracy than us, but the question remains about their overlap and the validity of their most recent results. Meanwhile, based on our evaluation of the NMRShiftDB we have worked on tweaking our algorithms for version 11 and I will have some new results to report soon. Stay Tuned.

On "the world's largest collection of NMR data" (>345,000 records of 13C NMR data):

If you go to Modgraph's website here, you will see how they handle predictions when there is an exact match in the database. You may also notice on the right pane that there are MANY exact matching records in the database. In fact, it turns out that there are 25 matching records in the database for Yohimbine within Modgraph NMRPredict version 3.8.22. However, when you drill down into the database, you will discover that these aren't exact matches at all, just different stereoisomers of Yohimbine, as well as from different sources with different experimental conditions.

But it did give us an idea on a quick experiment to run.

I ran a prediction for Benzene in Modgraph NMRPredict Version 3.8.22. The prediction was very accurate. It turns out that there are 56 exact structure matches for benzene in their prediction database. 56! And it appears to me that these matches are all logged as DIFFERENT database records.

Off the top of my head I ran a few more predictions:

38 exact structure matches for toluene

32 exact matches for acetone

39 exact matches for catechin

First of all, let me stress that it is very important to have different hits in the database from different sources. For example, entering different solvent data helps achieve solvent specific prediction. But my question is, are these indeed reported as different records as it appears? If so, this simply bloats the number of records in the database and can be perceived as a very misleading number to quote to the public when talking about NMR prediction. Is quoting the number of records really a useful statistic to measure?

I think that a user would be more interested in the number of unique compounds in a database.

In Modgraph’s defense, these records all seem to be different in some way. They have different references, are in different solvents, temperatures, concentrations, etc. Fair enough. But it should be known that in our software, all this information is added in one record (click image to enlarge):

15 different sources for the CNMR assignments of benzene. Different solvents, conditions, sources, references, etc. But keep in mind, all this information is stored in and counted as one record!

We currently have over 186,000 records in our CNMR database. But these represent unique structures in the database. If we have more than one source for a chemical structure (and there are many, many instances) it all goes in the same record. 

It seems like a good opportunity for me to explain exactly how ACD/Labs builds prediction databases and why we do not import databases from external sources.

Our database team culls the most recent literature, draws chemical structures, and tabulates NMR assignments in what we like to call the purgatory database. Once entered, these data are screened and prioritized in a way that ensures that the chemical shifts that have the most promise to improve your predictions are on the top of the list. These records then sit in limbo until one of our resident NMR spectroscopists is able to quality check, hence the name purgatory.

We are often asked whether or not we would be willing to import data from some other database such as SpecInfo, The Human Metabalome Project, NMRShiftDB, Aldrich, etc.

Our answer is generally no. Our response is not because we question the quality or validity of the data. It is simply because we have a purgatory database that literally consists of hundreds of thousands of compounds that are waiting for quality checking. This is data coming from the most recent literature that we currently do not have in our database that is prioritized in a way that is most beneficial to the structural diversity of our databases.

This is the standard we have put in place and we truly believe it is the best way for us to ultimately improve the prediction in our NMR software. If we didn't hold quality of science at a high standard within our company, we could double the size of the database tomorrow by importing the entire purgatory database as-is.

In closing, two conclusions from me:

1. I believe that we have created the best practice to follow for the validation of prediction accuracy on an independent data set. Modgraph has apparently chosen not to follow this practice and have quoted very questionable numbers, in my opinion. If they aren't going to produce those numbers in a consistent way, this is not a comparison and those numbers are meaningless with respect to ours. Again, we are open to suggestions on a better way or a different method. But those haven't come either. Database issues aside, it's the algorithms behind the predictions that are absolutely crucial. This is why the issue of overlap is so important. When ALL overlap is acknowledged and removed, it represents a fair measurement of algorithm performance.
2. Modgraph currently has the largest collection of records in the world. But what's important, the number of records, or the number of unique structures? For a database used to generate NMR predictions, the structural diversity of the database is key. While a large database of experimental values will likely improve NMR predictions, it has to be structurally diverse to be useful in the real world.

EDIT: This conversation has continued in the following entries (in order):

http://acdlabs.typepad.com/my_weblog/2007/07/final-note-on-t.html

 

Note from an NMRShiftDB Contributor

For those of you still interested in following the NMRShiftDB debate, one of their contributors Egon Willighagen (over 900 spectra, Bravo!) sounds off on his blog.

You can read his comments here:

http://chem-bla-ics.blogspot.com/2007/06/quality-of-chemical-database.html

EDIT: This conversation has continued in the following entries (in order):

http://acdlabs.typepad.com/my_weblog/2007/06/the_purgatory_d.html

http://acdlabs.typepad.com/my_weblog/2007/07/final-note-on-t.html

Use ACD/2D NMR Processor

There are other applications available, no doubt. You can also use TopSpin, VnmrJ, Delta, NUTS, Mestre-C, iNMR (if you are a Mac user)

But I am here to tell you that you should use ACD/2D NMR Processor...and it's not solely because I work for ACD/Labs.

When deciding which application to use, there are many things to consider. Of course capabilities and features are probably close to the top of the list and I am sure level of comfort and ease of use are very valuable to a user. But one question I think that you absolutely need to consider is, "how far is this software solution going to take me?"

Perhaps you use a different software right now. You've used it for over 20 years, you know all the shortcuts, and it is very easy to use. Even if there is something better out there...it isn't worth the dip you will have to face to learn a new interface and workflow.

I am not going to lie. There is a learning curve with 2D NMR Processor. But that learning curve generally comes because you are used to using another software. This learning curve does not mean the software is difficult to use. In my opinion it is dead easy. Many NEW users (when ACD/2D NMR Processor is their first encounter with NMR software) tell us that the software is incredibly easy to use. I'm a PC guy...so when I try and use my friend's Mac there is a learning curve just because I am used to doing things the PC way.

So the purpose of this blog entry is to convince you that it is worthwhile to get through the learning curve or Dip. If you use ACD/2D NMR Processor, it's worthwhile to get through the dip because I believe the rewards you reap and the opportunities it opens up are something that no other NMR software can provide.

Sure ACD/2D NMR Processor has it's own benefits over other software packages by itself:

• Support of Indirect Covariance
• The ability to electronically assign multiplets and correlations to atoms in a structure
• Vendor-neutral support
• Flexible reporting tools

NMR processing is a means to an end.

So certainly there are many different softwares that can do NMR processing. But NMR processing is only the beginning. Generally after running an experiment and processing the data, you have to interpret the spectra and report your results. What more can software provide?

Once ACD/2D NMR Processor is used exclusively you have the opportunity to use NMR prediction in a completely integrated way. That is, you can attach a structure and get immediate feedback on the consistency  between the  experimental spectrum and the structure.

Every time you process a data set in the software, you can very easily database all that information.

If you run a significant amount of 2D NMR experiments, my guess is that you do many structure elucidations. If that's the case, getting comfortable with 2D NMR Processor will in turn provide you with access to comfortably use ACD/Structure Elucidator.

Which brings me to my final point. ACD/Structure Elucidator is a terrific tool. It works very, very well when good quality data is obtained and prepared. However, it is not necessary to use Structure Elucidator for every dataset you encounter. Some of the problems you will face will be solvable in a matter of minutes. You may be able to solve the problem faster than it takes you to prepare the data and run it through the software. But it's those oh so difficult ones that you face once a month, three times a year, etc. where Structure Elucidator delivers. However, the point is, if you adopt 2D NMR processor now, and process all your data in it on the occasion that you hit the tough one, it's easy. Your data is already completely worked up and ready for Structure Elucidator if necessary.

Structure Elucidator has a learning curve associated with it as well. However, the learning curve with Structure Elucidator in fact, almost exclusively surrounds the transition an NMR spectroscopist has to make from their own processing software to ACD/Labs processing software. So if you adopt ACD/2D NMR Processor, get through "the dip" and use it on a regular basis, you now have the opportunity to explore so many of the other benefits that ACD/Labs provides to NMR spectroscopists, chemists, etc.

Steve Coombes talked about this very topic at our 5th Annual UK User's Meeting and sums it up much nicer than I have here. His presentation is available here.

The take home message for me, is that Structure Elucidator is not a black box where you input data and receive a structure as output. It's a software package that can complement a spectroscopist's elucidation workflow. I think Steve shows many of the additional benefits in his presentation.

Robien’s and Modgraph’s Response on NMR Prediction Validation

Wolfgang Robien and Modgraph Consultants, Ltd. have responded to my latest comments and findings on the evaluation of our NMR predictions vs. the NMRShiftDB. 

Let me first begin by saying that over the last 24 hours, ACD/Labs was informed by Modgraph representation that my claims on this blog regarding CSEARCH and Modgraph’s NMRPredict were not clear. Since this time it has since been explained that while Wolfgang’s CSEARCH program is indeed the basis of NMRPredict, the commercial product includes different underlying databases and datafiles as well as additional enhancements, such as 'auto-stereo recognition', different utilization of solvent-dependent predictions and 'BEST selection’, which MAY not be available in CSEARCH. 

It turns out that Robien’s average deviation of 2.22 ppm was based on his CSEARCH algorithm. So you may notice that in my previous entries, I have scratched out any inaccurate product mentions of NMRPredict. If you read Robien’s original posting you might be confused in the same way that I was since in Robien’s initial study (the study my response was based on) he referred to CSEARCH and NMRPredict as one and the same. As you can see in the comparison table, he is directly comparing CSEARCH/NMRPredict to NMRSHiftDB.

Additional comments from Robien:

“A few facts about CSEARCH/NMRPredict and NMRShiftDB" (notice he is linking to the NMRPredict product description on the Modgraph website!!!)

"My cooperation partners, where either CSEARCH-spectra are available or CSEARCH-technology has been implemented, also within CSEARCH/NMRPredict other collection like SPECINFO has been implemented"

Keep in mind, this is my own personal blog and I have taken on a personal responsibility to inform the public on news and events in the world of NMR software. I see the above text as written by Robien and make my own conclusions. I took what Robien wrote and proceeded with my thoughts. It was not clear in Wolfgang’s original article that he was not using Modgraph’s NMRPredict. If he had mentioned it, then perhaps it would have been less confusing. But there are several mentions and links to NMRPredict and Modgraph throughout his web pages. 

ACD/Labs has also made the appropriate revisions to the official validation document that I posted on my blog. The revised version is here. 

I want to stress that while this document mistakenly claimed a comparison with Modgraph’s NMRPredict, I believe that this document still represents a valid practice for the evaluation of prediction accuracy. Transparency within this validation study was the goal and I truly believe that this study represents a fair and unbiased way of properly performing a prediction accuracy validation on an independent data set. 

With that in mind, Robien has since collaborated with Modgraph Consultants, LTD. and produced what they believe is a TRUE prediction accuracy evaluation of the NMRPredict product. 

His findings reveal an overall average deviation of 1.40 ppm (compare to the ACD/CNMR Predictor deviation of 1.59 ppm).

So has Modgraph now definitely proven that indeed it is, “the most accurate carbon 13 NMR predictor in an independent evaluation?” 

The numbers suggest so, but please pay attention to the details within the study.   

“For this evaluation the combined databases from CSEARCH and SPECINFO holding a total of 345,308 reference spectra were used. Based on this higher number of reference spectra a somewhat higher structural overlap between our databases and the NMRShiftDB-test data has been detected.” 

In our document, we clearly state the structural overlap (57%) between our prediction database and the NMRShiftDB. This number is not provided by Modgraph. This number is extremely important to know, is it not? They mention “somewhat higher structural overlap” but what is it? 

They base their final results on the following: 

“In order to compensate for this, we have recalculated our overall average deviation of 1.40 ppm using the lower structural overlap as detected by ACD. The value of 1.40 ppm corresponds to 92,927 known carbon environments and 121,209 unknown carbon environments - without this compensation our overall average deviation would be slightly better, but a comparison with ACD's results would be impossible.” 

I do not believe that this is a very scientific way to produce these results. It is certainly not a valid practice to compare directly with the results produced by ACD/CNMR Predictor. Which compounds (or chemical shifts) did they remove for their analysis? 

For example, if they have 80% overlap between their database and the NMRShiftDB, which chemical shifts do they remove from their analysis to get to an overlap of 57%? (as they have to produce their final deviation of 1.40 ppm). 

Why not state what the overlap with NMRShiftDB is, and then perform a validation on completely novel chemical shifts like we have done? Transparency and valid science was our priority during the creation of the validation document. As a result we stated the database overlap up front, and conducted two separate studies to inform the public of our performance under both circumstances. 

So before we can make a final decision on performance, I think Modgraph needs to make very clear the following: 

1. What is the overlap between NMRShiftDB and Modgraph’s NMR prediction databases? Further, with several different database sources how much duplication of data exists across the databases and within the entire package?

2. Once that overlap is removed from the dataset, what is the final deviation produced by NMRPredict?

I think this information needs to be made very clear from Modgraph before they can claim to be, “the most accurate carbon 13 NMR predictor in an independent evaluation?”

We worked very hard to create a best practice for the validation of prediction accuracy on an independent data set. It is disappointing to see that the example was not followed in an attempt here by Modgraph to compare to our results.  Perhaps there is a better way. We are open for suggestions.

I hope that we are able to compare the performance in a truly fair and reliable way as to provide the public with the correct information!  

Finally, Jeff Seymour, Marketing Manager for Modgraph Consultants, Ltd. has issued a comment on my blog here:

http://acdlabs.typepad.com/my_weblog/2007/05/nmrshiftdb_acdl.html#comment-71790940

I want to highlight one comment specifically:

The average deviation in NMRPredict was 1.40 ppm compared to an average deviation of 1.59 ppm in ACD/CNMR Predictor version 10.5, already compensating for your somewhat smaller structural overlap.          

Again, I ask…what is the structural overlap and once you compensate for that and remove it from the test set, what is your average deviation? 

These numbers have NOT been published, and therefore this statement is dubious for the time being.

More information from Robien’s finding can be found via the link below:

http://nmrpredict.orc.univie.ac.at/csearchlite/Robien2Ryan_May31_2007.html

Modgraph’s announcement is here:

http://www.modgraph.co.uk/product_nmr_shiftdb.htm

EDIT: This conversation has continued in the following entries (in order):

http://acdlabs.typepad.com/my_weblog/2007/06/note-from-an-nm.html

http://acdlabs.typepad.com/my_weblog/2007/06/the_purgatory_d.html

http://acdlabs.typepad.com/my_weblog/2007/07/final-note-on-t.html

 

More Dialogue on NMRShiftDB Debate

Peter Murray-Rust and Tony Williams have added their two cents to this debate on their respective blogs.

Peter provides a great justification on providing open access to scientific information:

In the case of NMRShiftDB I am firmly of the opinion that it leads the way in opening access to scientific information. If the community wishes it can use it as a growing point to develop more and better data. If they don’t, they will continue to use existing non-open systems (or in most cases not use anything at all).

Tony goes into more depth about the details of the ACD/Labs validation study and specifically addresses Wolfgang's claim on large error:

What I do NOT mean is that a chemical shift at 120ppm is predicted to be at 80ppm and therefore there is a large error. No, the chemical shift at 120ppm could be experimentally correct but the prediction algorithm could fail to predict it correctly.

What I DO mean is that an assignment of a particular nucleus to 120ppm may be entered into the database but the ACTUAL shift should be 12ppm….that additional zero just showing up as an error during the submission process. So, the errors I am pointing to are those of incorrectly drawn structures, mis-assignments, transcription errors and other potential sources of error. My estimates refer to the number of significant assignment or structural errors that were glaringly incorrect and I was subjectively thinking of situations where the difference between the actual experimental shift value and the one assigned to nucleus was >20ppm….this does not mean that mis-assignments of even 1ppm are any less importance, just not necessarily as easy to detect and not part of my subjective criteria.

At present, the data have been examined in more detail and I believe I overestimated…a report of potential glaring errors has been returned to Christoph for him to examine and make changes to the database as he sees fit. Glaring errors are less than 250 in number based on my subjective criteria. Again, this does not mean that there aren’t hundreds or thousands of errors buried in the data…they are not obvious errors and require more manual examination.

Make sure you check out their blogs.

EDIT: This conversation has continued in the following entries (in order):

http://acdlabs.typepad.com/my_weblog/2007/06/robiens_and_mod.html

http://acdlabs.typepad.com/my_weblog/2007/06/note-from-an-nm.html

http://acdlabs.typepad.com/my_weblog/2007/06/the_purgatory_d.html