Hi ninaoki
I believe using the Lab approach with appropriate lighting is the correct and I believe you are on the right path with the correct math.
Better matches involve the correct numbers for the sample teeth and correct lighting/preparation for the actual patient.
Other then that, for product, you need to test your product against a large variety of patients and see if the visual match with the samples is the same as you workflow. When not, it means you need to tune up your product.
In making selections for analysis, avoiding specular areas is important. Also, if you use a PS tool that uses the tolerance option, it has inherent limitations as such a tool selects on an RGB math equation that do not exactly reflect what you see with you eyes (e.g. it is color data based not luminosity based. That means the blue component of RGB in selecting an area gets too much weight. Those are the fine tuning nuances that come with product development that are beyond the basic pointers from forum posting in my opinion.
If you need more help than just the basic pointers for you product in PS or Excel, it is probably an indicator you need to hire an expert to aid in achieving a quality product result as well as advice/course correction when you hit the inevitable bumps in the road during product development, testing, deployment, maintenance, and support.
Hoped that advice in the thread helped get you on your way and best wishes.
John Wheeler