First, the numbers in your spreadsheet do not match the numbers of the color sampler points in the image you provided. So that was a little confusing.
Responses:
1) I personally would measure both the 3D Lab and the 2D ab and the 1D L distances so you can see the information separated. You have embarked on a project where the differences in the luminosity and hues are closer to suble differences rather than clearly distinct. In such a case, the results will be very sensitive to the lighting and color accuracy of the original image relative to an absolute standard. That is not easy to do. You have already seen the issue of the specular highlights in you imges and I suspect matching up luminosities to a common standard is not easy as well.
Since the sample teeth are somewhat separated by Luminosity and Color, using the 2D and 1D distances might help.
In some respects, if when the shot of the tooth is taken, you had a reference tooth side by side with it, you might have some good info on how to adjust the image for the right color balance and luminosity. Just a thought.
2) The spreadsheet looks right to me other than the numbers don't match you picture example. The link to the formula is correct. To go to 2D you just eliminate one of the dimensions in the equation and for 1D just use one subtraction (no need to square and square root at that point)
3) Value, Hue and Chroma numbers can be calculated outside of Photoshop in Excel so not a must to have in Photoshop in my opinion
4) Since the spreadsheet you linked do not have the correct numbers from the image it is hard to comment. This gets back to having known and uniform lighting and color balance with relatively dry teeth I suspect so it would not be hard to off just due to those factors. Again, taking a picture with a reference tooth may be an idea.
If the number differences seem small, you can always sample with 16 bit values instead of 8 bit values and use the 16 bits numbers in the spreadsheet. I don't think it will change you answer in the spreadsheet you supplied to me though. It might make a difference differentiate between two close values in the table though. Having more digits in the number you use in Excel only helps if the color accuracy is spot on (key issue).
Note that you mention dark color. It is best to think of light and dark as Luminosity and brightness e.g. L channel and the color as the ab channels. In the image you provide, all three samples look very similar. That is not an issue of your sampling or calculations, that is just is what is in your picture. So if there is an issue, I suspect it is in how the image was shot/processed.
I suggest you try a number of samples before you conclude you have a problem.
----------------------
Hope the above information is helpful.
