Hi Ninanoki
Its not really a Photoshop question yet here are some pointers.
- The 3 points in 3D space create a triangle
- You already know how to calculate the length of each side of the triangle (just ∆E equation between two points)
- This link gives you all needed formulas for triangles:
http://www.ajdesigner.com/phptriangle/right_triangle_pythagorean_theorem_c.php
- In the scalene triangle section, given the length of the sides of the triangle you can calculate the angle between any two segments
- If either of the two angles between the two reference colors and your sample color is greater than 90 degrees, then the sample color is not between your reference colors
- The "height" or distance from the sample color to the line between the two reference colors is one of the equations given in the reference.
- From there, using the Pythagorean equation, you can use the calculated height and appropriate triangle side distance to calculate the distance from one of the reference colors to the ideal match point between the references
- The fraction of that distance from the reference color to the ideal color point, compared to the ∆E between the two reference colors is the same fractional point to pick between the L, a, and b components between the two reference colors to have the Lab coordinates of the closest match to your sample color point.
Those steps above could be combined into one single large equation yet thought stepping you through the geometry and providing the link to the equations would let you do the detailed work (i.e. teaching you to fish while not catching the fish for you).
Hope that does it for you.
PS - If you need a lot more you can always consider posting in the Fee for Service section:
https://www.photoshopgurus.com/forum/photoshop-freelance-work-fee-for-service-section-/