Personally I would stick with Lab since the color numbers represent a color whereas RGB numbers have to be paired to a specific RGB color space to know the color.
Also, this is only a good as the image colors representing the actual colors since getting a good calibration/match is an issue.
You also have to decide if you are looking just for a color/chroma etc match or if you want to include luminosity as well.
If you want to compare all three L,A,B components then here is the formula I would used to get the RMS distance of an image sample compared to a single palette sample
Li = L of the image sample
Ai = A component of the image sample
B = B component of the image sample
Lp = L component of palette
Ap = A component of palette
Bp = B component of palette
Distance = Square Root of ((Li-Lp)^2 + (Ai-Ap)^2 + (Bi-Bp)^2)
Do the above calculation for all palette values against the single image sample value and take the minimum of all calculations to have your closest match.
The calculation should be the same as ∆E by the way.
If you want to leave out the L part of the matching test, just leave out the L portion of the equation.
Hope that helps.
