The easy solution is to treat them as 3D vector and calculate the distance between to points, but the problem is that not all colour models are perceptually correct, and RGB, while is good for computers calculations, gives more space to some colours than others, when you talk in perceived colour terms.

A better approximation is to convert your colours to a different colour model, and the calculate the distance. The is a nice explanation in this online calculator http://colormine.org/delta-e-calculator