Colour classification

Hi guys I just need a step to step assistance on this one.
I want to deviate from a model i trained to classify types of soil by only the user using a photo taken in the app but now it might be a disadvantage to my research study.
So I want to go to a colour classification system that will classify the types of soil using their respective colours ( Munsell soil colour chart ).

But now I’m not sure should I trained the model based on pictures and say my headings for example red /brown /yellow /grey/ black and train it like that to classify the respective soils types.
It’s quite confusing guys but I need your help