My strong suggestion is that you work on the photography, particularly, the lighting, before you do anything else. It will make the processing vastly easier. With the proper lighting, you could achieve vastly better surface relief in the specimen. You want the light to be coming from all sides (ie, 360 degrees in azimuth), but just raking over the surface. Have the lowest angle of light be around 10 or 20 degrees, and the highest angle (ie, furthest above the surface) be about twice the minimum angle (ie, 20 to 40 degrees). The exact angles will depend on the particular sample you are photographing. If you know what a conventional light tent looks like, think of constructing one that is in the shape of a short cylinder where the height is only around half the diameter. The lighting is vastly more important than what camera you use, and it gives a good starting point for postprocessing.
Here is a quick example I threw together to illustrate the benefits of proper lighting. I didn't have a fossil, so instead, I used something of similar size and with a lot of surface relief, an old kitchen sponge.
First is a shot with all the light coming from near the camera. Note that the surface texture is dramatically suppressed.
01
Next is a shot with window light coming from one side at around the same height as the sample and a simple white reflector ( a piece xerox paper) on the opposite side to fill in the shadows. Now, the surface texture looks realistic.
02
BTW, both of the above photos were taken with my old iPhone, showing that you don't need a fancy camera to do this sort of photography. I didn't take any particular care to match the colors or brightness.
If you don't want to go to the trouble of making a lighting setup, but you have a flatbed scanner, you can use it instead of a camera to give good surface relief. It's not quite as good as a proper lighting setup, but it will certainly do the job:
03
More later on how to postprocess such images.
Cheers,
Tom M