This was tried perhaps 10+ years ago by an experimenter in Australia. I'm pretty sure he used a fluxgate sensor (which adds dead metal to the coil) and found that it didn't really help because he could still detect iron deeper with the detector than the mag sensor could identify.
Since then, I don't know of much effort in placing a mag sensor in a metal detector coil (at the hobby level, not the geophysics level as Ryan discusses above). We talked about it at White's but never had the time to do it. Now, mag sensors are much more sensitive and super tiny so it may make sense. I've been experimenting with a 3-axis magnetoresistive sensor that is 1.6mm square. Currently I'm working on a regular magnetometer (gradiometer) and it works just fine, but I need to write the software calibration to make it user friendly.
A potential problem with a mag sensor, especially in a metal detector where it is unlikely to be used as a gradiometer, is that there needs to be a baseline for detection. The sensor is always immersed in a magnetic field no matter where you are and this varies globally and locally. Probably the easiest solution is to use it only as a ferrous check feature, where you move the coil 8 inches off the target, press a cal button, and then center the coil over the target.
There is still some danger in using this feature, as you may detect a non-ferrous coin that has deeper piece of iron under it that the metal detector does not see, but the mag check will see. And it causes you to move on.