Many, MANY years ago i did something very similar on a real ST. See my old (german, google will help you) page at
http://www.harbaum.org/till/atari/multisync.html
This also includes links to schematics. This does:
* combine mono and color video signals into one rgb signal
* use a GAL to detect the video mode by measuring the hsync length and output color or mono based on this
* connect the mono sense pin with the gpio out on the video connector
Up to this point the whole setup behaves exactly like one of those old external "monitor switch boxes" that you could use to attach both a color and a mono screen at the same time and switch by sofware which would cause the ST to reboot with the new resolution.
But you could also use the "mono is active" signal generated by the GAL and feed that into the mono detect input on the video port. Then you could actually switch between all three video modes without the ST resetting/rebooting in between.