I have not heard of a way to modulate HDTV signals w/other video signals onto a single piece of coax. It is possible to do within a Crestron environment over a single piece of CAT-5. You can run 1 piece of CAT-5 to any room, use a breakout box, and have composite, s-video, and component video at that display location.
But, Crestron can get pricey to do this. I am not aware of Extron selling equipment that is a lot more affordable that can accomplish the same thing.
Traditionally, and most often, you actually run the cables you need for each video source to the room you need it in.
So, if you want composite video, s-video, and component video - that is 6 total pieces of RG-6 coax to each display. It is not terribly expensive, but is a lot of cable. I would also include a piece of cable for IR repeaters/control. I used 22/12 stranded wire instead of CAT-5 for my home because it is easier to work with.
There is no question in my mind that you quickly begin to lose quality as soon as you mutter the word: MODULATE
That is, you take the best possible video quality, compress it down to garbage, then wonder why it looks so bad when you decompress it at your television, when it looks so good right out of the box. Derrrr....
I ran 15 pieces of coax to my projector and to my plasma as well as having the option to run it upstairs to my master bedroom later on if I want to. This provides me with full PC, audio, component, s-video, and composite video connections if I want them at every location. I have some smaller displays that are only being setup with modulated video, but I acknowledge that is all they will ever display. Oh well.
2 * RG-6 + 2 * CAT5 is what is called a 'standard multimedia drop'. It gives you phone, ethernet, and 2 cable feeds at ONE location.
A. It's dumb - you really want your phone sitting on top of your TV?
B. It's dumb - you (specifically) aren't putting a cable box in every room.
Sounds like you need to sit down and talk to a consultant to get some things figured out.