Yes, it is possible.
What you do is your turn OFF HDMI-CEC, and you go buy a decent f'n universal remote control.
HDMI-CEC sucks, it doesn't read your mind, it has no idea what you want, and you can't change it or touch it or manipulate it to fit your standards. It is, one of the least thought out solutions that HDMI ever delivered, along with failing to include 2-channel stereo at all times.
You may use it the way it is, or you may get a remote that you can program to do exactly what you want.
Of course, you should never use your TV without using the decent speakers you purchased for it. People do this all the time, and it is their choice, but in no world does this ever make sense. Having a surround system on, at low volume should give better audio than the TV speakers ever will. We never, EVER, use the TV speakers, they just suck. Why not just get out of that habit and stick with your good stuff? Up to you of course.