If it occurs at particular frequencies, which is what it sounds like was described, I would lean toward voice coil.
Visually, if the surround isn't ripped, I would not expect the cone as an issue. I have had drivers with surrounds that had a small tear that wasn't visible until you pushed on it do this though. Is it a foam, paper or rubber surround?