If you feel you are going to struggle with math, you may discover much frustration with the technical side of loudspeaker design. Then again, there are plenty of loudspeakers that have been developed with "know-how" as opposed to PhD level calculus.
Nonetheless, you may find that your niche is more on the subjective side - the "art" side - wherein loudspeaker technical specs may not be as important as putting a good sounding system together based on your ears and experience. Along those lines, you might want to consider getting involved with systems design as opposed to loudspeaker design. It will still be a prerequisite (IMO) to be well-versed in the basic physics and math, but you don't necessarily need an engineering degree to put a wonderful sounding audio system together.
(As Don Davis said, "In audio and acoustics the fundamentals are not difficult; the physics are." Naturally, this is a double-edged sword...)
If you're going to consider formal education or training, you might look into professional training programs as opposed to a university degree. There are myriad options available in the US (NSCA, Syn-Aud-Con, CEDIA, etc.) - where are you located? There are also degree programs that are not as intense as, say, a EE degree from MIT. Again, which of these would be most appropriate may come down to where you are, or where you're willing to relocate to.
HTH.