Yes indeedy, they speak truth! I have built my own speakers and am building bigger ones from designs by Siegfried Linkwitz. It seems that I, like many other prospective speaker builders, thought that it would be a simple matter of finding the right size box and crossover points. Nothin' to it, right?
Ha! After reading on Linkwitz's site and in other places what is involved in designing a decent speaker it was quickly apparent that there's more to it than I'd be willing or able to learn! So, if DIY still appeals to you, stick to kits or to fully realized designs by genuine experts who have done the heavy mental lifting for you. That's what I did and am doing. Unless, of course, you'd like to independently pursue graduate-level studies in things like acoustics, filter theory, electrical engineering...
I have spent over a year on this stuff. I think I am an intelligent layperson, albeit math-impaired (and you'd better know
at least pre-calc from what I can discern). Compared to the people who design speakers for a living I am about at the level of a trained chimp...and unlikely to progress much further!
And as Jeff mentioned, these days there are enough really good speakers out there at moderate price points that DIY is no guarantee of getting a bargain. I and others pursue DIY only partly for the (hoped-for) savings and more for the fun of learning and building. For me, DIY is an excuse to buy cool tools!