That's a pretty thick piece of MDF you're using for the crossover, which might be part of the reason for your frustration. I suggest Masonite next time; stiff, reasonably easy to work, and much thinner.
Pegboard is OK but just, it's a very lightly pressed material that doesn't have much density. Easy to use, but maybe not ideal once it gets inside the speaker and is subject to pressure forces. The holes would help, but I feel it flexes too much.
3M makes hot glues and other adhesives that are electronics-safe. Epoxies are fine but a bit permanent. Ordinary hot glues, adhesives and RTVs may not be electronics safe and may lead to corrosion in the future.
Your build is well thought out and your woodwork is more than adequate. Good luck to you, hopefully it meets your goals when finished.