Hey, I'm really glad to hear you're making progress!
Yup, it's definitely trial and error, a lot of back and forth, and it can be a pretty big hassle. I don't blame you at all for being frustrated! Not everyone would have put in the time and been this patient, so I really commend you for that.
So...yes, having the cross-over on the subwoofer set to "in" instead of "out" would have been part of the problem. Basically, the cross-over is not a "brick wall". It is a filter that makes the sound get quieter and quieter above the frequency that is selected as the cross-over for the subwoofer. When you have the cross-over on the subwoofer set to "in", then it is applying its own filter ON TOP of the filter that the receiver is applying. So you get a super fast drop off when you have BOTH filters being applied. With the cross-over switch on the subwoofer set to "out", you only get one filter being applied - the one inside the Denon receiver. And that's the way it should be. Only one filter
Having a big mess of cables is also very common, and it can cause problems because you get different signals interfering with each other. Good cables have a nice, thick shield around them so that they block out any stray signals from other cables, wires or signal travelling through the air (radio, WiFi, etc.). Speaker wire typically does not use a shield because the signal going through a speaker wire is quite high in power. It is more likely to CAUSE a problem in other cables than to pick up noise FROM other cables itself. That said, actual electrical power cables are carrying the highest power of all, so if you have a speaker wire right on top of a power cord, you can often get a loud hum in the speaker, since the speaker wire is picking up the 60Hz hum from the power cord.
16 AWG speaker wire should be fine. You're in a small room, so you don't have any super lengthy runs of speaker wire. You might want to consider getting some new cables and taking a bit of time to better manage where all of your cables are placed though. If you go to
monoprice.com you can get really good, nicely shielded cables of every type for very low prices. I don't know for sure, but I'm thinking you might have a few very cheap, poorly shielded cables in your setup. I honestly still wonder about the subwoofer cable you are using because some of the problems you described before really, really sound like a cable defect problem. So maybe just a new
subwoofer cable will help you - although I can't be sure. They're very inexpensive from monoprice though, so it can't hurt
If your room is actually 15' x 15', that is definitely part of the problem! If you are familiar with any type of resonance, you will understand about standing waves. At certain frequencies, the sound waves will come out of the subwoofer and be at just the perfect wavelength where they propagate out from the subwoofer, reach a wall, bounce back off of that wall, and perfectly "double up" right back on top of the original wave. When you have two dimensions of your room being exactly the same - like 15' x 15', you end up with the exact same standing wave in two directions, which makes it even worse! That would explain why you hear such a loud thumping at certain frequencies and then everything else is much, much quieter. Unfortunately, there isn't really anything you can do to correct that, other than get a second subwoofer. And even that won't really help very much if your room is truly a square. About the best thing you could do would be to position the subwoofer as close to your seat as possible.
Putting the subwoofer into sealed mode - like you did - will help because it's in the lower frequencies where you get the most standing waves and build up of resonance in a small room. So limiting the very low bass output will help, and using the sealed mode does that.
If there is any chance of rearranging your room in the future, that's something you'll want to try. For now, I don't really know where else in the room you could possibly put the subwoofer. But you are for sure getting some standing wave problems. Like I say, if at all possible, just get the sub as close to your seat as you can.
I'm thinking you maybe didn't get the
full user manual for the Denon AVR-1712. They often only pack in a "quick start guide" in the box these days. So click that link I just made. That's the full user manual from Denon's website, and it's anything but short!

You might find some additional help in there.
Here's also the
full user manual for the HSU VTF-2 MK4. It's not nearly as long, but you might have just got a "quick start guide" in the box with the subwoofer as well. Most manufacturers seem to only be putting the full user manuals on their websites these days
Finally, it definitely sounds as though you got used to a WAY higher cross-over frequency with your old computer speakers, as expected

I would encourage you to run that THX Optimizer bass sweep and get as smooth and even a sound, all the way from 200Hz down to 20Hz as you can. And then just let yourself get used to the far more accurate sound of the bass.
It can be tough. Once you're used to a certain sound, naturally, you just come to expect it and it just sounds "right" to you. Folks who are really into car audio are often this way. They hear a genuinely accurate sound system and they think the bass is just way too quiet and nothing has the slam or impact that they are expecting.
Getting used to accurate sound reproduction is a lot like getting used to an accurate picture on a TV. The accurate picture often looks dull and too dim to a lot of people because they've been used to a much brighter, overblown image that so often comes right out of the box without any adjustment. But just like the more accurate picture, once you live with it for a while, you start to notice all the details that you could never see or hear before. All the stuff that was masked by the overblown image or sound.
I checked up on the Boston A25 speakers a little bit more (haven't heard them myself), and they do seem to be described as being a bit bass shy and not having a lot of impact or output in the bass. So a higher cross-over frequency of around 100 or 120Hz does actually seem about right. I think what you're finding with the 150Hz cross-over is probably closer to what you were used to with the computer speakers, but I highly doubt it's more accurate, or what the bass should actually sound like, to be honest
Again, use that THX Optimizer bass sweep. It's the easiest and best tool to quickly hear what's going on with your bass and what's going on in the transition between your speakers and your subwoofer. Try a bunch of different cross-over frequencies and run the THX bass sweep each time to hear the difference. Again, you're aiming for the smoothest, most even-sounding sweep from top to bottom. It should all sound exactly the same loudness throughout and be seamless from top to bottom.
I think you're on your way though! So maybe try some new cables from monoprice. Definitely try to clean up your cable mess. Don't have any cables or wires running right on top of or parallel with power cords. If a cable or wire needs to go over a power cord, run them at right angles to one another. Or just get as much space between them as possible.
You can read the full user manuals now. There might be some helpful info in them for you.
See if there's any way to try some different subwoofer positions, especially anything that puts the sub close to your listening position.
I think the sealed option is probably a good choice for your room. Make sure to keep the cross-over switch set to "out", not "in".
And get a THX labelled DVD or Blu-ray and use that THX Optimizer bass sweep! It'll be really easy to hear what the bass is doing in your room that way. Once you've got it as smooth and even and seamless as possible, just give yourself some time to get used to the more accurate bass. You might not notice the subwoofer as much as you used to with your old computer speakers. But that's as it should be! It should all sound seamless - as if those little A25 speakers were huge tower that can play right down to 20Hz all on their own! That's the illusion you should be after. And once you let yourself get used to it, you'll start to notice all the detail that was missing and just how overblown most other systems - and your old system - really are/were!