Attack on planet - no battle - all ship lost! - please help

battleships will do that when an invader ship moves on a planet. what I do is manually fly some battleships past the planet with the invader
to draw fire. To get the WPs usually means glassing the plant or close to it. Why the destroyed report I have no idea, but I play balance mod ?

Don't rely on a single troop transport for battle, especially not a frigate.

My suggestions for a troop transport.
Biggest ship possible that can mount 12 engines. (up to cruiser size)
Basics (bridge, etc)
12 engines.
Maybe 20% of space for cargo bays
A few PD to deal with seekers and units (maybe up to 10-20% of space? A mix is good, minimal 1 Meson, 1 AP and 1 Bomblet is an effective mix).
A few weapons (to be on the safe side, nothing major/fancy)
Combat Sensors so PDs can shoot seekers/units better
ECM to make ship harder to hit.
Some shields and plenty of armor, the more the merrier, use at least 1 phased shield if you have them.
Load up plenty of troops, plus a few fighters (maybe 20-30% of cargo space for fighters?).
Alternitive, don't use any weapons other than PD to allow for even more shields and armor. Basically want a sort of 'blockade runner' style ship.

Use at least 2 of such ships.
Their main goal is simply, live long enough to deliver their forces to the planet. Their survival is no longer an issue after they have deployed their forces, but it makes things easier if they do survive.

This is based on how I tend to defend planets, weapon platforms with a mixture of weapons, including a few weapons with the largest mounts possible, smaller weapons, PDs, and at least 500kt worth of missiles or torpedoes, backed up by fighters and troops on the planet to further complicate enemy invasions. Tends to make it hard for troop transports to get close safely.
Which brings the following phrase to mind...
"No Tresspassing. Violators will be shot. Survivors will be shot again."

Another factor is is when you actually get troops landed.
Militia.
The larger the population, the more militia will show up.
I don't think this is much of a fator. If I read the settings.txt file right, even a fully populated huge planet might only spawn a half dozen militia. Used to be alot more in older versions, iirc.

A side suggestion.
Bring along an extra transport or two (not for planet capture) with some weapon platforms, troops, fighters, and maybe also satellites and drones. Once a planet is captured, use the weapon platforms and units to garrison the planet, to hamper enemy attempts to recapture the planet, and also give them some peoblems if they try to simply counter attack to glass the planet.

Edit:
Almost forgot.

Were the Battleships confirmed destroyed?
As in there were none of the ships at/adjacent to the the sector of the planet you attacked?

Any chance that being 'spread out' at the border, with the troop transport destroyed, they turned around, and went after the planet to spread out (some turning more than others meaning longer until they accelerate to full speed), causing them to be spread out enough that the defenses wiped them out one at a time?

The strategies have a lot of factors that seem to trip over each other. I think someone made attempts to have ships 'soften up' enemy planets prior to invasion, just to have them apparently ignore some settings and simply glass the planet anyways.

As one person put it, damage on planets apparently works oddly.
Damage for the planet itself is based on population, which apparntly has no 'percentage' of damage, but is always 'max'.
So, with 4000 population, 2000 die, it doesn't view it as 2000/4000 (50% damage), but as 2000/2000 (0% damage), which means ships, even set to only do 10% damage or 90% damage will gleefully glass a planet.

So the question comes to be, how to get ships to attack the weapon platforms only, without the ships glassing the planet in the process?
Maybe slapping "all weapons gone" for damage amount onto everything relating to colonies under "target type an order settings"? (planet with colony, building, facility, weapon platform) and including "target type" under "targeting priority orders"?
Might work, might still glass the planet...
Unsure of how well that idea might work.