Congratulations on an ambitious program!! At 56, I admire your determination.
Many years ago I read a letter similar to yours in the Ann Landers advice column. A man of about your age wanted to become a physician, but he lamented that, by the time he finally got his M.D., he'd be 50. Ann's simple reply was, "And how old will you be in 8 years if you don't become a doctor?"
My humble opinion is this: Maybe you will find it hard to get work as a 50 year old astrophycisist, but without the PhD, you'll find it impossible.
What will your "fallback" position be if you give up the program now? Go back to banking?
As long as you can afford it, I say go for it. You will never never never look back with regret on the degree that you earned, but you may well rue the day that you gave up your dream.