I agree - go for it!

If you can, then why not?
And even if it doesn't lead to work in that field, then it will still enrich your life/knowledge base.
(I did my Physiscs degree many years ago, and don't use it in my work at all - but I'm still happy that I did it)