I’ve definitely gone down the same route as you - not reading books by straight, white men so I could give voices to POCs, women, the LGBT+ community. But then I got into a dilemma when there's a highly recommended read out there written by a man and I'm like - hmm am I going to miss out on this just because of my no men rule? I'm at a point now where I'm very conscious of the voices I'm reading and making sure I read a diverse range instead of rejecting a work on the basis of the author. But you're right - everyone should try it at least for a bit!
Two books I've read that have made me glad to "break" my rule are The Midnight Library and The House in the Cerulean Sea - wonderful reads and I've added those authors to my exception list for now.