Tell them to mind their own business, then get back to growing! You're a free man and it's your beard after all!