Trudeau refuses to refund $20,000 speaking fee to New Brunswick charity