How to get rid of the "hard part" of a hat?!?

on the front part of the baseball hat in the inside, the stitching and embroidering "white stuff" is molded and is hard and is very hard to move! how do i get it to fit to my head instead of my head fitting to it?