Too many people think "ums" are a sin, and let's include y'know like" as well!
Sure, in an ideal world, we would all speak fluently.
On the other hand, "ums" are part of conversational language, sprinkled liberally, even strategically. Your audience will not notice them, probably, forgive them if they do -
only if you are using WAAAY too many will they notice, and maybe wonder why, and be a little distracted, especially if our speech is not resonating in some way.
I usually advise that It's a risk worth considering in your rehearsal and practice times.
Start by listening to others - sports commentators, interviewees on television, anyone speaking publicly.
Listen for their fillers then you will learn to listen for your own.