Yep, this is totally dependant on how many fonts you have installed. If you have a lot, it will bog down. If there are no changes to those fonts, it will scan faster after that, but every time you install a new font, it will have to rescan them all, causing the slowdown.
The solution is to minimize.