Since 3.12 there is a new feature: Ability to select folders for the database to scan instead of the whole file system, in the database settings.
But I don't think it lets you exclude single tracks in a folder where the other tracks should be added to the database.
So put your audiobooks and podcasts in dedicated folders and select only your "music" (or whatever you call it) folder to be scanned.