but i have 10.13 and the new rockbox utility is for 10.15
Same question as above ... which version of Rockbox Utility are you using? Make sure you're using the latest version. In the past this error happened when using an old binary due to the website switching to SSL. Older versions won't work anymore.
If you're using the most recent version and still get an error: post the log (Help / System Trace) to pastebin or similar so people have a chance to figure what went wrong.
I also don't see a need to use a VM. If you us a VM I'd rather use a Windows or Linux VM, but the macOS binary should run on all version supported by the used Qt version, which would be 10.13 and up. Have you tried running it directly?
And as has been said: you can install everything manually. See the manual for details.