Compiling a regular build requires the build environment. Obviously, MacOS doesn't come with the rockbox build environment, so you can't do that without installing it.
The UI sim in linux is pretty simple, since it just uses GCC and a few libraries. Unfortunately, I don't know anything about MacOS, so I can't help you there. MIght want to search through the mailing list and see if anyone else has tried.