Are you using the gameboy viewer from within Rockbox? If so, then due to the relatively low power of the CPU (and the lack of documentation on the broadcom chip) and the size of the screen, the performance is not going to be great for all but the simplest games.
If you're using iBoy in iPod Linux then you're right, we can't help with that!