No, it is possible. The rockbox bootloader can load iPodLinux.
Simply rename the kernel .bin file to linux.bin, put it in your fat32 partition, and create an ext2 or ext3 partition as your third partition with the linux userland in it.
Then, when you boot if you hold Play, it'll load Linux 'stead of Rockbox.