I still have my 256GB SDXC upgrade that I use, and file copies have not been a copying issue (yet). Mostly AAC's so nothing too big, but on occasion I will put some FLAC files on it. I do generally use the dock for file copies, so be aware of that.
I have a Tarkan SD adapter btw. Only issue I've had with this setup is making sure that the bootloader files are copied early on. If I try to override them after 128GB+ is filled up, the bootloader fails. Rockbox files btw, can be overridden with no issues, it's just the bootloader files that have this issue (they don't really change anyway, unless you plan on compiling them yourself).
If you haven't already done so, perhaps try another USB cable, another USB port or try copying while in bootloader USB mode vs. Rockbox USB mode?