Is there any possible way round the FAT32 2TB capacity limitation?

[Frankenpod]

I'm getting to the point where I might upgrade a flash-modded ipod to greater than 2Tb total storage.  I realise there are some big problems in doing that - am curious whether they are likely to be insurmountable.

I'm frankly confused about the whole topic.  I gather FAT32 can cope with larger partitions, by using larger cluster sizes, but the problem is with the limitations of MBR?  And, I gather Rockbox itself would need major work to use GPT?  (and it's not likely to happen for a long time, if ever).  Or would using different cluster sizes in itself create a problem?

There's also the question of whether Windows could see the disk properly in order to sync files to it.

How about using more than one partition?  Would that be a possible workaround?  (Is the limit on partition size or on total disk size?)

I'll most likely just carry on increasing the ipods I have towards 2TB (upgrading the smallest one, then making a new one with the cards removed, and so on...), but every time 1Tb cards come down in price I get tempted and wonder about the possibility of exceeding that limit.

[amachronic]

The 2 TB limit is on total disk size and solving that would require a lot of work. Adding partitions won't help. We can only address 32 bits worth of sectors (= 2^32), and at 512 bytes per sector that's 2 TB.

That said, I think the iPod 6g has 4k physical sectors so it might be able to "cheat" and go up to 16 TB in theory. I have no idea if that would work as expected with MBR / FAT32.

[Frankenpod]

Hmmm, OK.  Probably will settle for 2Tb max for the foreseeable future, then.  Really wasn't sure whether the limit applied to partition size or total disk size.  I guess the crucial issue is the use of MBR?

With cards increasing in capacity, I presume this will eventually become an issue for other players, not just modded ipods? (Always assuming there is any future for standalone players and native-RB anyway)

[Frankenpod]

I notice there's comments on IRC about some work being done on possibly supporting >2TiB.

I confess I don't know enough to understand even what that discussion is about, or what targets it applies to.

But in case it's of any use to know, just thought I'd mention that I've experimented with that limit  bit with the quad iFlash and 7th gen Ipod, and the most I could ever get to work was a combo of micro SD cards - one labelled "1TB" (which seems to actually be 1000GB rather than 1024GB) + 512GB + 512GB + 128GB.

Though nominally > 2TB, that actually adds up to slightly under 2TiB, due to the (confusing) difference between GiB/TiB and GB/TB.  By my workings, the 2TiB limit actually comes out as 2200GB, going by the way card-manufacturers label their cards.

A combo that sometimes worked and sometimes didn't was 1TB+400GB+400GB+400GB (which comes out as almost exactly 2TiB, and seemed to be the highest you could get to before it all went haywire, presumably because anything beyond that exceeds the 2TiB limit - even going a few GB over that would fail to restore properly and come out as being just a few GB).

If it's any help with whatever the ongoing work about sector sizes relates to, at some point I might experiment again with going beyond that 2TiB limit, but it won't be till I build another iPod.

[philden]

There is a thread on head-fi.org where someone has been doing similar tests to find the maximum capacity, bearing in  mind the odd use of capacity labels:

https://www.head-fi.org/threads/official-ipod-video-classic-5g-5-5g-6g-6-5g-7g-ssd-mod-thread.566780/page-711

[speachy]

The simplest way to get >2TB is to just use an mSATA/m.2 SSD.  It will also be a lot more reliable, and probably use less power too.

Anyway.  As amacrhonic mentioned, there are two overlapping 2TB (well, 2TiB) limits in play.  The first is due to the classic MBR partitioning scheme, which is limited to 2^32 sectors on the whole drive.  Most stuff still uses 512-byte sectors at the hardware interface level, which equates to 2 TiB.

The second limit is due to FAT32.  the '32' here refers to its internal addressing; basically it too has a limit of 2^32(-1) sectors, and at 512 byte sectors, you're limited to 2 TiB (-512 byte) maximum partition size.

There is a newer partitioning system called GPT that ups the maximum drive size to 2^64 sectors.  But while that vastly increases the maximum overall disk size, we're still stuck with 2TiB maximum partition sizes due to FAT32's limitations.

I committed support for GPT partitioning back in May of 2023.  I have a patch (that needs much more testing before it can be committed) that reworks Rockbox's internals to support 64-bit sectors and extends the ATA and FAT32 drivers.  With these, it is possible to use drives larger than 2TiB.  Currently only ATA devices can work with this; this includes native SATA SSDs and _possibly_ the iFlash dual/quad adapters with sufficiently large cards stuffed into them.   I say "possibly" because we don't know if they will work properly or not.  [Edit:  by "work" I mean "do the iFlash adapters themselves follow the ATA specs and allow full LBA48 addressing versus truncating sector addresses to 32 bits" ]

What is _not_ part of this is support for SD cards >2TiB, ie SDUC cards, but I don't think any of those exist yet anyway.  SDUC support has to be added to every SD controller driver individually, and there may be technical limitations that preclude it from working on specific hardware.   Note that the iFlash SD adapters do (and will) _not_ support those SDUC cards!

That said, just because _Rockbox_ supports GPT doesn't mean that a given player will be able to boot off of a GPT system.  MBR and GPT can coexist but there's no guarantee that the device's low-level boot code will accept a hybrid GPT system.  Even if it does, crafting the necessary hybrid tables will be challenging!

However, even with >2TiB drives, you will have to create multiple FAT32 partitions in order to take advantage of that extra space. Rockbox has long supported "multivolume" arrangements, so that's not inherently a problem.  Support for >2TiB filesystems is done with something other than FAT32, typically exFAT -- and adding exFAT is a major undertaking and not likely to happen anytime in the foreseeable future. 

Similarly, it's theoretically possible to play games and have the low-level storage API lie about the sector sizes and tell the FAT (and USB storage) layers that sectors are something larger.  This would allow up to 16TiB FAT32 volumes, but making that possible will be a very disruptive change.

So tl;dr:  Support for >2TiB ATA drives is coming, though individual partitions will be still limited to 2TiB.  It will require updated bootloaders and main firmware.  It will likely result in the player's factory firmware becoming unusable if not outright unbootable.

Oh, one last comment -- this only applies for _native_ rockbox ports.  For hosted ports we're at the mercy of whatever (ancient) software the host platform uses, and the odds are very high that 2TiB is the upper limit for all of those.