Personal View site logo
Make sure to join PV on Telegram or Facebook! Perfect to keep up with community on your smartphone.
Please, support PV!
It allows to keep PV going, with more focus towards AI, but keeping be one of the few truly independent places.
What's so special about the GH2's preferred 128kB allocation unit size?
  • I wanted to share some findings with you that I found while experimenting with different methods to format a SDXC card before use in a GH2.

    While I cannot see yet an immediate beneficial use of my results, I thought that they may inspire other experiments or just add to the knowledge of the GH2 internals.

    At first I wanted to know what precisely a GH2 writes to my 64GB SDXC card when doing the usual "Format" before use. The answer:

    The GH2 creates a DOS-style partition table containing one primary partition that starts at an offset of exactly 32768 sectors (of 512 byte, each), or 16MB. This seems like a reasonable choice to me, as it makes sure the start of the filesystem is aligned to a 2^N byte offset that is almost certain to be a multiple of the erase block size of the MLC NAND flash memory that SD cards commonly consist of. The type ID of the primary partition is set to 7, a value used by Windows for NTFS as well as exFAT (and also for HPFS with OS/2, which may rest in peace). (Notice that with SD and SDHC cards, partition type 0xb is used, which is common for use with FAT file systems.) The unused 16MB before the first partition start may also help (when never having been written to) the SD card internal Flash controller and its wear level management to prepare completely empty erase blocks for writing. (NAND Flash can be read and written to when empty in small blocks such as 512 byte, but erasing NAND Flash memory is possible only in much larger "erase blocks" - such as 512k to 4MB - so the controllers in SD cards, USB sticks or SSDs need to employ some sector allocation management of their own to make sure they can erase blocks of that size before new data can be written.)

    The GH2 writes an exFAT filesystem to the primary partition with the following characteristics:

    • Volume serial number: A seemingly random number that changes with each format
    • exFat filesystem version: 1.0
    • Sector size: 512
    • Cluster size: 131072
    • First sector: 32768 (consistent with the start of the partition)
    • FAT first sector: 16384
    • FAT sectors count: 16384
    • First cluster sector: 32768
    • Root directory cluster: 4
    • Volume state: 0x0000
    • FATs count: 1
    • Drive number: 0x80
    • Label: (none, empty)

    This is the same filesystem layout that results if you format the partition using e.g. the following Windows command line (replacing X: with your actual drive letter):

    format X: /FS:exFAT /Q /A:128k

    The GH2 writes a directory structure and some files to the initialized exFAT filesystem:

    size name
         0 DCIM/
         0 MISC/
         0 PRIVATE/
         0 PRIVATE/AVCHD/
         0 PRIVATE/AVCHD/BDMV/
         0 PRIVATE/AVCHD/BDMV/PLAYLIST/
         0 PRIVATE/AVCHD/BDMV/CLIPINF/
         0 PRIVATE/AVCHD/BDMV/STREAM/
     504 PRIVATE/AVCHD/BDMV/INDEX.BDM
       66 PRIVATE/AVCHD/BDMV/MOVIEOBJ.BDM
         0 PRIVATE/AVCHD/AVCHDTN/
       72 PRIVATE/AVCHD/AVCHDTN/THUMB.TID
         0 PRIVATE/AVCHD/AVCHDTN/THUMB.TDT
         0 PRIVATE/AVCHD/IISVPL/
    607776 PRIVATE/FILEINFO.TBL
       76 PRIVATE/FILEMNG.DAT
    1672 PRIVATE/cache.dat
    

    If you archive this directory structure on a computer of yours, you can re-create a "blank" SDXC card for use with the GH2 at any time, just by creating the same filesystem type and copying these files to it. I tested that indeed, the GH2 behaves completely normal if you use a card prepared such, indistinguishable from an "original in-camera formatting".

    Now I wanted to know whether changing any aspects of this "formatting" would yield interesting results, and indeed, it does.

    First, I checked whether the offset of the first partition does make a difference to the GH2 when changed. Doubling or halving this offset didn't change anything, the GH2 seemed not to take notice of the difference.

    Next, I checked what would happen if I used a different allocation unit size, e.g. 256kB or 64kB. Such change causes the GH2 to display a warning when inserting the SDXC card, saying (as far as I can remember the wording): "This card has not been formatted with this camera. It will not be suitable for recording video." After displaying this message for a few seconds, the camera behaves normally regarding the taking of still pictures. When starting to shoot video, it first seems that all is normal, too, but whatever allocation unit size different from 128kB I tried, video recording will stop after a few seconds with the well known error message that "the cards write speed is insufficient". This happenes even when recording low-bandwidth video, so I wonder what special meaning the 128kB allocation unit size has for the GH2 that it is working so badly with other sizes.

    Notice that when I benchmark writing to the same card from my PC with allocation unit sizes other than 128kB (within reasonable limits), I do not measure significantly different performance. But of course, the SD card interface of the GH2 may be very different from the Kingston USB 3.0 card reader that I use with my PC.

    Well, that's about it with my experiments on this, so far.

    BTW: The "SD Formatter" offered at https://www.sdcard.org/downloads/formatter_3/ claims that it has one mode of operation where it pre-erases the Flash memory on the card, which would plausibly allow for faster writing, but only until all the blocks have been written to at least once. I also wonder under which circumstances this tool is able to issue the ERASE_WR_BLK_START/_END etc. SD protocol messages to a SDXC card that is connected via a card reader, the documentation of the formatter indicates that the tool falls back to just overwriting the blocks if it cannot erase them - but just overwriting blocks would not leave the card with empty erase blocks, so subsequent writes would require erase operations by the cards internal controller, resulting in no write performance benefit.

  • 17 Replies sorted by
  • (PS: The looks of my "quoting" kind of sucks. Can someone direct me to a documentation of this forum's syntax elements to do better?)

  • @karl

    Type Markdown in google

  • Ah, great, thanks Vitaliy, this "Markdown" stuff works just fine.

  • After reading some in the MMC/SD specs and in the Linux drivers/mmc source code, I think I can answer myself now:

    SD cards can indicate a preferred allocation unit size to the controller. Allocating and writing chunks of this size improves their performance.

    So it's probably not the GH2, but the SanDisk 64GB 95MB/s card that favors this 128k allocation unit size, and a different card may favor a different size, and I guess that the GH2 displays the warning message when the allocation unit size of the filesystem is not the same as the preferred size as indicated by the SD card.

    (To verify this theory, one would need a card that has a different allocation size preference - I currently do not have one.)

  • Interesting findings Karl. I recently bought a Sandisk Ultra 64GB miniSDXC card, but it was faulty so I sent it back & can't check the alloc size. I should get another soon-ish - when I do I'll report what it formats to.

    You might be right that the card is requesting a preferred alloc size, but it could also be that the GH2 is tuned for 128k writes.

  • I'll have one Friday : ).

  • Just today I got two more SanDisk Extreme Pro 64GB cards, so I could make one more experiment that I could not have done before: Dumping the parameters of the exFAT filesystem as it is formatted on a newly delivered card of that type (before the camera writes to it):

      Volume label                        
      Volume serial number      0x30373866
      FS version                       1.0
      Sector size                      512 
      Cluster size                  131072
      Sectors count              124702720
      Free sectors               124668672 
      Clusters count                486992
      Free clusters                 486987
      First sector                   32768
      FAT first sector               16384
      FAT sectors count              16384
      First cluster sector           32768
      Root directory cluster             4
      Volume state                  0x0000
      FATs count                         1
      Drive number                    0x80
      Allocated space                   0%
    

    No surprises in that.

    Of course, unlike after formatting in the camera, the exFAT filesystem is empty (no files or directories in it).

  • Another experimental result: I have tried to use the forementioned "SD Formatter" on two different operating systems using 3 different kinds of built-in and external card reader hardware.

    Alas, in no tested combination "SD Formatter" declared itself able to operate in the "erase"-mode - which is the only really interesting one, as it could actually provide some performance benefit for subsequent writing.

    Has anyone ever successfully used that "erase" mode?

    (BTW: A Realtek engineer recently announced on a mailing list that they are going to re-implement a Linux driver for their card readers - if that is based on Linux' generic SD driver framework as proposed, that would most likely allow to pre-erase SD cards connected to Realtek readers using the usual Linux tools. I hope this materializes...)

  • I got my 64GB Ultra microSDXC, same alloc size 128k shipped, same GH2 formatted.

    Unfortunately none of my partitioning programs (Easus Partition Master 9.1 Pro, Acronis Disk Director 11) recognize the exFat format properly, so I couldn't get the full FS specs.

    Here are the Atto results from my USB3.0 reader - Win7 confirms about 12.3MB/s copying large files over. Not superfast, but to my surprise it survived a 1min deathchart with my Sweet 16 mod (constant Q16).

    The read speed of ~43MB/s is a surprise, they're advertised as 30MB/s read.

    Cost ~£46 (UK Amazon, Play.com etc)

  • I got my 64GB Ultra microSDXC, same alloc size 128k shipped, same GH2 formatted.

    Unfortunately none of my partitioning programs (Easus Partition Master 9.1 Pro, Acronis Disk Director 11) recognize the exFat format properly, so I couldn't get the full FS specs.

    Here are the Atto results from my USB3.0 reader - Win7 confirms about 12.3MB/s copying large files over. Not superfast, but to my surprise it survived a 1min deathchart with my Sweet 16 mod (constant Q16).

    The read speed of ~43MB/s is a surprise, they're advertised on the packaging as 'up to 30MB/s' read.

    Cost ~£46 UK (eg. Amazon: http://www.amazon.co.uk/gp/customer-media/product-gallery/B007JTKM30/ref=cm_ciu_pdp_images_2?ie=UTF8&index=2&isremote=0)

    61uYwcz4LRL[1].jpg
    373 x 500 - 74K
    Sandisk miniSDXC Ultra 64GB (USB3.0 reader).gif
    476 x 607 - 27K
  • ... actually looking at the ATTO results, you can see that 128k is indeed a sweet spot! It peaks at 14.4MB/s there, nearly 1/2 meg faster than the next best, and nearly 1 meg faster than the surrounding values. That explains the decent GH2 performance.

  • The SD formatter doesn't work with the Flash: ERASE option here either (format fails). But it does support Overwrite: ERASE. - wouldn't that give the same benefit?

    I suspect the controller has to support the flash-erase commands.

  • I ran the ATTO test again, on two different USB 3.0 readers. This time both 64KB and 128KB come out ahead on both - the 2nd reader (attached) actually managed 15.7MB/s at 64k and only 14.7MB/s @ 128k. Both of those were still ~1 meg better than the next best value.

    So not an exact conclusion, but 64k and 128k do seem to be writing sweet spots.

    Sandisk-miniSDXC-Ultra-64GB-(USB3.0-reader2).gif
    475 x 607 - 26K
  • @_gl: "Overwrite: ERASE. - wouldn't that give the same benefit?" - No, that uses ordinary write operations to overwrite all blocks, and since the SD card cannot know that the written content is mostly irrelevant, all NAND Flash erase blocks will be "allocated" after this operation, so every subsequent write will (card-internally) have to erase a block before it can write to it.

    (Usually, Flash RAM controllers keep a spare area from which to allocate blocks for writing and re-assignment of defect blocks, so the SD-card will perform not as bad as if it has to erase one block for each single sector write - but it will still be slower than if the whole card was pre-erased.)

  • @karl, so there is no equivalent of TRIM on SD cards, ie. deleting a file (or quick-formatting the card) doesn't allow the card to wipe the allocated space?

  • @_gl: There is an equivalent to (the SATA protocols) "TRIM" for SD cards, but these erase commands need to be issued by the host, the SD card itself does not know about any filesystem structure, it does not know whether any of its blocks belong to a an actual file or just contain the left-overs of some long ago deleted file.

    It seems to me that the SD formatter utility tries to issue such TRIM-like commands when used with "Flash ERASE" mode, but it depends on the driver and the card reader hardware whether those commands actually make it to the card, if they don't the SD formatter reports an unsuccessful operation.

  • I'm using the Sandisk Extreme 32gb 45MB/s U1 class 10 cards, and I've constantly had issues with spanning on intra patches- have to use 24L to span. So after reading this thread, I've discovered the following:

    The card is formatted FAT32 by the GH2 with 32k clusters. So, I thought I'd try the advice given in this thread for formatting- I know these specs were given for 64gb cards, but I'd thought I'd give it a whirl.

    I'm running windows 7 64 bit. Card reader is a kingston usb 3.0 card reader.

    When formatting with exFAT 128k clusters, then copying over the file structure from a clean GH2 format, the camera states "memory card error. Format this card?" with yes/no prompts. Selecting "no" results in the camera stating "re-insert memory card" and no further actions can be taken.

    Formatting with FAT32 64k clusters and copying file structure back gives the same result.

    Formatting with FAT32 32k clusters and copying file structure back works- card can be written to.

    So, it looks like the camera does indeed have some type of spec it's either reading from the card or has embedded in it's firmware that doesn't allow the card to be used unless it's formatting meets same said specs. I was hoping to get over the spanning problem on the card in this fashion (at least with 64k clusters on FAT32), but I don't see how that's going to happen.

    Any thoughts?