Jump to content

>2TiB external USB drive and WinXP? Of course!


Recommended Posts

A Sata drive may be EITHER 512 or 512 AF or 4KN .

The 512 has 512 bytes sector.

The 512 AF has 4 Kb sectors BUT they are translated by the disk controller to "simulated" 512 bytes sectors (the OS can see them a 512 bytes each).

The 4KN (4 kb Native) has 4 kb sectors AND they remain 4 Kb (the OS sees them as 4 Kilobytes each).

sector sizes CANNOT be changed (a 512 or 512AE translated to 4 kb sectors is not a 4 kb Native).

An (external) enclosure with eSata or with USB connections may (or may not) provide an additional layer of translation, i.e. a 512 or 512AF disk in it may (or may not) be translated to 4 Kb sectors .

We have a documented specific case of one of such external enclosures that translates from 512 to 4 Kb  on the USB bus BUT NOT on the eSATA one:

https://msfn.org/board/topic/173265-formatting-an-external-drive-using-different-interfaces/

https://msfn.org/board/topic/173642-mkprilog-batch-to-access-a-same-disk-under-two-different-interfaces/

in practice the eSATA is "pass-through" and exposes the disk "as is", the USB is converting sector size.

Such a huge drive (14 TB) is very likely 4kN, it should be possible to partition and format it "normally", without needing the WD tool.

jaclaz

 

Link to comment
Share on other sites


5 hours ago, Comos said:

As I recall from the early days when I was investigating this, the sectors are actually 4K, but emulated into 512 and it looks like that some part of the role is playing the WD controller from the enclousure, because of you remove the drive from it and connect it to a regular SATA controller, you can only see a raw data.

Most drives today that are 3TB+ are 4kn, ie 4k native sectors.  And then most are 512e, ie 512k emulated sectors for compatibility.  (You can still get 512k sectored drives, but they're usually special order.  There's also 528k, 520k and some other formats of sectors used in enterprise storage.  In the same vein, you can also buy 4kn drives that fully report 4k sectors with no emulation, again usually special order.)

Most sas drives can be formatted to any sector size very easily using I think it is sg_util in linux (may be something different as I haven't done it myself).  And generally this is will work with sas drives that don't otherwise have a lock on changing the sector size.  From my reading and some experience, most sata drives cannot change their sector sizes like their sas counterparts--even on the sata versions of sas drives.

I believe the sector changing capability is an artificial limit imposed by manufacturers to make sure sata drives don't replace their more expensive sas counterparts in enterprise storage (as it could save companies literally hundreds of thousands of dollars), but that the best buy easystore drives are a bit of an exception wherein they can be changed to a 4k native mode.  I thought it was just the enclosure that was able to do this, so I tried an HGST 10TB drive that I know is 4kn in it and the WD utility did not allow me to change the sector size.

From my research, the My Book series of enclosures uses a encryption which causes a drive that is removed from an enclosure to appear as nothing/raw data when connected directly since the encryption is missing.  This isn't a problem if you're just shucking the drive from the enclosure or using the drive exclusively in the enclosure, but becomes a problem when a drive with data is moved in or out of the enclosure.  The same limitation doesn't exist on the easystore enclosure so that's why you find them for sale on ebay as they are a fine external enclosure for any drive, and can change certain select drives (who knows which ones?) from 512e to 4kn.

Link to comment
Share on other sites

11 hours ago, jaclaz said:

Such a huge drive (14 TB) is very likely 4kN, it should be possible to partition and format it "normally", without needing the WD tool.

 

You can, but I need to switch it into to the "XP Compatible" mode which that WD Formatter does.

Edited by Comos
Link to comment
Share on other sites

I don't know, maybe it is something specific to the WD disks. :dubbio:

A virtual disk with 4kb sectors can be partitioned/formatted just fine in XP, at the time I put together the mentioned dual mode batch it has been tested extensively and what I could do on the virtual disk has been replicated just fine on the real disk.

It is possible that you need to make the partition table manually, though,

@SamirD

You have to make up your mind, either most drives are 4kN OR most drives are 512AF (512e).

Generally speaking, and AFAIK, disk drives up to 4-6 TB are usually 512AF, larger one are 4 kN (but not always, only as a very rough "rule of thumb") , I would say that 512AF are much more common than 4kN.

Changing sector size on SAS (please read as SCSI) is only possible on certain models of disks, and the possibilities are usually between 512 and 520 or 528 bytes/ector:
https://linux.die.net/man/8/sg_format

https://forums.servethehome.com/index.php?threads/how-to-reformat-hdd-ssd-to-512b-sector-size.4968/

jaclaz

 

Edited by jaclaz
Link to comment
Share on other sites

I come to a resolution last evening.The WD enclousure controller has an onboard serial flash chip where in my case was disabled.With the disabled flash chip in behaves like a regular USB-SATA bridge and the WD Formatter simply can't see it as WD My Book.I need to get a spare controller,solder out the flash chip and see let's see how is the whole stuff encoded.Another option would be to get the controller from the new 14TB My Book, but that's a bit unlikely,since it exists on the market for a short time.

Link to comment
Share on other sites

On 11/7/2020 at 5:19 AM, jaclaz said:

A Sata drive may be EITHER 512 or 512 AF or 4KN .

The 512 has 512 bytes sector.

The 512 AF has 4 Kb sectors BUT they are translated by the disk controller to "simulated" 512 bytes sectors (the OS can see them a 512 bytes each).

The 4KN (4 kb Native) has 4 kb sectors AND they remain 4 Kb (the OS sees them as 4 Kilobytes each).

sector sizes CANNOT be changed (a 512 or 512AE translated to 4 kb sectors is not a 4 kb Native).

An (external) enclosure with eSata or with USB connections may (or may not) provide an additional layer of translation, i.e. a 512 or 512AF disk in it may (or may not) be translated to 4 Kb sectors .

We have a documented specific case of one of such external enclosures that translates from 512 to 4 Kb  on the USB bus BUT NOT on the eSATA one:

https://msfn.org/board/topic/173265-formatting-an-external-drive-using-different-interfaces/

https://msfn.org/board/topic/173642-mkprilog-batch-to-access-a-same-disk-under-two-different-interfaces/

in practice the eSATA is "pass-through" and exposes the disk "as is", the USB is converting sector size.

Such a huge drive (14 TB) is very likely 4kN, it should be possible to partition and format it "normally", without needing the WD tool.

jaclaz

 

If a drive is 512n, then it could not be changed to 4kn because the native sectors are 512k.  But it seems that the WD easystore drives do have the capability to change from 512e to 4kn, but not at the controller level as my HGST drive should have worked if that was the case.  I believe these easystore drives have the ability to change to 4kn and run 4kn.  I know I've read some online experiences of people running these 4kn in raid systems after they were shucked from the enclosures.

The usb enclosures can also provide translation and I remember those coming out around the same time the 3TB drives came out.

Most drives today, even super large ones like my 16TB Exos are still 512e.  Most every drive I have--8, 10, 12, 16TB--are all 512e.  The 4kn versions of these are usually special order from what I've found.

Link to comment
Share on other sites

6 hours ago, jaclaz said:

I don't know, maybe it is something specific to the WD disks. :dubbio:

A virtual disk with 4kb sectors can be partitioned/formatted just fine in XP, at the time I put together the mentioned dual mode batch it has been tested extensively and what I could do on the virtual disk has been replicated just fine on the real disk.

It is possible that you need to make the partition table manually, though,

@SamirD

You have to make up your mind, either most drives are 4kN OR most drives are 512AF (512e).

Generally speaking, and AFAIK, disk drives up to 4-6 TB are usually 512AF, larger one are 4 kN (but not always, only as a very rough "rule of thumb") , I would say that 512AF are much more common than 4kN.

Changing sector size on SAS (please read as SCSI) is only possible on certain models of disks, and the possibilities are usually between 512 and 520 or 528 bytes/ector:
https://linux.die.net/man/8/sg_format

https://forums.servethehome.com/index.php?threads/how-to-reformat-hdd-ssd-to-512b-sector-size.4968/

jaclaz

 

I believe it is something specific to these drives inside these specific enclosures.  4kn drives can easily be partitioned and formatted--both my WD easystore drives (10/14TB? can't remember) are MBR and have multiple 2TB FAT32 partitions.  They are recognized in almost every device too that can read FAT32 partitions, including nas units.

Most drives themselves are at the hardware level 4k.  Most drives are sold as 512e.  Hope that clears it up for you.

On SAS many users on the servethehome forum have easily moved between 512, 520, 528, and 4k.  You just have to know exactly what parameters on sg_format works for what drive, and while not all drives support it, the majority do.  The thread you linked to is old and most of the time the information about changing the sectors is in the individual FS or drive discussions, not in that thread.

 

Link to comment
Share on other sites

17 hours ago, SamirD said:

Most drives themselves are at the hardware level 4k.  Most drives are sold as 512e.

 

Sure :), that is exactly where the terminology is confusing.

A 4kn disk is  4k sectored AND exposes 4k sectors.
A 512e (512AF) is  4k sectored BUT exposes 512 bytes sector.

Nowadays anything bigger than - say - 500 GB or maybe 1TB is internally 4k (or at least this is what the manufacturers declare).

Actually 4kN disk are pretty much rare, compared with the much more common 512e (512AF).

jaclaz

 

 

Link to comment
Share on other sites

4 hours ago, jaclaz said:

Sure :), that is exactly where the terminology is confusing.

A 4kn disk is  4k sectored AND exposes 4k sectors.
A 512e (512AF) is  4k sectored BUT exposes 512 bytes sector.

Nowadays anything bigger than - say - 500 GB or maybe 1TB is internally 4k (or at least this is what the manufacturers declare).

Actually 4kN disk are pretty much rare, compared with the much more common 512e (512AF).

jaclaz

 

 

Yep!  Very much akin to the MB vs MiB issue that can be confusing as well.

I've seen that the cutoff for hardware 4k seems to be above 2TB.  Most of my 2TB drives are almost all 512n.

Yeah and because the 512e drives are just emulated 512k sectors, it really shouldn't have been made to be so difficult to make a drive 4kn.  I think the boot issues are what scared away most manufacturers, but those issues are almost a non-issue today except when trying to boot and run older software/OSes that need 512n.

The funny thing is, I have a maxoptix MO drive that uses 2048 sectors even back in the day.  Anyone that remember mscdex remembers how those also used larger sector sizes.

 

Link to comment
Share on other sites

BTW, now (well... actually for some time already) there is a 5 TB WD My Passport available on the market.
When (and if) any member here do decide to take the plunge, buy one and reformat it to 4 kiB sectors, please don't forget to tell us about the experience.  :angel

Link to comment
Share on other sites

3 hours ago, dencorso said:

BTW, now (well... actually for some time already) there is a 5 TB WD My Passport available on the market.
When (and if) any member here do decide to take the plunge, buy one and reformat it to 4 kiB sectors, please don't forget to tell us about the experience.  :angel

den,

the WHOLE point is that disk drives settings CANNOT be changed normally by the end user, if that WD 5 Tb is 4kN, it will remain 4kN, if it is 512e (512AF) it will remain 512e (512AF).

SATA disks DO NOT have an OS level tool to change the way sectors are exposed, some (not all) SAS disks (please read as Serial Attached SCSI) do have such a feature.

There could well be a manufacturer tool or some other third party software that can enable/disable the 4 kb to 512b translation in the controller (possibly via a TTL interface, not via the SATA, let alone via the USB), but IF it exists, I have never seen/heard of one.

jaclaz

 

Edited by jaclaz
Link to comment
Share on other sites

Of course! I just tried to change subjects and give a head's up regarding the newish USB 3.0 5 TB My Passport, which is smaller phisically than a My Book but *BIG* bytewise! Yet, I should have said I was trying to steer the thread back to USB external drives.

On 8/16/2020 at 8:07 PM, dencorso said:

[...] I have a NTFS formatted 4KiB sectored USB 3.0 WD My Passport using a single primary partition of 4TB for a long time already (since Mar 11, 2018, to be exact), with no issues at all. At the time I said:

Quote

I confirm it does work! I've just formatted a 4TB WD My Passport (manufactured in 2017, which came as GPT)
with the tool you indicated and lo: it's now visible to XP and declares 4KiB sectors and a capacity of
4,000,751,546,368 bytes, of which 4,000,561,778,688 bytes free!

And there is report by @Asp of having created a single 6 TB patition, too.

And there's also a report by @SamirD himself, of having changed sucessfully the apparent sector size to 4 kiB on two different 10TB Bestbuy WD Easystore drives, but IIRR, at that time he hadn't tried to format either as a single 10TB partitions...

wdbcka0100hbk-nesn.jpg

Just before his passing, RLoew was investigating the WD Quick Formatter and he said he had figured out which SCSI commands are used for the "sector size change", but his results also pointed out to the fact that the only HDDs he found that implemented such commands were the external USB WD drives. He also found out, although that's just a curiosity, that the "sector size" might also be set to other values than 4 kiB, but that, in fact, the NT-family OSes can use only 2 kiB (besides 512 bytes and 4 kiB), Yet, his son did not yet find any material about that reseach, which, in fact, may have been all done inside a debugger, so probably only some handwritten notes or text documentation may exist, if any. And no, he also determined that XP (SP1-3) cannot handle internal 4 kiB drives, be them SATA or IDE. Only the USB stacks are able to do that. It's possible that NT-family OSes v. 6.x may do that (10 certaily does), but that's beside the point, of course, since this is the XP forum, after all!

Quote

I think that's precisely what this version of the WD Quick Formatter, http://download.wdc.com/misc/WD_Quick_Formatter_Win_1_2_0_10.zip does: set some undocumented feature in the Drive, to change it from exposing 512e to exposing 4Kn.

The program requires no installation besides unzipping, FTW.
Some more info: https://support.wdc.com/knowledgebase/answer.aspx?ID=3868

3868_7.png

 

Link to comment
Share on other sites

I am not fully convinced by the last statement:

Quote

And no, he also determined that XP (SP1-3) cannot handle internal 4 kiB drives, be them SATA or IDE. Only the USB stacks are able to do that.

in the sense that it is probably very true that common drivers do not, but it is not a non-fixable issue, as said the Microsoft own VSS drivers work just fine with 512 and 4096 bytes/sector , and on the other hand there is the Paragon driver (that - with alternate results from reports of people who used it - *somehow* works ), so I would say that "a special driver is needed on XP" (it is the same thing, with a slightly more optimistic view).

About the switching between the exposed 512 and 4k sectors (on some WD disks) I am rather convinced that with the right (low level) tools it is entirely possible (in the disk controller itself or in the enclosure controller)  to change some firmware settings (as after all the exposing of 512 bytes sector in 512e/512AF disk drives happens to an added - inside the controller - "translation layer") the point is that - even if we find out the undocumented command it may be enabled or disabled (at the whim of the good WD guys) and if it is enabled it is (or will in a next release in case of a new disk/encloure model ) also be supported by the WD quick formatter, but that if it is disabled you will need anyway the hypoyhetical low level tool to enable it.

 Once said that, there is the additional issue (that we saw in the thread about the MBR 2.2 Tib+rest up to 4 Tib formatting style, that works on 7 but dosn't on XP) of some 32 bit addresses somewhere in some key (files) that unlikely (even if the issue is found) can be patched/corrected.

So, the "right" solution for bigger than 2.2Tib disks viable on XP is at the moment only about disks that expose (directly or through an enclosure controller) 4 Kb sectors,

Even if it would be a bottleneck/slowing down overlay, I wonder why third party USB enclosures with a selectable pass-through vs.  sector size conversion setting aren't available (or maybe they are and we don't know?[1] :dubbio:) , probably due to the thinning number of XP users, and while I am at it (dreaming) I don't see how a SATA to SATA converter (similar to the el-cheapo's IDE to SATA converters) capable of the sector size conversion cannot be made.

UNrelated, but not much:

https://www.sabrent.com/acronis/ssc/

jaclaz

[1] Possibly the ADU31ESP-X model of Addonics?

https://www.addonics.com/products/adu31esp.php

P.S.: I found hints - here and there - that the VLI (VIA) 701 is one of these chips that do 512->4K conversion (though I believe it is auto-switching with bigger than 2.2 Tib disk drives):

https://goughlui.com/2013/10/02/experiment-usb-to-sata-bridge-chips-and-2tb-drives/

https://www.usbdev.ru/files/vli/

and the "Intenso" controller (whatever it is):

https://superuser.com/questions/1271871/4k-emulation-sata-usb-controllers

 

Edited by jaclaz
Link to comment
Share on other sites

22 hours ago, jaclaz said:

I am not fully convinced by the last statement:

in the sense that it is probably very true that common drivers do not, but it is not a non-fixable issue, as said the Microsoft own VSS drivers work just fine with 512 and 4096 bytes/sector , and on the other hand there is the Paragon driver (that - with alternate results from reports of people who used it - *somehow* works ), so I would say that "a special driver is needed on XP" (it is the same thing, with a slightly more optimistic view).

I played with the Paragon GPT loader when I run into this issue on XP, that's already 8 years ago.The Paragon driver actualy works only on internal SATA controller where the > 2TB drive is connected, not over USB.Maybe I could find a discussion with the Paragon support somewhere in my email history.

Link to comment
Share on other sites

Well, while trying to do a condensed version of the main points in this vast subject (not all of them present in this thread alone), I realize I lost sight of the main post of my last two posts, which is: "There's a new, untested, generation of My Passport, which now has reached 5GB as their biggest USB external drive offer... so, does the old good WD format tool work with this generation, too?"
Although we know older WD My Books and Easystore drives exist in sizes bigger than 5GB, on which the old good WD format tool is known-for-a-fact to work, they're both physically bigger than the My Passports, which are much more portable, so a usable 5GB My Passport ought to be good news to the XP community. In fact, were it not the fact that Brazilian currency lost half its purchasing power since our current president's inauguration, and that Covid-19 ate about half the income of the least affected brazilians (all others lost more than that), I'd already have bought one 5GB My Passport, tested it myself and reported, but that's beyond me at the moment. [The last time I looked, São Paulo City, where I live, has already had 13128 people deceased due to Covid-19, more than most countries in the world will have till the end of this current pandemic. :( ].

As for the Paragon drive working with internal 4kiB drives, that's a welcome fact I had failed to notice, so thanks for the head's up!  :cheerleader:

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...