[gentoo-user] Partition of 3TB USB drive not detected

[gentoo-user] Partition of 3TB USB drive not detected

[Jörg Schaible]

Hi,

for my backups I use a 3TB USB drive (one big ext4 partition) without any 
problems. Just plug in the cable, mount it and perform the backup. The 
partition (sdi1) is detected an mountable without any problems:

=============== %< ==============
$ ls -l /dev/disk/by-id
total 0
lrwxrwxrwx 1 root root  9 Jul 31 13:17 ata-
Crucial_CT500MX200SSD1_161512468483 -> ../../sda
lrwxrwxrwx 1 root root 10 Jul 31 13:17 ata-
Crucial_CT500MX200SSD1_161512468483-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jul 31 13:17 ata-
Crucial_CT500MX200SSD1_161512468483-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jul 31 13:17 ata-
Crucial_CT500MX200SSD1_161512468483-part3 -> ../../sda3
lrwxrwxrwx 1 root root  9 Jul 31 13:17 ata-HL-DT-ST_DVDRAM_GH41N_K5Q9AN32423 
-> ../../sr0
lrwxrwxrwx 1 root root  9 Jul 31 15:20 ata-ST3000DM001-1CH166_Z1F45VR9 -> 
../../sdi
lrwxrwxrwx 1 root root 10 Jul 31 15:20 ata-ST3000DM001-1CH166_Z1F45VR9-part1 
-> ../../sdi1
lrwxrwxrwx 1 root root  9 Jul 31 13:17 usb-Generic-
_Compact_Flash_20060413092100000-0:0 -> ../../sdc
lrwxrwxrwx 1 root root  9 Jul 31 13:17 usb-Generic-_MS_MS-
Pro_20060413092100000-0:3 -> ../../sdf
lrwxrwxrwx 1 root root  9 Jul 31 13:17 usb-Generic-
_SD_MMC_20060413092100000-0:2 -> ../../sde
lrwxrwxrwx 1 root root  9 Jul 31 13:17 usb-Generic-_SM_xD-
Picture_20060413092100000-0:1 -> ../../sdd
lrwxrwxrwx 1 root root  9 Jul 31 13:17 usb-Generic_Flash_HS-
CF_26020128B005-0:0 -> ../../sdg
lrwxrwxrwx 1 root root  9 Jul 31 13:17 usb-Generic_Flash_HS-
COMBO_26020128B005-0:1 -> ../../sdh
lrwxrwxrwx 1 root root  9 Jul 31 13:17 usb-INTENSO_USB_AA04012900007518-0:0 
-> ../../sdb
lrwxrwxrwx 1 root root 10 Jul 31 13:17 usb-INTENSO_USB_AA04012900007518-0:0-
part1 -> ../../sdb1
lrwxrwxrwx 1 root root  9 Jul 31 15:20 wwn-0x5000c50065531ec5 -> ../../sdi
lrwxrwxrwx 1 root root 10 Jul 31 15:20 wwn-0x5000c50065531ec5-part1 -> 
../../sdi1
lrwxrwxrwx 1 root root  9 Jul 31 13:17 wwn-0x5001480000000000 -> ../../sr0
lrwxrwxrwx 1 root root  9 Jul 31 13:17 wwn-0x500a075112468483 -> ../../sda
lrwxrwxrwx 1 root root 10 Jul 31 13:17 wwn-0x500a075112468483-part1 -> 
../../sda1
lrwxrwxrwx 1 root root 10 Jul 31 13:17 wwn-0x500a075112468483-part2 -> 
../../sda2
lrwxrwxrwx 1 root root 10 Jul 31 13:17 wwn-0x500a075112468483-part3 -> 
../../sda3
=============== %< ==============
$ ls -l /dev/disk/by-label
total 0
lrwxrwxrwx 1 root root 10 Jul 31 13:17 boot -> ../../sda1
lrwxrwxrwx 1 root root 10 Jul 31 13:17 gentoo -> ../../sda3
lrwxrwxrwx 1 root root 10 Jul 31 15:20 intenso -> ../../sdi1
lrwxrwxrwx 1 root root 10 Jul 31 13:17 swap -> ../../sda2
=============== %< ==============

However, when I boot a rescue system from a USB stick, the partition on the 
USB is not detected. I already tried latest SystemRescueCD (default and 
alternate kernel), Knoppix and the Gentoo Admin CD. Nothing, the partition 
is not available.

What's the difference? Why does my kernel find this partition and the other 
one's do not? It's pretty silly to have a backup drive and cannot access it 
in question ;-)

- Jörg

Re: [gentoo-user] Partition of 3TB USB drive not detected

[Daniel Frey]

On 07/31/2016 06:37 AM, Jörg Schaible wrote:
> Hi,
> 
> for my backups I use a 3TB USB drive (one big ext4 partition) without any 
> problems. Just plug in the cable, mount it and perform the backup. The 
> partition (sdi1) is detected an mountable without any problems:
> 
> =============== %< ==============
> $ ls -l /dev/disk/by-id
> total 0
> lrwxrwxrwx 1 root root  9 Jul 31 13:17 ata-
> Crucial_CT500MX200SSD1_161512468483 -> ../../sda
> lrwxrwxrwx 1 root root 10 Jul 31 13:17 ata-
> Crucial_CT500MX200SSD1_161512468483-part1 -> ../../sda1
> lrwxrwxrwx 1 root root 10 Jul 31 13:17 ata-
> Crucial_CT500MX200SSD1_161512468483-part2 -> ../../sda2
> lrwxrwxrwx 1 root root 10 Jul 31 13:17 ata-
> Crucial_CT500MX200SSD1_161512468483-part3 -> ../../sda3
> lrwxrwxrwx 1 root root  9 Jul 31 13:17 ata-HL-DT-ST_DVDRAM_GH41N_K5Q9AN32423 
> -> ../../sr0
> lrwxrwxrwx 1 root root  9 Jul 31 15:20 ata-ST3000DM001-1CH166_Z1F45VR9 -> 
> ../../sdi
> lrwxrwxrwx 1 root root 10 Jul 31 15:20 ata-ST3000DM001-1CH166_Z1F45VR9-part1 
> -> ../../sdi1
> lrwxrwxrwx 1 root root  9 Jul 31 13:17 usb-Generic-
> _Compact_Flash_20060413092100000-0:0 -> ../../sdc
> lrwxrwxrwx 1 root root  9 Jul 31 13:17 usb-Generic-_MS_MS-
> Pro_20060413092100000-0:3 -> ../../sdf
> lrwxrwxrwx 1 root root  9 Jul 31 13:17 usb-Generic-
> _SD_MMC_20060413092100000-0:2 -> ../../sde
> lrwxrwxrwx 1 root root  9 Jul 31 13:17 usb-Generic-_SM_xD-
> Picture_20060413092100000-0:1 -> ../../sdd
> lrwxrwxrwx 1 root root  9 Jul 31 13:17 usb-Generic_Flash_HS-
> CF_26020128B005-0:0 -> ../../sdg
> lrwxrwxrwx 1 root root  9 Jul 31 13:17 usb-Generic_Flash_HS-
> COMBO_26020128B005-0:1 -> ../../sdh
> lrwxrwxrwx 1 root root  9 Jul 31 13:17 usb-INTENSO_USB_AA04012900007518-0:0 
> -> ../../sdb
> lrwxrwxrwx 1 root root 10 Jul 31 13:17 usb-INTENSO_USB_AA04012900007518-0:0-
> part1 -> ../../sdb1
> lrwxrwxrwx 1 root root  9 Jul 31 15:20 wwn-0x5000c50065531ec5 -> ../../sdi
> lrwxrwxrwx 1 root root 10 Jul 31 15:20 wwn-0x5000c50065531ec5-part1 -> 
> ../../sdi1
> lrwxrwxrwx 1 root root  9 Jul 31 13:17 wwn-0x5001480000000000 -> ../../sr0
> lrwxrwxrwx 1 root root  9 Jul 31 13:17 wwn-0x500a075112468483 -> ../../sda
> lrwxrwxrwx 1 root root 10 Jul 31 13:17 wwn-0x500a075112468483-part1 -> 
> ../../sda1
> lrwxrwxrwx 1 root root 10 Jul 31 13:17 wwn-0x500a075112468483-part2 -> 
> ../../sda2
> lrwxrwxrwx 1 root root 10 Jul 31 13:17 wwn-0x500a075112468483-part3 -> 
> ../../sda3
> =============== %< ==============
> $ ls -l /dev/disk/by-label
> total 0
> lrwxrwxrwx 1 root root 10 Jul 31 13:17 boot -> ../../sda1
> lrwxrwxrwx 1 root root 10 Jul 31 13:17 gentoo -> ../../sda3
> lrwxrwxrwx 1 root root 10 Jul 31 15:20 intenso -> ../../sdi1
> lrwxrwxrwx 1 root root 10 Jul 31 13:17 swap -> ../../sda2
> =============== %< ==============
> 
> However, when I boot a rescue system from a USB stick, the partition on the 
> USB is not detected. I already tried latest SystemRescueCD (default and 
> alternate kernel), Knoppix and the Gentoo Admin CD. Nothing, the partition 
> is not available.
> 
> What's the difference? Why does my kernel find this partition and the other 
> one's do not? It's pretty silly to have a backup drive and cannot access it 
> in question ;-)
> 
> - Jörg
> 
> 

I can only think of two reasons, the kernel on the livecd doesn't
support GPT (which is unlikely) or you're booting a 32-bit kernel live
USB. I am reasonably certain for drives > 2TB a 64-bit kernel and GPT
are required.

Dan

[gentoo-user] Re: Partition of 3TB USB drive not detected

[Jörg Schaible]

Hi Daniel,

thanks for your response.

Daniel Frey wrote:

[snip]
 
> I can only think of two reasons, the kernel on the livecd doesn't
> support GPT (which is unlikely)

That would be really strange. However, how can I prove it?

> or you're booting a 32-bit kernel live
> USB. I am reasonably certain for drives > 2TB a 64-bit kernel and GPT
> are required.

No, I've always chosen 64-bit kernels. I wonder what is so special about 
this partition ...

Cheers,
Jörg

Re: [gentoo-user] Partition of 3TB USB drive not detected

[james]

On 07/31/2016 08:37 AM, Jörg Schaible wrote:
> Hi,
>
> for my backups I use a 3TB USB drive (one big ext4 partition) without any
> problems. Just plug in the cable, mount it and perform the backup. The
> partition (sdi1) is detected an mountable without any problems:

this tells you the device is valid and working. good.
>
> However, when I boot a rescue system from a USB stick, the partition on the
> USB is not detected. I already tried latest SystemRescueCD (default and
> alternate kernel), Knoppix and the Gentoo Admin CD. Nothing, the partition
> is not available.

So there could be a multitude of reasons. Did thos systems have a bios 
that support booting from a usb device? Are the bios setting set to 
select the bios device correct?

I recently read somewhere the usb devices only support (2) types of 
image booting, but I cannot find that doc right now. rodsbooks is
an excellent resource for all the issues around booting and device
and various hardware/firmware issues. LikeWhoa has made booting the usb 
for gentoo, commonplace so search out those postings on the gentoo 
forums and in the wiki.


> What's the difference? Why does my kernel find this partition and the other
> one's do not? It's pretty silly to have a backup drive and cannot access it
> in question ;-)

It could be many things. You just have to ferret thru ideas until 
something works for your hardware, and it is a frustrating process.

make sure the image you are trying to boot is on a compatible partition 
and file system that is supported by the boot image.


hth,
James

[gentoo-user] Re: Partition of 3TB USB drive not detected

[Jörg Schaible]

Hi James,

james wrote:

> On 07/31/2016 08:37 AM, Jörg Schaible wrote:
>> Hi,
>>
>> for my backups I use a 3TB USB drive (one big ext4 partition) without any
>> problems. Just plug in the cable, mount it and perform the backup. The
>> partition (sdi1) is detected an mountable without any problems:
> 
> this tells you the device is valid and working. good.
>>
>> However, when I boot a rescue system from a USB stick, the partition on
>> the USB is not detected. I already tried latest SystemRescueCD (default
>> and alternate kernel), Knoppix and the Gentoo Admin CD. Nothing, the
>> partition is not available.
> 
> So there could be a multitude of reasons. Did thos systems have a bios
> that support booting from a usb device? Are the bios setting set to
> select the bios device correct?

Well, same machine, same bios.

> I recently read somewhere the usb devices only support (2) types of
> image booting, but I cannot find that doc right now. rodsbooks is
> an excellent resource for all the issues around booting and device
> and various hardware/firmware issues. LikeWhoa has made booting the usb
> for gentoo, commonplace so search out those postings on the gentoo
> forums and in the wiki.
> 
> 
>> What's the difference? Why does my kernel find this partition and the
>> other one's do not? It's pretty silly to have a backup drive and cannot
>> access it in question ;-)
> 
> It could be many things. You just have to ferret thru ideas until
> something works for your hardware, and it is a frustrating process.

Definitely annoying.
 
> make sure the image you are trying to boot is on a compatible partition
> and file system that is supported by the boot image.

It is. Thanks for the pointer to rodsbooks.

- Jörg

Re: [gentoo-user] Re: Partition of 3TB USB drive not detected

[Mick]

[-- Attachment #1: Type: text/plain, Size: 498 bytes --]

On Sunday 31 Jul 2016 19:14:45 Jörg Schaible wrote:
> Hi Daniel,
> 
> thanks for your response.
> 
> Daniel Frey wrote:
> 
> [snip]
> 
> > I can only think of two reasons, the kernel on the livecd doesn't
> > support GPT (which is unlikely)
> 
> That would be really strange. However, how can I prove it?

If after you boot your systemrescuecd you can list:

/sys/firmware/efi

you have booted into UEFI mode.  If not, you have booted into legacy BIOS 
mode.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[gentoo-user] Re: Partition of 3TB USB drive not detected

[Jörg Schaible]

Jörg Schaible wrote:

> Hi Daniel,
> 
> thanks for your response.
> 
> Daniel Frey wrote:
> 
> [snip]
>  
>> I can only think of two reasons, the kernel on the livecd doesn't
>> support GPT (which is unlikely)
> 
> That would be really strange. However, how can I prove it?
> 
>> or you're booting a 32-bit kernel live
>> USB. I am reasonably certain for drives > 2TB a 64-bit kernel and GPT
>> are required.
> 
> No, I've always chosen 64-bit kernels. I wonder what is so special about
> this partition ...

Currently I wonder, why my system can find the partition at all:

======================== %< ========================
# gdisk -l /dev/sdi
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: not present

Creating new GPT entries.
Disk /dev/sdi: 732566646 sectors, 2.7 TiB
Logical sector size: 4096 bytes
Disk identifier (GUID): 80C04475-9B51-4A44-A52F-1F165AE02695
Partition table holds up to 128 entries
First usable sector is 6, last usable sector is 732566640
Partitions will be aligned on 256-sector boundaries
Total free space is 732566635 sectors (2.7 TiB)

Number  Start (sector)    End (sector)  Size       Code  Name
======================== %< ========================

However, it's mounted successfully, see system logs:

======================== %< ========================
[22735.626752] sd 13:0:0:0: [sdi] 732566646 4096-byte logical blocks: (3.00 
TB/2.73 TiB)
[22735.629255]  sdi: sdi1
[23414.066315] EXT4-fs (sdi1): mounted filesystem with ordered data mode. 
Opts: (null)
======================== %< ========================

Has anyone ever tried the recovery option of GPT disk to rebuild GPT from 
MBR?

- Jörg

Re: [gentoo-user] Re: Partition of 3TB USB drive not detected

[Alarig Le Lay]

[-- Attachment #1: Type: text/plain, Size: 1592 bytes --]

On Sun Jul 31 19:56:33 2016, Jörg Schaible wrote:
> However, it's mounted successfully, see system logs:
> 
> ======================== %< ========================
> [22735.626752] sd 13:0:0:0: [sdi] 732566646 4096-byte logical blocks: (3.00 
> TB/2.73 TiB)
> [22735.629255]  sdi: sdi1
> [23414.066315] EXT4-fs (sdi1): mounted filesystem with ordered data mode. 
> Opts: (null)
> ======================== %< ========================

I always see such log when I mount a partition.

alarig@airmure ~ % dmesg | grep 'mounted filesystem'
[    3.145177] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: (null)
[    8.676445] EXT4-fs (sdg1): mounted filesystem with ordered data mode. Opts: (null)
[  777.844173] EXT4-fs (sdf1): mounted filesystem with ordered data mode. Opts: (null)
[  795.083528] EXT4-fs (sdf1): mounted filesystem with ordered data mode. Opts: (null)
[ 1264.427746] EXT4-fs (sdf1): mounted filesystem with ordered data mode. Opts: (null)
[87490.418548] EXT4-fs (sdf1): mounted filesystem with ordered data mode. Opts: (null)
[260291.765065] EXT4-fs (sdf1): mounted filesystem with ordered data mode. Opts: (null)
[346692.663156] EXT4-fs (sdf1): mounted filesystem with ordered data mode. Opts: (null)
[433094.161678] EXT4-fs (sdf1): mounted filesystem with ordered data mode. Opts: (null)
[519494.540467] EXT4-fs (sdf1): mounted filesystem with ordered data mode. Opts: (null)
[591990.890105] EXT4-fs (sdf1): mounted filesystem with ordered data mode. Opts: (null)

(sdf1 is daily mounted and umounted due to a cron)

-- 
alarig

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-user] Re: Partition of 3TB USB drive not detected

[james]

On 07/31/2016 12:56 PM, Jörg Schaible wrote:
> Jörg Schaible wrote:
>
>> Hi Daniel,
>>
>> thanks for your response.
>>
>> Daniel Frey wrote:
>>
>> [snip]
>>
>>> I can only think of two reasons, the kernel on the livecd doesn't
>>> support GPT (which is unlikely)
>>
>> That would be really strange. However, how can I prove it?
>>
>>> or you're booting a 32-bit kernel live
>>> USB. I am reasonably certain for drives > 2TB a 64-bit kernel and GPT
>>> are required.
>>
>> No, I've always chosen 64-bit kernels. I wonder what is so special about
>> this partition ...
>
> Currently I wonder, why my system can find the partition at all:
>
> ======================== %< ========================
> # gdisk -l /dev/sdi
> GPT fdisk (gdisk) version 1.0.1
>
> Partition table scan:
>   MBR: protective
>   BSD: not present
>   APM: not present
>   GPT: not present

If you have seen my recent thread, much of this automounting during 
boot(strapping) is flaky that is much of what I have been searching out
is a default (magical) partitioning schema that will eventually lead to
clear documents on the current state of affairs not only with old versus 
new motherboards (mbr-->efi) and disk (mbr < 2.2T and gpt >2.2T)
but including all sorts of new arm and other embedded (linux) boards.

Different forms of Solid State memory are next on my list, with usb (1.x 
--> 3.x) being top of the SS memory mediums..... (Sorry I do not have 
more atm).


>
> Creating new GPT entries.
> Disk /dev/sdi: 732566646 sectors, 2.7 TiB
> Logical sector size: 4096 bytes
> Disk identifier (GUID): 80C04475-9B51-4A44-A52F-1F165AE02695
> Partition table holds up to 128 entries
> First usable sector is 6, last usable sector is 732566640
> Partitions will be aligned on 256-sector boundaries
> Total free space is 732566635 sectors (2.7 TiB)
>
> Number  Start (sector)    End (sector)  Size       Code  Name
> ======================== %< ========================
>
> However, it's mounted successfully, see system logs:
>
> ======================== %< ========================
> [22735.626752] sd 13:0:0:0: [sdi] 732566646 4096-byte logical blocks: (3.00
> TB/2.73 TiB)
> [22735.629255]  sdi: sdi1
> [23414.066315] EXT4-fs (sdi1): mounted filesystem with ordered data mode.
> Opts: (null)
> ======================== %< ========================
>
> Has anyone ever tried the recovery option of GPT disk to rebuild GPT from
> MBR?

I see some sort of 'auto correction' by gpt technology to convert many 
forms of perceived mbr to gpt to be used by the booting process for 
spinning rust. So this issue is not limited to usb medium. I would also 
point out that I'd look deeply into the usb specs for the vendor of your 
usb sticks, as they do some 'funky things' at the firmware level inside 
many of the newer/faster/larger usb devices. It not just dumb memory 
like the early 1.x devices. Many are slanted to Microsoft business 
strategies. I'm not suggesting that is your current issues. I'm merely 
pointing out that some newer usb sticks are systems themselves complete 
with firmware so the devices looks like dumb memory. Furthermore, the 
silicon vendors provide firmware options to usb sticks vendors (like 
Texas Instruments) but also the vendor add to or change the hidden 
firmware as meets their multifaceted business objects. Sadly, the NSA is 
deeply involved here, as are many nation states and large corporations. 
You'd be surprised what youd find in a modern usb stick, should you take 
it into a class 6+ clean-room for analysis. The lower the particle count 
the more fantastic the tools
to open up silicon and look deeply into what is actually going on.
This is why folks love those classified research facilities that have 
govt contract and folks hanging around. Lots of very, very cool toys
you just do not hear about...... Way beyond microscopes built by 
physicist.....

Prolly not your issue, but still present. Cheap ass usb vendors often 
have corner issues that are unintentional, that is why well recognized 
vendors of SS memory are the best to deal with, for consistency of behavior.

I'd use as many different tools as you can find and read the vendor & 
silicon manufacturer's docs to see what you are really dealing with to 
ferret out this weirdness. (it's a darn time sync, just so you know).


[1] http://www.cleanroom.byu.edu/particlecount.phtml

hth,
James

[gentoo-user] Re: Re: Partition of 3TB USB drive not detected

[Jörg Schaible]

Hi Mick,

Mick wrote:

> On Sunday 31 Jul 2016 19:14:45 Jörg Schaible wrote:
>> Hi Daniel,
>> 
>> thanks for your response.
>> 
>> Daniel Frey wrote:
>> 
>> [snip]
>> 
>> > I can only think of two reasons, the kernel on the livecd doesn't
>> > support GPT (which is unlikely)
>> 
>> That would be really strange. However, how can I prove it?
> 
> If after you boot your systemrescuecd you can list:
> 
> /sys/firmware/efi
> 
> you have booted into UEFI mode.  If not, you have booted into legacy BIOS
> mode.


This machine has only plain old BIOS. The question is, why one kernel 
detects the 3TB partition and the the other one does not. How can I prove 
GPT support for the kernel itself.

Cheers,
Jörg

[gentoo-user] Re: Re: Partition of 3TB USB drive not detected

[Jörg Schaible]

james wrote:

> On 07/31/2016 12:56 PM, Jörg Schaible wrote:
>> Jörg Schaible wrote:
>>
>>> Hi Daniel,
>>>
>>> thanks for your response.
>>>
>>> Daniel Frey wrote:
>>>
>>> [snip]
>>>
>>>> I can only think of two reasons, the kernel on the livecd doesn't
>>>> support GPT (which is unlikely)
>>>
>>> That would be really strange. However, how can I prove it?
>>>
>>>> or you're booting a 32-bit kernel live
>>>> USB. I am reasonably certain for drives > 2TB a 64-bit kernel and GPT
>>>> are required.
>>>
>>> No, I've always chosen 64-bit kernels. I wonder what is so special about
>>> this partition ...
>>
>> Currently I wonder, why my system can find the partition at all:
>>
>> ======================== %< ========================
>> # gdisk -l /dev/sdi
>> GPT fdisk (gdisk) version 1.0.1
>>
>> Partition table scan:
>>   MBR: protective
>>   BSD: not present
>>   APM: not present
>>   GPT: not present
> 
> If you have seen my recent thread,

I saw it, but did not read it in depth, because I had the impression, it is 
mainly about EFI systems. I'll re-read it ...

> much of this automounting during
> boot(strapping) is flaky that is much of what I have been searching out
> is a default (magical) partitioning schema that will eventually lead to
> clear documents on the current state of affairs not only with old versus
> new motherboards (mbr-->efi) and disk (mbr < 2.2T and gpt >2.2T)
> but including all sorts of new arm and other embedded (linux) boards.
> 
> Different forms of Solid State memory are next on my list, with usb (1.x
> --> 3.x) being top of the SS memory mediums..... (Sorry I do not have
> more atm).
>
>> Creating new GPT entries.
>> Disk /dev/sdi: 732566646 sectors, 2.7 TiB
>> Logical sector size: 4096 bytes
>> Disk identifier (GUID): 80C04475-9B51-4A44-A52F-1F165AE02695
>> Partition table holds up to 128 entries
>> First usable sector is 6, last usable sector is 732566640
>> Partitions will be aligned on 256-sector boundaries
>> Total free space is 732566635 sectors (2.7 TiB)
>>
>> Number  Start (sector)    End (sector)  Size       Code  Name
>> ======================== %< ========================
>>
>> However, it's mounted successfully, see system logs:
>>
>> ======================== %< ========================
>> [22735.626752] sd 13:0:0:0: [sdi] 732566646 4096-byte logical blocks:
>> [(3.00
>> TB/2.73 TiB)
>> [22735.629255]  sdi: sdi1
>> [23414.066315] EXT4-fs (sdi1): mounted filesystem with ordered data mode.
>> Opts: (null)
>> ======================== %< ========================
>>
>> Has anyone ever tried the recovery option of GPT disk to rebuild GPT from
>> MBR?
> 
> I see some sort of 'auto correction' by gpt technology to convert many
> forms of perceived mbr to gpt to be used by the booting process for
> spinning rust. So this issue is not limited to usb medium. I would also
> point out that I'd look deeply into the usb specs for the vendor of your
> usb sticks, as they do some 'funky things' at the firmware level inside
> many of the newer/faster/larger usb devices. It not just dumb memory
> like the early 1.x devices. Many are slanted to Microsoft business
> strategies. I'm not suggesting that is your current issues. I'm merely
> pointing out that some newer usb sticks are systems themselves complete
> with firmware so the devices looks like dumb memory. Furthermore, the
> silicon vendors provide firmware options to usb sticks vendors (like
> Texas Instruments) but also the vendor add to or change the hidden
> firmware as meets their multifaceted business objects. Sadly, the NSA is
> deeply involved here, as are many nation states and large corporations.
> You'd be surprised what youd find in a modern usb stick, should you take
> it into a class 6+ clean-room for analysis. The lower the particle count
> the more fantastic the tools
> to open up silicon and look deeply into what is actually going on.
> This is why folks love those classified research facilities that have
> govt contract and folks hanging around. Lots of very, very cool toys
> you just do not hear about...... Way beyond microscopes built by
> physicist.....

Actually it is not that modern. ~5 year old Intenso 2GB. I'd be surprised if 
booting from the stick prevents partition detection of another USB drive, 
but who knows? Maybe I should burn the iso instead and boot that one ;-)

> Prolly not your issue, but still present. Cheap ass usb vendors often
> have corner issues that are unintentional, that is why well recognized
> vendors of SS memory are the best to deal with, for consistency of
> behavior.
> 
> I'd use as many different tools as you can find and read the vendor &
> silicon manufacturer's docs to see what you are really dealing with to
> ferret out this weirdness. (it's a darn time sync, just so you know).
> 
> 
> [1] http://www.cleanroom.byu.edu/particlecount.phtml
> 
> hth,
> James

Thanks,
Jörg

Re: [gentoo-user] Re: Re: Partition of 3TB USB drive not detected

[Mick]

[-- Attachment #1: Type: text/plain, Size: 1230 bytes --]

On Sunday 31 Jul 2016 22:38:22 Jörg Schaible wrote:
> Hi Mick,
> 
> Mick wrote:
> > On Sunday 31 Jul 2016 19:14:45 Jörg Schaible wrote:
> >> Hi Daniel,
> >> 
> >> thanks for your response.
> >> 
> >> Daniel Frey wrote:
> >> 
> >> [snip]
> >> 
> >> > I can only think of two reasons, the kernel on the livecd doesn't
> >> > support GPT (which is unlikely)
> >> 
> >> That would be really strange. However, how can I prove it?
> > 
> > If after you boot your systemrescuecd you can list:
> > 
> > /sys/firmware/efi
> > 
> > you have booted into UEFI mode.  If not, you have booted into legacy BIOS
> > mode.
> 
> This machine has only plain old BIOS. The question is, why one kernel
> detects the 3TB partition and the the other one does not. How can I prove
> GPT support for the kernel itself.


I see.  In this case have a look at /proc/config (it may be compressed) or 
depending on your version of sysrescuecd and kernel choice, have a look here:

https://sourceforge.net/p/systemrescuecd/code/ci/master/tree/

then compare your configuration to theirs.  The kernel module for GPT is 
'CONFIG_EFI_PARTITION' and it must be built in, rather than as a separate 
module.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-user] Partition of 3TB USB drive not detected

[J. Roeleveld]

On Sunday, July 31, 2016 03:37:55 PM Jörg Schaible wrote:
> Hi,
> 
> for my backups I use a 3TB USB drive (one big ext4 partition) without any
> problems. Just plug in the cable, mount it and perform the backup. The
> partition (sdi1) is detected an mountable without any problems:
> 
> =============== %< ==============
> $ ls -l /dev/disk/by-id
> total 0

<snipped bunch of 'ls' statements

> =============== %< ==============
> 
> However, when I boot a rescue system from a USB stick, the partition on the
> USB is not detected. I already tried latest SystemRescueCD (default and
> alternate kernel), Knoppix and the Gentoo Admin CD. Nothing, the partition
> is not available.
> 
> What's the difference? Why does my kernel find this partition and the other
> one's do not? It's pretty silly to have a backup drive and cannot access it
> in question ;-)

Which kernel do you boot?
The systerescue-cd has 4 kernels:
2 * 64bit and 2 * 32bit.

By default, it boots the "default" one for the architecture you are booting.
Have you tried booting the "alternate" kernel?

I have 1 system that I need to boot using the "alternate" kernel as the 
"default" one is too old. (Yes, by default it boots an old kernel)

It could easily be that the kernel you are using does not support your USB3 
adapter or something else you used.

Eg. apart from all the 'ls' statements, also check "uname" and 
"/proc/config.gz" for differences.

--
Joost

Re: [gentoo-user] Partition of 3TB USB drive not detected

[james]

On 08/01/2016 01:45 AM, J. Roeleveld wrote:
> On Sunday, July 31, 2016 03:37:55 PM Jörg Schaible wrote:
>> Hi,
>>
>> for my backups I use a 3TB USB drive (one big ext4 partition) without any
>> problems. Just plug in the cable, mount it and perform the backup. The
>> partition (sdi1) is detected an mountable without any problems:
>>
>> =============== %< ==============
>> $ ls -l /dev/disk/by-id
>> total 0
>
> <snipped bunch of 'ls' statements
>
>> =============== %< ==============
>>
>> However, when I boot a rescue system from a USB stick, the partition on the
>> USB is not detected. I already tried latest SystemRescueCD (default and
>> alternate kernel), Knoppix and the Gentoo Admin CD. Nothing, the partition
>> is not available.
>>
>> What's the difference? Why does my kernel find this partition and the other
>> one's do not? It's pretty silly to have a backup drive and cannot access it
>> in question ;-)
>
> Which kernel do you boot?
> The systerescue-cd has 4 kernels:
> 2 * 64bit and 2 * 32bit.
>
> By default, it boots the "default" one for the architecture you are booting.
> Have you tried booting the "alternate" kernel?
>
> I have 1 system that I need to boot using the "alternate" kernel as the
> "default" one is too old. (Yes, by default it boots an old kernel)
>
> It could easily be that the kernel you are using does not support your USB3
> adapter or something else you used.
>
> Eg. apart from all the 'ls' statements, also check "uname" and
> "/proc/config.gz" for differences.

I was just reading about "IOMMU" and how often, if it is not "correctly 
configured" in the kernel and other places, your memory map to other 
hardwawre, like USB, can be flaky or not work at all. Fixes often 
require loading the latest bios for your motherboard. It also matters 
the 'rev' of your motherboard and other details.

I have a Gigabyte GA-990FXA-ud3 that seems to be a victim of this bug. 
No, I have not had time to ferret out this issue, so here are a few raw 
links where it is talked about::

https://www.reddit.com/r/linux/comments/4ixnyg/question_about_iommu/

https://en.wikipedia.org/wiki/Input–output_memory_management_unit

http://developer.amd.com/community/blog/2008/09/01/iommu/

http://pages.cs.wisc.edu/~basu/isca_iommu_tutorial/

https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware


(more posted if you ask)...


hth,
James

[gentoo-user] Re: Re: Re: Partition of 3TB USB drive not detected

[Jörg Schaible]

Hi Mick

Mick wrote:

> On Sunday 31 Jul 2016 22:38:22 Jörg Schaible wrote:
>> Hi Mick,
>> 
>> Mick wrote:
>> > On Sunday 31 Jul 2016 19:14:45 Jörg Schaible wrote:
>> >> Hi Daniel,
>> >> 
>> >> thanks for your response.
>> >> 
>> >> Daniel Frey wrote:
>> >> 
>> >> [snip]
>> >> 
>> >> > I can only think of two reasons, the kernel on the livecd doesn't
>> >> > support GPT (which is unlikely)
>> >> 
>> >> That would be really strange. However, how can I prove it?
>> > 
>> > If after you boot your systemrescuecd you can list:
>> > 
>> > /sys/firmware/efi
>> > 
>> > you have booted into UEFI mode.  If not, you have booted into legacy
>> > BIOS mode.
>> 
>> This machine has only plain old BIOS. The question is, why one kernel
>> detects the 3TB partition and the the other one does not. How can I prove
>> GPT support for the kernel itself.
> 
> 
> I see.  In this case have a look at /proc/config (it may be compressed) or
> depending on your version of sysrescuecd and kernel choice, have a look
> here:
> 
> https://sourceforge.net/p/systemrescuecd/code/ci/master/tree/
> 
> then compare your configuration to theirs.  The kernel module for GPT is
> 'CONFIG_EFI_PARTITION' and it must be built in, rather than as a separate
> module.

Now it's getting weird.

My normal kernel (4.4.6) does not have that flag set. Nevertheless it 
detects the partition. The two kernels (both 4.4.12) of systemrescue have 
that flag. I've tested another machine with kernel that also has the flag 
and it does not detect this partition also.

However, I have another 6TB USB drive and that one has one big partition 
that is detected by both machines. The funny thing is, it reports to have 
only a MBR with one partition of 6TB (same output on both kernels):

================== %< ======================
~ # parted /dev/sde print
Model: WD My Book 1230 (scsi)
Disk /dev/sde: 6001GB
Sector size (logical/physical): 4096B/4096B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      8389kB  6001GB  6001GB  primary  ext4

~ # gdisk /dev/sde
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present


***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
typing 'q' if you don't want to convert your MBR partitions
to GPT format!
***************************************************************


Command (? for help): q
================== %< ======================

AFAICS this partition works fine, fsck does not report any problem. The 
funny thing is, it should not have been possible, because of the 2GB limit 
of MBR.

???

- Jörg

Re: [gentoo-user] Partition of 3TB USB drive not detected

[Andrea Conti]

Hi,

> ~ # parted /dev/sde print
> Model: WD My Book 1230 (scsi)
> Disk /dev/sde: 6001GB
> Sector size (logical/physical): 4096B/4096B

[...]

> AFAICS this partition works fine, fsck does not report any problem. The 
> funny thing is, it should not have been possible, because of the 2GB limit 
> of MBR.

The real limit of MBR is 2^32 sectors, which amounts to 2TB when using
512B sectors. Both your disks are using native 4kB sectors (look at the
logical sector size in the parted output), which effectively raises the
MBR limit to 16TB.

Not that this answers your question, as the Linux kernel has supported
4kB sectors for years and AFAIK it does not need any special
configuration options to do so...

andrea

Re: [gentoo-user] Partition of 3TB USB drive not detected

[james]

On 08/06/2016 02:21 AM, Andrea Conti wrote:
> Hi,
>
>> ~ # parted /dev/sde print
>> Model: WD My Book 1230 (scsi)
>> Disk /dev/sde: 6001GB
>> Sector size (logical/physical): 4096B/4096B
>
> [...]
>
>> AFAICS this partition works fine, fsck does not report any problem. The
>> funny thing is, it should not have been possible, because of the 2GB limit
>> of MBR.
>
> The real limit of MBR is 2^32 sectors, which amounts to 2TB when using
> 512B sectors. Both your disks are using native 4kB sectors (look at the
> logical sector size in the parted output), which effectively raises the
> MBR limit to 16TB.
>
> Not that this answers your question, as the Linux kernel has supported
> 4kB sectors for years and AFAIK it does not need any special
> configuration options to do so...
>
> andrea

It's always a good idea to check alignment on the newer disks. Here's a 
doc [1]; but it's not always clear how to ensure proper alignment. 
Supposidly the newer tools do this automation, but it's not 100%; ymmv.


[1] 
https://www.ibm.com/developerworks/linux/library/l-linux-on-4kb-sector-disks/

Many disk are hybrids of the 512/4906 sector size.



hth,
James