PDA

View Full Version : *RANT* NAS dropped raid5 array completely *RANT*


Malignant
11-07-10, 09:49
I just need to get some feelings of my chest here, and i hope they serve as a warning or wakeup call for other Qnap NAS users.

Yesterday i came home and i thought i'd watch a nice movie or something.

Booted up the AV center and sat down, navigated my Dune to the NAS, and it just wouldn't connect.
No panic here, happens sometimes because the dune sometimes needs a few minutes to get a DHCP addy.
Tried several more time's but still couldn't connect.
So i looked at my dune's settings and it had an IP addy.

So i dove under the closet to look at my NAS.
And then the horror hit me.

The VFD of my NAS was saying "choose hd setup".
*censored*, Choose HD setup ? I have a running raid 5 array wich worked properly this morning.

Rebooted the damn thing, and again "choose hd setup".

*censored* I've had a solo disk die on me once, now i have a friggin raid5 array wich is supposed to be safe. 2 HD deaths in one afternoon ?
No, that wasn't the case. All my HD's work flawless.

But then i noticed that my NAS had a different IP addy.
This isn't possible since i've set a static ip.

*censored*

Did some browsing around the web, and i've found a lot more people who have had this.

Seems like the Qnap can mess up it's internal config file completely on it's own, without any help.

This results in the breaking of the entire array and raid data tables.

*censored*

Raid 5 is supposed to be safe ? How the hell can this happen ?
How in gods name can a NAS corrupt it's self so bad that it loses all it's data tables.

Currently still waiting for the Qnap support guys.
It seems they can login remotely via SSH and fix the broken config file.

Let's just hope that they do, because my girlfriend is devestated.
I've had the vacation vid's on my NAS as well, as i thought they were safer there than on my PC.
Seems i was wrong.

So as a warning to all, do not think you're data is safe on your NAS.
If something as this can happen so easily i don't know what is safe any more.

Burning the *censored* on DVD's i guess from now on.
And reburning it every year, because all media that you can use to burn decays due to a chemical process being started after the burn.

Be warned everybody, you're NAS might kill you're entire array.

Damn that feels good, sorry if anybody is offended by this but i needed to get it off my chest.

HenkNL
11-07-10, 14:13
RAID IS NO BACKUP.

A backup is a storage on another data source. So you have to buy 2 nasses, exactly the same so even if one nas dies you still have the other one.

Henk

P.S. Could your nas be 'damaged' by the big thunder&lightning that visited the NL last night ?

Malignant
11-07-10, 14:40
I know RAID is no back-up.

But the redundancy of a raid 5 array should provide some extra form of safety.

But the problem with the Qnap is that it corrupts it's own config file so the array drops entirely.
I've read the qnap forums are there are a lot more reports of users having these problems.

Supposedly it's only the datatables in the config file that are corrupted.
The actual data is still there, the NAS just doesn't know how to mount it.

I find it shocking that a simple config file can cause all this.

P.S.

Seriously thinking about buying a second 4 bay raid 5 solution for back up of my current NAS.

P.P.S.

It wasn't lightning last night, all my other equipment is set on the same group and that all functions properly.
So no power surge here.

mysticc
11-07-10, 16:05
In my eyes, for simple backup RAID is overkill.

Was thinking on doing the same (Unraid) - however than thought "why would I need RAID for a backup?

I did not find any arguments justifyng the extra costs, so I just took an old machine standing in the corner, put old Vista on it and a bunch of disks and backuped up everything onto simple volumes, nothing special, so when a disk fails only data on this disk has to be copied again from the NAS.

The only thing is as my HDs in the Backup-machine can`t be as big as the volume on my NAS (5,7 TB) copying the NAS-volume to the several disks in the backup-machine has to be done in several steps.
Some loss of comfort, but I can live with it.

With PCI-4x-SATA-Controllers I can expand supported numbers of disks as soon as 7 drives (in my case) are no longer enough.
When putting bigger disk into my NAS all my older disks go into this machine.
When space in the case for disks gets low I will use 5,25-cases providing space for 5 3,5 disks (takes 3 5,25-slots)

Seems to be almost the cheapest solution to backup your NAS and your older disks still can be used.

In case your NAS has eSATA one could use an external eSATA-dockingstation, simply put in the naked disks one after one and copy back your data.
My Syno DS1010+ is providing eSATA, but I don`t like my backup-disks to be formatted with some Linux-FS.
For this reason I choosed a complete windows-machine for backups, so I can read the disks on my other windows-machine in case needed without the need of some additional FS-drivers.

In case you can go with Linux-FS and don`t mind fiddling around with the disks an eSATA-dockingstation really is the cheapest solution, but nevertheless pretty fast.

a5ian300zx
12-07-10, 11:20
Hi,

sorry to hear about your experience. just few questions

1) Does you router have static or dynamic IP address?

2) which firmware are you running on the NAS?

3) did you have any power cuts/surges which could have caused the system to reboot inc router?

a5ian300zx

PS: I have mine setup with dynamic IP addy on Router and NAS and I use DynDNS for remote access to the NAS.

Hi-Jack
12-07-10, 11:41
I take offense by such post :-) As a moderator, one needs to set an example for the rest to present a constructive post of issues and problems and we do not appreciate language like "shit" and "what the fuck" from any users, especially moderators, no matter how well we understand the issue at your place and the severity.

As for the QNAP, the config is stored on the hard drives so in any way, the config can be restored as long as no disks are broken. Untill this is 100% verified also by QNAP, I feel we may be jumping to conclusions... Besides the fact it may be an issue with the QNAP, it may also be "an additional component" you have installed besides the qnap supported ones, fiddling with telnet and ssh or by other means that cause instability, we'll never know.

If QNAP logs in remotely, I'm sure if no hardware failure has occurred, they are able to rebuild the original RAID5 without data loss... and as with all NAS units, you will find people having lost RAID's and so on... It's a risk one takes just as a single hard drive may fail... but with RAID5 there's less of a chance... and backups are out of the questions (2 NAS for the purpose of backups is overklill) however backing up "important" data you will feel extremely bad about is always an interesting recommendation.

PS: A table can only get corrupted if something is changed and it is badly written to disk... otherwise it's more of a read only file so something must have happened somehow or as a result to something...

Agree, QNAP should look into this to make sure there is no issue (backup system to restore a config file maybe?) and we will discuss with them. Keep us posted on your NAS when QNAP had a remote session... I'm pretty confident they will be able to recover it in case it is just the config file...

It is however a valuable lesson to all, that RAID5 and NAS are indeed not "failsafe" systems... They are more reasonable safety for bigger volumes...

twol
12-07-10, 11:44
I have just installed a USB 3.0 card in my PC and a device where you can just dock a drive. It is really fast so after reading this will start to plan to back up my NAS drives - although I hate the additional cost regardless of the "feeling safe(r) mindset

TheLastMohican
12-07-10, 19:57
If QNAP logs in remotely, I'm sure if no hardware failure has occurred, they are able to rebuild the original RAID5 without data loss

One question regarding the rebuild here, is this something the user can do himself
or something QNAP must do ?

Does the same go for other manufactures like Synology, Thecus, ReadyNAS, ... ?

I wonder since i known that for instance on a EONstor RAID you can do a rebuild
yourself whenever that's needed.

Or am i just asking a silly and stupid question now :?: :)

rudku
12-07-10, 23:43
@Malignant: Iīm sorry for casualty with your QNAP NAS.


For others I have some my advices about NAS with RAID5:
(I work only with QNAP, but these advices are valid generally for NAS of other producers)


1) run NAS only with UPS - best way is when UPS is supported by NAS, then can be properly shut-downed during power failure

2) use only RAID models of HDDs - every producer has special models of HDD with longer MTBF
- use the HDDs from the same delivery

3) backup your personal (sensitive) data - on QNAP NAS you can do it easy via function "Backup to External Drive" to any USB or eSATA external drive

4) backup your system settings of your NAS (on QNAP is it function "Backup System Settings").

5) be carefully with upgrading of FW on your NAS - donīt be hurry upgrade immediately after publishing a new version of FW. If you donīt have a special reason for any problem, if everything working, you donīt need a upgrade. (... NAS is not a Mediaplayer ...)

6) have a prepared a spare one HDD from same delivery



RAID5 is not a backup!
... but with these several rules will be care of your digital data more secure...

siamsquare
13-07-10, 00:49
Yo

Well keeping that in mind I copied all my home movie backups from my QNAP to my ReadyNAS Pro..just to be sure ;)

gr33tz

Malignant
13-07-10, 08:47
Problem is fixed.

Qnap logged into my NAS remotely and they reconstructed the array.

Seems there can arise problems when updating FW on your NAS.
They haven't found the exact source tho.
It occured with me when i updated to the 3.3.0 FW version.

Problem is that ALL settings are stored in a file in the root of the first HD.
After the update something got corrupted in this file ( uLinux.conf ) wich caused the Qnap to not recognize the entire array.

Just a BIG warning if this happens to you :
DO NOT RECONSTRUCT YOUR ARRAY, LEAVE IT BROKEN !

Qnap tech support logged in remotely and they reconstructed and remounted the array via putty.

Note that this is not something you can do on your own.
Believe me, i've seen commands flying by i've never even heard of.

Basically it just break's the array up in 4 different disks and since it was running Raid5 the NAS couldn't make anything of it.

There was no actual data lost during the proces ( thank god, my girlfriend almost wanted to kill me ).

I'm with Rudku on point 5. If it ain't broke don't update your FW.

My apologies for the strong language in the first post.
I was not in a correct state of mind at the time.

But for as far as i read on the Qnap forums this happens totally at random.
There are several cases of this happening, but i can't seem to draw one conclusion out of this. There is no single source i can piont to for this problem.

Just take note of my experience when this happens, and don't go messing around yourself.

P.S. Edited to topic title to better reflect it's contents.

greywall
13-07-10, 10:13
Problem is fixed.

Qnap logged into my NAS remotely and they reconstructed the array.

Seems there can arise problems when updating FW on your NAS.


Hi Malignant
Can I know how long it took to fix the problem with the team Qnap?
Because even I have a similar problem with my TS-509 pro.
Thank you .

Malignant
13-07-10, 10:22
I hopped on MSN to contact their tech support.
Got a reply in approximately 30 minutes.

They logged into my PC via TeamviewerQS.
Took about ~45 mins to reconstruct everything.

Just make the needed preperations for them to make to process go faster.
Have your desktop set up with a shortcut to the QnapFinder, Putty & WinSCP.

The major problem i had is that they are in the GMT+8 timezone.
I had to take a half day off.

Working hours for them are 9:30 till 24:00.
So for me that was 1:30 till 17:00.

greywall
13-07-10, 11:18
OK then is an operation that i can do it.
It only remains to be seen whether that problem will be rolled out again ...

I instead I decided to make backups of all data and reinitialize the NAS starting at 0. Useless that I'll tell you how many days of work I lost.

At this point we see for those of us, the array endures longer. ;)

Thanks again
greywall

Malignant
11-09-10, 12:59
Well, bumping up my old topic again unfortunately.

As of this morning my raid array is dropped again.
Haven't updated FW yet because i thought the corruption occurred during the FW upgrade i did.

But by the looks of it there is something seriously broken in FW 3.3.0.
Unfortunately it's weekend now so no way to get in touch with Qnap support.
When i do i'll try to get some more info from them regarding these issues.

For now i would suggest to stay away from FW 3.3.0 until everything is ruled out.

rudku
12-09-10, 14:32
Well, bumping up my old topic again unfortunately.

As of this morning my raid array is dropped again.
Haven't updated FW yet because i thought the corruption occurred during the FW upgrade i did.

But by the looks of it there is something seriously broken in FW 3.3.0.
Unfortunately it's weekend now so no way to get in touch with Qnap support.
When i do i'll try to get some more info from them regarding these issues.

For now i would suggest to stay away from FW 3.3.0 until everything is ruled out.

My experience with QNAP FW 3.3:
- I upgraded from FW 3.1.1 Build0815 to FW 3.3.1 Build0720 on end of July
- any problems during and after upgrade
- only issue with spindown of HDDs
- I upgraded to latest FW 3.3.2 Build0819 on end of August
- spindown of HDDs working now better
- any other issue