Jump to content
Scythe

Linux NTFS fileserver with rsync backup

Recommended Posts

Good day,

 

I'm trying to figure out a system so that I can use hardware RAID under Linux (probably FC6 or similar) and have the box act as a fileserver that works with Linux and Windows. I want the ability to read/write files in Windows and Linux from a different box on the workgroup.

 

I've read about the sofware RAID that I can do in Linux, but that doesn't give me the ability to read/write using a Windows box. And I've been trying to find RAID cards that are compatible with Linux under hardware RAID but haven't found any definitive answers. I was thinking about buying this one but of course none of the reviews say anything about using it in hardware RAID.

 

Much as I wish I could drop Windows, it's not an option. Can't play my games or do Photoshop and stuff in Linux (at least, I don't feel like messing with Wine or anything).

Edited by Scythe

Share this post


Link to post
Share on other sites

I have a sata/sis raid controller in my father's server and it is working with Linux. I'll have to connect and get the lspci so you can see what it is. It came in a Dell server I bought, but as far as I know it's a pci card unless I'm mistaken.

 

The cheaper cards you should stay away from, as they are mostly software raid embedded on a chip. I'll find out some more info later, and update.

Share this post


Link to post
Share on other sites

Sil3114 is software raid

cheap=software

 

I've been trying to find RAID cards that are compatible with Linux under hardware RAID

If it is hardware raid then it is OS independent, no?

 

I always see the same recommendation for HW raid in forums

3ware

but read the comments there

"Linux Software RAID vs. Hardware RAID" for a story

http://jeremy.zawodny.com/blog/archives/008696.html

 

People seems still happy with 3ware, I just give the link

because it is a healthy debate. Paranoia is good

 

otherwise look into HW linux databases

Share this post


Link to post
Share on other sites

Yeah, I read that page before. I've been looking for 3ware cards, but most are too expensive for my budget :( Guess I'll have to keep checking ebay.

Share this post


Link to post
Share on other sites

Providing there is a driver for the card in Linux. Whilst the raid is handled transparently, you might not actually be able to see any disks in Linux if the card has no module available to be added during install, or available in a kernel.

 

Sometimes, you can see the card, but you still see two separate disks. This also indicates that Linux is not seeing the card as it should be. Found this with a HP server recently, was seeing two disks, even though I configured mirroring. On booting with linux dd, and supplying the driver it saw just one disk.

Share this post


Link to post
Share on other sites

Let me see if I'm understanding this correctly - I'd have to install with the command #linux dd for it to recognize the RAID drives correctly? If I'm mistaken, how would I do this with, say, FC6?

Share this post


Link to post
Share on other sites

Scythe:

It depends on the RAID controller....

The terms simple and complex don't really fit because they swap over somewhat...

 

Perhaps the easiest way to explain is to take a step back....

Take a normal SCSI controller...

 

You can install onto a SCSI disk BUT you need the controller to be supported...not ALL SCSI controllers are HW... I once got one with an old scanner which was a SCSI equivalent of a win printer or winmodem.

 

However you plug in your SCSI devices to a real sCSI controller and the onboard firmware takes care of everything... for instance I have a AHA2940UW in my GF's machine... at the moment its only connected to a scanner. I can however attach an internal or external disk, CD writer or a whole plethora of stuff. Once I do it will just be recognised...

I can set it up outside of any OS at all, it has its own BIOS (which is firmware and upgradable) so long as I have the driver for it then the peripherals just work... I do however need SW.. so for my scannert to work I need a TWAIN driver and for the CD writer to write I need the libs for CD writing...

 

RAID controllers are pretty much the same... the best of them have their own LCD controls... you set the array and you can even do it not connected to the computer at all.

When you finished setting them up you just connect them to a SCSI controller and the SCSI controller SEE's them as a single disk. It even has a transposed geometry....

At the other end you have WinRAID, this is just SW RAID...

Inbetween you have lots which is less expensive than a stand alone cabinet (some RAID arrays are the size of a small room) ... some have the configuration stuff using an external program... so for instance you might need Windows to define the arrays but once they are defined they appear as a single disk presuming you have the SW...

 

In the end you might want to consider why you want a RAID array?

You are going to have to use a common filesystem... which is basically FAT32 or something like reiser... or you could use NTFS but its not dreadfully reliable...

FAT32 is very limited on permissions etc.

 

You could for instance use Xen and run SW RAID in linux with samba and then the Windows "machine" could use these disks... I do something not completely different but I only use Windows occaisionally so I have a VMWARE session (not good for gaming but that doesn't bother me) and I just use samba to read/write ...

Perhaps something along these lines but with Xen might provide closer to what you want?

Share this post


Link to post
Share on other sites

It would be a SATA raid card, just to clarify. Definitely don't feel like messing with SCSI.

 

I PMed ianw a few weeks back and he said if I used software RAID in Linux that Windows machines wouldn't be able to read/write to it (at least that's what I thought he said, correct me if I'm wrong ;))

 

I want the array simply for data security. I have at least 10 gigs of pictures, 20+gigs of music, tv shows, and isos that I'd rather not lose and backing up to DVDs does not appeal to me at all. Agreed that FAT32 and NTFS aren't the best filesystems, but it would probably end up being one of the two or Ext2/3. This is still in the planning stage so I can figure out the details as I go along. That's why I posted here ;)

Edited by Scythe

Share this post


Link to post
Share on other sites

raid is a good thing but it does not "dispence", should not stop

you from having backups if what you have is important

 

>>backing up to DVDs does not appeal to me at all

Add another IDE or USB HD (or better? IDE in USB enclosure) and run rsync with a cron job

 

consider the cost of one more HD against loss/time/money if everything goes missing

Share this post


Link to post
Share on other sites
It would be a SATA raid card, just to clarify. Definitely don't feel like messing with SCSI.

 

I PMed ianw a few weeks back and he said if I used software RAID in Linux that Windows machines wouldn't be able to read/write to it (at least that's what I thought he said, correct me if I'm wrong ;))

Yes he's absolutely correct.... although I guess its not impossible to write a driver for Windows that could read the linux metadevices... but you'd then need a compatible filesystem on top....

I want the array simply for data security. I have at least 10 gigs of pictures, 20+gigs of music, tv shows, and isos that I'd rather not lose and backing up to DVDs does not appeal to me at all. Agreed that FAT32 and NTFS aren't the best filesystems, but it would probably end up being one of the two or Ext2/3. This is still in the planning stage so I can figure out the details as I go along. That's why I posted here ;)

OK, that makes it much easier..... indeed it opens up lots of simple options....

 

The simpest by far is work out what *needs* backing up and what's convenient and the like...

For me the RAW files from my photo's are by far the most important along with documents and offical stuff and my mailbox...

However I only have about 300 GB of photo's whereas I have something over a TB of movies and audio... but worse comes to worse I'll just download it.or rebuy the DVD or CD.. (I really can't be bothered re-ripping my DVD's for archival) so I just use an external 500GB disk for my backups and a simple rsych script to copy new only files across every so often.

In between I unplug thr disk and it sits in a seal metal box in my drawer... if I were really paranoid I'd take it to a friends house in case my place burns down...

 

This has an added advantage in that I can take the disk offsite .... but the added disadvantage I need to remember to do it every so often....

RAID isn't foolproof either... RAID 1 is the closest to looking after you data but "wastes" half the diskspace anyway.... RAID 5 is usually badly implemnented, even by so called professionals....

in short RAID 5 is useless without the RAID metadata.... at protecting ALL your data.

So if you keep the metadata on the array itself (the most common) its useless if 2 disks fail at the same time... this might sound unlikely but its not.... the reasons being often the disks are all of a similar age and by definition (being in a RAID 5 array) amount of activity. However when one diosk fails in a RAID 5 array all that info has to be rebuilt and shuffled about.... if your 2nd disk was almost ready to fail then.... well now would be when its being pushed the hardest...

 

Another disadvantage of a RAID array is they take lots of planning and stuff but once they're set up then you more or less forget about them ... until they fail...

If you buy a cheap external disk .. say 2-3 times the capacity you need right now... (which seems to be about 100GB) you can buy this really cheap.... use it as a backup and then sooner or later it will start to fill up.... at his point... just buy a 200GB one.... (assuming you can still buy 200GB ones then)

transfer everything and you just created an extra backup.... (at least up to that date) you can drop round a friends house in case you have a fire or flood.

Advantage is disk prioces per GB are dropping every month... but if your going to make a decent RAID array you need to pay for that space up front... once the space you paid for is full... you have a few options but most of them involve... yep buying more space up front...

 

A modern unplugged disk you spin up once a month is incredibly unlikely to fail.... and its the backup disk.... so you already have a 2nd copy... a RAID array spinning 24x7... chance is it will eventually fail and you'll spend a long time reconstructing it.

If your paid to do this its one thing..when you just want your stuff back.... its another.

 

Personally I have 1 RAID array (a simple RAID 1) on my server for /var

I used to build enormous arrays for a job on unix... yet my array only looks after dynamic stuff for mysql and the like for my web pages... the rest is unimportant, its OS files and stuff I can reinstall anytime faster than rebuilding a failed array...

Share this post


Link to post
Share on other sites

rsync on a IDE in a USB drive enclosure that is my take

 

what about those NAS at 40£ + cost of HD? they spin down

 

Also law of series: get a raid1 and 2 HD from same manufacturer hot close to each other, one goes the other is likely to go as well

So much for raid1 if both dies

Share this post


Link to post
Share on other sites

Yeah, I was planning on RAID 1 with two 320 GB drives.

 

The external enclosure is something I've thought about before. It's looking more and more appealing (ie, its less expensive). Or I could opt out for something like this. That would give me the ability to turn the drive on and off when I'm not using it without having to hassle with the extra cables. I don't have enough plugs available as it is (definitely need to buy another power strip).

 

Any thoughts on using the Hard Drive rack as opposed to just an external? The obvious is that I can't take the drive out and use it with another computer easily, but that doesn't matter to me. Just wondering if anyone's had any experience with these and what to avoid/take into consideration.

 

Also, what are your thoughts on keeping the filesystem as NTFS for both the backup and the original drives? I could use the ntfs-3g driver in Linux, and since I probably won't be writing to the NTFS volumes that often within Linux I wouldn't think it would be a problem. Unless there's some reason Windows wouldn't be able to write to the drives when they're connected to the Linux box.

Edited by Scythe

Share this post


Link to post
Share on other sites
Any thoughts on using the Hard Drive rack as opposed to just an external?

not really, well yes faster than usb 2.0 hence better

 

my take is 2d highest risk of loosing data at home is theft/burglary and PC going walky.

So a separate HD is a plus, hence my view against raid 1

Share this post


Link to post
Share on other sites

Any thoughts on using NTFS for the storage drives with ntfs-3g? From what I've been reading about it, ntfs-3g is very good and it doesn't look like I'd have any problems using NTFS for Linux/Windows storage.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...