G9Online Notepad: Using Ubuntu Server to set up a home server and NAS

A reminder about screen

https://www.mattcutts.com/blog/a-quick-tutorial-on-screen/

Use screen to create terminal sessions that will persist when the terminal window is closed. Good when doing long copies or running a minecreaft server.

Change the SSH port

modify /etc/ssh/sshd_config

where it says:

# What ports, IPs and protocols we listen for

Port 22 <---change port to what you need it to be

then save and restart ssh serverSetup ZFS

http://www.howtogeek.com/175159/an-introduction-to-the-z-file-system-zfs-for-linux/

Install ZFS

Notes mainly taken from https://www.latentexistence.me.uk/zfs-and-ubuntu-home-server-howto/

$ sudo apt-get install zfsutils-linux

List disks by ID

The creators of ZFS on Linux recommend that you use disk names starting with /dev/disk/by-id/ rather than /dev/sda, /dev/sdb etc as they are more consistent (particularly the wwn identifier) so look in that folder to see what disk names you have.

$ ls -l /dev/disk/by-id/

List ZFS datasets

$ sudo zfs list

Create a two disk mirror

*NOTE* the below command is how it should be done but it fails due to a bug with ZFS (https://github.com/zfsonlinux/zfs/issues/3708)

sudo zpool create [pool_name] mirror /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N1VUU0LX /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N7FSX6F9

to make it work create the pool using /dev/sdb etc. but then:

sudo zpool create [pool_name] mirror /dev/sdb /dev/sdc

$ sudo zpool export [pool_name]
$ sudo zpool import -d /dev/disk/by-id [pool_name]

That will switch all /dev/sdx drives to the full ID.
Check the status of pools

$ sudo zpool status

Destroy a pool

$ sudo zpool destroy [pool_name]

While you can just start storing data in your newly-created filesystem (in /[pool_name] in our example) that isn’t the best way to use ZFS. Instead you should create additional filesystems within your storage pool to hold different types of data. This will allow you to do things like set compression, deduplication, quotas and snapshots differently for each set of data or backup an individual filesystem with zfs send. You use the zfs command to create your filesystems. Some examples:

$ sudo zfs create [pool_name]/music

$ sudo zfs create [pool_name]/videos

$ sudo zfs create [pool_name]/backups

The above examples will create filesystems in the pool and will automatically mount them as subfolders of the main filesystem. Note that the name is in the format pool / filesystem name and there is no leading slash on the pool name.

To list all the filesystems

$ sudo zfs list

List all drives

$ df -h

Add another 2 disk mirror to the ZFA Pool

Get the UUIDs of the drives

$ sudo blkid

$ zpool add unas mirror newDrive1 newDrive2

Sharing

Now we need to share the data, otherwise it’s not much of a server. ZFS will automatically manage sharing through NFS (Unix/Linux) or SMB (Windows) but you must first install the server software. For sharing to Windows clients use:

$ sudo apt-get install samba

To add NFS use:

$ sudo apt-get install nfs-kernel-server

You don’t need to configure much because ZFS handles most settings for you, but you might wish to change the workgroup name for Samba in /etc/samba/smb.conf.

To share a ZFS filesystem you change a property using the zfs command. For Windows clients:

$ sudo zfs set sharesmb=on tank/music

$ sudo zfs set sharesmb=on tank/videos

For Unix / Linux clients:

$ sudo zfs set sharenfs=on tank/backups

Or you can share the whole lot at once by sharing the main pool. The sub-filesystems will inherit the sharing property unless you turn them off:

$ sudo zfs set sharesmb=on tank

$ sudo zfs set sharenfs=on tank

$ sudo zfs set sharesmb=off tank/music

You can check whether your filesystems are shared or not:

$ sudo zfs get sharesmb,sharenfs

At this point you should be able to see your shares from other computers on the network but you probably won’t have permission to access them. You will need to ensure that the file permissions and owners are set correctly, and you will also have to add an account and set a password for use when connecting through Samba. If your username is ella then use:

$ sudo smbpasswd -a ella

to set your Samba password, and make sure that ella has permission to access all the files in your shared folders:

$ sudo chown -R ella:ella /tank/videos

Create a share that just shows photos for amazon cloud drive backups

Amazon cloud drive gives unlimited photo backups, so, to exclude video files and photoshop fiels etc, create a share that vetoes anything but a photo file:

Full article below, with details added in the comments. Search for Samba Share on the page:

https://havecamerawilltravel.com/photographer/amazon-cloud-drive-photos

[Pictures-Only]
comment = Pictures Only
path = /storage/dd5/files/documents/pictures
public = yes
read only = yes
veto files = /Video/_Lightroom 5/*.rar/*.ini/*.exe/*.db/*.PDF/*.pdf/*.MOV/*.MPG/*.AVI/*.mpg/*.avi/*.zip/*.3gp/*.3GP/*.MTS/*.mts/*.mp4/*.mov/*.flv/*.mkv/*.arw/*.doc/*.info/*.mp3/*.est/*.blend*/*.mxp/*.max/*.bak/*.mdf/*.xml/*.3ds/*.blend2/*.cdf/*.fbx/*.pz3/*.txt/

Mount USB drive

Find the /dev name of the USB drive once it is connected

Create a directory names, day, /usb and mount the usb device on the newly created /usb directory

$ mount /dev/sdd /usb/

Automatically mount the USB at boot and share it

1). Mount the drive at boot time:

* sudo vi /etc/fstab

* then add this line: /dev/sda3 /media/lnxBackUp ext4 users,user 0 0

In my case, sda3 is the drive that I would like to mount and ext4 is the partition of sda3.

2). Install samba server:

* sudo apt-get install samba

* sudo vi /etc/samba/smb.conf

then edit its config file by adding these information at the end of the page:

[lnxBackUp]

writable = yes

path = /media/lnxBackUp

guest ok = yes

create mask = 0777

comment = Backup drive

directory mask = 0777

* Create a folder in /media

* sudo mkdir /medir/lnxBackUp

* sudo mount -a

Format and Partition a 3TB external USB drive

The full article is:

https://joshstrange.com/ubuntu-formatting-a-3tb-drive/

If you get the error "The resulting partition is not properly aligned for best performance", you can use the Simple Instructions at: https://blog.hqcodeshop.fi/archives/273-GNU-Parted-Solving-the-dreaded-The-resulting-partition-is-not-properly-aligned-for-best-performance.html

The Simple Way involves using percentages

<parted> mklabel gpt
<parted> unit s
<parted> mkpart LVM ext4 0% 100%
<parted> quit

lsblk to find the drive name (will be something like /dev/sdX where X is a letter)
sudo parted /dev/sdX
In the parted interactive console type:
1. (parted) mklabel gpt
2. (parted) unit TB
3. (parted) mkpart primary 0.00TB 3.00TB (Changing 3.00TB to the size of your drive)
4. (parted) quit
sudo mkfs.ext4 /dev/sdX1 (Where X is the same as above and note the "1" now)
sudo tune2fs -m 1 /dev/sdX1
mkdir /Your/Mount/Point/Here
sudo blkid (Copy the UUID of your drive)
sudo nano /etc/fstab
Add the line UUID=YOUR-UUID-HERE-XXXX /Your/Mount/Point/Here ext4 defaults 1 2 then save and exit
sudo mount -a

NCDU Nice DU to show file sizes

Instal ncdu and use it show a tree of directory sizes that you can navigate with the cursor

Setup Samba

Not necessary as steps above cover this

https://help.ubuntu.com/community/How to Create a Network Share Via Samba Via CLI (Command-line interface/Linux Terminal) - Uncomplicated%2C Simple and Brief Way!

Install Avahi to announce the shared drive on the network

You have finished the most important part and basically you can use your file server now, but this little extra step is cool in my opinion. Instead of having to do Finder > Go > Connect to Server… Every time we want to connect to the server, why don’t we show up in Finder’s left bar?

To do that we need to install Avahi. Avahi is a simple file server daemon that announces to the network when it has connected, basically Bonjour. With this instead of telling OS X Mavericks to find the server, we can tell the server to tell everyone when it’s connected. In that way our server will appear as extractable media in the left bar of any finder Window, and it’s much easier and pleasant to interact with it using this method.

Luckily, installing and configuring Avahi is really fast and simple.

If you still have your su terminal window, type this in:

1	`apt-get` `install` `avahi-daemon avahi-utils`

Then open and edit the file

/etc/avahi/services/smb.service

(the file will be blank and that’s fine – you’re creating it from scratch):

1	`gedit` `/etc/avahi/services/smb.service`

And copy and paste this into it:

<?xml version="1.0" standalone='no'?>

<!DOCTYPE service-group SYSTEM "avahi-service.dtd">

<service-group>

 <name replace-wildcards="yes">%h</name>

 <service>

   <type>_smb._tcp</type>

   <port>445</port>

 </service>

 <service>

   <type>_device-info._tcp</type>

   <port>0</port>

   <txt-record>model=RackMac</txt-record>

 </service>

</service-group>

And that’s it! You don’t even need to restart Avahi. It will automatically announce to the network and your Mac will see your file server in Finder to the left.

Setup Simple eMail

This guide is for you, if:

You don’t want to run a mail server
You want to send email, and you don’t care about receiving it
You want people to receive the emails that your server sends

I’ve used this method with Linode, and it works perfectly.

Install mailutils

~ sudo apt-get install mailutils

When the setup wizard launches, choose the unconfigured option. You don’t need to do any special configuration to get this to work.

Install and configure sstmp

~ sudo apt-get install ssmtp
~ sudo vim /etc/ssmtp/ssmtp.conf
Hit “i” to enter Insert mode.
Uncomment FromLineOverride=YES by deleting the #
Add the following to the file:

AuthUser=<user>@gmail.com
AuthPass=Your-Gmail-Password
mailhub=smtp.gmail.com:587
UseSTARTTLS=YES
Save and close the file:

Hit Escape
Type :wq
Hit Enter

If you’re using two-factor authentication
Create a new application-specific password to use in the config file above. (If you’re using Gmail, you can manage those passwords here.)

Test it out
~ echo "This is a test" | mail -s "Test" <user>@<email>.com

Using a webmail service other than Gmail
You can follow the same pattern that I used above. You’ll need to:

Subsitute the SMTP address and port for your email service (e.g. Yahoo!) where it says smtp.gmail.com:587. (587 is the port number.)
Set up an application-specific password if your webmail provider allows it, and paste that into the password line, the way I did with Gmail. (Yahoo! appears to have something similar.)

SmartMonTools to monitor Hard Disk Drive Health

https://help.ubuntu.com/community/Smartmontools

sudo apt-get install smartmontools

Syncing Directories

rsync -avh --stats data/ /media/backups/data

rsync -avP --delete data/ /media/backups/data

The / on the end of data/ means the contents of directory data. This is necessary to prevent copying the directory data and creating the hierarchy /media/backups/data/data

-a is archive recursively and maintain all file timestamps and properties

-v is verbose

-P is for both -progress and --partial. The first of these gives you a progress bar for the transfers and the second allows you to resume interrupted transfers.

-n is for Dry Run to test the output of a command before executing it.

--delete is necessary if you want the files in the destination folder to be deleted if they have been deleted in the source folder. By default, rsync does not delete anything.

G9Online Notepad

Thursday, 2 June 2016

Using Ubuntu Server to set up a home server and NAS