mdadm - upgrading from 5 x 4TB to 5 x 10TB - pt 2

mdadm - upgrading from 5 x 4TB to 5 x 10TB - pt2

Sync all the drives

rsyncing all the drives, you should see this when examining the drives.

root@hal:~# mdadm --examine /dev/sda1
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6fc77ab1:cb9bdc74:2d2f3938:9ba4b4e7
           Name : HAL:HAL
  Creation Time : Tue Mar 11 23:31:15 2014
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 19532607488 (9313.87 GiB 10000.70 GB)
     Array Size : 15627540480 (14903.58 GiB 16002.60 GB)
  Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=261864 sectors, after=11718837248 sectors
          State : clean
    Device UUID : 78e8aac7:94c62194:39c4ce3b:f42b826d

    Update Time : Fri Oct 25 12:40:20 2019
  Bad Block Log : 512 entries available at offset 264 sectors
       Checksum : 3e06ccae - correct
         Events : 1872

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)

After removing the last WD Red 4TB drive, syncing with 5 x Tshiba N300s gave a rebuild speed of:

root@hal:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : active raid5 sdd1[9] sda1[6] sdf1[5] sdb1[8] sde1[7]
      15627540480 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UU_UU]
      [=>...................]  recovery =  7.0% (274570032/3906885120) finish=297.8min speed=203225K/sec
     
unused devices: <none>

Which is twice over twice as fast at rebuilding 50TB as it was for 20TB.
NICE!

Expand the /dev/md127 array

So after rebuilding, I'll need to expand the array so that it's using the full 10TB per disk rather than 4TB of the existing.

To do this I ran:

mdadm --grow /dev/md127 --bitmap none

mdadm --grow /dev/md127 --size max

 As runnig with bitmaps whilst resizing can be catastrophic, so I've read.

But after running the above, which gave this:

root@hal:/etc/sysctl.d# mdadm --examine /dev/sda
/dev/sda:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
root@hal:/etc/sysctl.d# mdadm --examine /dev/sda1
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6fc77ab1:cb9bdc74:2d2f3938:9ba4b4e7
           Name : HAL:HAL
  Creation Time : Tue Mar 11 23:31:15 2014
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 19532607488 (9313.87 GiB 10000.70 GB)
     Array Size : 39065214976 (37255.49 GiB 40002.78 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=261864 sectors, after=0 sectors
          State : active
    Device UUID : 78e8aac7:94c62194:39c4ce3b:f42b826d

    Update Time : Sun Oct 27 00:15:41 2019
  Bad Block Log : 512 entries available at offset 264 sectors
       Checksum : e00442d4 - correct
         Events : 2248

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)

and approx an 8 hour resync time

After this, check the file system is ok, before expanding it

xfs_repair -v /dev/md127

After any repairs, mount the array and xfs_grow it

root@hal:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.6G     0  7.6G   0% /dev
tmpfs           1.6G  2.6M  1.6G   1% /run
/dev/sdc1       222G  157G   54G  75% /
tmpfs           7.6G     0  7.6G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.6G     0  7.6G   0% /sys/fs/cgroup
tmpfs           1.6G     0  1.6G   0% /run/user/0
tmpfs           1.6G     0  1.6G   0% /run/user/1000
/dev/md127       15T   15T  292G  99% /mnt/md0
root@hal:~# xfs_growfs  /dev/md127
meta-data=/dev/md127             isize=256    agcount=32, agsize=122090240 blks
         =                       sectsz=512   attr=2, projid32bit=0
         =                       crc=0        finobt=0 spinodes=0 rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=3906885120, imaxpct=5
         =                       sunit=128    swidth=512 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 3906885120 to 9766303744
root@hal:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.6G     0  7.6G   0% /dev
tmpfs           1.6G  2.6M  1.6G   1% /run
/dev/sdc1       222G  157G   54G  75% /
tmpfs           7.6G     0  7.6G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.6G     0  7.6G   0% /sys/fs/cgroup
tmpfs           1.6G     0  1.6G   0% /run/user/0
tmpfs           1.6G     0  1.6G   0% /run/user/1000
/dev/md127       37T   15T   23T  40% /mnt/md0

mdadm - upgrading from 5 x 4TB to 5 x 10TB - pt 1

mdadm - upgrading from 5 x 4TB to 5 x 10TB - pt 1

This assumes you want to keep the existing data, and migrate from 20TB raw to 50TB raw.

This grows from 5 x 4TB drives to 5 x 10TB drives.

Wiping the 10TB drives (Toshiba N300)

using lsblk, to ID the drives then:

# parted /dev/sda
(parted) mklabel gpt
(parted) unit tb
(parted) mkpart primary 0tb 10tb 

(parted) set 1 raid on
(parted) align-check
alignment type(min/opt) [optimal]/minimal? optimal
Partition number? 1
1 aligned

(parted) print
Model: DAS TerraMaster (scsi)
Disk /dev/sdb: 10.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number Start End Size File system Name Flags

1 0.00TB 10.0TB 10.0TB primary

Adding the drive

Pull out one of the existing drives

Boot the NAS server, and the array won't come out due to the missing drive.

Bring it up via

mdadm --stop /dev/md127

Then

mdadm --assemble --scan

Add drive

mdadm /dev/md127 --add /dev/sdX1

Initially it was syncing at over 145k/sec (with 4 x WD Red 4TB drives + 1 x Toshiba N300 10TB)

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : active raid5 sde1[7] sda1[6] sdf1[5] sdb1[3] sdd1[2]
      15627540480 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [U_UUU]
      [>....................]  recovery =  0.6% (23770712/3906885120) finish=448.7min speed=144206K/sec
    

Details during first rebuild

root@hal:/# mdadm --detail /dev/md127
/dev/md127:
           Version : 1.2
     Creation Time : Tue Mar 11 23:31:15 2014
        Raid Level : raid5
        Array Size : 15627540480 (14903.58 GiB 16002.60 GB)
     Used Dev Size : 3906885120 (3725.90 GiB 4000.65 GB)
      Raid Devices : 5
     Total Devices : 5
       Persistence : Superblock is persistent

       Update Time : Thu Oct 24 17:57:47 2019
             State : clean, degraded, recovering
    Active Devices : 4
   Working Devices : 5
    Failed Devices : 0
     Spare Devices : 1

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : resync

    Rebuild Status : 31% complete

              Name : HAL:HAL
              UUID : 6fc77ab1:cb9bdc74:2d2f3938:9ba4b4e7
            Events : 1714

    Number   Major   Minor   RaidDevice State
       6       8        1        0      spare rebuilding   /dev/sda1
       1       8       65        1      active sync   /dev/sde1
       2       8       49        2      active sync   /dev/sdd1
       3       8       17        3      active sync   /dev/sdb1
       5       8       81        4      active sync   /dev/sdf1

With the new drive:

 mdadm --examine /dev/sde1
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x2
     Array UUID : 6fc77ab1:cb9bdc74:2d2f3938:9ba4b4e7
           Name : HAL:HAL
  Creation Time : Tue Mar 11 23:31:15 2014
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 19532607488 (9313.87 GiB 10000.70 GB)
     Array Size : 15627540480 (14903.58 GiB 16002.60 GB)
  Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
Recovery Offset : 0 sectors
   Unused Space : before=261864 sectors, after=11718837248 sectors
          State : clean
    Device UUID : 27c4f732:455e40e8:cb75bd34:12e0a46a

    Update Time : Fri Oct 25 07:19:29 2019
  Bad Block Log : 512 entries available at offset 264 sectors
       Checksum : f3835a55 - correct
         Events : 1808

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)

Once synced, pull out a drive, then add another 10TB one.

Using google mail servers as a smarthost from a linux server (ubuntu)

So you want to send mail securely using google's mail servers as a smarthost.

The first thing you'll need is an application specific password for exim.

Once that's created, I will assume the following:

• You are ok with Gmail rewriting your sender address or use this Gmail address.
• Your account name on your local Debian box: YOUR-USER-NAME
• Your host name in /etc/hostname: hostname1
• Your host name in the 127.0.1.1 line of /etc/hosts: hostname1.localdomain hostname1
• Your envelope address is SMTPAccountName@gmail.com for the outgoing SMTP connection to Gmail's SMTP server via Exim4
• Your Exim4 relays local SMTP connections coming with the mail envelope address: YOUR-USER-NAME@localhost and YOUR-USER-NAME@localhost.localdomain
• Your account name for SMTP at Gmail is SMTPAccountName@gmail.com
• Your password for SMTP at Gmail is y0uRpaSsw0RD
   

# dpkg-reconfigure exim4-config

• Choose "mail sent by smarthost; received via SMTP or fetchmail"
• Set to "localhost" for "System mail name:".
• Set to "127.0.0.1" for "IP-addresses to listen on for incoming SMTP connections" to refuse external connections.
• Leave as empty for "Other destinations for which mail is accepted:".
• Leave as empty for "Machines to relay mail for:".
• Set to "smtp.gmail.com::587" for "IP address or host name of the outgoing smarthost:".
• Choose "NO" for "Hide local mail name in outgoing mail?".
• Choose "NO" for "Keep number of DNS-queries minimal (Dial-on-Demand)?".
• Choose "mbox format in /var/mail/" for "Delivery method for local mail".
• Choose "YES" for "Split configuration into small files?".

vi /etc/exim4/passwd.client

add the following lines:

*.google.com:SMTPAccountName@gmail.com:y0uRpaSsw0RD

# chown root:Debian-exim /etc/exim4/passwd.client # chmod 640 /etc/exim4/passwd.client

Edit your aliases file

# newaliases

# update-exim4.conf
# invoke-rc.d exim4 restart
# exim4 -qff

 tada

mostly grepped from here 

Setting up OpenVPN on ubuntu / Mac os x pt2

Hardening OpenVPN on ubuntu / Mac os x

One of the often-repeated maxims of network security is that one should never place so much trust in a single security component that its failure causes a catastrophic security breach. OpenVPN provides several mechanisms to add additional security layers to hedge against such an outcome.

tls-auth

The tls-auth directive adds an additional HMAC signature to all SSL/TLS handshake packets for integrity verification. Any UDP packet not bearing the correct HMAC signature can be dropped without further processing. The tls-auth HMAC signature provides an additional level of security above and beyond that provided by SSL/TLS. It can protect against:

• DoS attacks or port flooding on the OpenVPN UDP port.
• Port scanning to determine which server UDP ports are in a listening state.
• Buffer overflow vulnerabilities in the SSL/TLS implementation.
• SSL/TLS handshake initiations from unauthorized machines (while such handshakes would ultimately fail to authenticate, tls-auth can cut them off at a much earlier point).

Using tls-auth requires that you generate a shared-secret key that is used in addition to the standard RSA certificate/key:

openvpn --genkey --secret ta.key

This command will generate an OpenVPN static key and write it to the file ta.key. This key should be copied over a pre-existing secure channel to the server and all client machines. It can be placed in the same directory as the RSA .key and .crt files.
In the server configuration, add:

tls-auth ta.key 0

In the client configuration, add:

tls-auth ta.key 1

proto udp

While OpenVPN allows either the TCP or UDP protocol to be used as the VPN carrier connection, the UDP protocol will provide better protection against DoS attacks and port scanning than TCP:

proto udp

user/group (non-Windows only)

OpenVPN has been very carefully designed to allow root privileges to be dropped after initialization, and this feature should always be used on Linux/BSD/Solaris. Without root privileges, a running OpenVPN server daemon provides a far less enticing target to an attacker.

user nobody
group nobody

Unprivileged mode (Linux only)

On Linux OpenVPN can be run completely unprivileged. This configuration is a little more complex, but provides best security.
In order to work with this configuration, OpenVPN must be configured to use iproute interface, this is done by specifying --enable-iproute2 to configure script. sudo package should also be available on your system.
This configuration uses the Linux ability to change the permission of a tun device, so that unprivileged user may access it. It also uses sudo in order to execute iproute so that interface properties and routing table may be modified.
OpenVPN configuration:

• Write the following script and place it at: /usr/local/sbin/unpriv-ip:

#!/bin/sh
sudo /sbin/ip $*

• Execute visudo, and add the followings to allow user 'user1' to execute /sbin/ip:

user1 ALL=(ALL)  NOPASSWD: /sbin/ip

You can also enable a group of users with the following command:

%users ALL=(ALL)  NOPASSWD: /sbin/ip

• Add the following to your OpenVPN configuration:

dev tunX/tapX
iproute /usr/local/sbin/unpriv-ip

Please note that you must select constant X and specify tun or tap not both.
• As root add persistant interface, and permit user and/or group to manage it, the following create tunX (replace with your own) and allow user1 and group users to access it.

openvpn --mktun --dev tunX --type tun --user user1 --group users

• Run OpenVPN in the context of the unprivileged user.

Further security constraints may be added by examining the parameters at the /usr/local/sbin/unpriv-ip script.

chroot (non-Windows only)

The chroot directive allows you to lock the OpenVPN daemon into a so-called chroot jail, where the daemon would not be able to access any part of the host system's filesystem except for the specific directory given as a parameter to the directive. For example,

chroot jail

would cause the OpenVPN daemon to cd into the jail subdirectory on initialization, and would then reorient its root filesystem to this directory so that it would be impossible thereafter for the daemon to access any files outside of jail and its subdirectory tree. This is important from a security perspective, because even if an attacker were able to compromise the server with a code insertion exploit, the exploit would be locked out of most of the server's filesystem.
Caveats: because chroot reorients the filesystem (from the perspective of the daemon only), it is necessary to place any files which OpenVPN might need after initialization in the jail directory, such as:

• the crl-verify file, or
• the client-config-dir directory.

More info here:
http://openvpn.net/index.php/open-source/documentation/howto.html#security 

tftpboot not working after upgrade?

check permissions on /var/lib/tftpboot
and check your default options in /etc/default


TFTP_OPTIONS="--secure"
TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/var/lib/tftpboot"
TFTP_ADDRESS="0.0.0.0:69"

Setting up OpenVPN on ubuntu / Mac os x

Since PPTP is inherently insecure, moving to something SSL or IPSEC based is far more perferrable....

OpenVPN on the Gateway

We'll begin with installing openVPN on the gateway:

sudo apt-get install openvpn easy-rsa

After that, make sure our openvpn config is always used:

Edit: /etc/default/openvpn

## Comment out everything and add:
# Start our openvpn.conf automatically:
AUTOSTART="all"

We'll need to edit some openvpn configuration files and generate some keys for the people we want to have access. There are some nice example configs, so we'll be using them. We'll need root access for the next dozen of instructions, so let's su and head to the openvpn config directory:

cd /etc/openvpn/
cp -r /usr/share/doc/openvpn/examples/easy-rsa/ .

Note the above directory may not exist in ubuntu 14, it may be in /usr/share/easy-rsa
 

Now that we got some examples to work with, let's edit them.

Edit: easy-rsa/2.0/vars

## Comment the line that starts with:
## export EASY_RSA
## And add this below:
export EASY_RSA=/etc/openvpn/easy-rsa/2.0

## You can also set some defaults for your certificates.
## Find the following variables and set them appropriately.
export KEY_COUNTRY=UK
export KEY_PROVINCE=SY
export KEY_CITY=London
export KEY_ORG="OpenVPN"
export KEY_EMAIL="waga@an.example"

Time to source that information into our current shell session:

source ./easy-rsa/2.0/vars

And build the necessary certificates. (Certificate Authority, Server key, Client key and Diffie Hellman.) You can probably rename your server and client(s) however you want, but need to remember the names for later configuration. Answer yes when asked if you want to sign the certificates and want to commit them.
When it asks for a Challenge Password, you can just leave it blank and enter.
The Common Name that is asked when building a server key needs to be the same as the argument given to the build-key-server command ('server' in our example).
The Common Name that is asked when building a client key needs to be the same as the argument given to the build-key command ('client1' in our example). Watch for the errors, and resolve using the prompts it gives you

./easy-rsa/2.0/clean-all
./easy-rsa/2.0/build-ca
./easy-rsa/2.0/build-key-server server
./easy-rsa/2.0/build-key client1

(to add another user simply change client1 for e.g. the username phil, bob, dave etc)

./easy-rsa/2.0/build-dh

We'll copy the generated keys over when we get to the Client steps. It's time now to set up the openvpn.conf we told OpenVPN to autostart:

Edit: /etc/openvpn/openvpn.conf

## Add: 
dev tun
proto tcp
## Change this port number if you want a non-standard port.
port 1194

ca /etc/openvpn/easy-rsa/2.0/keys/ca.crt
cert /etc/openvpn/easy-rsa/2.0/keys/server.crt
key /etc/openvpn/easy-rsa/2.0/keys/server.key
dh /etc/openvpn/easy-rsa/2.0/keys/dh1024.pem

user nobody
group nogroup
## Leave this as-is; it is the VPN virtual network address.
# i've set this to something else (192.168.xxx.xxx) since my 3rd DC uses this range

server 10.8.0.0 255.255.255.0

persist-key
persist-tun

status openvpn-status.log
log-append /var/log/openvpn
verb 3
client-to-client
# disable this or ALL traffic will be going through your VPN

#disabling allows split routing/split tunnelling
#push "redirect-gateway def1"
# enter your local LAN subnet here.... 

push "192.168.x.x 255.255.255.0"

# Remove comment if you want lzo compression
#comp-lzo

OpenVPN on the Client

On the client we need to install an OpenVPN client, for Mac OSX I use Tunnelblick which is currently hosted at Google code. After opening the .dmg file, copy the Tunnelblick.app file to your Applications. When this is done, let's transfer the client certificates from the gateway to the client, I prefer to use SCP for this.
You will probably need root ssh access to the file. If you don't have that, make sure you've copied the keys to your homedir instead. If you have created multiple client certificates, or changed the names of the certificates, then don't forget to scp those over as well.

scp root@your.gateway:/etc/openvpn/easy-rsa/2.0/keys/{ca.rt,client1.{crt,key}} ~/Library/openvpn && sudo chown $USER ~/Library/openvpn/{ca.rt,client1.{crt,key}}

Now that we have a client, we will need to configure it. Be sure to set your Gateway server's IP where it says YOUR-GATEWAY-IP-GOES-HERE and if you also picked a non-standard port, change that in this config as well. The same applies for any client certificates you've created on the gateway.

Create: ~/Library/openvpn/openvpn.conf

# We are a client, not a server/gateway.
client

# We use a TUN interface
dev tun

# We use the TCP protocol
proto tcp

# Name of the gateway's certificate
## 'server' if you followed our example.
remote-cert-tls server

# The hostname/IP and port of the server.
# You can have multiple remote entries
# to load balance between the servers.
## If you chose a non-standard port, change 1194 to the port you picked.
remote YOUR-GATEWAY-IP-GOES-HERE 1194          

# Keep trying indefinitely to resolve the
# host name of the OpenVPN server.  Very useful
# on machines which are not permanently connected
# to the internet such as laptops.
resolv-retry infinite

# Most clients don't need to bind to
# a specific local port number.
nobind

# Downgrade privileges after initialization (non-Windows only)
user nobody
group nobody

# Try to preserve some state across restarts.
persist-key
persist-tun

# SSL/TLS parms.
# See the server config file for more
# description.  It's best to use
# a separate .crt/.key file pair
# for each client.  A single ca
# file can be used for all clients.
## If you picked different names for your client certificates, change here
ca ca.crt
cert client1.crt
key client1.key

# Enable compression on the VPN link.
# Don't enable this unless it is also
# enabled in the server config file.
# comp-lzo

# Set log file verbosity.
verb 3

# Make sure the right gateway settings are used.

push "dhcp-options DNS 8.8.8.8"
#push "dhcp-options WINS 10.8.0.1"
#push "redirect-gateway"

Key files:

Here is an explanation of the relevant files:

Filename	Needed By	Purpose	Secret
ca.crt	server + all clients	Root CA certificate	NO
ca.key	key signing machine only	Root CA key	YES
dh{n}.pem	server only	Diffie Hellman parameters	NO
server.crt	server only	Server Certificate	NO
server.key	server only	Server Key	YES
client1.crt	client1 only	Client1 Certificate	NO
client1.key	client1 only	Client1 Key	YES
client2.crt	client2 only	Client2 Certificate	NO
client2.key	client2 only	Client2 Key	YES
client3.crt	client3 only	Client3 Certificate	NO
client3.key	client3 only	Client3 Key	YES

Routing issues after network switching:

I've encountered some problems when switching between local networks after having had the VPN on. It seems that after I've switch to a different LAN, I can't reach the gateway host from the new local network, thus not allowing me to set up the VPN from the second VPN. My guess is that not all routes are reset after disconnecting.

To find out what route seems to be incorrect use:

netstat -rn

In my case it was w.x.y.z/32, where w.x.y.z is my gateway's IP.
You can remove it via:

route delete w.x.y.z/32 

infact, I've changed the Applications/Utilities/Tunnelblick.app/Contents/Resources/client.down.tunnelblick.sh file so that the post script removes the VPN gateway from teh routing table.

IP=`ping -n -c 1 vpn-gw.mydomain.com | grep icmp_seq | cut -d " " -f 4 | cut -d ":" -f 1`
route delete $IP

References:
Ubuntu / mac os x openvpn