Saturday 18 October 2014

Creating RRD graphs for Raspbery Pi / DS18B20


So I wanted to move the TME sensor I bought in my previous post to the bitcoin mining room, to measure the temperature there.

I still needed to monitor the temp in the comms cupboard where I have 4 HP Microserver NASs, UPS, firewall, HP managed switch etc etc to make sure it isn't going to melt.

I saw this excellent article here that uses simple off the shelf electronics to accomplish this. Following this and buying the parts listed from Maplins and Pimori and ebay.

Maplins parts list:
Resistor pack (you never know when you might need more)

Pimori parts list:
Hook up wire (looked cool with all the colours)
ADA fruit pi - t cobbler
Adafruit raspberry pi perma proto
Raspberry Pi GPIO cable

Ebay parts list:
ds18b20t (waterproof version)
Total cost of parts incl Raspberry pi b model < £90

So I did some basic config following the article there and basically adding the following modules to /etc/modules
w1-gpio
w1_therm
then in /sys/bus/w1/devices/ should have the sensors with their individual serials there

phil@temp-comms /sys/bus/w1/devices $ ls -al
total 0
drwxr-xr-x 2 root root 0 Oct 18 00:13 .
drwxr-xr-x 4 root root 0 Oct 18 00:13 ..
lrwxrwxrwx 1 root root 0 Oct 18 00:13 28-0000062d26d0 -> ../../../devices/w1_bus_master1/28-0000062d26d0
lrwxrwxrwx 1 root root 0 Oct 18 00:13 28-0000062e10ae -> ../../../devices/w1_bus_master1/28-0000062e10ae
lrwxrwxrwx 1 root root 0 Oct 18 00:13 28-0000062e511a -> ../../../devices/w1_bus_master1/28-0000062e511a
lrwxrwxrwx 1 root root 0 Oct 18 00:13 w1_bus_master1 -> ../../../devices/w1_bus_master1
phil@temp-comms /sys/bus/w1/devices $ 



For each sensor there will be a 28-xxx directory, and under each directory 280000062xxxx there is is a file called w1_slave. Cat that and it'll have the temperature
phil@temp-comms /sys/bus/w1/devices/28-0000062d26d0 $ cat w1_slave
c7 01 4b 46 7f ff 09 10 01 : crc=01 YES
c7 01 4b 46 7f ff 09 10 01 t=28437
phil@temp-comms /sys/bus/w1/devices/28-0000062d26d0 $
t=temperature, but it's 1000 higher than it should, so just divide it by 1000 to get the temp in degrees celcius.



and then followed this article to use multiple temp sensors with the wiring like so




So what this article doesn't tell you is how to produce the graphs.
So obviously setting up cron with 1 minute intervals like so in /etc/cron.d/logs
* * * * * root /usr/local/bin/temp.sh 

with temp.sh containing
#!/bin/sh
file=/var/www/temp.txt
echo -n spare \ > $file
cat /sys/bus/w1/devices/28-0000062e511a/w1_slave | grep t | awk '{print $10}' | cut -d = -f 2 >> $file

echo -n comms_top \ >> $file
cat /sys/bus/w1/devices/28-0000062d26d0/w1_slave | grep t | awk '{print $10}' | cut -d = -f 2 >> $file

echo -n comms_bottom \ >> $file
cat /sys/bus/w1/devices/28-0000062e10ae/w1_slave | grep t | awk '{print $10}' | cut -d = -f 2 >> $file
Note the serials for the temp sensors were ascertained by individually plugging them in, making a note and appying labels physically to the device so I knew what I was monitoring.

So after the the data was grabbed and stored in a text file, you have to process it. I decided on PHP, cos it's easy and can be used easily on apache, which I can use to publish the data for view via a browser.
So after installing apache2, libapache2-mod-php5, php5-rrd,  rrdtool etc you need to input the data into an RRD

Using PHP and 
#!/usr/bin/php

 // Fetch current time
    $t = time();
    $filename1="/mnt/ramdisk/temp-spare.txt";
    $filename2="/mnt/ramdisk/temp-bottom.txt";
    $filename3="/mnt/ramdisk/temp-top.txt";

    $handle = fopen($filename1, "r");
    $contents1 = fread($handle, filesize($filename1));
    fclose ($handle);
    $contents1=(float)$contents1/1000;

    $handle = fopen($filename2, "r");
    $contents2 = fread($handle, filesize($filename2));
    fclose ($handle);
    $contents2=(float)$contents2/1000;

    $handle = fopen($filename3, "r");
    $contents3 = fread($handle, filesize($filename3));
    fclose ($handle);
    $contents3=(float)$contents3/1000;

   $option = array ("$t:$contents1:$contents2:$contents3");
        echo "$t ,$contents1, $contents2, $contents3 ";
    $ret = rrd_update("/mnt/ramdisk/temp-comms.rrd", $option);

    if (! $ret) {
              echo "<b>Graph error comms: </b>".rrd_error()."\n";
    }

then you've gotta create the GIFs
index.php
// for comms cupboard
function create_graph_temperature_comms ($output, $start, $title)
{
    $options = array(

    "--slope-mode",
    "--start", $start,
    "--title=$title",
    "--vertical-label=Temperature in oC",
    // oops got the top and bottom values mixed up on input
    "DEF:temp_spare=/mnt/ramdisk/temp-comms.rrd:spare:AVERAGE",
    "DEF:temp_top=/mnt/ramdisk/temp-comms.rrd:temp_bottom:AVERAGE",
    "DEF:temp_bottom=/mnt/ramdisk/temp-comms.rrd:temp_top:AVERAGE",

    "DEF:temp_sparemin=/mnt/ramdisk/temp-comms.rrd:spare:MIN",
    "DEF:temp_sparemax=/mnt/ramdisk/temp-comms.rrd:spare:MAX",
    "DEF:temp_topmin=/mnt/ramdisk/temp-comms.rrd:temp_top:MIN",
    "DEF:temp_topmax=/mnt/ramdisk/temp-comms.rrd:temp_top:MAX",
    "DEF:temp_bottommin=/mnt/ramdisk/temp-comms.rrd:temp_bottom:MIN",
    "DEF:temp_bottommax=/mnt/ramdisk/temp-comms.rrd:temp_bottom:MAX",
   
    "CDEF:ttemp_spare=temp_spare,1,*",
    "CDEF:ttemp_top=temp_top,1,*",
    "CDEF:ttemp_bottom=temp_bottom,1,*",
   
    "VDEF:vtemp_spare=temp_spare,AVERAGE",

    "LINE:ttemp_spare#00aa00:Spare:",
    "GPRINT:temp_spare:AVERAGE:Avg temp \\t %6.2lf",
    "GPRINT:temp_spare:LAST:Last temp \\t %6.2lf \\n",

    "LINE:ttemp_top#cc0000:Top:",
    "GPRINT:temp_top:AVERAGE:Avg temp \\t %6.2lf",
    "GPRINT:temp_top:LAST:Last temp \\t %6.2lf \\n",

    "LINE:ttemp_bottom#000088:Bottom:",

    "COMMENT:\\n",
    "--alt-autoscale",
    "--alt-y-grid",
   

   
  
    "GPRINT:vtemp_spare: test %6.2lf",
    );

    $ret = rrd_graph($output, $options);
        if (! $ret) {
        echo "<b>Graph error for temp comms</b>".rrd_error()."\n";
        }
}
And you'll get something like this:




Tuesday 16 September 2014

List users on a terminal services server

qwinsta /server:servername1

 SESSIONNAME       USERNAME                 ID  STATE   TYPE        DEVICE
 services                                    0  Disc
 console                                     1  Conn
                   bob5                      2  Disc
 rdp-tcp#1         bob4                      3  Active
                   bob3                      4  Disc
 rdp-tcp#4         bob2                      5  Active
 rdp-tcp#0         bob                       6  Active

Tuesday 9 September 2014

Disabling clipboard, Drive Mappings, client redirection etc within a RDP session

So you want to disable
* clipboard copy/paste
* Drive mappings
* printer mappings
fDisableClip , fDisableCdm, fDisableCpm, 
and you don't have the PC/Server on the domain?

Easiest thing to do is set it manually using reg query/add like so:
echo off
set server=%1
set passwd=%2
echo =====================================================================
echo %server% %passwd%
psexec \\%server% -u %server%\administrator -p %passwd% reg query "HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services" | findstr /i fdisable

psexec \\%server% -u %server%\administrator -p %passwd% reg add "HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services" /v fDisableClip /t REG_DWORD /d 0x1 /f
psexec  \\%server%  -u %server%\administrator -p %passwd% reg add "HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services" /v fDisableCdm /t REG_DWORD /d 0x1 /f
psexec \\%server% -u %server%\administrator -p %passwd% reg add "HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services" /v fDisableCpm /t REG_DWORD /d 0x1 /f
psexec \\%server% -u %server%\administrator -p %passwd% reg add "HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services" /v fDisableLPT /t REG_DWORD /d 0x1 /f
psexec \\%server% -u %server%\administrator -p %passwd% reg add "HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services" /v fDisablePNPRedir /t REG_DWORD /d 0x1 /f

psexec \\%server% -u %server%\administrator -p %passwd% reg query "HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services" | findstr /i fdisable

 And if for some reason, domain PCs don't have it applied:

echo off
set server=%1
echo =====================================================================
echo %server%
psexec \\%server% reg query "HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services" | findstr /i fdisable

reg add "\\%server%\HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services" /v fDisableClip /t REG_DWORD /d 0x1 /f
reg add "\\%server%\HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services" /v fDisableCdm /t REG_DWORD /d 0x1 /f
reg add "\\%server%\HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services" /v fDisableCpm /t REG_DWORD /d 0x1 /f
reg add "\\%server%\HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services" /v fDisableLPT /t REG_DWORD /d 0x1 /f
reg add "\\%server%\HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services" /v fDisablePNPRedir /t REG_DWORD /d 0x1 /f

psexec \\%server% reg query "HKLM\Software\Policies\Microsoft\Windows NT\Terminal Services" | findstr /i fdisable
 
 

Saturday 30 August 2014

Using google mail servers as a smarthost from a linux server (ubuntu)

So you want to send mail securely using google's mail servers as a smarthost.

The first thing you'll need is an application specific password for exim.

Once that's created, I will assume the following:
  • You are ok with Gmail rewriting your sender address or use this Gmail address.
  • Your account name on your local Debian box: YOUR-USER-NAME
  • Your host name in /etc/hostname: hostname1
  • Your host name in the 127.0.1.1 line of /etc/hosts: hostname1.localdomain hostname1
  • Your envelope address is SMTPAccountName@gmail.com for the outgoing SMTP connection to Gmail's SMTP server via Exim4
  • Your Exim4 relays local SMTP connections coming with the mail envelope address: YOUR-USER-NAME@localhost and YOUR-USER-NAME@localhost.localdomain
  • Your account name for SMTP at Gmail is SMTPAccountName@gmail.com
  • Your password for SMTP at Gmail is y0uRpaSsw0RD
     
# dpkg-reconfigure exim4-config
 
  • Choose "mail sent by smarthost; received via SMTP or fetchmail"
  • Set to "localhost" for "System mail name:".
  • Set to "127.0.0.1" for "IP-addresses to listen on for incoming SMTP connections" to refuse external connections.
  • Leave as empty for "Other destinations for which mail is accepted:".
  • Leave as empty for "Machines to relay mail for:".
  • Set to "smtp.gmail.com::587" for "IP address or host name of the outgoing smarthost:".
  • Choose "NO" for "Hide local mail name in outgoing mail?".
  • Choose "NO" for "Keep number of DNS-queries minimal (Dial-on-Demand)?".
  • Choose "mbox format in /var/mail/" for "Delivery method for local mail".
  • Choose "YES" for "Split configuration into small files?".
vi /etc/exim4/passwd.client
 
 
add the following lines:
*.google.com:SMTPAccountName@gmail.com:y0uRpaSsw0RD
# chown root:Debian-exim /etc/exim4/passwd.client # chmod 640 /etc/exim4/passwd.client
 
Edit your aliases file
# newaliases
 
# update-exim4.conf
# invoke-rc.d exim4 restart
# exim4 -qff
 
 tada
 
mostly grepped from here 
 
 

Tuesday 26 August 2014

List installed windows programs on a pc remotely via command line


install the sysinternals process tools from here

psexec \\COMPUTERNAME -u administrator -p password reg query "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall" /s | findstr /B ".*DisplayName"
 voila
Note, there are other methds to query via WMI, but this only works with MSI installed packages.
 

Sunday 24 August 2014

Creating RRD graphs for TME Ethernet thermometer TME_C_EU pt2

 Ok, so last post I created the graphs, but they don't look that good. They don't seem to scale right. They scale from 0 to about 30oC


So it's showing values 0-30oC, which doesn't show the zoomed in detail, as the fluctuations are only 27oC +- 3oC. This is to do with auto-scaling and that stacked area graphs start from 0.
The detail we need is actually in the 25-30 range.

So I've had to change the index.php script so that it auto scales. It doesn't look as fancy, but at least you can see the temperatures better.
    "--slope-mode",
    "--start", $start,
    "--title=$title",
    "--vertical-label=Temperature in oC",
    "DEF:temp=/mnt/ramdisk/temp.rrd:temp:AVERAGE",
    "DEF:tempmin=/mnt/ramdisk/temp.rrd:temp:MIN",
    "CDEF:ttemp=temp,1,*",
    "LINE:ttemp#E54400::",
    "COMMENT:\\n",
    "--alt-autoscale",
    "--alt-y-grid",
    #"--rigid",
    "GPRINT:temp:MAX:Max temp %6.2lf",
    "GPRINT:temp:MIN:Min temp %6.2lf",
    "GPRINT:temp:AVERAGE:Avg temp %6.2lf",

To make this prettier, you could also define the max/min values for the entire graph
$options = array(
    "--slope-mode",
    "--start", $start,
    "--title=$title",
    "--vertical-label=Temperature in oC",
    "DEF:temp=/mnt/ramdisk/temp.rrd:temp:AVERAGE",
    "DEF:tempmin=/mnt/ramdisk/temp.rrd:temp:MIN",
    "DEF:tempmax=/mnt/ramdisk/temp.rrd:temp:MAX",
    "CDEF:ttemp=temp,1,*",
    "VDEF:ttempmin=tempmin,MINIMUM",
    "VDEF:ttempmax=tempmax,MAXIMUM",
    "HRULE:ttempmin#4444FF::dashes=2,2",
    "HRULE:ttempmax#FF5500::dashes=2,2",
    "LINE:ttemp#000000::",
    "COMMENT:\\n",
    "--alt-autoscale",
    "--alt-y-grid",
    #"--rigid",
    "GPRINT:temp:MAX:Max temp %6.2lf",
    "GPRINT:temp:MIN:Min temp %6.2lf",
    "GPRINT:temp:AVERAGE:Avg temp %6.2lf",
        );
This will draw a horizontal ruler at the min and max values with blue and red dashed lines, respectively.

Saturday 23 August 2014

Creating RRD graphs for TME Ethernet thermometer TME_C_EU pt1

So I decided to buy one of these network temperature sensors from Papouch here to monitor my 19" rack at home. I wanted to graph the temperature variations from it, as it can get quite hot when all the other servers are on.

Setup Apache with mod_php5 + rrd+ php5_cli

There are many articles on setting up apache + mod_php5 + rrdtools + php5_cli, so i won't bother you with the details here.

Create the RRD

One you've installed rrdtool and php5 cli, create the below file and chmod a+x so that it is executable.
Once run, this will create your RRD to input data called temp.rrd
#!/usr/bin/php
<?
echo "Creates blank temp graphs";
$options = array (
 "--step", "60",            // Use a step-size of 1 minute
 "--start", "-1 day",     // this rrd started 6 months ago
 "DS:temp:GAUGE:100:U:U",
"RRA:MIN:0.5:60:8800",
"RRA:MAX:0.5:60:8800",
"RRA:AVERAGE:0.5:2:328800",
);


$ret = rrd_create("./temp.rrd", $options);
if (! $ret) {
 echo "<b>Creation error: </b>".rrd_error()."\n";
}

?>

Create the graphs

This should be your index.php This should be in your /var/www directory
<?

create_graph_temperature("/var/www/temp-hr.gif","-1h","Hourly temperature");
create_graph_temperature("/var/www/temp-2hr.gif","-2h","Bi-Hourly temperature");
create_graph_temperature("/var/www/temp-daily.gif","-23h","Daily temperature");
create_graph_temperature("/var/www/temp-weekly.gif","-1w","Weekly temperature");
create_graph_temperature("/var/www/temp-monthly.gif","-1m","Monthly temperature");
create_graph_temperature("/var/www/temp-yearly.gif","-1y","Yearly temperature");

function create_graph_temperature ($output, $start, $title)
{
    $options = array(
    "--slope-mode",
    "--start", $start,
    "--title=$title",
    "--vertical-label=Temperature in oC",
    "DEF:temp=/mnt/ramdisk/temp.rrd:temp:AVERAGE",
    "CDEF:ttemp=temp,1,*",
    "CDEF:ttempq=temp,6,/",
    "AREA:ttempq#F31616::STACK",
    "AREA:ttempq#F03D11::STACK",
    "AREA:ttempq#ED650D::STACK",
    "AREA:ttempq#EA8D08::STACK",
    "AREA:ttempq#E7B504::STACK",
    "AREA:ttempq#E5DD00::STACK",
    "COMMENT:\\n",
    "GPRINT:temp:MAX:Max temp %6.2lf",
    "GPRINT:temp:MIN:Min temp %6.2lf",
    "GPRINT:temp:AVERAGE:Avg temp %6.2lf",
        );
    $ret = rrd_graph($output, $options);
        if (! $ret) {
        echo "<b>Graph error for temp</b>".rrd_error()."\n";
        }
}


?>

<img src=temp-hr.gif><img src=temp-2hr.gif></br>
<img src=temp-daily.gif><img src=temp-weekly.gif></br>
<img src=temp-monthly.gif><img src=temp-yearly.gif></br>
Note:you may need to chgrp www-data and chmod g+w on the /var/www directory, so that apache can write the GIFs to there. 

Update data into RRD

Do to this, I wrote a small PHP script
#!/usr/bin/php
<?
 // Fetch current time
    $t = time();
    $filename="/mnt/ramdisk/temp.txt";

    $handle = fopen($filename, "r");
    $contents = fread($handle, filesize($filename));
    fclose ($handle);
    $contents=(float)$contents/10;
    $option = array ("$t:$contents");
    echo "$t ,$contents";
    $ret = rrd_update("/mnt/ramdisk/temp.rrd", $option);

    if (! $ret) {
    mail ("blah@gmail.com","temp insert rrd error","trying to insert temp from temp.txt into temp.rrd");
        #echo "<b>Graph error: </b>".rrd_error()."\n";
    }


?>

Crontab

I've set up my TME_C on IP address 192.168.2.120
Note I've saved the output to a ramdisk (as my machine boots from a USB key)
 crontab -e on root
* * * * * /usr/local/bin/update_temp_script.sh
Note you should be calling a script which does the update, then inserts it into the RRD in sequence. If you don't it maybe updating the RRD with blank values.
update_temp_script.sh
/usr/bin/curl -s http://192.168.2.120/fresh.xml | grep sns  | awk '{print $10}' | cut -d "\"" -f 2 > /mnt/ramdisk/temp.txt 
/usr/local/bin/update_temp.php
That's it - it's that simple.
You should have graphs looking similar to this:


Now you can also setup a daily cron job to tar.gz these files to the local file system instead of leaving it on a ramdisk or what not. If you leave it on the ramdisk, you'll lose it if the server powers it off.

part 2 here

NOTE - Have corrected this as i was only storing about 1/5 of the data, so have upped the data storage points on the RR. This was calculated at one data point every 60 seconds. So 60 seconds in 24 hrs = 1440 points.
Hourly points for a year = 366 *24 = 8784 points. I rounded this to 8800


Friday 22 August 2014

PuTTY with tabs!

Now there are several programs that do this, but the best I've found is SuperPutty (here).

With this it acts more like the Mac OSX terminal, where you can press CTRL + [ and CTRL + ] to go to previous and next tabs.

Just remember to change the selection tab to be 'compromise' rather than windows., cos windows right clicking pops up a sodding menu.


Thursday 5 June 2014

Cogito Classic Smartwatch - initial review

Cogito Classic smart watch - initial review

 So am out in Taiwan for Computex and came across this in a whole range of colours - the Cogito smartwatch - It's one fo the first smartwatches I've handled in real life, that doesn't look like a smartwatch, which is a good thing.
So first impressions (i've only had the watch for 3 days) - it looks good, and the strap is rubbery (silicon actually). Thought I've noticed it smells of burnt rubber - odd. Of the the things that attracted me to the watch was the overall looks and the fact it's water resistant to 100m (10ATM), so I can swim and snorkel with it, unlike most other watches.

Cogito Classic compared to my Omega Planet Ocean Chrono

So what do you get in the box? the standard affair of manual and watch - nothing else, and to be honest you don't need anything else.

Specifications

Out of the box I have
Software revision: 2.00.0044
Hardware revision: 2.00.0000
Firmware revision: 1.03.0002

Pairing

First what you have to do is pair it. I have a HTC One, and whatever you don't pair it using the built in android system - it won't allow the App to detect the watch.
Download the app from the Appstore (cogito watch, connected devices), and pair it with that.

Allow the app and the service to accept notifications (it will do that from the app) or it will never be able to send notifications to the watch.

Once paired, it will have listed the watch as both 'Connected' and 'Paired'. If you do it with just the android system, it will list it as 'paired' but it never connects.

It will also auto set the time

The phone App

Ok, so the app for android is pretty simple.
The first page lists all the notifications, and you can filter it with the buttons on the left hand side.
Android App home screen

Under watch settings, I've found that music controls do not work with google music :(

Annoyances

  • If the caller ID is longer than (I think 10 characters), it will not display the caller name.
  • If multiple notifications appear (e.g. phone, mail, social), it will display only the total of all notifications. You cannot select the notification and find out how many there are of a certain type, with the left/right button
  • Cannot individually select a notification and acknowledge that.
  • The App should auto clear all notifications above 100. What happens if you receive over 10,000 notifications? Will the app crash?
  • Cannot set date on watch to DD/MM (default set to MM/DD)

Issues

So this has only happened once to me, where the watch crashes, and it only displays the last notification. I wasn't so sure if it was the app on my phone crashed or what? Doing a soft reset (pressing the four buttons all at once with 3 seconds), didn't resolve it, so I had to open the back and pull the main cr2032 battery out, and short some pins, then re-pair it.

Not a great start.

The customer support email address from within the app gave me SMTP 550 error - the recipient does not exist! W T F!

Also contacting them on twitter has yielded no response either (so far 2 days).




Saturday 29 March 2014

Using a CF adapter on a SATA interface - DMA timeout / issues / linux

Using a CF adapter on a SATA interface but with DMA/Timeout Errors

So you've decided to use a SATA to CF adapter, and when you boot linux, you're getting various timeouts when accessing the CF card along the lines of:

 ata3.00: status: { DRDY ERR }
 ata5.00: status: { DRDY ERR }

or maybe even this:

Nov 27 15:26:09 Tower kernel: ata5.00: status: { DRDY } (Drive related)
Nov 27 15:26:09 Tower kernel: ata5: hard resetting link (Minor Issues)
Nov 27 15:26:09 Tower kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) (Drive related)
Nov 27 15:26:09 Tower kernel: ata5.00: configured for UDMA/133 (Drive related)
Nov 27 15:26:09 Tower kernel: ata5.00: device reported invalid CHS sector 0 (Drive related)
Nov 27 15:26:09 Tower kernel: ata5: EH complete (Drive related)

But basically, something to do with DMa and your disk timing out?
First of all, replace your SATA cable - it could be borked.
If you're still getting timeout message regarding DMA, it's probably because your CF card doesn't support it.

If that is the case, you'll need to add a kernel parameter to your /etc/default/grub options or whne you set up linux/ubuntu.

The option you will need is 
libata.dma=3


libata.dma=0 Disable all PATA and SATA DMA
libata.dma=1 PATA and SATA Disk DMA only
libata.dma=2 ATAPI (CDROM) DMA only
libata.dma=4 Compact Flash DMA only 
so libata.dma=3 enables DMA for disks and CDROMs, but not CFs.

Sunday 16 March 2014

mdadm with 4tb HDDs + kernel tweaks for improved speeds pt3 (samba config)

So it's all done right?

Well I thought I'd share this lil snippet.

When deleting a file on a network share e.g. Samba shares delete, deletes permanently. It doesn't move it to a 'trash can' or 'recycle' bin.

To enable undelete, use this tweak under your samba share declaration:

[share] 
vfs object = recycle
        recycle:repository = .deleted/%U
        recycle:keeptree = Yes
        recycle:touch = Yes
        recycle:versions = Yes
        recycle:maxsixe = 0
        recycle:exclude = *.tmp
        recycle:exclude_dir = /tmp
        recycle:noversions = *.doc

Where the file will be moed to the .deleted folder under your username.
 

Speeding up SSH SCP connections

Speeding up SCP copy operations


So you're quickly copying a ~12GB MKV file from your download server to your archive server and are only getting a paltry 25MB/s.... obviously it isn't the network or disk as you've tested the dual gig connection at around 80-90MB/s via samba. You can't be arsed using rsync so you use scp to copy it but it's pretty damn slow.

file.mkv  12% 1893MB  22.0MB/s   10:23 ETA

Sound familiar? Well, you'll need to change the encryption options and turn off compression.

You can either do this on the command line (scp -c arcfour). Arcfour is plain RC4 with a 128-bit key (RFC4253).

or your could change to ~./ssh/config file to something line this:
Host hal
        Compression no
        Ciphers arcfour

After these changes you should be getting around double to triple the SCP copy speed

      100%   15GB  50.3MB/s   05:10

Please note this is for internal connections only. I wouldn't recommend this for external connections.

Wednesday 12 March 2014

mdadm with 4tb HDDs + kernel tweaks for improved speeds pt2 (tweaking)

Creating a linux software RAID with HDDs greater than 4tb and kernel tweaking for increased speeds pt2 (tweaking)

 Continuing from pt1

I've created a raid device (/dev/md0) and it's currently rebuilding the array with a whopping 16TB cpacity.
You can examine this via here:

mdadm --examine /dev/sda1
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6fc77ab1:cb9bdc74:2d2f3938:9ba4b4e7
           Name : HAL:HAL  (local to host HAL)
  Creation Time : Tue Mar 11 23:31:15 2014
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 7813771264 (3725.90 GiB 4000.65 GB)
     Array Size : 15627540480 (14903.58 GiB 16002.60 GB)
  Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 77542ec8:7c4efdea:9510ca31:fd4bb187

    Update Time : Wed Mar 12 00:16:15 2014
       Checksum : e038ac94 - correct
         Events : 14

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAAA ('A' == active, '.' == missing)


Now the speed you're getting will be the basic speed that'll use up to around 40% of your CPU (YMMV). To speed up the rebuild speed, i've used this kernel tweak:

echo 1024 > /sys/block/md0/md/stripe_cache_size
This has increased my rebuild speed double, from ~56k/s to around 96k/s
root@HAL:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sde1[5] sdd1[3] sdc1[2] sdb1[1] sda1[0]
      15627540480 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_]
      [========>............]  recovery = 42.7% (1669179940/3906885120) finish=388.2min speed=96060K/sec
     
unused devices: <none>
CPU usage during rebuild.
top - 07:57:38 up  8:30,  2 users,  load average: 1.29, 1.23, 1.15
Tasks: 107 total,   2 running, 105 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.1%us, 28.0%sy,  0.0%ni, 67.0%id,  0.0%wa,  0.0%hi,  3.8%si,  0.0%st
Mem:   8047068k total,   528472k used,  7518596k free,      672k buffers
Swap:        0k total,        0k used,        0k free,   339700k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                
 2046 root      20   0     0    0    0 S   39  0.0 121:38.13 md0_raid5                                                                                                                               
 2048 root      20   0     0    0    0 D   23  0.0  73:15.66 md0_resync   
 
iostat during rebuild
(notice odd tps for sda1 - not sure what that is about ....)
Every 1.0s: iostat -k 1 2                                                                                                                                                     Wed Mar 12 08:18:06 2014

Linux 3.11.0-15-generic (HAL)   12/03/14        _x86_64_        (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.62    0.19    5.44    0.50    0.00   93.25

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             219.66     56178.28         0.00 1789479436         74
sdb             110.27     56178.21         0.00 1789477229         74
sdc             110.26     56178.13         0.00 1789474572         74
sdd             110.27     56177.87         0.00 1789466360         74
sde             111.24         0.08     56176.92       2529 1789436170
sdf               0.45         7.86        13.15     250385     418979
md0               0.01         0.02         0.00        704          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.10    0.00   30.39    0.00    0.00   68.51

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             343.00     87048.00         0.00      87048          0
sdb             172.00     85000.00         0.00      85000          0
sdc             177.00     87560.00         0.00      87560          0
sdd             193.00     95240.00         0.00      95240          0
sde             191.00         0.00     95240.00          0      95240
sdf               0.00         0.00         0.00          0          0
md0               0.00         0.00         0.00          0          0

Now, the best file system to use is xfs, as it will auto calculate the stripe size you need to get the best perfomance.

When i tried this, it complained that it already contained a partition table, so I overwrote it.

root@HAL:~# mkfs.xfs /dev/md0
mkfs.xfs: /dev/md0 appears to contain a partition table (gpt).
mkfs.xfs: Use the -f option to force overwrite.
root@HAL:~# mkfs.xfs /dev/md0 -f
log stripe unit (524288 bytes) is too large (maximum is 256KiB)
log stripe unit adjusted to 32KiB
meta-data=/dev/md0               isize=256    agcount=32, agsize=122090240 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=3906885120, imaxpct=5
         =                       sunit=128    swidth=512 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0



Once mounted, using mount /dev/md0 /mnt/<your dir>

you should be able to see this when you do a df -h
 root@HAL:/etc/samba# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdf1       7.5G  1.3G  6.2G  17% /
udev            3.8G  8.0K  3.8G   1% /dev
tmpfs           1.6G  912K  1.6G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.9G     0  3.9G   0% /run/shm
/dev/md0         15T   95G   15T   1% /mnt/md0



Don't forget to use blkid to add the newly mount raid array to fstab!
root@HAL:/etc/samba# blkid
/dev/sda1: UUID="6fc77ab1-cb9b-dc74-2d2f-39389ba4b4e7" UUID_SUB="77542ec8-7c4e-fdea-9510-ca31fd4bb187" LABEL="HAL:HAL" TYPE="linux_raid_member"
/dev/sdb1: UUID="6fc77ab1-cb9b-dc74-2d2f-39389ba4b4e7" UUID_SUB="20a6a078-6ae1-309a-408e-44bb1f2bb18c" LABEL="HAL:HAL" TYPE="linux_raid_member"
/dev/sdc1: UUID="6fc77ab1-cb9b-dc74-2d2f-39389ba4b4e7" UUID_SUB="f4fa566f-7bf7-aecc-db62-3db7e33ee637" LABEL="HAL:HAL" TYPE="linux_raid_member"
/dev/sdd1: UUID="6fc77ab1-cb9b-dc74-2d2f-39389ba4b4e7" UUID_SUB="d55da20b-16e7-e3b1-14b7-4d4069ca14f7" LABEL="HAL:HAL" TYPE="linux_raid_member"
/dev/sde1: UUID="6fc77ab1-cb9b-dc74-2d2f-39389ba4b4e7" UUID_SUB="65c1c74a-94f4-a448-69f2-0cee3b237402" LABEL="HAL:HAL" TYPE="linux_raid_member"
/dev/sdf1: LABEL="HAL" UUID="ec1e225b-ea2b-41f8-a2c9-199bdcb8d541" TYPE="xfs"
/dev/md0: UUID="8cc6eaad-e7c1-4e0d-820c-3b6334406011" TYPE="xfs"


I've already installed samba and am copying files from another NAS to my new one.
With this, I am maxing out my gig ethernet on my windows machine at around 88MB/s, which is almost reaching the limit on my gig network.

using iostat -k 1 2
Every 1.0s: iostat -k 1 2                                                                                                                                                     Wed Mar 12 20:05:50 2014

Linux 3.11.0-15-generic (HAL)   12/03/14        _x86_64_        (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.57    0.07   13.06    1.13    0.00   85.17

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             207.90     52663.69       412.26 3913864509   30638377
sdb             105.77     52663.40       412.90 3913842698   30686017
sdc             105.76     52664.02       412.61 3913888889   30664137
sdd             105.81     52664.60       413.06 3913931749   30698045
sde             105.43        13.69     53063.40    1017416 3943569905
sdf               0.35         4.31         9.65     320101     716898
md0               6.60         0.05      1623.15       3551  120629217

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.62    0.00   12.42   46.58    0.00   40.37

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              59.00         0.00     14764.00          0      14764
sdb              41.00         0.00     18976.00          0      18976
sdc              46.00       288.00     19240.00        288      19240
sdd              44.00       264.00     18976.00        264      18976
sde              41.00         0.00     18976.00          0      18976
sdf               0.00         0.00         0.00          0          0
md0             154.00         0.00     38916.00          0      38916


I am getting these speeds. Although I haven't tested the raid speed using bonnie, I am confident that I can saturate my gig ethernet link, and once I've bonded the dual NICs I have, we'll see how much I can push to my NAS

Final stats using vnstat, using a test 100gb
                           rx         |       tx
--------------------------------------+------------------
  bytes                   109.78 GiB  |      768.67 MiB
--------------------------------------+------------------
          max          779.37 Mbit/s  |     6.16 Mbit/s
      average          593.73 Mbit/s  |     4.06 Mbit/s
          min               0 kbit/s  |        0 kbit/s
--------------------------------------+------------------
  packets                   80111899  |         9964991
--------------------------------------+------------------
          max              67732 p/s  |        8634 p/s
      average              51651 p/s  |        6424 p/s
          min                  3 p/s  |           0 p/s
--------------------------------------+------------------
  time                 25.85 minutes








mdadm with 4tb HDDs + kernel tweaks for improved speeds pt1 (RAID creation)

Creating a linux software RAID with HDDs greater than 4tb and kernel tweaking for increased speeds pt1 (RAID creation)

So I've run out of storage space at home, and decided to build another NAS box with another HP Microserver N54L.
These cheap and cheerful servers come in at around £99 (with cashback) and HP are always doing cashback with these servers it seems.
I upgraded the memory to 8gb with these Crucial memory DIMMS (Part Number: CT2KIT51272BD1339 - 8GB Kit (4GBx2), 240-pin DIMM) for £85 inc shipping.
I am also using a 8GB USB key to boot from and a dual port gig NIC from intel.

So my drive config is 5 x 4TB HDD (WD Reds -WD40EFRX), but the N54L has a limitation where the 5th SATA connector is for the CDROM drive, so it's purposely crippled by default. To unleash the full speed of 3gb/s SATA, you have to do a simple BIOS update. Instructions are here

Once that's done, you will have hot-swap and a 5th SATA port running at 3gb/s!

First - you must *not* use fdisk, as this will only recognise the first 2tb of the drives. I only realised this after I had created the RAID and then did a mkfs.xfs, and found out I only had 8TB or so of space.... like W T F?!
You will need to partition it with GPT and set the flag to RAID. This can be done with gparted. There is an ISO you can download from the gparted site.

Once there, just partition with gparted, and set flag=raid. Then apply.
Restart and then build your array with the command

mdadm --create --verbose --metadata=1.2 /dev/md0 --level=5 --raid-devices=5 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 --name=HAL
At this current time, mine is still rebuilding, but hopefully this should work!

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sde1[5] sdd1[3] sdc1[2] sdb1[1] sda1[0]
      15627540480 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_]
      [>....................]  recovery =  3.9% (154246500/3906885120) finish=1105.3min speed=56583K/sec
     
unused devices: <none>