RaspberryPi

From HerzbubeWiki
Jump to: navigation, search

This page has my notes on fiddling with the Raspberry Pi.


References

General information


Tutorials and HOWTOs


Download links:


Glossary

NOOBS 
New Out Of the Box Software. An operating system installer software that is used to download an OS image
Pi 
Short for Raspberry Pi
Raspbian 
A Debian-based operating system for the Raspberry Pi


Prepare the SD card on the Mac

The information in this section is largely a repeat of what you can read in the "HOWTO install NOOBS on a Mac" that is referenced at the top of this page.

Before you begin, make sure that the Raspberry Pi is properly connected to some hardware:

  • A mouse and a keyboard (both USB)
  • A monitor (you need a HDMI cable)
  • A network (Ethernet cable) with connection to the Internet
  • Have the power supply ready, but don't connect the Pi yet (or it will try to boot)


The first thing to do is to set up the SD card. You do this on your Mac:

  • Download and install the SDFormatter app (link is in the "References" section)
  • Launch the SDFormatter app. For some reason the app wants you to enter administrator credentials - apparently this is normal and no cause for worry
  • Insert the SD card into the SD card reader of your Mac. The SDFormatter app should automatically notice that the card was inserted and update its UI accordingly.
  • Enter a name for the SD card (e.g. "RASPBERRYPI"). Note that spaces and probably other special characters are not allowed in the name (the app will tell you so only after you attempt to format the card with the illegal name), and that lower-case letters will be converted to upper-case.
  • Select "Quick Format" (this is much faster than the "Overwrite Format" option)
  • Click the "Format" button
  • When formatting is complete, the SD card should become available as a volume in the Finder


Copy NOOBS to the SD card

  • Download NOOBS from the Raspberry Pi Foundation website (link is in the "References" section). A few notes:
    • The "NOOBS Lite" image is much smaller than the regular NOOBS image because it doesn't include an operating system, it contains only NOOBS. The regular image contains NOOBS plus the Raspbian operating system.
    • So if you don't want to install Raspbian then the Lite image is a better download choice now. Later on when you run NOOBS you will get the choice which operating system to download.
    • Whether you download Rasbian now together with NOOBS in one image, or later via NOOBS (roughly a 3 GB download), does not matter - the image you get is the same, and it always includes the fancy GUI desktop even though you may want to set up your Raspberry Pi as a server machine only.
  • Unzip the NOOBS archive
  • Copy the unzipped files to the SD card volume. Make sure that you copy the unzipped files to the root of the volume.
  • Eject the SD card in the Finder, then remove it from the Mac


The next step is to install the operating system on the Raspberry Pi.


Install the operating system

Installation of the OS is quite simple and painless:

  • Insert the SD card into the Raspberry Pi
  • Connect the power supply
  • The Pi now boots from the SD card and automatically runs NOOBS. After a moment you should see the NOOBS graphical UI.
  • Select Raspbian as the operating system, then proceed with the installation
  • Wait until NOOBS has downloaded and installed everything


After NOOBS has finished installing the operating system it will reboot the Pi into the graphical desktop of the operating system. The next step is to configure the OS (Raspbian in my case). If you ever need to launch NOOBS again, you can reboot the Pi and hold down the shift key while the machine boots.


General Raspbian configuration

The physical console

The Pi boots into the graphical desktop and logs you in automatically as the default user pi. You can switch to console-mode via the time-honored keyboard shortcut Ctrl+Alt+F1. This will display the first console where the default user pi is also already logged in.

Switching to other consoles is possible with the usual keyboard shortcuts Alt+F[1-7]. On consoles 2-6 you will get a login prompt. On console 7 (Alt+F7) you will switch back to the graphical desktop, from where you have to break out again with a Ctrl+Alt+F* shortcut.


Change keyboard layout

Run 

sudo raspi-config

From the interactive menu select "Internationalisation Options" followed by "Change Keyboard Layout", then choose the new keyboard layout. After that it's probably simplest if you reboot to activate the new layout - I'm pretty sure there are commands that allow to activate the layout without a reboot, but I was unable to find out which ones these are.

raspi-config made the following simple change: 

pi@raspberrypi1:/etc $ git diff default/keyboard
diff --git a/default/keyboard b/default/keyboard
index b6e6925..5287ac5 100644
--- a/default/keyboard
+++ b/default/keyboard
@@ -3,7 +3,7 @@
 # Consult the keyboard(5) manual page.
 
 XKBMODEL="pc105"
-XKBLAYOUT="gb"
+XKBLAYOUT="ch"
 XKBVARIANT=""
 XKBOPTIONS=""

After the reboot, this file was also changed: 

console-setup/cached_UTF-8_del.kmap.gz


Timezone

Setup the timezone by running 

sudo dpkg-reconfigure tzdata


Change default password for user pi

As mentioned above, the default user is named 

pi

The default password for that user is 

raspberry

It is a good idea to change that default password to something else, just in case.


Root access

Raspbian is configured to disallow root login. This is explained very well in this Raspberry Pi Stackexchange post.

Instead the idea is to use sudo. The default user pi is a sudoer who is allowed to run all commands as root, even su root.


System name

The default system name is raspberrypi. To change the system name, edit the following file 

/etc/hostname

This change will take effect only when you reboot the system. To also change the name in the current session, run this command 

sudo hostname <new name>


SSH access

The SSH daemon is already running by default and accepts connections for the default user pi. I like to have public key auth, so I'm scp'ing the appropriate public key to 

~/.ssh/authorized_keys

I have not tried this out, but apparently SSH can be enabled or disabled using 

sudo raspi-config


Disable default login upon boot

As mentioned above, the Pi auto-logins both on console 1 and on the GUI console after a boot. To disable this, run 

sudo raspbi-config

then select "Boot options". Here you can define the desired login options. I prefer "Text console, requiring user to login".


This causes one systemd file to be changed, and a new file to be created. I am not knowledgeable when it comes to systemd, so I won't pretend that I understand the changes. Here they are, unfiltered, for those who might be able to make head or tail of it: 

pi@raspberrypi1:/etc $ git diff /etc/systemd/system/getty.target.wants/getty@tty1.service
diff --git a/systemd/system/getty.target.wants/getty@tty1.service b/systemd/system/getty.target.wants/getty@tty1.service
index 196d01b..facee85 120000
--- a/systemd/system/getty.target.wants/getty@tty1.service
+++ b/systemd/system/getty.target.wants/getty@tty1.service
@@ -1 +1 @@
-/etc/systemd/system/autologin@.service
\ No newline at end of file
+/lib/systemd/system/getty@.service
\ No newline at end of file


pi@raspberrypi1:/etc $ cat /etc/systemd/system/default.target
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Multi-User System
Documentation=man:systemd.special(7)
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes


Disable WLAN and Bluetooth

TODO


CAcert certificates

My public server pelargir.herzbube.ch runs on a diet of SSL certificates that are issued by CAcert. To be able to connect to the various services of pelargir, the CAcert root certificates must be added to the Raspbian system-wide trust store. The wiki page CAcert explains how to do this in general for Debian systems.

This section is obsolete, I no longer use CAcert certificates.


Install the HTB

Follow instructons on the wiki page HTB to install the HTB (herzbube's toolbox).


Switch to vim

vim-tiny is already installed, but I like the benefit of directly opening .gz files with vi, so I install the Debian package 

vim


Now run the following interactive command to switch the default text editor from nano to vim: 

sudo update-alternatives --config editor


btrfs support

I am going to try to use the filesystem btrfs on various external USB disks that I will attach to the Pi. Unlike the default filesystem ext4, some things still have to be configured for btrfs.


First, load the btrfs kernel module: 

sudo modprobe btrfs

Next, install the user-space utilities that allow us to create or otherwise handle a btrfs file system 

btrfs-tools


Raspbian legacy distribution support

When I installed Raspbian the Debian release "jessie" was the current thing. I never updated to a newer Debian release, so when jessie eventually was ditched from the official package mirrors I could no longer install packages.

To fix the problem I had to manually replace the package mirror in /etc/apt/sources.list. Old mirror: 

deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi

New mirror: 

deb http://legacy.raspbian.org/raspbian/ jessie main contrib non-free rpi

According to this StackExchange post yet an alternative mirror seems to be http://archive.volumio.org. Note: You can't browse the mirror content with a web browser, all directory listings return a "File not found" page, but accessing the actual mirror/package files seems to work. I never tried out the mirror for real, though.


Integration into LAN

In my case, all I have to do is configure the Fritz!Box, which acts as the DCHP server in my LAN, to assign a name to the device: 

raspberrypi1


Fileserver configuration

Prepare a hard disk

Connect an external USB hard disk to the Pi. Format it for btrfs like this: 

sudo mkfs.btrfs -f /dev/sda1

Run blkid to find out the new filesystem's UUID: 

pi@raspberrypi1:~$ sudo blkid 
[...]
/dev/sda1: UUID="98a93ee5-02a2-4d48-8e76-517861803ed8" UUID_SUB="0a19a860-e481-4aa7-9f2d-1f4a7c2e7f14" TYPE="btrfs" PARTUUID="cbdf9638-01"

Add an entry to /etc/fstab: 

UUID=98a93ee5-02a2-4d48-8e76-517861803ed8  /mnt/fileserver        btrfs   defaults   0   2

Finally, mount the file system (create the mount point first if it doesn't exist yet): 

sudo mount /mnt/fileserver


Prepare filesystem for "Scan to Network Folder" function

The household's current HP printer has a "Scan to Network Folder" function that allows the printer to place scanned documents into a predefined network folder. In this step we create the filesystem infrastructure that is necessary for this to work.


First, we need the actual filesystem folder that will receive the documents: 

mkdir -p /mnt/fileserver/daten/Temp/scanning


Next we create a symlink to this folder that we can then use to set up a Samba network share with non-privileged access: 

ln -s /mnt/fileserver/daten/Temp/scanning /mnt/fileserver/scanning


Last but not least, because several unrelated users need to have write access to this folder, we need to make it world-writable: 

chmod 777 /mnt/fileserver/daten/Temp/scanning


Install samba

Install the package 

samba

to install the Samba file server daemon. See the Samba wiki page for details on how to configure Samba.


Here are my changes to the default configuration in /etc/samba/smb.conf: 

[global]
   # Don't sync because I want the system user "pi" to have a different
   # password than the Samba user "pi". The Samba user's password is much
   # simpler.
   unix password sync = no

[daten]
   comment = Haushalt Daten
   path = /mnt/fileserver/daten
   read only = no
   browseable = yes
   create mask = 0700
   directory mask = 0700
   valid users = pi

[alles-andere]
   comment = Alle anderen Daten
   path = /mnt/fileserver/alles-andere
   read only = no
   browseable = yes
   create mask = 0700
   directory mask = 0700
   valid users = pi

# We don't want the scanner device to be able to access the entire fileserver 
# share - there are too many sensitive documents on it! If the printer 
# device is compromised in a currently unforeseen attack, then we want to 
# limit the attacker to a fileserver area that is as small as possible. This 
# is the rationale behind this separate "scanning" network share. 
# In order to make the limitation effective, we must of course also have a 
# separate user (piscanner) that the printer device can use, but which has
# access only to the "scanning" network share. The regular "pi" user is 
# trusted, so it can have access as well.
[scanning]
   comment = Scanning Daten
   path = /mnt/fileserver/scanning
   read only = no
   browseable = yes
   create mask = 0666
   directory mask = 0777
   valid users = pi, piscanner

Notes:

  • Not visible above: I have commented out all other shares


Create system user for scanner device, but prevent it from logging in on the console. The commands used here are documented slightly better on the BasicSystemConfiguration# wiki page. 

adduser piscanner
passwd -l


Finally, add Samba users: 

sudo smbpasswd -a pi
sudo smbpasswd -a piscanner

Notes:

  • The Samba service must be running or the command will fail
  • Each user you create must correspond to an existing system user or the command will fail


Backup jobs

Summary

The Raspberry Pi is an integral part of my backup solution. It does the following things:

  • Create an off-site copy of the backup data set that is created every night on the dedicated server machine pelargir
  • Create a copy of some data sets that are stored on the intranet file server
  • Create snapshots of these copied data sets using a time-machine like tool


The following sections document how the system needs to be configured for this to work:

  • Connect 2 external USB hard disk drives to the Pi
  • Install a few Debian packages to provide the necessary software
  • Configure a cron job that runs the shell scripts that perform the two jobs


Prepare 2 hard disks

Connect 2 external USB hard disks to the Pi, then proceed to make them available in the system in the same way as outlined above in the "Fileserver" section. When everything is ready the hard disks appear something like this: 

pi@raspberrypi1:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
[...]
/dev/sda1       1.9T  2.0G  1.9T   1% /mnt/backup-copy
/dev/sdc1       1.9T   17M  1.9T   1% /mnt/backup-snapshot


Install keychain

Install the Debian package 

keychain

This is necessary so that automated cron scripts can use RSA/DSA authentication for passwordless logins to remote servers. See the OpenSSH wiki page for details.


bup

Although Raspbian includes a Debian package for bup, at the time of writing the package is severely out of date: Version 0.25-1 which is from December 2013 vs. the most recent release 0.28.1 which is from June 2016. In the last few versions bup has made important advances that I do not want to do without, because of that a manual build of an up-to-date clone of the bup repository is necessary.

As user pi run these commands: 

# Install build dependencies. An up-to-date list of packages can be found in
# the README of the GitHub repository.
sudo apt-get install python2.7-dev python-fuse python-pyxattr python-pylibacl linux-libc-dev acl attr python-tornado

# Optional, with this present the make process will create man pages
apt-get install pandoc

# Optional, this is used for writing parity information so that bup
# may be able to recover from some amount of repository corruption:
apt-get install par2

# Get the sources
mkdir ~/build
cd ~/build
git clone https://github.com/bup/bup.git

# Build, test and install
cd bup
make
make test
sudo make install DESTDIR=/usr/local PREFIX=''

Note: Two of the tests fail, but this is not an error. The reason is that the tests erroneously try to run bup restore on a repository with an absolute instead of a relative path. Of course, the repositories which the tests expect to exist in the root directory of the system do not exist there.


cron configuration

Here's the cron configuration snippet that runs the scripts: 

pi@raspberrypi1:~$ cat /etc/cron.d/backup-scripts 
# /usr/local/htb/bin is required to find HTB scripts
# /usr/local/bin is required to find bup
PATH=/usr/local/htb/sbin:/usr/local/htb/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# ----------------------------------------------------------------------
# Create copies of data sets
# ----------------------------------------------------------------------
# Run at 03:00 am every Monday and Thursday.
# The on-site backup job runs at 01:00 am every day, but
# that's irrelevant because we don't copy the data produced
# by that job. Instead we copy the data after logrotate has
# run. logrotate runs at 06:25 am. This means that we are
# actually copying the data from the previous night.
00 3 * * 1,4 pi /usr/local/htb/bin/htb-mkbackupcopy.sh -i "*.1" -i "*/" -e "*" root@pelargir.herzbube.ch:/var/backups/ /mnt/backup-copy/backupcopy.pelargir/ >>/mnt/backup-copy/backupcopy.pelargir.log 2>&1

# Run two hours after the previous copy job. This should
# be sufficient time for the previous job to complete.
# Even if it isn't, there's no harm done when the two jobs
# run in parallel - all that will happen is that the two
# jobs take a little longer because they both write to the
# same file system.
#
# The data set copied here is less than 10 GB in size.
00 5 * * 1,4 pi /usr/local/htb/bin/htb-mkbackupcopy.sh /mnt/fileserver/daten/ /mnt/backup-copy/backupcopy.daten/ >>/mnt/backup-copy/backupcopy.daten.log 2>&1

# Run 5 minutes after the previous copy job. This should
# be sufficient time for the previous job to complete.
#
# The data set copied here is less than 200 GB in size.
#
# When last measured, a full copy was 124 GB in size and
# took 7.5 - 8 hours to complete (slightly more than
# 0.25 GB per minute), and an incremental copy on those
# 124 GB with zero files to copy took less than 1 minute
# to complete.
05 5 * * 1,4 pi /usr/local/htb/bin/htb-mkbackupcopy.sh -e /Filme /mnt/fileserver/alles-andere/Media/ /mnt/backup-copy/backupcopy.media/ >>/mnt/backup-copy/backupcopy.media.log 2>&1

# Run 15 minutes after the previous copy job. This should
# be sufficient time for the previous job to complete.
# 15 minutes is sufficient to copy roughly 3-4 GB of data.
#
# The data set copied here is less than 100 GB in size.
#
# When last measured, a full copy was 50 GB in size, and
# and an incremental copy on those 50 GB with zero files
# to copy took less than 30 seconds to complete.
20 5 * * 1,4 pi /usr/local/htb/bin/htb-mkbackupcopy.sh /mnt/fileserver/alles-andere/Backup/Snapshots /mnt/backup-copy/backupcopy.backupsnapshots/ >>/mnt/backup-copy/backupcopy.backupsnapshots.log 2>&1

# Run 25 minutes after the previous copy job. This should
# be sufficient time for the previous job to complete.
# 25 minutes is sufficient to copy roughly 5-7 GB of data.
#
# The data set copied here is less than 10 GB in size.
#
# When last measured, a full copy was 2.7 GB in size and
# took 10 minutes to complete, and an incremental copy
# on those 2.7 GB with zero files to copy took less than
# 10 seconds to complete.
45 5 * * 1,4 pi /usr/local/htb/bin/htb-mkbackupcopy.sh -i /mailman.tar.gz -i "/OldWindowsData" -i "/Work" -i "/facebook-herzbube102.zip -e "/*" /mnt/fileserver/alles-andere/Archiv/ /mnt/backup-copy/backupcopy.archiv/ >> /mnt/backup-copy/backupcopy.archiv.log 2>&1

# ----------------------------------------------------------------------
# Create snapshot of data set copies
# Creating the initial snapshot may take substantial time, but once the
# initial snapshot exists subsequent snapshots typically are created in
# under a minute - unless, of course, the data set has seen a large
# turnover of files since the last snapshot. This means that the snapshot
# jobs can be timed relatively close after each other.
# ----------------------------------------------------------------------
# Run at 03:00 am every Sunday.
# This creates a snapshot of the off-site data that was
# copied on Thursday.
00 3 * * 0 pi /usr/local/htb/bin/htb-mkbackupsnapshot.sh /mnt/backup-copy/backupcopy.pelargir /mnt/backup-snapshot/backupcopy.pelargir.bup >>/mnt/backup-snapshot/backupcopy.pelargir.bup.log 2>&1
10 3 * * 0 pi /usr/local/htb/bin/htb-mkbackupsnapshot.sh /mnt/backup-copy/backupcopy.daten /mnt/backup-snapshot/backupcopy.daten.bup >>/mnt/backup-snapshot/backupcopy.daten.bup.log 2>&1
20 3 * * 0 pi /usr/local/htb/bin/htb-mkbackupsnapshot.sh /mnt/backup-copy/backupcopy.media /mnt/backup-snapshot/backupcopy.media.bup >>/mnt/backup-snapshot/backupcopy.media.bup.log 2>&1
30 3 * * 0 pi /usr/local/htb/bin/htb-mkbackupsnapshot.sh /mnt/backup-copy/backupcopy.backupsnapshots /mnt/backup-snapshot/backupcopy.backupsnapshots.bup >>/mnt/backup-snapshot/backupcopy.backupsnapshots.bup.log 2>&1
40 3 * * 0 pi /usr/local/htb/bin/htb-mkbackupsnapshot.sh /mnt/backup-copy/backupcopy.archiv /mnt/backup-snapshot/backupcopy.archiv.bup >>/mnt/backup-snapshot/backupcopy.archiv.bup.log 2>&1

Notes:

  • The shell scripts executed by cron are documented on the wiki page BackupScripts
  • The copying script requires that a working keychain / ssh-agent configuration is in place. See the OpenSSH wiki page for details.


Troubleshooting

System reboots into emergency mode

The system probably recommends to issue the command journalctl -xb to investigate the problem. This dumps a large log into your lap from which it can be very difficult to find out the root cause. This slightly modified command lists only log messages that have some sort of error: 

journalctl -xb -p3

In my case the root problem was that I had an entry in my /etc/fstab which referred to a removable USB drive which was no longer present at boot time. Uncommenting the entry and then rebooting solved my problem. Other solutions (which are all not very satisfying) are

  • Use the mount option "nofail". The problem here is that you don't get any errors if you really want to mount the drive but it doesn't work.
  • Use the mount option "noauto". The problem here is that the drive not only is not mounted at boot time, but also if you attach it later when the system is already running.
Retrieved from "https://wiki.herzbube.ch/index.php?title=RaspberryPi&oldid=7003"