BasicSystemConfiguration

tasksel

Debian package: tasksel

Let's the user choose from a task-oriented list what she wants to do with the system (examples: "desktop environment", "web server", "mail server", etc.). Each entry in the list corresponds to an entire collection of Debian packages that is installed when the user selects that entry.

It is probably not a very good idea to run tasksel after the system has been customized to some extent.


base-config

Debian package: base-config

From the man page:

base-config is the program that was run when you first rebooted into your newly installed debian system. It walks you through setting up the system and down-loading additional software, and so forth. The program can be run at any later date to walk you through essentially the same process again.


Locales

Having the proper locales is very important for working on the command line. An UTF-8 locale is especially important so that umlaut ("ä") and other non-ASCII characters are handled properly. Entering an umlaut without a locale simply does not work!

The locales present on a system also determine which languages are supported in gettext-based PHP applications. For instance, Gallery requires that one of the various "de_DE" locales is installed before the German language can be used in the web interface.


The package

locales

is responsible for managing locales. Add/remove locales and define a default locale using the following command:

dpkg-reconfigure locales

To see which locales currently are present on the system:

locale -a


Setting a default locale causes the environment variable LANG to be set to that locale upon login. The default locale is written to

/etc/default/locale

In a quick test I tried to manually change this file. This seemed to work: When I did my next login, the LANG envvar was properly set to the new value. However, I would still recommend setting the default locale by running dpkg-reconfigure - I simply don't know enough about the system,


Keyboard configuration

The keyboard configuration matters when you have physical access to the system and want to login at the console. When you administrate the system remotely via SSH the computer you're sitting at determines the keyboard layout.


Use loadkeys to load the kernel keymap for the console. Available keymaps can be found in /usr/share/keymaps.

At boot-time the script /etc/init.d/keymap.sh loads the keymap file /etc/console/boottime.kmap.gz.

It should be possible to change this setting by issuing the command dpkg-reconfigure console-common and answering the subsequent debconf questions. If that doesn't help you can manually copy the keymap to /etc/console.

Note: It is possible that the boot-time keymap file includes another file that is not yet available when booting reaches the keymap-loading stage, because the /usr filesystem has not yet been mounted. A solution for this is to create a specific keymap loader script in /etc/init.d that will be executed at a later time when /usr has become mounted.


/etc/motd

The content of /etc/motd is printed to the console or remote terminal session screen when a user logs in. The default in Debian is no fun, so change this to something more suitable.


Note: Around the release of Debian etch, the file to edit was /etc/motd.tail.


Time zone

The system's current time zone is stored in

/etc/timezone

Time zone data is maintained in the Debian package tzdata. Change the system's time zone with this command:

dpkg-reconfigure tzdata


Shell environment

/etc/profile.d

Files present in the folder

/etc/profile.d

are sourced by the system-wide /etc/profile. This happens for all users when they log in.


For instance, to enable the HTB (see the wiki page HTB for details) I have this:

root@pelargir:~# cat /etc/profile.d/pelargir_htb.sh 
HTB_BASE_DIR=/usr/local/htb
. $HTB_BASE_DIR/etc/htb-profile.sh


/etc/bash.bashrc

This file is sourced by /etc/profile when the user logs in. It is also read by each new invocation of bash, as long as the new sub-shell is interactive (i.e. the file is not read when a script is run).

Currently I have added the following stuff:

# ------------------------------------------------------------
# BEGIN pelargir
# ------------------------------------------------------------
HTB_BASE_DIR=/usr/local/htb
. $HTB_BASE_DIR/etc/htb-bashrc.sh
# ------------------------------------------------------------
# END pelargir
# ------------------------------------------------------------

See the wiki page HTB for details.


Automatically display compressed files with less

The Debian package less provides two useful shell scripts /usr/bin/lesspipe (this is the one I prefer) and /usr/bin/lessfile. As advised in /usr/share/doc/less/LESSOPEN:

You just need to put the following in your .zlogin/.login/.bash_profile/whatever:

eval $(lesspipe)
 or
eval $(lessfile)

and from now on you can directly open any compressed text file with less as if the program itself were able to read the compressed file. For instance:

less /usr/share/doc/less/changelog.Debian.gz

Note: I put the "eval" above command into

/etc/profile.d/pelargir_lesspipe.sh


Automatically display compressed files with vi

See VimStuff.


User/group management

Add/remove users and groups

The following are low-level tools to manage users and groups. They are present on any Linux system.

useradd
userdel
usermod
groupadd
groupdel
groupmod

The following are Debian specific front-ends to the above low-level tools. They should be used on Debian systems because they conform to Debian policies.

adduser
deluser
addgroup
delgroup

Note: Both the low-level tools and their Debian specific front ends manipulate system files (/etc/passwd, /etc/group, /etc/shadow). If the user database is located in an LDAP directory, they cannot be used anymore.


Disable an account

To disable an account so that the user can no longer log in:

passwd -l

This changes the password hash in /etc/shadow to a value that is not possible to generate by entering any password. In other words, the account still exists and the user can try to log in, but she will never succeed.


Log file rotation

Debian package

Log file rotation is done by the Debian package

logrotate


Configuration

The package's configuration file is /etc/logrotate.conf, which in turn includes all the package specific files located in

/etc/logrotate.d

The configuration file format is explained in

man logrotate


Manually running logrotate

logrotate is usually executed periodically by cron (by default it is part of /etc/cron.daily), although it can be manually called like this

logrotate /etc/logrotate.conf


Possible options are:

  • --force: Force log rotation even if logrotate doesn't think it needs to actually rotate a file.
  • --debug: Print debug output to stderr. No changes will be made to the logs or to the state file.
  • --state /path/to/file: Use the status file specified here. The default status file is stored in /var/lib/logrotate/status.


To test a configuration you can use the following endlessly looping shell script snippet:

while test true; do
  echo "rotating..."
  echo "foo" >/path/to/file
  logrotate --force --state /tmp/logrotate-test.status /tmp/logrotate-test.conf
  sleep 1
done


Configuration on pelargir

Here is a nifty configuration to rotate backup files in several stages: Daily (7x), weekly (4x) and monthly (12x):

root@pelargir:~# cat /etc/logrotate.d/pelargir-backupninja
# ----------------------------------------
# This configuration rotates backup files in 3 stages:
# - Daily, for the last 7 days
# - Weekly, for the last 4 weeks
# - Monthly, for the last 12 months
#
# After one year backups are rotated out of existence.
# ----------------------------------------


# ----------------------------------------
# Daily rotation stage
# ----------------------------------------
/var/backups/debconfsel.txt
/var/backups/git/*.tar
/var/backups/hardware.txt
/var/backups/home-directories/*.tar
/var/backups/ldap/*.ldif
/var/backups/mysql/sqldump/*.sql
/var/backups/postgres/*.sql
/var/backups/sysreport.txt
/var/backups/usr-local/*.tar
/var/backups/var-www/*.tar {
  daily
  missingok
  # Rotate 7+1 days
  rotate 8
  compress
  # The most recent backup remains uncompressed so that it can
  # be transferred to an off-site backup location, and further
  # processed by a tool that supports de-duplication (e.g. bup).
  # De-duplication works best on uncompressed data.
  delaycompress
  # These are backup files, not log files, so it wouldn't make
  # sense to re-created the original file after it was rotated
  nocreate
  # Rename the oldest rotation so that it can enter the weekly
  # rotation stage
  postrotate
    OLDEST_ROTATION_FILE="$1.8.gz"
    NEXT_STAGE_FILE="$1.weekly.gz"
    if test -f "$OLDEST_ROTATION_FILE"; then
      mv "$OLDEST_ROTATION_FILE" "$NEXT_STAGE_FILE"
    fi
  endscript
}

# ----------------------------------------
# Weekly rotation stage
# ----------------------------------------
/var/backups/debconfsel.txt.weekly.gz
/var/backups/git/*.tar.weekly.gz
/var/backups/hardware.txt.weekly.gz
/var/backups/home-directories/*.tar.weekly.gz
/var/backups/ldap/*.ldif.weekly.gz
/var/backups/mysql/sqldump/*.sql.weekly.gz
/var/backups/postgres/*.sql.weekly.gz
/var/backups/sysreport.txt.weekly.gz
/var/backups/usr-local/*.tar.weekly.gz
/var/backups/var-www/*.tar.weekly.gz {
  weekly
  missingok
  # Rotate 4+1 weeks
  rotate 5
  # The file is already compressed
  nocompress
  # Rotated files keep the compression extension .gz. This results
  # in "foo.weekly.1.gz" instead of the default "foo.weekly.gz.1".
  extension .gz
  nocreate
  # Rename the oldest rotation so that it can enter the monthly
  # rotation stage
  postrotate
    #     $1 = foo.weekly.gz
    # Oldest = fook.weekly.5.gz
    #   Next = foo.monthly.gz
    OLDEST_ROTATION_FILE="$(echo "$1" | sed -e 's/\.gz$/.5.gz/')"
    NEXT_STAGE_FILE="$(echo "$1" | sed -e 's/\.weekly\.gz$/.monthly.gz/')"
    if test -f "$OLDEST_ROTATION_FILE"; then
      mv "$OLDEST_ROTATION_FILE" "$NEXT_STAGE_FILE"
    fi
  endscript
}

# ----------------------------------------
# Monthly rotation stage
# ----------------------------------------
/var/backups/debconfsel.txt.monthly.gz
/var/backups/git/*.tar.monthly.gz
/var/backups/hardware.txt.monthly.gz
/var/backups/home-directories/*.tar.monthly.gz
/var/backups/ldap/*.ldif.monthly.gz
/var/backups/mysql/sqldump/*.sql.monthly.gz
/var/backups/postgres/*.sql.monthly.gz
/var/backups/sysreport.txt.monthly.gz
/var/backups/usr-local/*.tar.monthly.gz
/var/backups/var-www/*.tar.monthly.gz {
  monthly
  missingok
  # Rotate 12 months
  rotate 12
  nocompress
  extension .gz
  nocreate
  # No further rotation stages
}


Here is the configuration to rotate the log files of all Apache virtual hosts:

root@pelargir:/etc/logrotate.d# cat pelargir-apache-vhost 
# This configuration closely resembles the one that the apache
# Debian package provides in /etc/logrotate.d/apache2.
#
# A notable difference is that we don't need a prerotate
# directive here, we rely on the fact that such a directive
# already exists in /etc/logrotate.d/apache2
# (pre-rotation triggers awstats log file evaluation)
#
# Additional notes:
# - The permissions specified by the "create" directive
#   (see below) normally would prevent the awstats update
#   script from accessing the log files, because the update
#   script runs as user www-data. To circumvent that problem
#   user www-data has been added to group adm. See the
#   awstats wiki page for a discussion of other solutions,
#   and why this solution is probably the best.

/var/log/apache2/*/*.log {
  weekly
  missingok
  rotate 52
  compress
  delaycompress
  notifempty
  create 640 root adm
  sharedscripts
  postrotate
          if /etc/init.d/apache2 status > /dev/null ; then \
              /etc/init.d/apache2 reload > /dev/null; \
          fi;
  endscript
}


Here is the configuration to rotate the log files created by rsyslog for named services (see this wiki page for details):

root@pelargir:/etc/logrotate.d# cat pelargir-rsyslog-services
# This configuration closely resembles the one that the rsyslog
# Debian package provides in /etc/logrotate.d/rsyslog.

/var/log/gitolite/gitolite.log
/var/log/imapd/imapd.log
/var/log/slapd/slapd.log {
  size 1000k
  missingok
  rotate 10
  compress
  delaycompress
  notifempty
  nocreate
  postrotate
    invoke-rc.d rsyslog rotate > /dev/null
  endscript
}


Manage SysV services

An interactive front-end to enable/disable SysV services on the terminal is

sysv-rc-conf

A command line utility to do the same, but with more control over when exactly a service is started or stopped (i.e. the sequence number can be specified):

update-rc.d

The latter is especially useful if the rc script contains an LSB-style header. In this case, it should be sufficient to say

update-rc.d <myservice> defaults


Partitioning and file system management

Partitioning and formatting

There are two commonly used partitioning utilities:

  • fdisk: Is used if the partition table format is MBR. Does not support the partition table format GPT.
  • parted: Is used if the partition table format is GPT (possibly MBR, too).

For the MacMini I had to use parted.


Both utilities work interactively. Important: Specify the device that represents the entire hard disk drive.

fdisk /dev/sda
parted /dev/sda


Once partitions have been created, they need to be formatted. The following example formats the first partition on /dev/hda using the ext3 file system:

mke2fs -j /dev/hda1


File system management

Parameters of an existing ext[234] file system can be changed. The file system must be unmounted for this to work. Use the following command:

tune2fs

For instance, to optimize lookups in large directories with many files:

tune2fs -O dir_index /dev/hda7
e2fsck -D /dev/hda7

(the second command changes existing directories to use the hashed b-tree that were enabled by tune2fs)


File system layout

Mount point Size In use Remark
/ 2 GB 250 MB The current size seems to be grossly oversized, but on the other hand we really want to be on the safe side.
/tmp 4 GB 150 MB Exists so that an out-of-control program does not crash the system by filling up the root file system.
/usr 20 GB 2.7 GB Contains the main system installation, including /usr/local. Very stable, not much fluctuation.
/home 20 GB 1.4 GB Contains the user home directories (excluding the directory of the super user). At the moment does not contain much data except mail.
/var 20 GB 2.5 GB Contains data and all kinds of stuff that is variable and volatile. Things that use up a lot of space have been "outsourced" to their own file systems.
/var/backups 100 GB 194 MB Has been separated from /var so that a rampaging backup script does not crash the system by filling up /var.
/var/cache 50 GB 14 GB Mainly exists as a separate file system because BitTorrent files require a lot of space.
/var/samba/daten 100 GB 7.0 GB File server #1, contains files that are not media.
/var/samba/media 450 GB 377 GB File server #2, contains media files that require a lot of space (e.g. music, images, movies).
swap 4 GB - Ditched the old rule "swap = double the RAM size" - same as RAM size should be sufficient.


Mount file systems based on UUID

Traditionally file systems in /etc/fstab are referenced using a device node under /dev. Example:

/dev/sda5       /mnt/foo     ext4    defaults     0       2

This becomes a problem if the file system is located on a removable medium such as an USB stick or an external USB hard disk drive. If the medium is disconnected, then reconnected at a later time, the kernel will not assign the same device node to the file system if in the meantime another device has been connected to the host.

The solution to this is to reference file systems using their UUID. Type the following command to get a list of all devices that are currently connected to the host, including their UUIDs:

blkid

Note that if blkid is run without root privileges it will merely print cached information. If blkid prints nothing at all the cache is still empty and you'll need to run the command once with root privileges (e.g. using sudo).

Once you have obtained a file system's UUID you can rewrite the file system's entry in /etc/fstab like this:

UUID=ca8f1d49-429f-4f12-a587-79e299cb4f67       /mnt/foo     ext4    defaults     0       2

Important: A file system's UUID is randomly created when the file system is created, so if you reformat a partition for any reason the partition will get a new UUID and any /etc/fstab references to the old UUID will have to be updated.


Information on /etc/fstab

Lines in /etc/fstab have the following format:

/dev/sda5       /mnt/foo     ext4    defaults     0       2

Discussion:

  • Column 4 lists mount options. The string "defaults" says that the default mount options for the file system type apply.
  • Column 5 indicates whether the dump command needs to dump the file system. Value 0 = don't dump.
  • Column 6 is a numerical value that indicates the order in which order file systems have to be checked when the system boots. Value 0 = don't check. Value 1 = check file system (used for root file system). Value 2 = check file system (used for non-root file systems); file systems on different drives are checked in parallel.


Mount options (cf. man mount):

  • user = Non-root users can mount the file system. If specified this implies noexec, nosuid, and nodev (unless overridden by subsequent options).
  • exec = Programs stored on that file system can be executed
  • suid = Allow set-user-identifier or set-group-identifier bits to take effect
  • rw = Mount file system read-write
  • dev = Interpret character or block special devices on the filesystem
  • async = I/O to the filesystem should be done asynchronously
  • auto / noauto = File system is mounted / not mounted when mount -a is used
  • defaults = rw,suid,dev,exec,auto,nouser,async


Working with btrfs

If it is not already loaded, load the btrfs kernel module

modprobe btrfs

Install the user-space utilities that allow us to create or otherwise handle a btrfs file system

btrfs-tools

Create the filesystem

mkfs.btrfs /dev/sda1


cron

The Debian package for running the cron service is, unsurprisingly,

cron


The cron daemon looks for things to execute in the following places

/var/spool/cron/crontabs 
The traditional user-based crontab files which can be managed using the crontab command line utility
/etc/crontab 
A single file that lists commands to execute by user. The format of the file is similar to the traditional crontab format, with the addition of a "user" field that indicates the username that each command should be executed as. The cron daemon monitors /etc/crontab for changes.
/etc/cron.d 
An entire directory that may contain many files, in the same format as /etc/crontab. The cron daemon monitors /etc/cron.d for changes. The man page discourages using this directory.


It gets interesting if we look at the default content of /etc/crontab:

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user	command
17 *	* * *	root    cd / && run-parts --report /etc/cron.hourly
25 6	* * *	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6	* * 7	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6	1 * *	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

This illustrates that there are a number of directories where we can drop scripts that are then run at specific times, either hourly, daily, weekly or monthly.


Init scripts

An init script is a script that should be executed during system boot, or when a switch to a different run level occurs. Init scripts must be placed into the folder

/etc/init.d

The classic way to enable an init script is to create a symlink in one of the runlevel specific folders (e.g. /etc/rc2.d). The number used in the symlink name determined the order in which scripts where executed. Debian used to create these symlinks with the command update-rc.d, however since Debian Squeeze the new command to enable a script is

insserv foobar

insserv parses the LSB headers of an init script and then calculates the proper start sequence by resolving the dependency list found in all init scripts. The symlinks in the runlevel specific folders are then (re)created as necessary. For this to work, init scripts must be LSB compliant and contain LSB headers so that they can specify their dependencies.

For details see

  • This status page with details about the initiative to migrate Debian to dependency-based booting
  • How to LSBize an Init Script
  • /etc/init.d/skeleton which might serve as a template for a completely new init script (the template contains useful Debian-specific stuff for starting/stopping a daemon process)


This is how the LSB header of my script /etc/init.d/pelargir looks like:

#! /bin/bash

### BEGIN INIT INFO
# Provides:          pelargir
# Required-Start:    $network
# Required-Stop:     $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Host specific small stuff
# Description:       Currently this script only manages a few (important!)
#                    iptables rules.
### END INIT INFO

# Notes
# - See http://wiki.debian.org/LSBInitScripts for a primer on LSB compliance
# - "Provides" is set to the script name, as recommended in the primer
# - Currently this script only manages iptables rules, and iptables is in /sbin,
#   so there is no need to depend on either $local_fs or $remote_fs

[...]