ISFDBPersonalMac

About

This page provides instructions on how to install a local copy of the ISFDB database and software on a Mac. The instructions were successfully tested on Mac OS X 10.10, but there is no reason to assume that they won't work on newer Mac OS X versions as well. Older Mac OS X versions may be a problem, though - for instance, if a pre-2.4 version of Apache is installed on the system, the Apache configuration snippets will not work.

Instructions for Mac OS X 10.5 are available if you view an older version of this page in the page history.

Required software

Summary

In a nutshell, the following software is required to run an instance of ISFDB:

Python: The ISFDB software is written in this language, therefore a Python interpreter is required.
Apache web server: The ISFDB software expects to be run in a web environment. Apache is the best choice for this on Mac OS X.
MySQL database: The ISFDB data is stored in this database backend.
MySQLdb: The ISFDB software accesses the database via this specific (and no other) Python module.

In addition to the above, a couple of indirect dependencies also need to be installed:

Apple's Developer Tools (Xcode): This is required because there is no ready-made, pre-compiled version of MySQLdb available for Mac OS X, so you need to compile it yourself
The CVS version control software is required to download the ISFDB source code
The Homebrew package manager: Some of the software packages required by the ISFDB are already present on Mac OS X, but others need to be manually installed. Homebrew is a modern and easy-to-use package manager that lets you install the missing software packages with a few simple commands.

Why not MAMP?

MAMP is a nicely integrated solution that brings together almost all the major components required for running an instance of ISFDB: Apache, MySQL and Python. What's missing is the Python module MySQLdb, and unfortunately I have not found an easy way to add that module to MAMP.

Which user?

To perform the installation steps described in the following sections, log in with an administrator account.

Developer tools

The main suite of Apple's developer tools are bundled under the name "Xcode". I recommend downloading and installing Xcode via the Mac OS X App Store because this really is the simplest and most convenient option. As an alternative you can also download Xcode from Apple's Developer Connection, but you still need to register an account.

Once you have Xcode on the system, launch it and install the command line tools:

Open the Preferences dialog
Go to "Downloads > Components"
Select the entry "Command Line Tools"
Click the install button.

Homebrew

The procedure for installing Homebrew is dead simple, just follow the instructions on the Homebrew website. If necessary more documentation can be found here.

Python

You don't have to install Python, you can use the Python version that comes preinstalled with Mac OS X. On Mac OS X 10.10.5 this is Python 2.7.10.

MySQL

Install the MySQL database server and client packages like this:

brew install mysql

MySQLdb

Download the latest version of MySQLdb from here, then unpack, build and install with these commands (change the tarball and directory name if you have a newer version of MySQLDb):

tar xfvz MySQL-python-1.2.3.tar.gz
cd MySQL-python-1.2.3
python setup.py install

The last command produces a flurry of output, but one of the last few lines should look something like this:

Installed /Library/Python/2.7/site-packages/MySQL_python-1.2.3-py2.7-macosx-10.10-intel.egg

Apache

You don't have to install the Apache web server, you can use Apache that comes preinstalled with Mac OS X. On Mac OS X 10.10.5 this is Apache 2.4.16.

CVS

Older versions of Mac OS X had CVS preinstalled, but this is no longer the case. Xcode 4 and older also included CVS, but Apple's relentless drive for modernization has eliminated that as well. So now, Homebrew to the rescue - install CVS like this:

brew install cvs

Configure the servers

Which user?

To perform the configuration steps described in the following sections, log in with an administrator account.

MySQL

The MySQL server process is not yet running at this point, but in order to perform the next configuration steps you have to start it. You have two choices, but I recommend the second one (i.e. run the server in the background):

Manually run the server using this command: mysql.server start
Run the server in the background now, and also restart it again when you login the next time: brew services start mysql

MySQL is now running, but without a password for the root database user. You should run the following command to set a root password, and to also remove other security risks. Just follow the on-screen instructions.

mysql_secure_installation

From now on you can connect to the database with this command, you will then be prompted for the password for the user "root" that you have just set:

mysql -uroot -p

Apache

Edit this file in your favorite text editor

/etc/apache2/httpd.conf

Search for the following three lines and make sure that they are uncommented, i.e. that they do not have a hash character (#) at the beginning of the line:

LoadModule userdir_module libexec/apache2/mod_userdir.so
Include /private/etc/apache2/extra/httpd-userdir.conf
LoadModule cgi_module libexec/apache2/mod_cgi.so

Add the following line at the bottom of the file

Include users/*.conf

Save the changes. Now load the following file in the text editor. Replace "USERNAME" with the name of the user that should run the ISFDB website:

/etc/apache2/users/USERNAME.conf

If the file does not exist, create it. Make sure that the file contains the following lines. Again, replace "USERNAME" with the appropriate user name.

<Directory "/Users/USERNAME/Sites/">
    Options Indexes MultiViews
    AllowOverride None
    Require all granted
</Directory>
<Directory "/Users/USERNAME/Sites/isfdb/cgi-bin/">
    Options +ExecCGI
    AddHandler cgi-script .cgi
</Directory>

Save the changes. If this is the first time that you are doing this, you are done (because you will start the Apache server in the next section). If not then you probably want to restart the Apache server now so that it picks up your changes:

sudo apachectrl restart

Run the servers

Which user?

To perform the server launch steps described in the following sections, log in with an administrator account.

Apache

Launch Apache with this command

sudo launchctl load -w /System/Library/LaunchDaemons/org.apache.httpd.plist

Note: From now on, the Apache web server is automatically started each time the system is rebooted.

If you point your browser to http://localhost you should now see the message "It works!".

MySQL

If you followed the configuration instructions further up, then MySQL should already be running at this time. If not then execute the following command now:

brew services start mysql

Note: From now on, the MySQL server is automatically started each time the system is rebooted.

Obtain the ISFDB files

Which user?

To perform the download steps described in the following sections, log in with the account that will host the local ISFDB website.

All-in-one download script

The ISFDB consists of three sets of files:

The Python source code
The most recent database backup
The most recent backup of all cover scans

The 100% easiest way to get all these files is with a download script that you can get from this ISFDB wiki page. Copy the script from that page, paste it into your favourite text editor and save the script somewhere on your machine. For instance:

~/Downloads/isfdb_download.sh

Make the script executable:

chmod +x ~/Downloads/isfdb_download.sh

Run the script:

~/Downloads/isfdb_download.sh ~/Downloads/isfdb

When you get a password prompt, just press ENTER. The longest part is until all the cover scans have been downloaded. The good news is that you can run the script repeatedly and it will download only those things that have changed since the last time.

Manually download data files

If you don't want to use the convenient download script from the previous section, then you can find weekly database and cover scan backups on the ISFDB wiki. Download and uncompress the latest files.

Manually get the source code

If you don't want to use the convenient download script from the previous section, you can download the source code yourself. On this ISFDB wiki page you can see which version of the software the ISFDB website is currently running. However, as far as I know there are no official packages to download for any specific software version, so you simply have to download a snapshot of the current development tree from here:

http://isfdb.cvs.sourceforge.net/viewvc/isfdb/

Select "Download GNU tarball", then after the download is complete unpack the tarball like this:

mkdir -p ~/Downloads/isfdb/sources
cd ~/Downloads/isfdb/sources
tar xfvz /path/to/isfdb.tar.gz

Populate the MySQL database

Which user?

To perform the steps described in the following section, log in with the account that will host the local ISFDB website.

Load the ISFDB data file into the database

Uncompress the database backup file that you downloaded in the previous section

unzip ~/Downloads/isfdb/database/BACKUP-FILE.zip;

Launch the MySQL console. The "-p" option causes MySQL to prompt for a password.

mysql -uroot -p

If you want to refresh your database from a more recent backup file, type the following at the MySQL command prompt. Warning: This will destroy anything you have previously entered into your local database!

drop database isfdb;

Type the following commands at the MySQL command prompt to load the previously downloaded database backup file.

create database isfdb;
use isfdb;
source ~/Downloads/isfdb/database/BACKUP-FILE;
quit

Note: The information in this section was taken from this ISFDB wiki page.

Install the ISFDB source files

Which user?

To perform the steps described in the following sections, log in with the account that will host the local ISFDB website.

Prepare web space

Create folders that will receive the ISFDB source files. The "Sites" folder is the default name on Mac OS X for the folder where a user can place his or her files that should be accessible via web browser.

mkdir -p ~/Sites/isfdb
mkdir -p ~/Sites/isfdb/cgi-bin

Limitations

The ISFDB software has the following limitations which we'll try to work around by editing/fixing various files:

The software is built on the assumption that it runs at the root level of the website, i.e. normally it's not possible to have the files below a user directory (e.g. http://localhost/~USERNAME/) or some other folder (e.g. http://localhost/isfdb/)
Various build scripts contain a hard-coded path for the Python interpreter (/usr/bin/python). If a different Python interpreter should be used (e.g. one that has been installed by Homebrew to a /usr/local/bin/python) then those scripts need to be manually edited to make the installation process work. Known scripts:
- biblio/local.mk
- edit/local.mk
- mod/local.mk
- rest/local.mk
Other scripts may also contain a hard-coded Python interpreter path. Currently known:
- scripts/create_user.py

Fix index.html

Edit

~/Downloads/isfdb/sources/isfdb2/css/index_stub

Replace all occurrences of HTMLHOST with HTMLLOC, e.g.

http://HTMLLOC/cgi-bin/index.cgi

Edit

~/Downloads/isfdb/sources/isfdb2/css/index_install.py

Replace all occurrences of HTMLHOST with HTMLLOC, e.g.

image = string.replace(image, 'HTMLLOC', HTMLLOC)

Fix CSS

Edit

~/Downloads/isfdb/sources/isfdb2/common/SQLparsing.py

On the line that references "biblio.css", replace HTMLHOST with HTMLLOC, e.g.

print '  @import url("http://%s/biblio.css");' % (HTMLLOC)

Do the same for the following files

~/Downloads/isfdb/sources/isfdb2/biblio/common.py
~/Downloads/isfdb/sources/isfdb2/edit/isfdblib.py
~/Downloads/isfdb/sources/isfdb2/mod/isfdblib.py

Fix favicon

Edit

~/Downloads/isfdb/sources/isfdb2/css/Makefile

Add the following new line to the list of targets

favicon.ico \

Note: Use the same type of indentation as the other lines, notably the first character on the line must be a TAB!

Edit

~/Downloads/isfdb/sources/isfdb2/biblio/common.py

On the line that references "favicon.ico", replace HTMLHOST with HTMLLOC, e.g.

print '<link rel="shortcut icon" href="http://%s/favicon.ico">' % (HTMLLOC)

Fix logo

Edit

~/Downloads/isfdb/sources/isfdb2/common/navbar.py

On the line that references "isfdb_logo.jpg", replace HTMLHOST with HTMLLOC, e.g.

print '<img src="http://%s/isfdb_logo.jpg" width="130px" alt="ISFDB logo" >' % (HTMLLOC)

Fix login

Edit

~/Downloads/isfdb/sources/isfdb2/common/login.py

Rewrite the line that references "submitlogin.cgi" so that it looks like this:

print '<form METHOD="POST" ACTION="http:/%s/submitlogin.cgi">' % (HTFAKE)

Prepare for installation

Edit ~/Downloads/isfdb/sources/isfdb2/common/localdefs.py as follows. Replace "USERNAME" with the name of the user that should run the ISFDB website, and "SECRET" with the password used to access the MySQL database:

HTMLLOC         = "localhost/~USERNAME/isfdb"
HTFAKE          = "/localhost/~USERNAME/isfdb/cgi-bin"
DBASEHOST       = "localhost"
HTMLHOST        = "localhost"
COOKIEHOST      = "localhost"
LOCALFILES      = "directory where files generated by the night job reside"
WIKILOC         = "www.isfdb.org/wiki"
USERNAME        = "root"
PASSWORD        = "SECRET"
DBASE           = "isfdb"
UNICODE         = "iso-8859-1"
DO_ANALYTICS    = 0

Then edit ~/Downloads/isfdb/sources/isfdb2/INSTALLDIRS:

INSTALL_CGI     = ~/Sites/isfdb/cgi-bin
INSTALL_HTML    = ~/Sites/isfdb

Make copies of the two files in a safe place. If you do a "cvs update" on your source folder later on, and someone has checked in new versions of those files, you'll need to merge in your settings before installing again.

Installation

Run these commands:

cd ~/Downloads/isfdb/sources/isfdb2
make -B install

Note: The option "-B" means "Unconditionally make all targets". It must be specified because without it the installation fails. Reason: The file biblio/.TARGETS is not created for some unknown reason; this then causes an error because biblio/local.mk wants to include biblio/.TARGETS.

Enjoy!

Enjoy the fruit of your labor and point your browser at this URL (replace "USERNAME" with the appropriate user name):

http://localhost/~USERNAME/isfdb

Enable ISFDB editing

The ISFDB relies on MediaWiki to register users. In the personal setup without the Wiki, this is not possible. A script is provided to update the appropriate tables to allow editing permissions. After everything is installed and you are able to view pages in the ISFDB, do the following

~/Downloads/isfdb/sources/isfdb2/scripts/create_user.py LoginName password

Note 1: You must specify a password because the login python script does not allow logins without a password.

Note 2: At the moment, logging in does not work with Firefox for some obscure, cookie-related reason. In Safari, the ISFDB code tries to set the login cookie and the browser accepts it, but in Firefox there is not even an attempt to set the cookie. Cookie-related code is in login.cgi, but sprinkling it with print statements has not brought enlightenment so far.

ISFDBPersonalMac

Contents

About

Required software

Summary

Why not MAMP?

Which user?

Developer tools

Homebrew

Python

MySQL

MySQLdb

Apache

CVS

Configure the servers

Which user?

MySQL

Apache

Run the servers

Which user?

Apache

MySQL

Obtain the ISFDB files

Which user?

All-in-one download script

Manually download data files

Manually get the source code

Populate the MySQL database

Which user?

Load the ISFDB data file into the database

Install the ISFDB source files

Which user?

Prepare web space

Limitations

Fix index.html

Fix CSS

Fix favicon

Fix logo

Fix login

Prepare for installation

Installation

Enjoy!

Enable ISFDB editing

Navigation menu

Search