Crystal Runs Your Super Threaded Archive of Lists

Crystal is a web based mailing list archive. It is still under development, and some interesting features (including some of those that will make it different from other archives ...) are still missing. But it should run stable already, so you might as well try it now. :-)

Highlights

Archive incoming mails and make them accessible by date, author, subject, thread, topic, etc.
Do not require the archive to be updated regularly by a special command. Instead, each mail is visible as soon as it's received.
Make attachments accessible (supports MIME, also nested, and uuencode).
Search for text, subject, message id and topic.
Allow authenticated and anonymous access (as configured by the administrator).
Let users send mails to the list via WWW (optionally anonymously).
Provide user administration where users can change, add and remove email addresses and change their passwords and other settings via WWW (optionally).
Can be configured either via config files or in the source. Config files support variables and includes to simplify administration of multiple archives.

Crystal is free software (GNU GPL). Crystal's WWW home page is http://fjf.gnu.de/crystal/.

If you would like to see crystal in action before trying it, you can have a look at the GNU Pascal mailing list archive at http://www.gnu-pascal.de/crystal/gpc/en/.

Update notes

Previous versions supported a UsersAllowed setting. However, it turned out that this was not good enough (with some tricks an attacker could circumvent it). Therefore, now it is suggested to create a group containing all user ids that should be able to run crystal, including the web server, the mailer daemon and human administrators. Since by default crystal is now installed group-, but not world-executable, an attacker who does not belong to this group cannot do anything with crystal. When building, you can specify the group as CRYSTAL_GROUP. The default is crystal. The setting UsersAllowed does not exist anymore, and crystal will complain if it is found in your config file.

With crystal 0.996, you should rebuild your databases since there have been several changes in the internal table format:

crystal --action log-crystal

Requirements

To compile crystal you need:
- GNU make, GCC and flex. These programs are pre-installed on many modern systems.
- A quite recent version of GPC (see the crystal source on exactly which GPC version is required).
- Some GPC units that crystal needs (e.g., for MySQL, email, HTTP, CGI) are included with the crystal sources.
- A libc with a crypt() function to encrypt passwords. If you want to password protect crystal (and possibly other web pages), this function must be compatible with the encryption used by your web server. (Apache uses the standard Unix crypt() function for AuthUserFile, so if you use Apache, the encryption is compatible.) Most libc's have such a function. If yours doesn't (e.g., due to legal restrictions), try to get a replacement library (possibly cryptographically weakened). Don't ask me for one.
To use crystal you need:
- Apache as your web server. Basic familiarity with Apache configuration is expected – learn from the Apache docs and examples otherwise. Other web servers that support the CGI protocol should work as well, possibly with small changes to crystal, but this hasn't been tested.
- A MySQL database server. No special knowledge about MySQL is required (you only have to install the server, and set up a MySQL user which is briefly described below). Other SQL databases can probably be supported as well, but might require more than trivial changes to crystal because some queries might use MySQL specific features. (I didn't check this in each case.)
- Sendmail, Exim, or qmail as your mail server. Most systems have one of them installed or readily installable, so you'll rarely need to download one. Besides setting up mail aliases for crystal and (optionally) the list, no further knowledge about them is required. Other mail servers can be supported as well, probably with none or very small changes to crystal. (Sendmail and Exim work the same as far as crystal is concerned, and the only difference between the two and qmail that's relevant to crystal is that qmail gives the envelope and return path in environment variables UFLINE and RPLINE, while the others send them at the beginning of the mail.)
- bzip2, gzip, zip, uudecode, file (standard on most systems)
- convert (part of ImageMagick)
- antiword; version 0.33 or higher is required, with data files installed system-wide (i.e., in /usr/share/antiword/, so that antiword will find them)
- mpack

Installing crystal

Not discouraged yet? Good! :-)
Download the crystal sources and extract them, of course.
Configure crystal. That's not automatic, unlike most other programs, since you have to provide some data about your (planned) installation:
- Copy the config file crystal-skel.conf to crystal.conf in the directory where you will install crystal and fill in the settings as described in the file. Please check all settings and adjust them when necessary. Some have reasonable defaults, others must be filled in by you and crystal will not run without them.
- Alternatively, you can also configure crystal in the sources. This is a little more work than changing a config file, but it is a little faster (since it does not have to parse the config file on each invocation) which might be relevant on a busy server, and it allows for more flexibility (there are some hooks that can be filled in with your own code doing arbitrary things that the config file doesn't support).
  To do this, copy the file crystal-<LANG>.pas (where LANG is the language you want) to a new source file, remove the call to ReadConfigFile, and insert your wanted configuration settings. See crystal-main.inc for descriptions. (Most of them are the same as those in the config file.) Note: Since the configuration includes the MySQL password, you must keep it secret, i.e. in the first case the file crystal.conf, in the second case the new source file and the executable, must not be world-readable.
Compile crystal:
- English version, using a config file:
```
make
```
- German version, using a config file:
```
make de
```
- Using source configuration:
```
make custom CUSTOM_SRC=<your-source-file>.pas
```
  You can override the executable name with EXENAME=<exename>.
Copy the executable (crystal-<LANG>) to its directory. It is made setuid by default. So you should not have compiled crystal as root (you should never compile any foreign program as root!). Install it in <BaseDir> such that it's executable by Apache, but preferably setuid to another user to which the mailbox and data files generated by crystal will belong (either your regular user ID or, for increased security, a special user ID just for crystal).
Note: If you want to run several crystal installations on one server (for different mailing lists), you have to use a separate directory for each of them. If you use config files, you can use the same crystal executables for all of them. If you use source configuration, however, you will have to compile a (slightly differently configured) executable for each of them. You can, however, run different crystal executables that use the same mail data (e.g., different language versions) in the same directory.
Make sure that your MySQL server was configured with
```
--enable-local-infile
```
(if it's a recent version; otherwise it doesn't have this option and doesn't need it). Otherwise try adding the option
```
--local-infile=1
```
in the start script for mysqld, or re-build MySQL with
```
--enable-local-infile
```
. If you fail to do so, you will note the problem when you get an error like this on standard error when importing a mailbox or in the log file when handling mail:
```
ERROR: DBHCommand: The used command is not allowed with this MySQL version (LOAD DATA LOCAL INFILE ...)
```
Set up your MySQL server (if not done yet), and create a user for crystal, e.g. by entering the following command in the mysql client as the MySQL administrator (fill in the settings as in your crystal configuration):
```
GRANT ALL ON <DBDatabase>.* TO <DBUser>@<DBHost> IDENTIFIED BY "<DBPassword>";
```
The database and tables will be created automatically by crystal during initialization.
Create the mail alias for crystal (cf. MailingListCrystalAddress).
- Sendmail, Exim:
  In an aliases file, e.g. /etc/aliases, add something like:
```
# crystal mailing list archive
    <alias>: "|<full-path-of-the-crystal-executable> --import-sendmail"
```
  Then run newaliases (if needed).
- qmail:
  Create a config file .qmail-<alias> in the appropriate directory containing something like:
```
|<full-path-of-the-crystal-executable> --import-qmail
```
Initialize crystal:
```
./crystal-<LANG> --init
```
- To import an existing mailbox do:
```
./crystal-<LANG> --import-mailbox=<file>
```
  (Note: There are some scripts included with crystal, in particular mailbox-fix that might help you fixing some minor problems in mailbox files.)
- If you have used a previous version of crystal and the internal data format has changed (so you cannot just continue using its data files and database), you can do:
```
./crystal-<LANG> --action=<old-log-file>
```
Configure Apache so that a whole directory (in URL space) will be served by crystal. E.g., add something like the following in the appropriate place of your Apache config file(s), and reload or restart or kill -HUP Apache.
```
ScriptAlias <base-url-dir> <full-path-of-the-crystal-executable>
```
If you want to require authorization, put the following into a section specific to the crystal file, directory or location. (Note that crystal generates and updates the files users and group that can be used for Apache authorization.)
```
Options None
  AuthType Basic
  AuthName <textual group description>
  AuthUserFile <crystal-dir>/users
  AuthGroupFile <crystal-dir>/group
  Require group <GroupName>
```
The file mailinglist, generated by crystal can be used as the distribution list for the mailing list (if all subscribers of the list will be administered through crystal). You can install it, e.g., as a forward file (e.g., .forward) or via an aliases file (Sendmail, Exim) or as .qmail-<list-alias> (qmail) in the appropriate respective directories. Don't copy or hardlink it because crystal will update and replace it when necessary. Make a symlink, or use PostProcessUpdateUsers in your crystal configuration.
This step is optional. You can also administer your list by other software (e.g., majordomo). Then the user data settings (if any) WRT the mailing list made within crystal are ignored. (Or you can hack something up, based on PostProcessUpdateUsers, to transfer changes made in crystal to your other mailing list software, however you like ...)
Try to access crystal via WWW, add some users if you like, try sending mails to the list, etc. Check if everything works.
Enjoy!
Report any bugs and suggestions to me (frank@pascal.gnu.de).

To do

Draw a nice logo (if you can draw one, please send it to me :-)
Let users choose which of their adresses to use for their replies via crystal
More advanced searching
Recognize replies in the text and provide links to them
Recognize topics through replies
Let users set topics via WWW
Show the different writers by different colors
Translate the messages in the HTML output into more languages besides English and German
Write more and better documentation
Write a (semi-)automatic install program

Acknowledgements

G-N-U GmbH who supported several enhancements to crystal.
Eike Lange who wrote the MySQL unit used by crystal
Peter Gerwinski who wrote parts of the CGI unit used by crystal helped inventing the name CRYSTAL and contributed some ideas and code to crystal
Nicola Girardi and Jan Andres who contributed ideas to the EMail unit used by crystal
For ideas regarding the design and features of crystal and bug reports:
- Anja Gerwinski
- Benedikt Wildenhain
- Simon Orlob
- Tim Eisenhardt