Project Dune:InstallGuide

From PDune

"Project Dune" Installation Manual

Requirements

In order to install Project Dune, you will have to install other REQUIRED software first and confirm it is functioning correctly:

Java Development Kit 1.5, or later
Either Postgres 8.1.4 or later OR MySQL 5.0 or later
Apache Tomcat 5.0, or later

The following software is OPTIONAL, but if it is not installed, you can't use all the features of the project:

GraphViz, preferably the last version,
Either CVS or Subversion
A mail server. The suggestion is PostFix if you run Linux. You could also decide to run a POP3/IMAP server like "Courier", which comes along with Ubuntu. Windows users might like to try hMailServer or James.

Project Dune is downloadable from SourceForge. Make sure to always get the most recent distribution, it will have less bugs and more features. The main binary download package is dune-install-x.x.x.war. This should get your demo going in about 5 minutes (not including the time required for installing the database servers and depending services).

 The pdune-libs package is used to decrease the overall size of the download. The libraries do not change as much, but are substantial in size. You should always use the same or next lesser version of the
 pdune-libs package. Not every release of Project Dune releases a new pdune-libs package.

Installation

Installing third party software

All third party software described in the Requirements have good reference guides with better installation instructions than we could ever provide here, so if the instructions do not allow you to get any further, consult the website where the software was downloaded from.

Installing Java

Install Java 1.x first. The Windows installation comes with an installer program that allows you to change the installation path amongst other things. Installation of Java on Linux may in certain cases be more complex. On certain distributions of Linux for example, you may need to deal with the so-called "alternatives" configuration, to prefer the Sun installation over the gcj. If you search for the keyword update-alternatives, you should find out how to do that. Test the preferred java alternative on your system by running the command "java -version". It should show the Java output with "Sun" as the provider, version 1.5 or later.

Installing Graphviz

Graphviz is a library for generating diagrams on the fly using a simple ASCII file. It is only used for reports at the moment, so if you decide not to use the dependency diagram reports, you don't need to install this package. It can be downloaded from the [1] site. Linux users on Debian or Ubuntu should use apt to get the graphviz package.

Installing Apache Tomcat 5.0

It is time to install Tomcat 5.0 (or later). Get the version of Tomcat for your platform from [2] For windows users, this is a Windows installer and very easy to execute. For Linux, this is either provided by your Linux distribution or a .tar.gz file. Extract the archive to the desired location where you wish the application to reside. When the files have extracted, you should see a directory called "bin". Now, if you type "./startup.sh" (or double-click the "start.bat" file on Windows), Tomcat should automatically find your Java installation and startup.

 If the default port 8080 for the Tomcat installation is already in use, the startup will give errors. In this case, you should attempt to modify the listening port first, then attempt the restart.

You probably want to modify the listening port to a different port than 8080. This can be done through the "server.xml" file that you will find in the "conf" directory. This file is an XML file, which contains a tag called a "connector". This tag has a "port" attribute set to "8080" by default. That's where you change the port. For security reasons and possible image caching later on, many organizations prefer to put Apache in front of Tomcat and run Apache from port 80. This is making use of a mod_jk module, which connects to Tomcat on behalf of the user. There are plenty of resources on the Internet that describe how to set up a system to use this, just search for mod_jk and tomcat.

Installing Postgres 8.1.4

Install Postgres 8.1.4 or later. On Windows, this should be a simple executable installation file. On Linux, this may either be provided by your distribution (Debian, Ubuntu, SuSE, RedHat all have this), otherwise you have to download the .tar.gz from the Internet and install it from there. Follow the instructions from the postgres website for installation.

 A pre-requisite to use Postgres is that the user you'll use in the installation process to create the database and user can login through a standard TCP connection. You should attempt to connect with that
 user through -h localhost and confirm it's able to connect prior to starting up the dune installer.

Another issue on postgres is that it uses sequences that need to be manually reset just after dune was installed. This is not part of the automatic installation process (another option is to keep clicking "save" when the data save fails in the application, as this will cycle through the first already used primary keys). To reset the sequences, use the following pl/sql procedure.

 CREATE OR REPLACE FUNCTION resetsequences() RETURNS integer AS $$
 DECLARE sequences 	REFCURSOR;
 		seq_rec		TEXT;
 		table_rec	TEXT;
 		sql			TEXT;
 		number		integer;
 BEGIN
 	OPEN sequences for SELECT c.relname AS sequencename FROM pg_class c WHERE (c.relkind = 'S');
 	LOOP
 		FETCH sequences into seq_rec;
 		
 	   	if not found then
 			exit;
 	   	end if;
 
 		table_rec := substr( seq_rec, 5 );
 		
 		sql := 'select ( count( * ) + 1 ) FROM ' || table_rec;
 		EXECUTE sql INTO number;
 
 		sql := 'SELECT setval ( || seq_rec || , ' || number || ')';
 		execute sql;
 
 	END LOOP;
 	RETURN 0;
 END;
 $$ LANGUAGE plpgsql;

Save this file in your home directory as sequence-reset.sql. Then load it in the database as follows. The createlang utility is provided by postgres as a separate binary and is executed on the shell.

 gt@amsterdam # createlang plpgsql <db-name>
 gt@amsterdam # psql -d <db-name>
 <db-name>=# \i /home/gt/sequence-reset.sql
 CREATE FUNCTION
 <db-name>=# select resetsequences();
  resetsequences 
 ----------------
               0
 (1 row)
 
 <db-name>=#

Installing MySQL 5.0

On Windows, this should be a simple executable installation file. On Linux, this may either be provided by your distribution (Debian, Ubuntu, SuSE, RedHat all have this), otherwise you have to download the package from the Internet at the mysql.com site. Follow the instructions from the website for installation.

Installing Project Dune

Libraries

To reduce the size of each update and to reduce unnecessary load on download servers, we have provided the libraries as a separate download, as these do not change as often. The packaging process will request you to provide the location of this file, so that it can include the libraries. You should always use the version of the libraries that is equal to your dune binary package version or the next lesser version. There are two ways to install this. You can either upload it when the install step is ready for it, or you can drop the package on the server beforehand and then refer to it by absolute path. Of course, the library package must be readable by the Tomcat process to enable this.

 The default installation of Tomcat on Ubuntu and Debian use the SecurityManager of Java. Dune cannot yet supply the configuration required to install Dune on a JVM that is secured with a SecurityManager. 
 For that reason, we recommend you turn that feature off. Please verify with your system administrator if that is a recommended approach. There is a request ongoing to specify the required permissions for 
 the installation process, but it's unclear how this should be handled.

Copy the dune-install-x.x.x.war package to the webapps directory of Tomcat.
Start up Tomcat. Verify that the package deployed properly.
With any browser, go to the following url: http://<server-name>/dune-install-<x.x.x>, replacing x.x.x with the version number you downloaded.
Follow the steps in the wizard. Supply information where necessary. When you are at the final screen, leave that open in the browser.
As the screen indicates, copy the chosen .war files from the "bin" directory to your "webapps" directory.
Wait for Tomcat to auto-deploy (if configured) or restart Tomcat. Inspect the logs to make sure the modules are deployed properly.
Login with admin/admin. You'll need to create a user that has a role of project manager first. Then create a project, a release for that project and you can start creating issues.
Enjoy! Any questions, please post them on http://www.projectdune.net or see http://www.projectdune.org for the wiki.

If you are running postgres, you must grant the correct permissions on the objects. This is still a lot of work to achieve automatically in the installation process. After installation, you can use the following statements to quickly generate these for you. Run them through postgres, then deploy the war files.

 select 'GRANT SELECT, UPDATE, INSERT, DELETE ON ' || c.relname || ' TO <your-user>;' 
 from pg_class AS c LEFT JOIN pg_namespace n ON n.oid = c.relnamespace where c.relkind = 'r' 
 and n.nspname NOT IN('pg_catalog', 'pg_toast') and pg_table_is_visible(c.oid);
 
 select 'GRANT SELECT, UPDATE ON ' || c.relname || ' TO <your-user>;' 
 from pg_class AS c LEFT JOIN pg_namespace n ON n.oid = c.relnamespace where c.relkind = 'S' 
 and n.nspname NOT IN('pg_catalog', 'pg_toast') and pg_table_is_visible(c.oid);

Installing the daemon application

Project Dune regularly needs to send emails and execute other jobs. The jobs are scheduled and managed by the dune-daemon application in the bin directory in the dune files area. This application creates its own log file and shows the progress. The bin directory already contains a pre-configured batch file or shell script. This script is called daemon-run.sh on UNIX or daemon-run.bat on Windows. The jobs and job schedule are controlled in the DUNEJOB table in the database and comes preconfigured to send out email every five minutes. A description of each job is included in the DUNEJOB table. The settings for the schedule follow the same methodology as "cron" on UNIX. The daemon actually uses the quartz Java library, which should contain more detailed explanations if necessary.

The DuneDaemon application can be run with the option -single so that it will execute all jobs once and then immediately terminate. For regular usage, no parameter should be provided.

 <user>@<host>:/usr/local/pdune-<VERSION>/bin$ ./daemon-run.sh -single

Verifying the deployment of the war file

After the file dune.war was copied by the web installer to the webapps directory of your Tomcat installation, it may auto-deploy if Tomcat has been configured to do so. In other cases, you may have to deploy manually by running some commands. You can verify if Tomcat attempted to deploy by reading the catalina.out file in the logs directory of Tomcat. That file contains all messages related to correct deployment of the dune.war file. The final messages in the log should look more or less like this:

 <SOME-DATE> INFO  [LocaleFilter] Initialising LocaleFilter
 <SOME-DATE> org.apache.coyote.http11.Http11BaseProtocol start
 <SOME-DATE> INFO: Starting Coyote HTTP/1.1 on http-<PORT>

For a final test, attempt to access the application with the web browser. Use the URL: http://<host>:<port-number>/<instance-name>

The home page should come up and a login box should appear.

Default login

You may now log in with "admin/admin". Modify the password after this startup using the menu setting on the right. Congratulations!

Finalizing the installation

The system is now set up, but the temporary files directory fills up with temporary files in a couple of days. When people print reports, upload attachments or when the system sends email, it uses this temporary directory for conversions and storage of uploaded files.

On UNIX, you can set up a crontab to get rid of files of a certain age as follows:

/15 * * * * find <location-to-workspace>/temp -amin +15 -exec rm -f {} \;

This command for the crontab will remove all files that are 15 minutes old. Of course, you can determine your own settings and time-frame, as you desire. If you are a Windows administrator and have the instructions for doing the same thing on Windows, I'd be grateful if you send me the instructions.

Migration from Bugzilla or Mantis

Before attempting any migration, please review the migration guide that comes with the distribution for information on how the mappings are done from a stock distribution and how you can modify the migration process yourself to change these mappings. This item will only explain how to execute the migration process and assumes that you have already properly configured the migration process and mappings in the source files.

The migration process can be run through Java. Please note that actually running the start command does not securely start the process ( the username and password are readable from the command line ), so you probably only want to do this on an isolated machine if your process is going to run a while. You need to comply with the following pre-requisites:

Java is installed
You have read and understood the migration guide
Any required modifications/configurations to the source code for migration have been applied.
You have modified the database URL, username and password settings to contact the to-be-migrated database on the command line in the mig-run.sh script.

Then it is possible to run the migration process through Java as follows:

 <user>@<host>:/usr/local/pdune-<VERSION>/bin$ ./mig-run.sh

Integration with source code repositories

This chapter will highlight configurations and issues about integration with source code repositories. There are three sections at the moment, which detail the common configurations and issues that you should be aware of and then a detailed section for either CVS or Subversion.

Common issues

Project Dune offers integration with two source code control systems. The source code control system will invoke so-called "hook" scripts, which sends ChangeLogs from the code repository to the Project Dune server. The other way around, Project Dune uses its internal configuration to connect to the repository, but it will just read files for inspection purposes.

The hook scripts and the methods used to send the ChangeLogs may differ slightly per repository system. Subversion is already getting more mature and has interesting features and transactions that CVS does not offer. CVS is however around for longer and it is seen as necessary to integrate with it.

For whatever system that is in use, a script is in use that will verify the log message and search for an Issue ID. It will send this information to the Project Dune server, which will validate the issue exists, it is assigned to the user attempting to change those files for that purpose and that the issue is in the correct state.

The other script is invoked after the files have been changed. It will send the changed files and respective versions to the Project Dune server, which will store these changes in the database.

 If the webserver goes down after the pre-commit script, but before the post-commit script sends the changed files to the server, the registered changes will go lost, because the post-commit script will not
 abort the processing. However, the system will have generated an "inspection request", so an inspection will be ready. It will not have all the files associated in this case, so a manual reconciliation is 
 needed.

CVS integration

The access from the Project Dune server to the CVS repository is created by means of a local checkout, on the server of Project Dune, from the repository that you wish to integrate with. The username, password and url should be configured in the database (PROJECT table) so that this access is possible. The config parameters can be inspected from the Project list.

When checking out the remote project on the server, ensure that you check out from the root and not an individual module. This means a checkout line like:

cvs co :<protocol>:<user>@<hostname>:/cvsroot .

This way, all the relative paths can easily be resolved by the server, which otherwise becomes very confusing.

The integration with CVS occurs in two scripts: "verifymsg" and "loginfo". The verifymsg script is generally used to verify the log message and bug ID's for example. The loginfo script is generally used to provide mailings or external logs about modified files. Copy the files for CVS integration from the scripts/cvs directory into the CVSROOT directory of the repository on the CVS server, not the local repository checkout.

Then you will need to modify some scripts of the CVS installation, so that these scripts get called. Open up verifymsg and add a line at the end:

 DEFAULT $CVSROOT/CVSROOT/cvs-verify-msg.pl $USER %l

Similarly, for the loginfo script, add a line at the end as follows:

 ALL (echo ""; cat) | \
 $CVSROOT/CVSROOT/cvs-pre-commit.pl $CVSROOT %p $USER %{sVv}

Now, edit both the cvs-pre-commit.pl and cvs-verify-msg.pl script. There is a setting that starts with my $url =, marked conspicuously with remarks, which should point to the location of your Project Dune server. Verify that the scripts installed can be executed. Use the chmod command to adjust, if necessary.

As a final step, log into the database and verify that the settings for accessing the repository are correct. Manually check these settings from a shell to be sure. The class you will need to reference in the project screen (when creating a new project) is:

 net.sf.pdune.custom.repository.CVSRepository

The URL used in this configuration would be the same URL that the commandline client would use in order to access that repository.

 There may be some problems due to the version of CVS you are using. Some versions of CVS are still including comma's when outputting modified files and their versions. Other CVS versions may have also slight
 differences that make text parsing difficult on the server. In case you think you have these problems, contact us on the site Project Dune community site and log a suspected bug report.

Subversion integration

Integrating with subversion is slightly easier, because you do not need to checkout the project in order to communicate with the server. Make sure the URL that is configured in the database is correct (test it manually), then simply configure the SubversionRepository breakout in the create project screen and verify the settings for username & password in the project configuration to access the repository.

The subversion server must have two hooks installed. One to be invoked at pre-commit time, the other at post-commit. In order to install, log in to the server where the subversion repository is located, find the hooks directory inside the repository and copy the files from scripts/subversion into this directory. Ensure that the files you are copying do not have an extension. Also make sure that the access permissions for these scripts are correct, so that the subversion daemon (www-server?, svn server?) can execute these files.

Now, edit both scripts. There is a setting that starts with my $url =, marked conspicuously with remarks, which should point to the location of your Project Dune server. The class you will need to reference in the project screen (when creating a new project) is:

 net.sf.pdune.custom.repository.SubversionRepository