Fatih Mehmet Güler

Overview

This page contains step by step instructions to setup a development server with Subversion, Trac,
Jenkins. This is how I usually setup a development server on an Ubuntu Server. I hope you find it useful.

Please note that some parts might be aged, or may not suit to your needs. If you have a question or
a comment, you can always reach me at fatih[at]fmguler.com

And don't forget to change johndoe with your username ;)

1.Initial User Management

After OS setup, first add a new user, restrict sudoers to admin group, create admin group, and add this user to admin group in order to make it a sudoer, finally disable root;

adduser johndoe
vi /etc/pam.d/common-password //can set min=8 for security
visudo
%admin  ALL=(ALL) ALL //add this line under user privilidge spec.
groupadd admin
adduser johndoe admin //add the new user to admin group which will make it sudoer
//login as johndoe
sudo passwd -l root //disable root account
sudo dpkg-reconfigure tzdata //set time zone to your place

Next, restrict ssh access to sshlogin group and add this new user to that group;

//restrict ssh login to sshlogin group users
sudo vi /etc/ssh/sshd_config
//add these to end
#restrict ssh access
AllowGroups sshlogin
//add user johndoe to sshlogin group so that we can login with ssh
sudo groupadd sshlogin
sudo adduser johndoe sshlogin
sudo /etc/init.d/ssh restart

Then add a restricted user: devuser which will run all restricted applications. Note that this user cannot connect with ssh, and owns all the application folders. If you need to modify the files within /development you need to change user to this (sudo su devuser).

sudo adduser devuser

REMINDER: You can install applications or do root privilidged operations with the sudoer user (johndoe or another sudoer). You can modify files in /development with the restricted user (devuser).

2. Install Dropbox

Install Dropbox, with the restricted user; (http://wiki.dropbox.com/TipsAndTricks/TextBasedLinuxInstall)

//login as devuser or change user to devuser (sudo su devuser)
cd /home/devuser
wget -O dropbox.tar.gz http://www.dropbox.com/download\?plat=lnx.x86 
tar -xzvf dropbox.tar.gz 
cd .dropbox-dist/ 
./dropboxd

Exit dropboxd. Create the /development folder layout (link to dropbox folders);

//as a sudoer (e.g. johndoe)
cd /
sudo mkdir development
sudo chown devuser:devuser development/
sudo su devuser
cd development
//link dropbox folders to /development counterparts
ln -s /home/devuser/Dropbox/somefolder/bin/ bin 
ln -s /home/devuser/Dropbox/somefolder/backup/ backup 
ln -s /home/devuser/Dropbox/somefolder/meta/ meta

Disable dropbox lan sync. feature to prevent listen port 17500;

cd /development/bin
chmod +x *.py
chmod +x *.sh
./dropbox.py stop
./dropboxp2p.py -d
./dropbox.py start

NOTE: You can find dropbox.py here.

3. About Folder Layout

The folder layout of the development infrastructure is like below;

/development: root folder
/development/bin: binary folder, contains start and backup scripts.
/development/backup: backup folder, where the backups are taken to.
/development/meta: configuration file templates and related infrastructure files.
/development/svn: subversion root folder, contains svn repos.
/development/trac: trac installation folder, contains trac binary and trac projects under scripts folder.
/development/hudson: hudson root folder, contains hudson jobs and binaries.
/development/jetty: jetty root folder, contains java web applications.

4. Installations

The list of programs and services utilized;

dropbox
subversion
trac
hudson
openvpn
nginx

Below are the setup process for each of them;

dropbox: already defined above. It is used for backup and central storage of scripts and configuration templates.

subversion: install subversion package with apt-get;
```
sudo apt-get install subversion
sudo su devuser
cd /development
mkdir svn
```
Then copy subversion repositories to this svn folder from backup (as devuser user). Later, test subversion by starting the svn start script: /development/bin/start-svn.sh. You may need to change listen address from this script to 127.0.0.1 since openvpn is not installed yet. Forward 3690 port with putty. Test with repo browser. After testing close it, because we will start this process by another start script.

trac: to install trac follow these steps;
```
sudo apt-get install python-setuptools
sudo apt-get install python-subversion
sudo apt-get install language-pack-tr
sudo easy_install Trac==0.12
sudo easy_install http://trac-hacks.org/svn/tracwysiwygplugin/0.12
sudo su devuser
cd /development
mkdir trac
```
Then copy trac installations to this trac folder from backup (as devuser user). Later, test trac by starting the trac start script: /development/bin/start-trac.sh. Forward 8000 port with putty. Test with browser. After testing close it, because we will start this process by another start script. While testing if it says this, do it so: The Trac Environment needs to be upgraded. Run "trac-admin /development/trac/YourProject upgrade"

install account manager plugin : account manager plugin makes it easy to manage user accounts. To install it do;

sudo easy_install http://trac-hacks.org/svn/accountmanagerplugin/trunk

Then enable these modules from trac admin page: AccountManagerAdminPage , AccountManager , HtDigestStore , HtDigestHashMethod , AccountModule , LoginModule

Disable LoginModule from trac components. From Accounts>Configuration menu make filename: /development/trac/users.htdigest and realm: trac then save. trac-start.sh shouldn't have the --auth part. Restart trac.

trac post commit hook : used to give auto reference from post messages to trac tickets;

cp post-commit.tmpl post-commit 
chmod 755 post-commit 
vi post-commit 
/usr/local/bin/trac-admin /development/trac/xyz changeset added "$1" "$2"
//if you migrate from windows you shoud do in vim;
:set ff=unix

Then enable tracopt.ticket.commit_updater.* under trac components from admin page.

trac notifications & email : to enable mail sending from trac, for notifications and password resetting, change smtp address to 127.0.0.1 from trac.ini and do;

sudo apt-get install postfix
sudo dpkg-reconfigure postfix
//select internet site from the list
//now can test mailing with;
sudo apt-get install bsd-mailx
echo test | mail -s "test mail sent to external" [email protected]
//and check mail by typing;
mail
//forwarding incoming mail can be done with;
http://www.cyberciti.biz/faq/linux-unix-bsd-postfix-forward-email-to-another-account/

hudson: Hudson is the build server used to build projects and store binaries. It is a Continuous Integration (CI) tool which builds the project on every commit. The steps to install and configure hudson;
```
cd /development/hudson
wget http://hudson-ci.org/latest/hudson.war
sudo apt-get install openjdk-6-jre-headless
//can start hudson by the script or java -jar hudson.war
/development/bin/start-hudson.sh
```
After that, install required programs to /development/hudson/prog (or somewhere else) and configure paths inside hudson admin web interface.
```
cd /development/hudson/prog/setup
wget http://.../jdk-6u21-linux-i586.bin
wget http://.../jdk-1_5_0_22-linux-i586.bin
wget http://download.oracle.com/otn-pub/java/j2sdk/1.4.2_19//j2sdk-1_4_2_19-linux-i586.bin
wget http://mirror.veriportal.com/apache//ant/binaries/apache-ant-1.8.1-bin.zip

//then unzip/run these, and move extracted folders to /development/hudson/prog
cd /usr/bin
sudo ln -s /development/hudson/prog/apache-ant-1.8.1/bin/ant ant 
```
For .NET and Win32 projects the mono alternative clearly sucks. Do not waste your time trying to build your .NET projects with mono and xbuild. You cannot build Win32 projects anyway. Instead you can run .NET and Win32 projects in a windows Hudson slave. For this, set up a windows machine (virtualbox, or vmware is fine), install jre, and login to hudson (master hudson, e.g. hudson.example.com) from this machine. Then add a slave node from manage hudson -> slaves. Finally, run jnlp slave agent, and that's it. Now you can set from the job configuration "Restrict where this project can be run" to the slave's name, and these jobs will run at the slave. You shoud install .net sdk, msbee to the slave machine (the easiest is to install visual studio). The latest quirk is to remove msbuild configuration from manage hudson, so that hudson will not look for msbuild.exe on the linux master machine. To start java jnlp slave agent from command line, the command is;
```
java -Duser.language=en -jar slave.jar -jnlpUrl http://hudson.example.com/computer/hudsonwin/slave-agent.jnlp
 -jnlpCredentials hudson:hudsonpassword
```

openvpn: Open VPN is used to access services which are not exposed to public IP, instead run on local IP's. Follow these steps to install; (https://help.ubuntu.com/10.04/serverguide/C/openvpn.html)

sudo apt-get install openvpn
//init CA
sudo mkdir /etc/openvpn/easy-rsa/
sudo cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0/* /etc/openvpn/easy-rsa/
sudo chown -R johndoe /etc/openvpn/easy-rsa/
cd /etc/openvpn/easy-rsa/
vi vars //edit KEY_COUNTRY, KEY_CITY, etc at the bottom.
source vars
./clean-all
./build-dh
./pkitool --initca
./pkitool --server "John Doe Development Server"
cd keys
openvpn --genkey --secret ta.key
sudo cp John\ Doe\ Development\ Server.crt John\ Doe\ Development\ Server.key ca.crt dh1024.pem ta.key /etc/openvpn
cd ../../
/rename John Doe Development.* to server.*
//edit server conf
sudo cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz /etc/openvpn/
sudo gzip -d /etc/openvpn/server.conf.gz
sudo vi /etc/openvpn/server.conf
sudo /etc/init.d/openvpn restart

client configuration : First generate keys for the client;

cd /etc/openvpn/easy-rsa/
source vars
./pkitool hostname
//copy these files to client's ;
/etc/openvpn/ca.crt
/etc/openvpn/easy-rsa/keys/hostname.crt
/etc/openvpn/easy-rsa/keys/hostname.key
/etc/openvpn/ta.key

In the client install open vpn client for windows (http://openvpn.net/index.php/open-source/downloads.html ) and put the files specified above to: C:\Program Files\OpenVPN\config. Then copy client.ovpn from sample-config folder in the client to the config folder. Edit it's server address (remote dev.yoursite.com 1194). Run OpenVPN GUI as administrator, click connect.

site to site vpn : To access the development server private services everyone must install openvpn client. To prevent this, we can make site to site vpn and route all traffic to the vpn site from a single vpn client. In this scenario, we have a ubuntu server box in the local area network which will be the vpn box. Just insall openvpn with "sudo apt-get install openvpn". Then copy the client configuration files to this server. After connecting as a regular client, we will route all the traffic to the vpn network from this client. In order to forward traffic we must enable packet forwarding. Here are all the steps to establish site to site vpn;

//In the ubuntu server vpn box;
sudo apt-get install openvpn
cd /home/johndoe
mkdir openvpn-10.8.0.0
cd openvpn-10.8.0.0
//download here vpn client files from My Dropbox » dev.yoursite.com » meta » conf-templates » openvpn-client
sudo openvpn --config client.conf

//In another screen (assuming you are working in screen, use ctrl+c)
sudo vi /etc/sysctl.conf
//uncomment this line to enable packet forwarding
net.ipv4.ip_forward=1
sudo su
echo 1 > /proc/sys/net/ipv4/ip_forward
exit
//packet forwarding done

After connecting from a single client and enabling packet forwarding, we can connect from other machines through the vpn machine. But the last problem is that our gateway should know to route 10.8.0.0 traffic through 10.0.0.x machine. Either do that from the firewall or add this to the connecting machines (assuming windows);

//In the windows machines add this route:
route add 10.8.0.0 MASK 255.255.255.0 10.0.0.52

References for site to site vpn: site to site , packet forwarding

nginx: Nginx is a lightweight web server and reverse proxy server. We use it as a reverse proxy server to redirect multiple sub domains to various web applications. The steps to install nginx and configure it to proxy trac.yoursite.com to 127.0.0.1:8000 are;

sudo apt-get install nginx
cd /etc/nginx/sites-available
sudo vi default
//to return 444 for any unknown domain
//change server name to _ and add return 444 t
server_name _;
return 444;
//exit with :x

//copy the trac configuration file from meta folder
sudo cp /development/meta/conf-templates/nginx\ \(etc-nginx-sites-available\)/trac.yoursite.com ./
cd ../sites-enabled/
sudo ln -s /etc/nginx/sites-available/trac.yoursite.com trac.yoursite.com
sudo /etc/init.d/nginx restart
//now you can access to trac as http://trac.yoursite.com/

//copy the hudson configuration file from meta folder
sudo cp /development/meta/conf-templates/nginx\ \(etc-nginx-sites-available\)/jenkins.yoursite.com ./
cd ../sites-enabled/
sudo ln -s /etc/nginx/sites-available/jenkins.yoursite.com jenkins.yoursite.com
sudo /etc/init.d/nginx restart
//now you can access to hudson as http://jenkins.yoursite.com/

jetty: Jetty is an embedded java servlet container. It is preferred over tomcat because of easy management and lower memory consumption. Java web applications (.war files) are deployed to jetty. The steps to install jetty are;

wget http://download.eclipse.org/jetty/stable-7/dist/jetty-distribution-7.2.0.v20101020.tar.gz
tar xzvf jetty-distribution-7.2.0.v20101020.tar.gz
java -Xmx100m -jar start.jar

//edit etc/jetty.xml to change listening port or thread pool
//also you can configure nginx to forward external traffic (same as trac/hudson)

5. Firewall (Important!!)

Since none of the ports of linode servers are firewalled, we have to (we'd better :)) install a firewall, since we might misfollow the instructions and not close some listening ports. For this, the command which lists the listening ports is our precious friend;

sudo netstat -antp | grep LISTEN

Here 0.0.0.0:something is very dangerous. If you do not understand these at all, do yourself a favor and install a firewall;

sudo apt-get install ufw
//add all of the ports to be served publicly
sudo ufw allow 22
sudo ufw allow 80
sudo ufw allow 443
sudo ufw status //still inactive
sudo ufw enable
sudo ufw status verbose

6. Fail2Ban (SSH Brute Force Security)

It is hard to force everyone to connect to ssh with keys (but must be done some day, if that day ever comes, just edit /etc/ssh/sshd_config and turn off password authentication like: PasswordAuthentication no). Until then, it is easier to install fail2ban which will ban brute force IP's (3 failed ssh attempts will be banned for 10 minutes). To install fail2ban;

sudo apt-get install fail2ban
cd /etc/fail2ban
sudo cp jail.conf jail.local
sudo vi jail.local
//change destemail to [email protected]
//change action = %(action_)s to action = %(action_mw)s
//change ssh maxretry to 3
//save
sudo vi /etc/rsyslog.conf
//change $RepeatedMsgReduction off
//save
sudo /etc/init.d/rsyslog restart
sudo /etc/init.d/fail2ban reload

Then try to enter with invalid passwords. Fail2ban should send email with whois information to the specified email and ban you for 10 minutes.

Too many fail2ban emails? Add allowed ssh IPs to hosts.allow;

sudo vi /etc/hosts.allow
//add>
sshd: X.Y.Z.D, A.B.C.*
sudo vi /etc/hosts.deny
//add>
sshd: ALL

Be the change you want to see in the world