Playing with SVN
Posted by DineshPremalal on Mon 20 Mar 2006 at 10:15
Today I started to set up a SVN repository for our final year project. I tried to setup a SVN server using Apache2 so that the SVN repository is available to the client through the WebDAV/DeltaV protocol. Read on for a trial-and-error introduction.
The Version Control with Subversion book (by Ben Collins-Sussman, Brian W. Fitzpatrick & C. Michael Pilato) was very useful to me when I struggled with SVN. The e-version of the book also available for free.
After installing subversion I found it inside the directory /usr/share/doc/subversion/book.
First I installed apache2 using apt-get
#apt-get install apache2
Then subversion:
#apt-get install subversion
In order to setup subversion with apache2, we need to use the mod_dav_svn module. Here I tried to use the pre-compiled version from the Debian repositories because otherwise the process get more complicated.
/** * If you really need to compile apache2 and subversion your self the following * link might help you. But I *didn't get dirty with it. :) * http://www.linuxjournal.com/article/7655 */
Now I need to install mod_dav_svn module. I tried serveral searches using aptitude. Then I found out that libapache2-svn package provides mod_dav_svn plugin, so I installed it.
#apt-get install libapache2-svn
The most interesting part is, when we install pre-compiled package using apt-get or aptitude they edit configaration files accordingly.
When we install apache2 it installs configuration files in the /etc/apache2 directory. Modules are placed inside /usr/lib/apache2/modules.
In normal installation we found only httpd.conf for managing httpd server's configuration but in Debian we manipulate two configuration files namely "apache2.conf" and "httpd.conf".
As the book stated I edited httpd.conf as following:
LoadModule dav_module /usr/lib/apache2/modules/mod_dav.so LoadModule dav_svn_module /usr/lib/apache2/modules/mod_dav_svn.so
After restarted httpd server:
debian:/etc/apache2# /etc/init.d/apache2 restart Forcing reload of web server: Apache2[Sun Mar 05 20:21:41 2006] [warn] module dav_module is already loaded, skipping [Sun Mar 05 20:21:41 2006] [warn] module dav_svn_module is already loaded, skipping [Sun Mar 05 20:21:42 2006] [warn] module dav_module is already loaded, skipping [Sun Mar 05 20:21:42 2006] [warn] module dav_svn_module is already loaded, skipping
It seems that module already loaded then, so I commented out that newly added two lines (little devation from the book).
debian:/etc/apache2# /etc/init.d/apache2 restart Forcing reload of web server: Apache2.
Since my intention was to have more repositories in the svn, I added following lines to the apache2.conf:
<Location /svn> DAV svn # any "/svn/foo" URL will map to a repository /home/dinesh/svn/foo SVNParentPath /home/dinesh/svn (absolute path to repository) </Location>
Then I created repository using 'svnadmin':
$svnadmin create file:///home/dinesh/svn/repos
Then created directory in that repository:
$svn mkdir file:///home/dinesh/svn/repos/guththila
First I tried to acces repository using the http:// protocol but it didn't work :
dinesh@debian:~/tmp$ svn checkout http://127.0.0.1/svn/repos/guththila project svn: PROPFIND request failed on '/svn/repos/guththila' svn: Could not open the requested SVN filesystem
Then tried to use the repository using web http://127.0.0.1/svn/repos/guththila:
<d:error> <c:error> <m:human-readable errcode="160029"> Could not open the requested SVN filesystem </m:human-readable> </c:error> </d:error>
The Apache error log (in /var/log/apache/error.log) said:
[error] [client 127.0.0.1] (20014)Error string not specified yet: Berkeley DB error while opening 'nodes' table for filesystem /home/dinesh/svn/repos/db: Permission denied
It seems that there was a problem in permissions. I set permissions of the svn repository to the 777
#chmod -R 777 svn
Then I was able to acces the repository through the web browser using the URL http://127.0.0.1/svn/repos/.
At one point I came across a database error. The Apache error log said:
[error] [client 127.0.0.1] (20014)Error string not specified yet: Berkeley DB error while opening environment for filesystem /home/dinesh/svn/repos/db: DB_RUNRECOVERY: Fatal error, run database recovery
After seaching on the web found a similar problem it said to goto the repository db folder and run db_recovery.
$ cd ~/svn/repos/db dinesh@debian:~/svn/repos/db$ db4.2_recover
This recovered the database.
Now everything works fine but when I tried to commit into the repository:
$svn commit -m "test" svn: Commit failed (details follow): svn: MKACTIVITY of '/svn/repos/!svn/act/3033bdb6-400e-0410-9aed-b2743e91598e': 500 Internal Server Error (http://localhost)
This happened because of I didn't have authentication method for my svn repository.
# htpasswd -cm /etc/svn-passwd dinesh New password: Re-type new password: Adding password for user dinesh # htpasswd /etc/svn-passwd -m thilina New password: Re-type new password: Adding password for user thilina
I added the following lines to the apache2.conf:
<Location /svn> DAV svn SVNParentPath /usr/local/svn AuthType Basic AuthName "Subversion repository" AuthUserFile /etc/svn-passwd </Location>
With this method the password is transmitted as plain text over the network.
Now I'm able to acces and commit SVN repository over the network!
I still need to configure the SVN repository to use SSL. I'll post the steps that I followed to configure SVN with SSL also.
If you interested please look at my weblog for original article
The subversion book is pretty sloppy at telling you how to manage the repository ownership, which tends to bite new admins, especially when using file or ssh+svn access. It doesn't help that the debian package doesn't step up to the plate and do the right thing.
Basically, you have to make sure that the repository files and directories are writable by all the users with commit access to it. In your case—access via Apache only—this means you should set the repository to be owned by the web server user (ie www-data), (as root):
chown -R www-data:www-data /home/dinesh/svn/repos
find /home/dinesh/svn/repos -type d chmod 2755 "{}" /;
find /home/dinesh/svn/repos -not -type d chmod 0644 "{}" /;
It's probably a smart move to shift this repository to something like /var/lib/svn/repos, too.
The svnserve case is much the same, as the repository only has one system user, however if you want to use file: and svn+ssh: access methods instead or as well, you need to do a bit more work:
- Set up a group for all your subversion users with write access, say svn-users. Put all the users in it, including the svnserve and apache user if you want access via those methods.
- Change the group ownership on the whole repository to svn-users
- Make sure the group write bit is set on all the repository files and directories
- Make sure the set group id bit is set for yourrepos/db and it's subdirectories, so that new files created will have the correct group ownership set (svnadmin does this for you automatically)
- When using the file: or svn+ssh: access methods, make sure the umask has been set to 0002 or similar. Debian's default umask is 0022, which filters out the group write permission. A wrapper script is probably the best way to handle this.
This applies to Subverson 1.1.x in sarge. I think these issues might have been addressed in 1.2 and greater that's in etch. I haven't really tinkered with it recently.
[ Parent | Reply to this comment ]
Good comments jwm,
I used your find/chmod lines on some other data, and I am just not sure if they work? I run bash and they might work in other shells.
Here is what works for me
find path/to/change -type d -exec chmod 2755 \{} \;
find path/to/change -not -type d -exec chmod 0644 \{} \;
[ Parent | Reply to this comment ]
Ah—I made a brain-o on the last slash.
[ Parent | Reply to this comment ]
-- use -fsfs
Sure the bdb package is faster, but as you show in your example it can easily be corrupted. Using -fsfs makes a tar backup simple and reliable.
-- add "SVNPathAuthz off" to your Apache2 location directive
If you don't need per-directory security, turn off the checks to improve the speed.
-- using SSL
Someone mentioned having trouble with the 'SSLRequire' directive. The easiest way to force SSL is to add a RedirectMatch to your normal, HTTP (80) virtual host:
RedirectMatch (.*) https://svn.mydomain.org$1
SSL and Apache2 work well and allow one to control access using authentication methods that might already be in use. They are slower though and can be CPU intensive. If you don't need the SSL, you might consider avoiding it in order to improve speed. mod_auth_digest offers a more secure method, but I've never tried it with Subversion clients.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
you can use fsfs backend instead of the legacy Berkley-DB backend for data storing.
One of the motivation to move from BDB to fsfs is the speed at which BDB backend tends to lock, keeping admin on the edge to restore it.
There's less problem when migrating from one version of BDB to another (as Subversion did in 1.3 if I am correct).
See http://svnbook.red-bean.com/nightly/en/svn.reposadmin.html#svn.re posadmin.basics.backends for more informations
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
Thanks for sharing your experiences. I am also about to go through this, although not for the first time. I intend to do a more complex setup and try to write about it.
I just want to point out a few things.
Instead of editing :
As the book stated I edited httpd.conf as following:
LoadModule dav_module /usr/lib/apache2/modules/mod_dav.so LoadModule dav_svn_module /usr/lib/apache2/modules/mod_dav_svn.so
You can under Debian run
a2enmod dav a2enmod dav_svn
That will set the appropriate links between /etc/apache2/mods-available/ and /etc/apache2/mods-enabled/
I am still a bit undecided how to properly add the subversion location. I have used the conf file in /etc/apache2/mods-availible, but maybe sites-enabled can be used too. (I am thinking for multi-site apache)
Have a look at the file
/etc/apache2/mods-available/dav_svn.conffor an example. I use the below structure but the mentioned file have more information. (Please observe that the location apache tags were stripped by the blogging software)
DAV svn SVNPath /srv/svn/project AuthType Basic AuthName "Subversion Repository" AuthUserFile /etc/apache2/dav-svn.passwd AuthzSVNAccessFile /etc/apache2/dav_svn.authz Require valid-user
For access configuration have a look in
/usr/share/doc/libapache2-svn/INSTALL.authz.gz
Lastly the permission should be handled differently. It is not good to set the repository writable by anyone. I can not find the reference right now but on earlier recomendation I have configured my running repository as
chown -R www-data.www-data /srv/svn/project
I am not sure how well that works with other access methods, but I will investigate that in the comming months.
It would be good if you can investigate these things and update the article accordingly so that the article shows as much as possible the Debian way. That is at least what I like about http://www.debian-administration.org/
[ Parent | Reply to this comment ]
As for the perms I set everything to be writable by svn.
[ Parent | Reply to this comment ]
Any special reason for building it from source ?
I am just curious if I am missing out on some feature that is missing in the precompiled debian package.
Or has this to do with AD (Active Directory?). Is that the reason you have to build from source ?
[ Parent | Reply to this comment ]
I tend to use packages when possible with the exception of apache. I tend to build apache from source because there are often many instances of apache running on one box (apache dev boxes).
AD(Active Directory) may have been part of the reason but the real reason(s) have been lost to time...
Should have included them in my notes and I could have gotten it from svn. :-)
[ Parent | Reply to this comment ]
It contains non-ascii characters (they are french, so the log there commits in french) : all my accentued characters where replaced by "?\195" or "?\162", or "\?....", etc., instead of "é, è, ç, ..."
For those who wants to fix the problem, consider setting the LANG environement variable to something like "fr_FR@euro" (adjusts it to your locale), as in (in the /hooks/post-commit script) :
LANG=fr_FR@euro /usr/lib/subversion/hook-scripts/commit-email.pl "$REPOS" "$REV" test@xxx.xxxx
There maybe better solutions (you need to set this fix for every hook of every repository :/ ), which I'll be glade to know :)
[ Parent | Reply to this comment ]
You can add SSLRequireSSL
DAV svn
SVNParentPath /usr/local/svn
AuthType Basic
AuthName "Subversion repository"
AuthUserFile /etc/svn-passwd
SSLRequireSSL
Then only ssl acces is allowed. SSL should work fine in the default apache2 instalation.
[ Parent | Reply to this comment ]
http://projects.edgewall.com/trac/wiki/TracOnDebianSarge
which worked like a charm :)
(Trac btw. might be worth taking a look at as well!)
How ever, I'm new to debian and apache2 so I would like to know the debian way of apache2 configuration... i just did what's in the tutorial.
Your hint to use SSLRequireSSL didn't work for my default apache2 installation (not even after: a2enmod ssl). What else am I missing?
Would be great to have a comprehensive debian subversion setup, config and maintain articel here.
Regards
[ Parent | Reply to this comment ]
http://www.debian-administration.org/articles/349
[ Parent | Reply to this comment ]
OUCH!
How about chown -R www-data:www-data svn; chmod -R 770 svn ?
With 777 you're givng permission for everything to write directly on the svn repository files, something that will most likely cause corruption, sooner or later. (the apache web daemon runs as the www-data system user by default in Debian)
Also, what are your reasons to go with the berkelydb format?
I have seen zero problems with the fsfs backend so far (9 months, 1800 commits, ~100MB).
I choose fsfs originally because it's filesystem-atomic, meaning that you can just tar up the svn dir at any time and have a consistent backup.
For SSL encryption (sometimes a must to get arround corporate firewalls/proxies, not many of them understand and/or allow svn!), just make apache2 use ssl the regular way, there are no svn implications and most clients appear to support it.
PS: it's worth noting that the latest svn version (not in stable atm) allows for true dav autoversioning.
This allows you to do stuff with a regular (non-svn) dav client, like the windows "mount remote location" thingy. Every change will be committed in svn, and is tracable & reversable with an svn client.
I haven't tried this out yet and I expect it to be slow, but ... it sure sounds great!
[ Parent | Reply to this comment ]
File /etc/apache2/mods-available/dav_svn.conf distributed as part of libapache2-svn package contains the following lines:
# Set this to the path to your repository
SVNPath /var/lib/svn
Second line should probably read
SVNParentPath /var/lib/svn
Otherwise, the error prevents from accessing the repository
[date] [error] [client x.x.x.x] Could not open the requested SVN filesystem [500, #2]
Semen Trygubenko
[ Parent | Reply to this comment ]
I found this 'Playing with Subversion' article very useful, so thanks to Dinesh for posting it.
I used the article to help me set up my own subversion repository, and I have documented my procedure at http://www.iau.ukfsn.org/iau/linux/subversionSetup.html. I'll update this document as my Subversion usage gets more ambitious (and as my understanding improves).
I don't know if I've got everything quite right, however. In particular I have two areas of doubt:
1. AuthzSVNAccessFile
I commented out the AuthzSVNAccessFile field in <location /svn>. I need AuthzSVNAccessFile only if I want to control access per directory, is that right? Actually I have no idea what Authz is for.
2. Permissions
The original poster gave everything 777 permissions. I can see this is a bad idea. Two corrections were offered:
(a) debian administration
# find /var/svn-repos/ -type d -exec chmod 2755 \{} \;
# find /var/svn-repos/ -type f -exec chmod 644 \{} \;
(b) http://trac.edgewall.org/wiki/TracOnDebianSarge
# find /var/svn-repos -type d -exec chmod 2770 \{} \;
# find /var/svn-repos -type f -exec chmod 660 \{} \;
My understanding of permission numbers is fairly rudimentary so I have two questions here:
- why are files and directories given different permisions?
- what is the difference between the two approaches?
I would appreciate any help on these two issues, and I'll update my subversionSetup page accordingly.
Thanks.
Best wishes
Ivan
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]