URL rewriting is the method of converting complex URL parameters into more human readable format to allow more simple and memorable URLs. This can be an important function if you start using frameworks or content management systems which automatically generate long and at times cryptic URLs. While URL rewrite on the more popular Apache HTTP Server is relatively easy to set up using the default mod_rewrite module, reproducing this functionality on Tomcat requires a little more work.

Standard URL: http://www.example.com/list.cfm?product=fruit&page=1&order=asc&perpage=30
Rewrite URL: http://www.example.com/list/fruit/asc/30/1

Installation

A downloadable copy of URLRewrite can be found from one of 2 sources. The outdated website at http://www.tuckey.org/urlrewrite/ lists version 3.2 as the most recent version. But there is a more recent and in my testing still stable 4.0 beta at Google Code http://code.google.com/p/urlrewritefilter/downloads/list that contains some critical bug fixes. This article will assume you have downloaded the 4.0 beta and not the 3.2 stable.

Extracting the downloaded URLRewrite archive reveals a single WEB-INF folder which contains a lib folder and the file urlrewrite.xml. Both these items will need to be copied to the WEB-INF folder of your Tomcat server root directory. For example if example.com was located in /var/www/www.example.com or c:\www\www.example.com the lib folder and urlrewrite.xml would go in /var/www/www.example.com/WEB-INF/ or c:\www\www.example.com\WEB-INF.

Content of Tuckey URLRewrite 4.0's archive

Content of WEB-INF with URLRewrite

Content Of A Clean WEB.XML

Generally for most default installations of Tomcat the WEB-INF folder will only contain the single file web.xml. We will need to edit web.xml using a text editor to enable URLRewrite on Tomcat but because it is an XML text file. It can be machined parsed so I’d recommend editing it using a source code text editor such as NotePad++ on Windows or Textmate on OS/X.

Add the following code to web.xml anywhere contained within the <web-app></web-app> tags.

<!-- URL ReWriter -->
<filter>
<filter-name>UrlRewriteFilter</filter-name>
<filter-class>org.tuckey.web.filters.urlrewrite.UrlRewriteFilter</filter-class>
<!-- set the amount of seconds the conf file will be checked for reload
can be a valid integer (0 denotes check every time,
empty/not set denotes no reload check) -->
<init-param>
<param-name>confReloadCheckInterval</param-name>
<param-value>0</param-value>
</init-param>
<!-- you can disable status page if desired
can be: true, false (default true) -->
<init-param>
<param-name>statusEnabled</param-name>
<param-value>true</param-value>
</init-param>
<init-param>
<param-name>logLevel</param-name>
<param-value>DEBUG</param-value>
</init-param>
<init-param>
<param-name>statusEnabledOnHosts</param-name>
<param-value>localhost</param-value>
</init-param>
</filter>

<filter-mapping>
<filter-name>UrlRewriteFilter</filter-name>
<url-pattern>/*</url-pattern>
<dispatcher>REQUEST</dispatcher>
<dispatcher>FORWARD</dispatcher>
</filter-mapping>

Edited WEB.XML

Let’s quickly go through these settings.

confReloadCheckInterval is a numeric value in seconds that tells how frequently URLRewrite should check your urlrewrite.xml rules for any changes. Normally with Tomcat the modification of an XML configuration file requires a restart before the changes are reflected. You can set this value to -1 to disable automatic checking, while our value of 0 will mean that URLRewrite will check the urlrewrite.xml on every HTTP request. It is a great setting while testing and developing but a resource waste if used on a production server.

statusEnabled is a Boolean value that enables a URLRewrite status page that is reachable via a web browser at http://www.example.com/rewrite-status. It is probably best to disable this feature on production servers.

logLevel sets how much logging should be produced by URLRewrite. While the default setting is INFO, I suggest using DEBUG while you are testing. Through in a production environment you will probably want to use ERROR or FATAL to limit logging as URLRewrite can generate some large log files very quickly with more verbose log settings.

statusEnabledOnHosts allows you set which IP addresses and hosts that have access to the URLRewrite status page previously mentioned.

Finally the <filer-mapping> tag tells what kinds of methods to pass through via URLRewrite. The tags < url-pattern ></ url-pattern> should be left as is to apply URLRewrite to the whole site. While the 2 < dispatcher ></ dispatcher > tags mean that URLRewrite should be used for all HTTP REQUESTS and HTTP internal FORWARDing.

Once done, save your web.xml file and restart your Tomcat server. If all goes well you should be able to point your browser to http://www.example.com/rewrite-status and an UrlRewriteFilter 3.2.0 build 1 configuration overview should be shown. Yes that 3.2 version number is incorrectly listed in 4.0. Point your browser to http://www.example.com/test/status/ and you should be automatically forwarded to /rewrite-status. When this works then congratulations as you now have URLRewrite enabled on your server. Now I will give you some helpful example rules that may come in use. These rules all go in-between the <urlrewrite></urlrewrite> tags located in the urlrewrite.xml file within the WEB-INF folder. Whenever a page is requested on your Tomcat server the URLRewrite application will in a sequential order process ALL the rules contained in the urlrewrite.xml.

Pretty URL, SES Friendly URL Pass-Through

The most common use of URLRewrite would probably be to enable a 3rd party framework or CMS to use pretty URLs. The rule below is a generic setup that could be adapted for many uses. Generally speaking this should always be the LAST rule listed in your urlrewrite.xml rule set. The rule passes all URL requests to the index.cfm file except requests with URLs pointing to files or folders listed in the <condition></condition> tag regular expression value. So with this rule the URL http://www.example.com/list/apples would be displayed as is in the user’s browser but URLRewrite will actually pass http://www.example.com/index.cfm/list/apples to the Tomcat server. You do want to make sure that the page that contained within the <to></to> tag value is also listed in the (not equal) <condition></condition> value otherwise you could run into an infinite loop.

<rule enabled="true">
<name>Generic Pretty URLs Pass-through</name>
<condition type="request-uri" operator="notequal">^/(index.cfm|robots.txt|osd.xml|flex2gateway|cfide|cfformgateway|railo-context|admin-context|files|images|jrunscripts|javascripts|miscellaneous|stylesheets)</condition>
<from>^/(.*)$</from>
<to type="passthrough">/index.cfm/$1</to>
</rule>

Permanent Redirection

This rule is specifically for when you want to do a permanent redirection using the HTTP code 301. If we break this rule down, the <rule enable=”"> Boolean enables you to selectively turn off this rule without the need to comment it out. The <name></name> tags contains the label you wish to use to describe the rule. <from></from> tag contains a regular expression to forward all requests for the documents.cfm page plus any URL parameters. While <to></to> is the new URL to redirect to. The attribute type tells URLRewrite to send a permanent direct code to the browser requesting the URL, while the attribute last = true tells URLRewrite not to process any further rules for this page request.

<rule enabled="true">
<name>Permanent redirect example</name>
<from>^/documents.html(.*)$</from>
<to type="permanent-redirect" last="true">/file/list/document</to>
</rule>

Selective HTTPS Enforcement

If you have HTTPS setup on your server you can use URLRewrite to enforce certain folders, URL paths or files to only be served on an encrypted HTTPS protocol. The <condition></condition> tag is used to enforce additional policies as to when the rule should be implemented. The attribute type with a value of scheme and the attribute operator with a value of equal states that when the URL scheme (http, https, ftp, etc) is equal to HTTP then apply this rule.

<rule enabled="false">
<name>Force HTTPS example</name>
<note>Automatically redirects adminstration requests to a secure protocol.</note>
<condition type="scheme" operator="equal">^http$</condition>
<from>^/CFIDE/administrator/(.*)$</from>
<to type="permanent-redirect" last="true">https://www.example.com/CFIDE/administrator/$1</to>
</rule>

Railo HTTPS Enforcement railo-content.

<rule enabled="false">
<name>Force HTTPS example</name>
<note>Automatically redirects adminstration requests to a secure protocol.</note>
<condition type="scheme" operator="equal">^http$</condition>
<from>^/railo-context/admin/(index|web|server).cfm$</from>
<to type="permanent-redirect" last="true">https://www.example.com/railo-context/admin/$1.cfm</to>
</rule>

Conditions Based On URL Parameters

You can also apply conditions to user supplied URL parameters. In the example below the condition looks for the URL parameter named fruit and sees if its value is either kiwi, apple or orange. If the values match then it redirects to a replacement URL which also incorporates the parameter. The URL request http://www.example.com/list.html?fruit=apple would forward to http://www.example.com/list/fruit/apple.

<rule enabled="true">
<name>Selective fruit example redirect</name>
<condition type="parameter" name="fruit" operator="equal">(apple|kiwi|orange)</condition>
<from>^/list.html(.*)$</from>
<to type="permanent-redirect" last="true">list/fruit/%{parameter:fruit}</to>
</rule>

This article was written in mind with the CFEclipse add-on running off Eclipse Helios. I have not tested these steps with Adobe’s proprietary CFBuilder.

Eclipse: http://www.eclipse.org/downloads/compare.php
CFEclipse: http://cfeclipse.org

Let’s start.

Visit Apache Tomcat and download your appropriate edition under the Core section. Windows users should download either the 32-bit or 64-bit Windows zip but not the Service Installer.

Tomcat 7: http://tomcat.apache.org/download-70.cgi
Tomcat 6: http://tomcat.apache.org/download-60.cgi

Extract the downloaded zip containing Apache and then move the created apache-tomcat-7.*.* directory to a location of your choice. In this example I will move it to my server collection under W:\Dev\Server\.

Go to the Railo download page and fetch the latest Current stable release Railo Custom WAR Archive under the ALL OS.

Railo: http://www.getrailo.org/index.cfm/download

Using 7-Zip or some other WAR compatible archive tool extract the downloaded Railo-3.*.* WAR archive.

7-Zip: http://www.7-zip.org

Now create a working resource directory for your CFEclipse project. I will use W:\Railo\Test\.

Copy the content of the extracted WAR archive to your project resource directory.

Start Eclipse.

From the pull-down menu select Window > Preferences then Server > Runtime Environment and select Add.

In the New Server Runtime Environment dialog select your appropriate Apache Tomcat version and select Next >. If you cannot see an Apache folder containing various Tomcat versions you will need to click the Download additional server adapters link and follow the Install New Extension wizard.

Give your Tomcat Server a name and point it to your Tomcat installation directory.

Click the Installed JREs… button and in the Installed JREs dialog select Add. In the Add JRE wizard select Add and point the JRE home: input to your local Java Runtime Environment location. This is often either C:\Program Files\Java\jre6 or C:\Program Files (x86)\Java\jre6. If you downloaded the 64-bit edition of Tomcat you will need to point to the 64-bit Java RE that is installed on your machine. Once done select the Finish button and then the OK button in the Installed JREs dialog.

Java Runtime Environment: http://java.com/en/download/manual.jsp

Now make sure the New Server Runtime Environment has your installed JRE selected and not the Workbench default JRE and press Finish. Press OK in the Server Runtime Environments dialog.

Now we need to bring up the Eclipse Servers view. From the Eclipse pull-down menu select Window > Show View > Other > Server > Servers and press OK. The Servers tab should now be selected, right click on the Servers pane and select New > Server.

In the Define a New Server dialog select your Apache Tomcat server type. Give the server a Server name, keep the host name as localhost, make sure the Server runtime environment is correct and select Finish.

You should have a new entry in your Servers tab. Double left-click it to bring up the server’s Overview tab.

Under Server Locations select the Use Tomcat installation option. The Deploy path should be set to webapps and then save the changes using Ctrl-S.

Now select the Modules tab in the Overview and press the Add External Web Module… button. Point the document base to your Railo working resource directory in my case it is W:\Railo\Test\ but keep the path as / and press OK. Save the updated Web Modules using Ctrl-S.

Back to the Servers tab select your server, right click and select Start. Eclipse should automatically switch tabs to the Console as the server starts before reverting you back to the Servers. The server should have a revised status saying Started, Synchronized.

Point your web browser to http://localhost:8080/and you should have the Railo Welcome page. To shut down your server select it under the Eclipse Servers tab, right-click and select Stop. Congratulations you have a working embedded Tomcat Railo server operating within Eclipse.

Next we will create a CFEclipse project and a simple Hello World application otherwise you can end the tutorial here.

In Eclipse pull-down Window and select Show View > Project Explorer.

Right click on the Project Explorer pane and select New > Project to bring up the New Project wizard. Open the CFEclipse folder and selected CFML Project and press Next.

Give your CFEclipse Project a Project name and then deselect the Use default location checkbox. Select the location of your working resource directory where you placed your Railo installation. For me it was W:\Railo\Test\ and then Finish.

Open your newly created project from the Project Explorer, right-click and select New > File. Create a new file named hello.cfm in your project and Finish. Edit the hello.cfm and add the following mark-up and save using Ctrl-S.

<cfdump var=”Hello world!”>

In the Servers tab start your server. Once it has loaded select from the Eclipse pull-down menu Window and then Show View > Browser View. Under the Project browser tab enter the URL http://localhost:8080/hello.cfm and you should see a dump of text Hello world!.

Now you are ready to develop, test and troubleshoot your ColdFusion Mark-up all contained within the Eclipse environment.

Cygwin is an awesome tool for agnostic operating system users who regularly use both Windows and Linux but want the power of a Linux shell and its tools running within Windows. It is a great replacement to the primitive Windows command prompt without the need to learn a new collection of commands or a new scripting language. A requirement which is needed for the Microsoft developed and recommended command prompt alternative PowerShell.

Cygwin is POSIX (Portable Operating System Interface for uniX) complaint which basically means it shares a common API with all the other complaint operating systems. It does not mean applications compiled for other POSIX systems such as Linux software will automatically work within Cygwin. To add new POSIX software you still need to compile the software source code under Cygwin using its own compilers and libraries. This process is a requirement with most POSIX complaint operating systems e.g. software complied for FreeBDS (Unix) will not work under Debian/Ubuntu (Linux).

Cygwin will work on any x86 32-bit or x86-64 editions of Windows NT from NT 4.0 Service Pack 4 (1998) onwards to Windows 7 including 2000, 2003, 2008, XP and Vista. There is no support for the Windows 16/32-bit hybrids such as 95, 98 or ME.

Installation

So lets get started. Visit http://cygwin.com/install.html, download then run the setup.exe. This file is a Windows based package manager for Cygwin. By default it will download and install the latest minimal install of Cygwin but on future launches it will also enable you to add new packages and update your existing Cygwin software. So after you install Cygwin do not delete or loose the location of setup.exe!

Cygwin Net Release Setup Program (setup.exe).

For Choose A Download Source select Install from Internet.

Select Root Install Directory.

By default Cygwin will install to C:\cygwin but you can change this to any directory of your choosing. It is highly recommended that you do not choose a directory path which contains spaces in the name, so both C:\Program Files\cygwin or C:\Program Files (x86)\cygwin are out.

Select Local Package Directory.

Cygwin setup will save all downloaded packages to this folder after it has finished installation. You can then either delete them, leave them in this directory or back them up to a USB stick for future use on this or other computers.

Select Your Internet Connection.

Unless otherwise needed you should leave the Internet connected selection with the default selection of Direct Connection.

Choose A Download Site.

Choose A Download Site lets you select a file repository from where Cygwin setup will download the the packages and files. It is preferable to pick a download location from a site this is geographically close to your own. But if you can not determine this from the country codes in the URL or can not find a suitable mirror then simply select the first item.

Selecting Packages

Select Packages.

This Select Packages screen is the most important part to Cygwin setup. It is the Cygwin package manager interface, equivalent to Synaptic Package Manager in Debian and Ubuntu or YUM Extender in Fedora Linux. The screen capture has a number of arrows pointing to features of the interface but as a first time user you can completely ignore all of these and press Next to accept the default installation.

• The dark green solid arrow points to the search dialog enabling you to quickly filer packages by name. Unfortunately the search does not work on the package descriptions.
• The solid black arrow has 4 radio buttons enabling you to mass-apply installation restrictions on all packages.
• Keep tells Cygwin setup not to update any installed packages.
• Prev is short for previous which reverts all packages back one version.
• Curr is the default and recommended option as it installs the current version of all selected packages.
• Exp will install the most current or experimental version of all selected packages.
• The red arrows point to the manager categories which can be expanded to reveal individual packages.
• The purple arrow points to version numbers which belong to packages that are already installed onto Cygwin.
• Orange arrow enables you to manually override the global package version (black arrow radio buttons).
• Skip means the package has not been installed and setup will ignore it.
• Keep means the package has been installed but setup will ignore it.
• Reinstall will overwrite an existing package installation with the same version from a fresh download.
• 1.0-1 or some other version number means setup will install this version of the package for the first time.
• Source means setup will ignore the installed package but will download the source codes.
• Uninstall removes the installed package from your Cygwin installation.
• The yellow arrow points to a Binary check-box while the light green points to a Source check-box. These check-boxes only appear only when available to the package New column.
• The light blue arrow is the package download size in Kilobytes.
• Dark blue contains the package name and a brief description.

Progress.

Cygwin setup will now download and install all the default-install packages. Wait until the Progress screen is complete and go onto the Next screen.

Create Icons & Install Complete.

Create Icons enable you to add Desktop and Start Menu shortcuts. Once finished use either shortcut to launch Cygwin.

Bash Terminal

Welcome to Bash.

Welcome to your Bash prompt in Windows! You can test this by running Bash version.

bash --version

Display your version of Bash.

The first thing I’d recommend doing is customising your bash shell to implement some aliases for basic colouring for some staple Cygwin commands.

dir -A ~

Will reveal some hidden setting files in your home directory which you can edit these using a text editor, but please avoid using the standard Windows Notepad. I use Windows Notepad++ and would edit w:\cygwin\home\Ben\.bashrc but you would probably need to edit c:\cygwin\home\[User name]\.bashrc. Windows may have issues opening a file which it sees as only having an extension but lacking a file name. If this is the case just use the Open With dialog but make sure you have the Always use the selected program to open this kind of file check-box left unchecked.

The yellow highlights the settings I would recommend to un-comment.

Bash settings file after the comments have been removed.

Once you have finished your changes you can apply them by reloading Bash.

bash --login

Test your new alias and option defaults.

cd /
ll

You now see a colourised directory and file listing of the Cygwin root.

Line list of root.

This is the same directory as you chose during the Cygwin setup installation which by default is c:\cygwin. You can use either Windows File Explorer, Windows command prompt or Cygwin to modify the contents of the root. For more details on how Cygwin handles POSIX permissions within Windows and other potential incompatibilities I’d recommend reading the Using Windows Security in Cygwin chapter in the manual.

Tips

By default there is no Cygwin clear command (cls in command prompt) but the easiest way to clean your terminal is to press Ctrl l.

Use the Tab key on your keyboard to auto-complete commands, file and directory names. This Bash feature is superior to the basic Tab auto-complete used by Windows command prompt.

Did you also know you can chain 2 or more commands together using a semi-column ; character as the separator.

cd /; echo "Root Directory"; ls; cd ~; echo "My Home Directory"; ls

And if you didn’t realise in the command above, change directory forward-slash cd / takes you to the Cygwin root, while change directory tilde cd ~ takes you to your user home directory.

Drill Down of the Cygwin Root

• Cygwin.bat is the Windows batch file used to launch a Cygwin Bash prompt within a Windows command prompt.
• Cygwin.ico is a standard Windows icon file attached to the Cygwin batch file.
• bin/ is a directory containing binary (execute) applications that have been compiled for use on Windows under Cygwin. In here are all the unique POSIX-like commands and applications that give you all that extra functionality that a bare-Windows install often lacks.
• cygdrive/ contains a list of mounted Windows drive letters. From here you can access any drive partition that is viewable to Windows. For example cygdrive/c/Windows/ would be the location of most peoples Windows installation.
• dev/ is short for devices. Traditionally for POSIX this is where your detected hardware devices are listed but in Cygwin its usage is mainly there for software compatibility and can be ignored. http://cygwin.com/cygwin-ug-net/using-specialnames.html#pathnames-posixdevices
• etc/ contains all your Cygwin related configuration files. Unlike the Windows registry these should all be plain text files that can be edited within any Windows text editor (such as NotePad++) or a Cygwin editor such as VM or Nano.
• home/ contains the user directories which are not unlike the Windows User directories. This is where your non-Cygwin files such as documents, file downloads should be stored.
• lib/ is an abbreviation of library. These are compiled, shared components that are used by various Cygwin commands and applications but you can ignore this directory.
• proc/ is a virtual directory that allows you to display various aspects of your system using the cat command.
• cat /proc/cpuinfo Displays your computer CPU details.
• cat /proc/meminfo Displays your Windows memory information.
• cat /proc/mounts Lists your disk mounts within Cygwin.
• tmp/ is the Cygwin temporary directory.
• usr/ is the bread and butter of your Cygwin install.
• usr/bin/ is identical to bin/ that was listed previously.
• usr/include/ are programming header files which are somewhat like shared libraries, you can ignore this.
• usr/local/ is a location where you can compile applications for yourself, again if this does not interest you it can be ignored.
• usr/sbin/ contains system binaries and applications tools for Cygwin system administration.
• usr/share/ are platform independent read-only shared files such as documentation. For example /usr/share/fonts contain a collection of system fonts. /usr/share/doc contains application documents.
• usr/src/ is a location for storing application source code.
• usr/ssl/ is a location for storing SSL (Secure Socket Layer) and TLS (Transport Layer Security) certificates which are used for Internet communication encryption and security.
• usr/tmp/ is the same temporary folder as /tmp.
• var/ is where variable data files are stored. These are files generated by yourself or applications. For example all log files would go in /var/log. If you ran an Apache HTTP daemon the files it hosted would be contained in /var/www.

POSIX and Cygwin Commands

Now the usage of a Bash shell or POSIX-like commands is out of the scope of this article. You can run dir /bin to see all the .exe execute commands available to you within Cygwin. Like Windows command prompt you do not need to include the file extension when running an application, dir works the same as dir.exe.

If you need quick help on the usage of a command, use [command] –help e.g. dir –help. Many commands share the same options.

If you think the default Cygwin bash shell is a bit ugly then you are not alone. Microsoft does not give the default command prompt much love and unfortunately this is what Cygwin relies on for its own terminal interface. You can easily change this though by again running Cygwin setup.exe. When you are back in the Select Packages screen do a search for mintty. Select to install the mintty binary package and once complete go to your Cygwin start menu. There should be a new mintty option in addition to the Cygwin Bash Shell.

Install Mintty Terminal.

Cygwin under command prompt is not that attractive.

Cygwin under Mintty allows for font smoothing.

From now on use Mintty for a cleaner terminal interface to Cygwin.

Services and Daemons

Want to to implement a service or daemon into your Cygwin install? I will show an basic example using Lighttpd, a light-weight HTTP server that is an alternative to Apache2 HTTP (which also can be installed to Cygwin).

Run Cygwin setup.exe again to search and install lighttpd and make sure you say yes to install any resolving dependencies. Once installed you will find the application residing in the usr/sbin/ directory as lighttpd.exe. An example configuration file lighttpd.conf.default can be found in etc/lighttpd/. We will create an empty configuration file using the touch command.

cd /etc/lighttpd
touch lighttpd.conf

Now use your Windows notepad application to edit c:/cygwin/etc/lighttpd/lighttpd.conf and copy the settings listed below.

server.document-root = “/var/www/servers/www.example.org/pages/”

server.port = 80

server.username = “www”
server.groupname = “www”

mimetype.assign = (
“.html” => “text/html”,
“.txt” => “text/plain”,
“.jpg” => “image/jpeg”,
“.png” => “image/png”
)

static-file.exclude-extensions = ( “.fcgi”, “.php”, “.rb”, “~”, “.inc” )
index-file.names = ( “index.html” )

$HTTP["host"] == “www2.example.org” {
server.document-root = “/var/www/servers/www2.example.org/pages/”
}

This config we are using is taken from the Lighttpd Configuration Tutorial.

Now attempt to launch lighttpd with your new configuration file.

/usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf

Our attempt will fail because the web server document root directory does not exist, so lets create that and lets also make an index.html document and try it again.

mkdir /var/www ; mkdir /var/www/servers ; mkdir /var/www/servers/www.example.org ; mkdir /var/www/servers/www.example.org/pages
touch /var/www/servers/www.example.org/pages/index.html

In a Windows notpad application edit c:/cgywin/var/www/servers/www.example.org/pages/index.html and copy/paste the following HTML code.

<html>
<title>Hello World</title>
<body>
<h1>Hello from Lighttpd running in Cygwin!<h1>
</body>
</html>

index.html content.

Lets retry running Lighttpd again.

/usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf

Once Lighttpd has loaded you may need to press the Enter key to restore your Bash prompt but Lighttpd will continue to run in the background. Now point your local Windows web browser to 127.0.0.1 or localhost.

Chrome browser viewing our primitive HTML file.

When you want to shutdown Lighttpd run find the PID (Process IDentifier) and kill it.

ps -s
kill [PID number]

Discover and shutdown of the Lighttpd service.

Congratulations! You have successfully implemented a Bash shell environment on your Windows operating system that also has a robust but simple HTTP server.

Popular Cygwin Applications

Other popular Cygwin packages that maybe of interest.

Servers services such as Apache, BIND, PostgreSQL, Pure-FTPd, OpenSSH.
Programming languages such as C, C++, Perl, Python, Ruby, TCL.
Remote log-in tools SSH, RSH, Telnet.
File transfer tools FTP, SCP, rsync, unison, rtorrent.
In addition to an extensive range of network, system and file diagnostics tools.

More information on Cygwin can be found in its User’s Guide.

Despite its name SFTP is not an extension to the 1985 FTP (File Transfer Protocol – RFC 959) which was in common use until the late 1990s to share files online. FTP usage has fallen out of favor for Internet use in the 2000’s due to the protocol’s inherent insecurities in addition to competition from newer protocol such as the bandwidth friendly BitTorrent protocol. The most glaring insecurity with FTP is that it requires log-in user names and passwords to be communicated between the client and the server using unencrypted plain-text.

Fortunately in recent years many of the more popular FTP clients have implemented complete support for SFTP making the end-user transition from insecure to secure transfers seamless. Older, now redundant stop gaps for secure FTP such as FTPS/FTP-SSL which is now referred to as FTP with TLS (RFC 4217) were confusing to use and difficult to setup correctly.

It is now gotten to the point where people probably would not know the difference between browsing a secure SFTP site to browsing an open, insecure FTP site. Personally my favourite FTP/SFTP client is the multi-platform, open sourced FileZilla but there are many others such as SmartFTP, WinSCP or FireFTP (for Firefox).

Okay to the task at hand we will do everything in CLI (command line interface) aka shell mode.

First make sure your repository is up to date.

sudo apt-get update

Now install OpenSSH server software.

sudo apt-get install openssh-server

We then create a user group for SFTP access, I will be calling it sftponly. For security I think it is best practice not to allow accounts with SFTP access additional admission to the server using secure shell (SSH) remote log in.

sudo groupadd sftponly

Run the following to display your new group. It will be listed as the last entry.

cat /etc/group

Cat allows you to quickly display a text file while /etc/group is the file that defines the groups on the server. You should see something like this.

Output of cat /etc/group.

Each line is an individual group, you can see the name, the password which is set to x which means none, the numeric group id and users who are associated with the group. For sftponly there are currently no assigned users.

Take note of the group id, in this screenshot it is the value 1001.

We now add a new user that we will use exclusively for SFTP access.

sudo useradd [user name] -d / -g [sftponly group id] -M -N -o -u [sftponly group id]
sudo passwd [user name]

The arguments we used.

• -d is the user home directory which needs to be set to / (root).
• -g is the user group id to assign which in our example needs to be assigned to sftponly.
• -M stops the useradd command creating a home directory.
• -N useradd by default creates a group with the same name as the new user, this disables that behaviour.
• -u is the user id, which in our case needs to be the same id value as sftponly.
• -o allows duplicate, non-unique user ids.
• The passwd command sets an encrypted user password.

Add a user of your choice, I will use ben_example. To display your users.

cat /etc/passwd

Output cat /etc/passwd.

We now backup and edit the SSH Daemon configuration file.

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
sudo nano +76 /etc/ssh/sshd_config

The line

Subsystem sftp /usr/lib/openssh/sftp-server

Needs to be replaced with

Subsystem sftp internal-sftp

Now go to the end of the document, the key combination Alt / should take you there or you could simply use the Page Down key. After UsePAM Yes add the following lines to configure our sftponly group permissions and settings. The ChrootDirectory setting will confine all sftponly users to this directory. Otherwise sftponly will have access to your server root which you do not want. /var/www is often the default Debian/Ubuntu location for web servers to place their assets such as HTML, CSS files and images. Though you can use a different directory for ChrootDirectory such as /var/sftp.

Match group sftponly
ChrootDirectory /var/www
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp

Once finished use the key combination Ctrl O to save and then Ctrl X exit.

Output of sudo nano /etc/ssh/sshd_config.

Now make sure the directory you assigned to ChrootDirectory actually exists and if it does not then create it. Also the directory group and owner need to be root which it should by default if you use the command.

sudo mkdir /var/www

Output of SFTP root directory.

Now for later testing create the first of 3 directories within your ChrootDirectory.

cd /var/www
sudo mkdir test_readonly
sudo chmod 755 test_readonly

If you do not understand the chmod command shown we will quickly go through it but it is beyond the scope of this article. Change mode (chmod) parameter 755 is a permission code in octal notation. You break it up into 3 parts, 7/5/5.

1. 1st part is root permission for this directory.
2. 2nd is group permission.
3. 3rd is everyone else.

The value of the individual part grants permissions for its user. A value of 0 grants no permission, a value of 1 grants execute permissions, 2 grants write and 4 grants read. These values can be summed to create multiple permissions. So 1 (execute) + 2 (write) + 4 (read) equals 7 which grants execute, write and read aka total access. chmod 755 test_readonly means the root has total access. While users associated with the directory’s group and everyone else only have execute and read access, 4 (read) + 1 (execute) = 5. More can be read about Linux permissions here http://en.wikipedia.org/wiki/File_system_permissions#Symbolic_notation.

sudo mkdir test_readwrite
sudo chown root:sftponly test_readwrite
sudo chmod 775 test_readwrite

The above commands creates a test_readwrite directory whose owner is root and group is sftponly. Both root and sftponly members have full access permissions within test_readwrite, which allows the creation and deletion of files and sub-directories.

To remove browsing access to a directory you remove the read permission as shown below. 1 (execute) + 2 (write) = 3, execute and write access but critically no read which leaves the other two permissions redundant.

sudo mkdir test_noaccess
sudo chmod 733 test_noaccess

Restart the SSH server.

sudo /etc/init.d/ssh restart

If you don’t know your server IP address.

ip -o -f inet addr

Discover your server's IP address.

lo is your local host, eth0 is your Ethernet cable connected address, wlan0 is probably wireless.

Connect to your SSH server using an SFTP client such as FileZilla. Make sure you use the correct IP address and port number which by default is 22.

Filezilla site manager connecting to my example SFTP server.

You should be connected to the directory assigned to ChRootDirectory (var/www) in your SSHD configuration and that doubles as the SFTP client root folder. Also listed should be the test_readonly, test_readwrite and test_noaccess directories that we created earlier. Play or navigate around, hopefully you can upload files and create/delete directories within test_readwrite. While test_noaccess should be displayed but limited to browsing or download access.

Browsing your SFTP root in Filezilla.

In the screen capture below we have Filezilla’s message log with 3 sections highlighted. The orange section shows my failed attempt at accessing test_noaccess. The purple is the successful attempt at accessing test_readonly, but a failure in creating a New directory sub-folder within. While the green section shows access into test_readwrite as well as being able to create a New directory sub-folder and its subsequent removal.

Colour-coded message log from browsing our SFTP service in FileZilla.

Congratulations you now have a working example of a SFTP service running on your server. It should be mentioned that a number of these instructions were originally learnt from the blog post SFTP on Ubuntu and Debian in 9 easy steps and the reader comments.

Attempting to list a file in Linux, a case-sensitive operating system.

According to the official documentation latin1_swedish is the MySQL equivalent of the Windows cp1252 character set. For those of you who do not know Windows cp1252 character set is basically the same as the universal ISO 8859-1 / Latin1 character set.

The default MySQL 5.x collation for text.

If the above paragraph meant very little to you, a character set or character encoding scheme is a means of representing alpha-numeric and punctuation characters in an alternative form. The character A encoded in the Morse code character set is represented by a dot and dash.

The same character A as a capital in the 1963 ASCII character encoding scheme is represented by the hexadecimal 0041, the decimal 65 and the octal of 101. More importantly it also states that the character capital A is represented as the 7-bit binary code 100 0001. The basis of this 1963 uniformity is still adhered to today and allows ASCII text documents written on ancient computers to be usable on modern machines. It permits this text file http://www.textfiles.com/apple/parameters.txt from 1982 which was written on an Apple II home computer to be read in a modern web browser. B.t.w. the odd formatting in the text file is due to the Apple II’s archaic capability of only handling upper-casing letters with 40 character strings per line.

So back to the latin1_swedish, CP1252 and ISO 8859-1 character sets which I should add are for the most part interchangeable. They are sets of numeric codes (remember how we said A is represented as 41 hexadecimal or 65 decimal) which can handle the majority of characters needed by Western European languages that use the Latin alphabet. So French, Spanish, Italian and English are in but the Greek alphabet is not. You can find a list of all the characters that make up the ISO 8859-1 character set with their numeric, HTML and binary codes on ascii-code.com.

Lets go through some of these acronyms and technical terms.

• ASCII (American Standard Code for Information Interchange) is an early computing standard that was developed by the Washington based American Standards Association to exclusively handle English data communication between different computers and telecommunication equipment.
• ANSI (American National Standards Institute) escape codes were formalised in 1976 as an extension of ASCII to include colour, formatting and controls for computer display terminals (display monitors). Confusingly ANSI escape codes are also known as ECMA-48 and as ISO 6429. Former Microsoft DOS users may remember the need for the ANSI.SYS driver to handle ANSI documents.
• The ISO 8859 (International Organisation for Standardisation) is a mid-1980s European standard with a similar purpose to ASCII except that it introduced additional character sets to handle non-English languages.
• While CP1252 (Code Page) is part of a Microsoft developed guideline designed for use in the Windows operating system. Non-standard compliant code pages were initially developed by IBM as CP437 to be a collection of graphical glyphs for use on their first personal computer, the 1981 IBM-PC in the US market. Other vendors later modified or created new code pages for various non-US regions.

The ISO 8859 and Code Page series of 8-bit (1 byte) character sets were only designed to handle 256 unique characters. They had to include upper and lower casing for each character, separate characters for accents, as well as  control codes for new lines, tabs, spaces, printing etc. This size limitation meant different languages and their derivatives need their own unique 8-bit character sets. It is why there are numerous legacy sets and character schemes out in the wild including 16 different parts to ISO 8859.

Unfortunately these different sets are not interchangeable outside the basic ASCII English alpha-numeric and punctuation characters. As the sets generally assign different characters to the same identifiers. So someone in the Baltic (ISO 8859-4) reading a document written in the UK (ISO 8859-1) would see £10 displayed as Ŗ10. In ISO 8859 both £ and Ŗ share the same hexadecimal of 00A3, decimal 163, octal of 243, binary of 10100011. It means you can never have both these characters displayed together in the same document when using ISO 8859.

The solution to this glaring issue was to create a new character encoding set without the inherit 8-bit / 1 byte, 256 character limit. One that could handle all characters in a unified setup without the need to switch between different groupings of characters. The solution was the Universal Character Set but more commonly referred to as Unicode. Unicode solves all these problems with a simple and fairly obvious solution. It assigns a unique code point to every character it recognises and supports. Capital A is given U+0041, the pound sign £ is always U+00A3 and capital R with cedilla Ŗ is guaranteed to be U+0156.

These code points are converted into binary using the Unicode Transformation Format character encoding schemes. There are a number of these schemes including UTF-16 and UTF-32 but the most common is UTF-8 due to its popularity on the Internet. It partly won acceptance due to its backwards compatibility with the original ASCII standard and its variable-width encoding. Which makes it size/bandwidth friendly for many Western European texts in addition to the majority of Internet scripting and programming languages which are based on English.

The ASCII compatibility means that the 8-bit (1 byte) ASCII binary representation for capital A 01000001 is also the same UTF-8 binary representation for the code point U+0041 (capital A in Unicode). The first 127 code points in UTF-8 are all 1 byte in size. The following 1920 points are 2 bytes, then the next 65,346 are 3 bytes while the remaining 1million+ are 4 bytes. Which means both Unicode and UTF-8 can represent over 1 million unique characters yet most Latin-1 characters remain 1 byte in size.

In contrast UTF-32 which is a fix-width encoder requires 32-bits or 4 bytes of binary data for all code points including our example capital letter A (00000000 00000000 00000000 01000001). In UTF-16 the capital A in binary is 00000000 01000001, so it requires 2 bytes but there are many other code points that require a pair of 16-bit  or 2 byte values making them 4 bytes in size.

3 identical Western Latin-1 HTML files encoded using ANSI, UTF-8 and UTF-16. The green numbers are the file sizes shown in bytes.

Here is an example where UTF-8 is less efficient than UTF-16. These 3 files contain an identical Haiku in plain text saved in Japanese Hiragana script.

My advise as to which collation you should choose for your data depends on the column’s purpose. If potentially a group of international users are going to be submitting data such as their personal details, addresses, names, signatures etc. to your system then you should at least use UTF-8 to store that data.

A MySQL warning caused by attempting to save the name 'Jackie Chan' in Simplified Chinese Unicode characters with a column using the latin1_swedish_ci collation.

An attempt at displaying the Simplified Chinese Unicode.

The Unicode encoded name displays correctly when the username column is collated to utf8_unicode_ci.

On the other hand if you’re going to be keeping the data submission in-house or you know that you are not going to encounter any non Western Latin-1 characters then you should be okay sticking to the MySQL’s defaults.

Alternatively another good use for UTF-8 is where you are pulling data off the Internet and saving it locally to your MySQL database. With data fetched from website HTTP headers, RSS feeds or interactions with 3rd party API’s. You never know what data your application may run into so your system and database should be prepared for anything. But remember that each step of your setup needs to fully support UTF-8, for example …

HTML form text input encoded as UTF-8 → PHP data processing in UTF-8 → MySQL UTF-8 storage.

Additional links.

• Character Sets and Collations That MySQL Supports
• Conversion Table – Decimal, Hexadecimal, Octal, Binary
• Graphemica
• Binary Convert
• Unicode FAQ
• ANSI Character Set
• ASCII Character Set

Every WordPress blogging system provides the option to import and export data between other WordPress blogs. This is not only limited to the blog entries themselves but can also include the post’s categories, tags, comments, drafts and even spam! It does all this using the WordPress Extended Rss document format, WXR.

The WXR format is based on the Really Simple Syndication or Rss specification which is a very popular dialect of XML. It has been designed as a syndication format for websites who wish to share and serialise some of their data. http://www.rssboard.org/

A web syndication specification might seem an odd choice for a blog exporting tool but Rss popularity on today’s Internet, its simplicity and its expandable format through the use of 3rd party extensions make it a great choice. Being an XML dialect also means you can open up any text editor and have complete access to all blog data in a mark-up format that is human readable (in a layout not too different from a HTML file.)

To create a WXR export file of your own you need to login into your WordPress Dashboard, scroll down to Tools and select Export. A filter option allows you to drill down to specific data to trim your export file size. If you are exporting the complete site I’d recommend changing the Statuses filter to ‘Published’. If left as ‘All Statuses’ all the blog’s redundant auto-saved entries will be included, which ineffectively duplicate the published articles.

Once you have pressed Download Export File button and it has finished downloading you should have an XML document with the name of wordpress-[yyyy]-[mm]-[dd].xml. You can open this with any text editor or even Windows Notepad. But it is preferable that you use a text editor that can parse the XML document for colourisation as it makes the document much easier to read. NotePad++ http://notepad-plus-plus.org/ is a good choice for Windows users while TextMate http://macromates.com/ is probably the best choice for OS/X.

As the title suggests in this post I will attempt to decode the content of the WordPress Extended Rss document. This means I will list in published order the Rss elements contained within a standard export and briefly describe their purpose.

This will not be a tutorial on XML or Rss and I will assume you have some understanding of either. However if this is not the case things should not be too hard to follow especially for those people familiar with HTML documents.

<!-- This is a WordPress eXtended RSS file generated by WordPress as an export of your blog. -->

At the top of the WXR file there is a large commented section explaining the purpose of the document and in case you have forgotten instructions on how to import the file into a WordPress blog.

Beyond the comments is the required <rss> element containing 5 namespace extensions as well as the Rss version as a numeric value. The extensions listed include the RDF site summary content module, the well-formed web comment API, the Dublin Core metadata element set and 2 WordPress extensions. If this isn’t making too much sense then don’t worry as it is not really important unless you are developing an Rss parser.

The namespaces listed are unique, with each serving specific functions that the base Rss specification does not cover. Each XML namespace starts with xmlns: and is followed by an abbreviated title of the namespace which is usually an acronym. The URL that follows each title is a requirement and should point to a webpage that provides further information on the namespace.

Xmlns:dc="http://purl.org/dc/elements/1.1/"

Is an example of the Dublin Core element set namespace.

Below the <rss> element is the <channel> container element. This holds all the child elements and data related to the WordPress blog. You can find the closing </rss> element at the bottom of the Rss document. At the top of the <channel> we have the elements that are associated with the WordPress blog metadata.

<title> Contains the site title of the blog.

<link> Is the URL of the blog as determined by WordPress.

<description> Is a tagline that can be modified in the Dashboard under General Settings.

<pubDate> Was the time and date that the WXR document was created. It is in the RFC-822 format http://asg.web.cmu.edu/rfc/rfc822.html as required by the Rss standard. The format should be self explanatory except for the last numeric value which represents the local differential from GMT using a +/-hhmm format. Plus 2 hours from GMT would be represented as +0200. The WordPress time zone can be changed in the Dashboard under General Settings, Timezone.

<generator> Is the name or a URL pointing to the homepage of the application that was used to create the Rss document.

<language> Is the primary language the blog is written in as determined by General Setting, Language in the WordPress Dashboard. A list of valid codes used to represent the language can be found at http://www.rssboard.org/rss-language-codes.

<wp: wxr_version> This is our first example of an extended Rss element. We can recognise that it does not belong to the Rss specification as the element contains a colon. Left of the colon contains the elements extension while right is the element name. wp:wxr_version is the version number for the WordPress extension Rss.

<wp:base_site_url> Is the root URL  of the WordPress hosting provider.

<wp:base_blog_url> Is the root URL of the WordPress blog.

<wp:category> Contains a complete collection of categories associated with the blog. You can view and edit the list within the Dashboard under Posts, Categories. Each category is given its own <category> element and contains the following 3 child elements.

• <wp:category_nicename> Is the category name in a URL friendly format.
• <wp:category_parent> If the category belongs to a hierarchy then the parent category is listed.
• <wp:cat_name><![CDATA[]]> The original name of the category contained within a <![CDDATA[   ]]>. The CDATA or character data enclosure tells the XML/Rss parser not to process the text contained within. This is a safety measure in case the text contains any illegal characters that could generate errors. http://www.w3schools.com/xml/xml_cdata.asp

<wp:tag> Contains a complete collection of the blog post tags. You can view and edit the post tags within the Dashboard under Posts, Posts Tags. It contains the following 2 child elements.

• <wp:tag_slug> Is the URL friendly name of the tag.
• <wp:tag_name> Is the original name of the tag contained within a character data enclosure.

<cloud> Is a pointer to the RssCloud API which is a blog monitoring service supported by WordPress.com. It enables a supporting client to receive instant notification when the blog is updated. http://www.rssboard.org/rsscloud-interface

<image> Is a logo belonging to the site that can be displayed by Rss clients. You can modify the logo under the General Settings, Blog Picture / Icon dialog in the Dashboard. There are strict size and image formats requirements imposed by the Rss standard. http://www.rssboard.org/rss-specification#ltimagegtSubelementOfLtchannelgt

<atom:link rel=”search”> Is a URL pointing to the Open Search description document supplied by WordPress. It enables supported Rss clients and web browsers an easy means to provide search terms to the blog and receive results in a standardised XML format. http://www.opensearch.org/Specifications/OpenSearch/1.1#OpenSearch_description_document

<atom:link rel=”pub”> Is a URL pointing to the Google designed pubsubhubbub notification service that is supported by WordPress. In my opinion this is easier to implement and use then the alternative <cloud> service that offers similar functionality. http://code.google.com/p/pubsubhubbub/

That is the end of the Rss metadata related elements. Below are the list of child elements contained within the <item></item> elements. Items are repeated multiple times as each item holds a single blog post, article or page.

<title> Title of the blogpost or page.

<link> URL to the blogpost or page.

<pubDate> Time and date that the post was posted online.

<dc:creator> Lists the author of the post. The element is a Dublin Core Rss extension as the Rss specification doesn’t contain any suitable elements for this role.

<category> Each category associated with the blog is given 2 category elements. The first element contains just the category as a name, while the second element contains both the category name and the URL friendly nicename attribute.

<guid> Is the globally unique identifier used for the identification of the blog post by Rss and WordPress clients. The isPermaLink=false attribute just means that this identifier is not a legitimate website URL and is not usable in a web browser.

<description> In Rss documents this element contains the synopsis of the item but in WXR it is left blank.

<content:encoded> Is the replacement for the restrictive Rss <description> element. Enclosed within a character data enclosure is the complete WordPress formatted blog post, HTML tags and all.

<excerpt:encoded> This is an unknown elementThis is a summary or description of the post often used by RSS/Atom feeds..

<wp:post_id> This is an auto-incremental, numeric, unique identification number given to each post, article or page.

<wp:post_date> Time and date that the post was published.

<wp:post_date_gmt> Time and date in GMT that the post was published.

<wp:comment_status> A value stating whether public access for posting comments is opened or closed.

<wp:post_name> Is a unique, URL friendly nicename based on the post title.

<wp:status> Publish status of the post with the options; ‘publish’, ‘draft’, ‘pending’,’private’.

<wp:post_parent> The numeric identification number if the post’s parent. This I think is applicable to WordPress pages which can be nested within each other.

<wp:menu_order> I assume is related to menu navigation of nested pages.

<wp:post_type> Post type either ‘post’, ‘page’,’media’.

<wp:post_password> A non-encrypted password used by WordPress to restrict reading access to the post.

<wp:is_sticky> A numeric Boolean value (0 = false, 1 = true) to determine if the post as a sticky. A sticky post means the post will always be displayed at the top of any list of posts.

<wp:postmeta> Are containers for newer additions the WXR document format that have been introduced  after the original WXR specification. Each <wp:postmeta> element contains 2 child elements.

• <wp:meta_key> Is the reference key for the meta data element.
• <wp:meta_value> Is the value for the meta data element contained within a character data enclosure.

Below is a list of the <wp:meta_key> references currently used by WXR.

delicious; is data related to the Delicious social bookmarking web service. http://www.delicious.com/

geo_latitude; is the positioning location of the author when submitted the post. The value is the latitude in degrees using the World Geodetic System 1984 (WGS84) datum. It seems to be based on the Google Gears Geolocation API. http://code.google.com/apis/gears/api_geolocation.html

geo_longitude; is the positioning location of the author when they submitted the post. The value is the longitude coordinates.

geo_accuracy; is the horizontal accuracy of the above positioning values in metres.

geo_address; is the address determined by the above geolocation data.

geo_public; is a Boolean numeric value that determines if the geolocation data should be displayed in the post.

email_notification; is an unknown value related to the email notification service for posting comments.

_wpas_done_yup; is an unknown numeric Boolean value.

_wpas_done_twitter; is an unknown numeric Boolean value related to Twitter.

reddit; is data related to the reddit social news web service. http://www.reddit.com/

_edit_last; is an unknown reference.

_edit_lock; is an unknown reference.

<wp:comment> Is a child element for the post item that contains 12 sub-elements listed below. These sub-elements belong to the a single post comment contained within a <wp:comment> element set.

• <wp:comment_id> This is an auto-incremental, numeric, unique identification number given to each comment.
• <wp:comment_author> The name of author who submitted the comment. The name value is contained within a character data enclosure.
• <wp:comment_author_email>An e-mail address provided by the author of the comment.
• <wp:comment_author_url> The URL of the author’s website provided by the author of the comment.
• <wp:comment_author_IP> The IP address belonging to the author of the comment. The IP address is automatically recorded by WordPress.
• <wp:comment_date> The date and time local to the blog that the comment was posted.
• <wp:comment_date_gmt> The date and time at GMT that the comment was posted.
• <wp:comment_content> The comment text enclosed within a character data enclosure.
• <wp:comment_approved> A numeric Boolean value to determine if the comment is displayed.
• <wp:comment_type> The type of comment. If left blank it is classed as a normal comment otherwise a value of ‘pingback’ means it is a post request notification link. http://en.wikipedia.org/wiki/Pingback
• <wp:comment_parent> The numeric identification of the parent comment used when the comment is a response to a pre-existing comment.
• <wp:comment_user_id> A numeric identification belonging to the author if they were logged in when they submitted the comment.

Hopefully that extensive list helps you out. It should be current with all the main elements in a standard WordPress Extended Rss document been covered. If you find any mistakes, errors or know the purpose of any of the unknown elements please leave a comment.

Software

Due to the potentially large collection of DVD discs many people may own this project had few important requirements for the software.

• All DVD copy protection and region encoding needed to be removed.
  Readers in the USA should note that under the Digital Millennium Copyright Act it is illegal for most consumers to bypass or break any form of digital copy protection. Legally there is no personal fair use granted to consumers in the USA to bypass or break DVD copy protection schemes for the discs they own. This means that by following these instructions, technically readers located in the USA will most likely be breaking the law.
• Only the movie and primary audio tracks were to be ripped to the PC with out any extras or menus.
• The movie data needed to copy over as quickly as possible. We needed to transfer the movie in its original format with no transcoding or additional compression.
• Streamlined ripping interface to keep user prompts to a minimum.
• Free.

I ended up going with DVDFab HD Decrypter software by Fengtao Software which can be found at http://www.dvdfab.com/hd-decrypter.htm. DVDFab is an easy to use tool for DVD/Blu-ray ripping and conversion. While most the DVDFab suite is shareware you do not need to pay for software as the HD Decrypter component we use is freeware.

Requirements

• A Windows PC with a DVD or compatible disc drive which is able to read your DVD discs.
• DVDFab HD Decrypter software by Fengtao Software mentioned previously which can be found at http://www.dvdfab.com/hd-decrypter.htm.
• A large hard drive big enough to hold your transferred collection. Two identically sized drives would be preferred, one to hold the movie collection and one to clone the hard drive for use as a backup. You would hate to transfer over 500 discs and then loose it all to a faulty disk drive! How much space you’ll need depends on your collection. On average a movie will require between 3.5 to 7 GB though some longer movies (The Godfather Part II, Lord of the Rings etc) do span multiple discs. As a conservative estimate you’ll need about 5 GB per movie or a terabyte for every 200 movies.
• Media centre or Home Theatre PC software to organise and play your collection. This software needs to be able to read the DVD VOB container format (http://en.wikipedia.org/wiki/VOB) when stored on a computer hard disk. My recommendations would include Boxee (http://www.boxee.tv/), VLC Player (http://www.videolan.org/vlc/) or XBMC (http://xbmc.org/).

Install and Run

1. Download and install DVDFab from http://www.dvdfab.com/hd-decrypter.htm.
2. Start up DVDFab.
3. On the DVDFab Welcome Screen check Do not show again and click Start DVDFab. We will not be using any of the trial software for this project.

Settings and Customise

1. Select the green Settings button that contains a tick.
   
2. Go to General. You may want to check Memorize User Interface Options. Change your Output directory. I am using M:\New Rips but you could use something that is easy to remember such as C:\Movies. Uncheck Creates files in subfolders of output directory.
3. If you have a graphics card with a GPU by ATI/AMD or Nvidia you can use these to improve the ripping throughput. While you are in General, select the sub-menu A/V Codex. Under the Video Decoder section find MPEG2 and change its mode from software to either DXVA for ATI/AMD cards or CUDA for Nvidia cards.
4. Go to DVD Copy and change the Default output size from DVD5 to DVD9. DVD5 refers to a single sided disc and is only useful if you want to burn your movie to a DVD-R. As we are just ripping to a hard disk we do not need to shrink or split the DVD rip to a smaller factor. Otherwise this could be a time consuming and a not very productive process.
5. Change the Default output type from DVD writer to DVD folder.
6. If by default you do not want any subtitles. Go to DVD / Blu-ray, Streams, Subpicture and check Do not preselect any subpicture track.
7. Then press OK to save your changes and close the settings dialog.

Ripping A Disc

1. Now you can either rip the Full Disc using the current default selection. Or we can save hard disk space and time by just extracting the movie with selected audio tracks by ignoring all the extra features, advertising, menus, etc. To rip the movie rather than the whole DVD select Main Movie from the DVD Copy menu on the left.
   
2. Change the Target from the default to your preference.
   
3. Insert the DVD disc you wish to copy into your computer’s DVD drive.
4. If you get a Windows AutoPlay prompt when you insert the disc I’d suggest turning it off.
   
   
5. When prompted by DVDFab select the DVD disc region. Feel free to check the Do not show again. If you ever need to change this in the future you can go to Settings, DVD / Blu-ray, Protection, DVD.
6. DVDFab will spend a few moment scanning the disc.
7. In my example I am happy with the default options. DVDFab is intelligent enough to select the movie on the disc as well as the English audio and sound tracks. Your mileage maybe different depending on which DVD you use.
8. DVDFab lets you know how much hard disk space the rip is going to use. You can decrease this size by removing audio tracks you don’t need. If you want an explanation of the most common types of audio tracks, see the Audio Tracks paragraph near the end of this article.
9. When you are ready press Next.
   
10. Now on the review screen check to make sure your Volume Label is the actual title of the movie. Some discs use very generic labels and this will be used by DVDFab for the movie’s folder name. I recommend using the following naming format .. Title.(Year).Part
    i.e. Die.Hard.(1988), The.Godfather.Part.II.(1974).cd1, The.Godfather.Part.II.(1974).cd2.
    You will generally find the movie’s year of theatre release on the back of the DVD case.
    

Ripping Process

1. Once you are ready press Start.
2. Depending on your computer hardware and the movie length the ripping process should take somewhere between 10-20 minutes.
3. When your rip is complete DVDFab will play a notification audio and eject the DVD disc. From here you can select Finish to start the ripping process over again with a new movie.
4. You can use Windows explorer to go to your movies folder to browse over your completed DVD transfers.
   

Media Centres – Boxee

If you want a media centre platform to display and play your DVD rips I’d recommend Boxee for first time users. The Boxee software (http://www.boxee.tv/make) is quite well known due to the recently released and uniquely designed Boxee Box by Logitech (http://www.boxee.tv/box). While the Boxee Box hardware is geared towards fetching and streaming online content for use on a TV. The standalone Boxee software is a less focused, more all round media centre which is what you will probably want. One potential negative is that while the Boxee software is free does require you to create a user login account to run the required social elements of the software.

Media Centres – XBMC

The alternative to Boxee is the free and open source XBMC (http://xbmc.org/). XBMC was once an acronym for XBox Media Centre but now days it is primarily a piece of PC software since the original XBox has long fallen out of use. Boxee is actually a software fork of XBMC and so in many ways the two are quite alike. XBMC is much more customisable but the trade off is greater complexity.

If you do plan on using XBMC I’d recommend changing the default Confidence skin to something else. Confidence and a couple of other skins such as Transparency! and Xpreience do not understand the folders generated by our ripping process.

To change the XBMC skin use your keyboard arrow keys to navigate down to System and press Enter. Then navigate to Appearance, Skin, then use the right arrow key and enter or mouse to select Skin again. In the skin dialog select Get More… and install the following skins, Back Row, Mediastream Redux, Night, pm3.hd, Shade to try each of them out. Remember just press Esc a few times to quickly return to the XBMC Home screen.

If you need to add a folder on your computer to the library goto System, Video, Library, Import video library.

The Settings system menu in XBMC is a bit confusing at first but you get the hang of it eventually.

Region Locking Issues

1. I won’t go into the history of region locking other than to say that it was annoying attempt by Hollywood Studios to stagger DVD pricing and release dates for various international markets (http://en.wikipedia.org/wiki/DVD_region_code).
2. If part of your DVD collection does not match the region setting of your computer’s DVD drive you may find those mismatched discs cannot be ripped by DVDFab.
3. The simplest way to get around this problem is to buy a cheap, secondary DVD-ROM drive and use Windows to change the region setting on it. Internal Desktop DVD-ROM drives can be had for $15 USD and slower USB external drives at about double that.
   
   
4. Otherwise you could try convert or hack the DVD drives Region Playback Control but this is potentially risky if you are not comfortable with what you are doing. I do not recommend attempting to flash a laptop DVD drive. If the drive flash fails you could be left with an expensive piece of hardware to replace or dead weight in your machine. Flashing the drive ROM means you replace the internal software (firmware) that operates the drive. A corrupt or broken firmware usually means your DVD disc drive will be irreversibly broken. More information on DVD RPC replacements can be found at http://www.doom9.org/index.html?/rpc1.htm.

Audio Tracks

Audio on DVD can be a pretty complicated and confusing topic as the DVD specification allows for 4 different formats. PCM, DTS, MPEG-2 Audio Layer II (MP2) and Dolby Digital AC-3. In addition to these you may have different tracks of the same format included for different speaker setups. It is common to see AC-3 5.1 and AC-3 2 tracks on the same disk. AC-5.1 is optimised for 5 speakers in surround (the .1 is a sub-woofer) while AC-3 2 is geared towards a traditional stereo 2 speaker setup. Older cinema releases (those prior to mid-1990s) will often use AC-3 2 as the original theatre stereo track or AC-3 1 as the original theatre mono track.

Dolby Digital AC-3 is the most common and supported format. It can be the considered the lowest common denominator and should be supported by all audio hardware. If you are trying to optimise space or to keep things simple go for the AC-3 5.1 track or AC-3 2 if no 5.1 track is available. If you are converting an older movie that was released to theatres in stereo or mono you may wish to include both the remixed for DVD surround sound track as well as the original theatre track.

MPEG-2 Audio Layer II otherwise known as MP2 is a rare alternative to the Dolby standard only occasionally found on PAL region 2/4 DVDs. Technically its specs are greater than those of AC-3 but you will need to make sure it is supported by your audio hardware. Personally I would choose AC-3 over MP2 just for the greater compatibility.

DTS is a well supported alternative format to AC-3. DTS is commonly found in cinemas and has been designed to scale well on many different audio playback setups. If given the choice I would choose DTS over AC-3 though the trade-off is that significantly greater hard disk space is needed for your movie. Whether you actually notice an improvement of your audio playback in DTS over AC-3 is debatable and is probably dependant on your audio hardware.

PCM is an exotic format that is rarely found in movie DVD releases though occasionally found in music video titles. Unlike the other 3 standards PCM does not use lossy compression (http://en.wikipedia.org/wiki/Lossy_compression) so in theory is a higher quality and more pure audio format. It is not unlike the audio only compact discs though PCM on DVD has the capacity for a much higher playback quality than CDs. If given the choice of a compressed format over an uncompressed, I will take the uncompressed most times. Unfortunately just like DTS you will trade-off hard disk space for possibly only minor perceivable audio improvements which also will be dependant on your hardware. Also it should be noted that while PCM supports up to 6 audio channels (surround sound) it is usually only implemented as a 2 channel stereo track.

Well I hope this was of some help to the many avid DVD collectors out there who may want to move their collections over to a more flexible Home Theatre PC system. If you have any questions please post them in the comments section.

In File Explorer you need to find the Mass Effect launcher …

C:\Program Files (x86)\Steam\steamapps\common\mass effect\Binaries\MassEffect.exe

Right click the ‘MassEffect.exe’ program and bring up Properties.

• Select the Compatibility tab.
• Within the Compatibility mode section select the Run this program in compatibility mode for check-box.
• Make sure the pull-down list has Windows XP (Service Pack 2) selected.
• Now select Run this program as an administrator check-box in the Privilege Level section.
• Press OK.

You should now be able to run the game from within Steam without any problems. Once you have launched Mass Effect for the first time I found you could disable the Run this program as an administrator check-box, that way you are not prompted for administrator rights every time you run the game.

On visiting with a standard 32-bit edition of Internet Explorer, Firefox, Chrome, etc you’re only be given the option of downloading the 32-bit compatible plug-in. The probable reasoning behind this was to avoid confusion with 64-bit Windows owners. Who might not realise that their browsers are 32-bit programs running within a 64-bit operating system, making the 64-bit plug-ins useless.

As of writing there are currently are no active 64-bit Windows editions of Chrome, Firefox or Opera. Microsoft conveniently includes a 64-bit copy of Internet Explorer 7/8 in all 64-bit editions of Windows Vista and 7. Doing a quick Windows search for ‘Internet Explorer’ or ’64-bit’ should reveal it and with that you should be able to obtain a 64-bit edition of Java. You should also be able to install and run both editions of Java on the same operating system.

Now I have the correct edition of Java I can run the 64-bit version of Eclipse 5.6 (Helios) without any of the cryptic “Failed to load the JNI shared library” errors!

We will require the popular Url Rewrite Filter (http://tuckey.org/urlrewrite/).

cd ~
wget http://tuckey.org/urlrewrite/dist/urlrewritefilter-2.6.zip
unzip -x urlrewritefilter-2.6.zip -d urlrewrite
cd urlrewrite/WEB-INF
ls

Now you should see listed a lib directory and the file urlwrite.xml. Both these need to be copied into your website’s root /WEB-INF directory.

cp -r * /var/www/mysite.com/WEB-INF
cd /var/www/mysite.com/WEB-INF
nano web.xml

Copy and insert this code below into your web.xml. It will tell your server to load the Url Rewrite Filter and to recheck your urlwrite.xml every 60 seconds for changes. This is important in case in the future you need to re-enable the Railo administration without the requirement of restarting your server.

<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    version="2.5">

    <!-- URL ReWriter -->

    <filter>

        <filter-name>UrlRewriteFilter</filter-name>

        <filter-class>org.tuckey.web.filters.urlrewrite.UrlRewriteFilter</filter-class>

        <!-- set the amount of seconds the conf file will be checked for reload

        can be a valid integer (0 denotes check every time,

        empty/not set denotes no reload check) -->

        <init-param>

            <param-name>confReloadCheckInterval</param-name>

            <param-value>60</param-value>

         </init-param>

         <!-- you can disable status page if desired

         can be: true, false (default true) -->

         <init-param>

            <param-name>statusEnabled</param-name>

            <param-value>false</param-value>

         </init-param>

     </filter>

     <filter-mapping>

        <filter-name>UrlRewriteFilter</filter-name>

        <url-pattern>/*</url-pattern>

     </filter-mapping>

</web-app>

Now edit your urlwrite.xml to look something like this.

<?xml version="1.0" encoding="utf-8"?>

<!DOCTYPE urlrewrite PUBLIC "-//tuckey.org//DTD UrlRewrite 2.6//EN"

        "http://tuckey.org/res/dtds/urlrewrite2.6.dtd">

<!--

    Configuration file for UrlRewriteFilter

    http://tuckey.org/urlrewrite/

-->

<urlrewrite>

<rule enabled="true">

	<name>Disable Railo</name>

	<from>^/railo-context/admin/(.*)$</from>

	<to>null</to>

	<set type="status">403</set>
</rule>

</urlrewrite>

You can now restart your server and then try to access http://www.mysite.com/railo-context/admin/web.cfm or http://www.mysite.com/railo-context/admin/server.cfm. Hopefully if all went well both pages will return a HTTP status 403. If you need to re-enable these pages, edit your urlwrite.xml and change the rule enabled attribute value to false.