Working with remote branches

Note: Better description here.

Our current project is held in a git repository. It is set up for anonymous (cvs/subversion style) push/pull operation using an ssh keypair so that we don’t need to enter password.

Recently we have been experimenting with branches. Two main branches exists: origin (which represent the serverside copy) and master (which represents the clientside copy). In git, ordinary branches exists only on the clientside and are not transferred when pushing. The solution is to use remote branches. A few commands for listing available branches:

(A) git branch
(B) git branch -r
(C) git branch -a

They list local branches (A), remote branches (B) and both local and remote branches (C). An asterisk is used to mark the current branch.

One does not work directly on a remote branch. Instead you create a corresponing local branch which tracks the remote branch. The syntax is

git checkout --track -b LOCAL_BRANCH_NAME origin/REMOTE_BRANCH_NAME

where LOCAL_BRANCH_NAME is the name of the local tracking branch and REMOTE_BRANCH_NAME is the name of the the remote branch.

Pulling works as expected, but pushing to a remote branch is a bit more complicated than one might expect. The syntax is

git push origin LOCAL_BRANCH_NAME:REMOTE_BRANCH_NAME

I’m sure there exists some fancy way to store this information so that a ‘git push‘ would do as expected, but at the moment we will cope.

Full description at this page.

Man pages of interest:
git branch
git push
git checkout

References:
Checking out a remote branch
Pushing to a remote branch (see examples)

WordPress and source code inclusion

Unless you know how to bend this trick, wordpress will do a very decent job at messing up everything resembling source code. This is especially true if the language you are using depends heavely on indentation. Python springs to mind.

The trick is – in the ‘Visual’ editing mode – to encapsulate the code in these tags

…

While entering these tags, the visual editor should report ‘Path: p‘. The value of the language attribute should – of course – be chosen to match the style of the code being included. The first link on this page lists the supported languages.

Leading spaces seems to be removed in the visual editor, but not in the code editor. So one has to do some flip-flopping … be carefull though, the interface is very eager to forget about these spaces.

Running CGI as a different user under Apache 2

The following was written with RedHat Fedora Core 6 (FC6) in mind, but should not represent too much misguidance for most other Linux distributions.

The problem: We have some CGI which has to be executed as a particular user, which is not the same as the one running the web server.

The solution: Configure apache to have per-user cgi-bin’s and use suexec.

First lets install Apache and set it up to start at boot time. In RedHat apache is called httpd (as if there isn’t any other webservers out there), so the commands you are looking for are:

root@server$ yum install httpd httpd-manual httpd-devel
root@server$ /sbin/chkconfig httpd on
root@server$ /sbin/service httpd start

To get some info on the apache installation run the following command:

root@server$ /usr/sbin/apachectl -V

In particular, notice the SUEXEC_BIN option. If a suexec executable is found at this location all per-user CGI and SSI is run as the appropriate user. According to this (section 16.1) suexec on Fedora Core 6 is set up for a document root (DOCROOT) of /var/www. There might be a problem using different DOCROOT’s.

To make sure that the server actually started:

root@server$ ps auxwww | grep apache

As the default FC6 apache is configured to pre-fork you should see a lot of /usr/sbin/httpd processes.

Configure apache (/etc/httpd/conf/httpd.conf) to have per-user public_html dirs by modifying the ‘IfModule mod_userdir.c’ section to contain:

<IfModule mod_userdir.c>
#
# UserDir is disabled by default since it can confirm the presence
# of a username on the system (depending on home directory
# permissions).
#
# UserDir disable#
# To enable requests to /~user/ to serve the user's public_html
# directory, remove the "UserDir disable" line above, and uncomment
# the following line instead:
#
UserDir public_html

</IfModule>

Also add per-user cgi-bin's to the same file:

#
# Per-user cgi-bin
#
<Directory /home/*/public_html/cgi-bin/>
Options ExecCGI
SetHandler cgi-script
</Directory>

Make sure that the directories for the appropriate user exists:

user@server$ mkdir ~/public_html
user@server$ mkdir ~/public_html/cgi-bin

Restart apache to make changes take affect:

root@server$ /etc/init.d/httpd restart

Testing: Place the following in a file called /home/user/public_html/cgi-bin/whoami.cgi:

<pre>#!/usr/bin/python

import cgi
import os

# for form access
import cgitb; cgitb.enable()

form = cgi.FieldStorage()

# get username
execution = os.popen('whoami', 'r', 0)
whoami = "".join(execution.readlines())
execution.close()

# print header
print "Content-Type: text/html"     # HTML is following
print                               # blank line, end of headers

# print html
print "<html>"
print "  <head>"
print "    <title>Whoami</title>"
print "  </head>"
print "  <body>"
print "    <h1>Whoami</h1>"
print "    "+whoami
print "  </body>"
print "</html>"</pre>

Give it the right permissions:

user@server$ chmod a+x /home/user/public_html/cgi-bin/whoami.cgi

And direct your browser to:

http://server/~user/cgi-bin/whoami.cgi

McCabe-style cyclomatic complexity in Python code

Cyclomatic complexity is a software metric for measuring source code complexity by analyzing the number of linearly independent paths through a function (or full program). This meassure corresponds to the effort needed to do a full path by path test of the code. If such tests are not done (which – without automation – clearly will be the case for pieces of code with large number of paths) it will be degraded to a nice indicator for the trustworthiness of the code.

Such a complexity analyser has been written for python (in perl). The files has to be saved under the correct names (complexdef, complexdefclass, complexity and tab) and made executable.

One of these scripts depends on ksh. This is not part of a default Ubuntu Gutsy Gibbon installation but can easely be added by running:

sudo apt-get install ksh

Another issue seems to be that the perl syntax for the sort function has evolved, so that lines of the form

open (T,"|$dn/tab|sort -n +5") || die "no tab?";

has to be changed to

open (T,"|$dn/tab|sort -n -k 5") || die "no tab?";

This issue is present once in complexdef and twice in complexdefclass. For some reason there seems to be a syntax error in the definition of the usage function of the tab file. Replacing from the ‘sub usage {' line to the end with the following seems to be working:

sub usage {
print “dummy value\n”;
}

A nice wrapper script has been written in python. When this is configured (by changing PATH_TO_COMPLEXITY) it can be run by

./complexity.py PATH_TO_PY_FILES

Creating SSH tunnels from within Python

The following script creates a secure tunnel over localhost (could be any ssh-capable host you have a login to) pointing to port 80 at google.com. The target could be any port at any host reachable from our gateway host.

We are using the pexpect module to do bidirectional communication with the ssh-command itself. The command basically translates to this one:

ssh tmp@localhost -L 4000:google.com:80 -N

This command will ask for the password of tmp@localhost and we will need to interprete the output of the command in order to figure out if it was successful (e.g. the port could be in use). Hence the need for two-way communication.

If the port turns out to be in use we scanforward through the range [4000,4050]. And the code:

#!/usr/bin/python</code>

# http://pexpect.sourceforge.net/pexpect.html
# http://www.thescripts.com/forum/thread26248.html

import pexpect
import time

thetimeout = 2
port_lower = 4000
port_upper = 4050

def create (via_host, via_user, via_pwd, to_host, to_port):
  for via_port in range(port_lower, port_upper+1):
    cmd = 'ssh '+via_user+'@'+via_host+' -L '+str(via_port)+':'+to_host+':'+str(to_port)+' -N'
    child = pexpect.spawn(cmd, timeout=thetimeout)
    r = child.expect(['.ssword:*', 'Privileged ports can only be forwarded by root.*', pexpect.EOF, pexpect.TIMEOUT])

    if r == 0:
      time.sleep(0.1)
      child.sendline(via_pwd+'\n')
      r = child.expect(['bind: Address already in use.*local forwarding', pexpect.EOF, pexpect.TIMEOUT])

      if   r == 0 or r == 1:
        pass
      elif r == 2:
        return (child, via_port)
      else:
        print "Error: Unknown result from SSH tunneling attempt."
        pass
    else:
      print "Error: You are trying to open a privileged port as an unprivileged user."
  print "Error: Unable to bring up a tunnel within port range ["+str(port_lower)+","+str(port_upper)+"]."
  return (None, -1)

child, port = create("localhost", "tmp", "tmp", "google.com", 80)
print "port = "+str(port)
while (1):
  time.sleep(2)

It seems that WordPress does an excellent job at messing up code inclusions and we are not allowed to upload anything but jpg, jpeg, png, gif, pdf, doc, ppt, odt files. Since the above is python code (where indentation matters) you will have to be creative, sorry . Found a way around this.

Flymake for emacs python-mode

I accidentally stumbled upon this blog entry pointing to an EmacsWiki page explaining how to set up emacs to do on-the-fly checking of python code. Basically what it does is to pass the buffer contents to an external program and, based on the result, change the background color to indicate which lines resulted in warnings/errors. Pretty sweet!

Perhaps we should consider going back to emacs …

Howto install Oracle 11g on RedHat Fedora Core 6 (x86)

This guide describes the issues we encountered while installing Oracle on Redhat FC6 (which is not certified by Oracle). We found a related guide for Redhat Enterprise 4 here and another one for FC7 here.

Preparation
First, download “Oracle Database 11g Release 1 (11.1.0.6.0)

Standard Edition, Standard Edition One, and Enterprise Edition” for linux x86 from this page (1.7GB). Installation will take up approx 3.3 GB + 1.5 GB for some sample database.

The installation program won’t run as root, so you have to be an ordinary user. It will, however try to access /etc/oraInst.loc (an act which few ordinary users are allowed to). So lets say your ordinary user is testdb then, as root, run:

touch /etc/oraInst.loc
chown testdb /etc/oraInst.loc
chgrp testdb /etc/oraInst.loc

To start the installation program run:

unzip linux_11gR1_database.zip
cd database
./runInstaller

When checks fails

After having done most of the installation, the installer began doing some checks. Two of these failed for us. The description was:

Output generated from configuration assistant "Oracle Net Configuration Assistant":
Command = /mnt/sdc/testdb/oracle/product/11.1.0/db_1/bin/netca /orahome /mnt/sdc/testdb/oracle/product/11.1.0/db_1 /orahnam OraDb11g_home1 /instype typical /inscomp client,oraclenet,javavm,server,ano /insprtcl tcp /cfg local /authadp NO_VALUE /nodeinfo NO_VALUE /responseFile /mnt/sdc/testdb/oracle/product/11.1.0/db_1/network/install/netca_typ.rsp
UnsatisfiedLinkError exception loading native library: njni11

Configuration assistant “Oracle Net Configuration Assistant” failed
—————————————————————————–
The “/mnt/sdc/testdb/oracle/product/11.1.0/db_1/cfgtoollogs/configToolFailedCommands” script contains all commands that failed, were skipped or were cancelled. This file may be used to run these configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.—————————————————————————–Output generated from configuration assistant “Oracle Net Configuration Assistant”:
Command = /mnt/sdc/testdb/oracle/product/11.1.0/db_1/bin/netca /orahome /mnt/sdc/testdb/oracle/product/11.1.0/db_1 /orahnam OraDb11g_home1 /instype typical /inscomp client,oraclenet,javavm,server,ano /insprtcl tcp /cfg local /authadp NO_VALUE /nodeinfo NO_VALUE /responseFile /mnt/sdc/testdb/oracle/product/11.1.0/db_1/network/install/netca_typ.rsp
UnsatisfiedLinkError exception loading native library: njni11

Configuration assistant “Oracle Net Configuration Assistant” failed
—————————————————————————–
The “/mnt/sdc/testdb/oracle/product/11.1.0/db_1/cfgtoollogs/configToolFailedCommands” script contains all commands that failed, were skipped or were cancelled. This file may be used to run these configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.—————————————————————————–

The .../configToolFailedCommands file included the following explanation:

UnsatisfiedLinkError exception loading native library: njni11

java.lang.UnsatisfiedLinkError: jniGetOracleHome
at oracle.net.common.NetGetEnv.jniGetOracleHome(Native Method)
at oracle.net.common.NetGetEnv.getOracleHome(Unknown Source)
at oracle.net.ca.NetCA.main(Unknown Source)
Oracle Net Services configuration failed. The exit code is -1

While not very informative UnsatisfiedLinkError and Jni (Java Native Interface) indicated that the issue had something to do with a C/Java binding of some sort. In the Enterprise 4 guide (see top of this note) a list of required packages (with very specific version numbers) were listed. Among those was libaio and libaio-devel, which contains Kernel Asynchronous I/O (AIO) Support for Linux.

This is definitely C territory and would need some binding to be made available to Java. So we decided to install these packages. As root, run

yum install libaio-devel libaio

And everything seems to be spinning.

Pending warnings

We still have an unresolved warning on not having enough swap space. Got plenty of the to add a swapfile it it becomes an issue.