Monologue

I've just completed the 19th new skin of my campaign, mitchinson-golden, which can be seen on demo.mojoportal.com

One more to go and I'm finished with skinning for a while!

I've just completed the 18th new skin for mojoPortal, far exceeding my original goal of 10 new skins. This one is named mitchinson-khaki, and the one below is styleshout-coolwater, which I completed yesterday. You can see these skins now on demo.mojoportal.com. I still have 2 more designs that I want to try and make skins with today while I have skinning momentum. Tomorrow I will package a new release of mojoPortal to include all these new skins. I've been working full blast on skins since April 23rd until today. I'm very happy with the results, but ready to get working on other priorities.

Long story

Short story

FOSS Factory

I've completed the second skin of 3 for the 11th design of my 11 design campaign, styleshout-envision which can be seen on demo.mojoportal.com. Of course since its a public demo site, someone may come along and change it, but you can change it back or checkout the different skins by (key icon) Administration Menu > Site Settings.

Coming up next,

.

I've completed one skin of 3 for the 11th design of my 11 design campaign, styleshout-refresh which can be seen on demo.mojoportal.com. Of course since its a public demo site, someone may come along and change it, but you can change it back or checkout the different skins by (key icon) Administration Menu > Site Settings.

There are 2 more skins coming, styleshout-coolwater, and styleshout-envision, but they are really just variations on the same design.

This will be in svn trunk later today for developers and will be in the next release of mojoPortal for everyone else.

Most compilers have some (or in some cases many) intrinsic functions. HotSpot has a number of them (see here, search for "intrinsics known to the runtime") as does the CLR JIT. IKVM has had a couple as well (System.arraycopy(), AtomicReferenceFieldUpdater.newUpdater(), String.toCharArray()). These were sort of hacked into the compiler and I finally decided to clean that up a little and add more scalable support for adding intrinsincs. The trigger to do this was that I added four more intrinsics: Float.floatToRawIntBits(), Float.intBitsToFloat(), Double.doubleToRawLongBits() and Double.longBitsToDouble().

Benchmark

Here's a micro benchmark:

public class test {
  public static void main(String[] args) {
    long sum = 1;
    long start = System.currentTimeMillis();
    for (int i = 0; i < 10000000; i++) {
      sum += Double.doubleToRawLongBits(sum);
    }
    long end = System.currentTimeMillis();
    System.out.println(end - start);
    System.out.println(sum);
  }
}

Here are the results:

Since the x86 .NET results are highly sensitive as to whether the double on the stack happens to be aligned or not, I included both results.

Implementation

Here's the MSIL that IKVM generates for the loop:

IL_000b:   ldloc.2
IL_000c:   ldc.i4    0x989680
IL_0011:   bge       IL_0028
IL_0016:   ldloc.0
IL_0017:   ldloc.0
IL_0018:   conv.r8
IL_0019:   ldloca.s  V_3
IL_001b:   call      int64 [IKVM.Runtime]IKVM.Runtime.DoubleConverter::ToLong(float64,
                     valuetype [IKVM.Runtime]IKVM.Runtime.DoubleConverter&)
IL_0020:   add
IL_0021:   stloc.0
IL_0022:   ldloc.2
IL_0023:   ldc.i4.1
IL_0024:   add
IL_0025:   stloc.2
IL_0026:   br.s      IL_000b

The conversion isn't actually inlined, but instead a local variable of value type IKVM.Runtime.DoubleConverter is added to the method and a static method on that type that takes the value to be converted and a reference to the local variable is called. Here's the code for IKVM.Runtime.DoubleConverter:

[StructLayout(LayoutKind.Explicit)]
public struct DoubleConverter
{
  [FieldOffset(0)]
  private double d;
  [FieldOffset(0)]
  private long l;

  public static long ToLong(double value, ref DoubleConverter converter)
  {
    converter.d = value;
    return converter.l;
  }

  public static double ToDouble(long value, ref DoubleConverter converter)
  {
    converter.l = value;
    return converter.d;
  }
}

It uses the .NET feature that allows you to explicitly control the layout of a struct  to overlay the double and long fields. Note that this construct is fully verifiable.

For comparison, the standard System.BitConverter.DoubleToInt64Bits() uses unsafe code and looks something like this:

public static unsafe long DoubleToInt64Bits(double value)
{
  return *((long*)&value);
}

For some reason (probably because it isn't verifiable) the JIT doesn't like this so much and doesn't inline this method.

JIT Code

Here's the x86 code generated by the .NET 2.0 SP1 JIT:

049E15CE  cmp    ebx,989680h
049E15D4  jge    049E1600
049E15D6  lea    ecx,[esp+8]
049E15DA  mov    dword ptr [esp+10h],esi
049E15DE  mov    dword ptr [esp+14h],edi
049E15E2  fild   qword ptr [esp+10h]
049E15E6  fstp   qword ptr [esp+10h]
049E15EA  fld    qword ptr [esp+10h]
049E15EE  fstp   qword ptr [ecx]
049E15F0  mov    eax,dword ptr [ecx]
049E15F2  mov    edx,dword ptr [ecx+4]
049E15F5  add    eax,esi
049E15F7  adc    edx,edi
049E15F9  mov    esi,eax
049E15FB  mov    edi,edx
049E15FD  inc    ebx
049E15FE  jmp    049E15CE

Here's the x64 code generated by the .NET 2.0 SP1 JIT:

00000642805B8A90  cmp        ecx,989680h
00000642805B8A96  jge        00000642805B8AB1
00000642805B8A98  cvtsi2sd   xmm0,rdi
00000642805B8A9D  lea        rax,[rsp+20h]
00000642805B8AA2  movsd      mmword ptr [rax],xmm0
00000642805B8AA6  mov        rax,qword ptr [rax]
00000642805B8AA9  add        rdi,rax
00000642805B8AAC  add        ecx,1
00000642805B8AAF  jmp        00000642805B8A90

In both cases the construct is inlined properly. It is also obvious why the x64 code is so much faster, it uses SSE (as we've seen before) and only uses one memory store/load combination.

HotSpot

For completeness, here's the code generated by HotSpot x64:

0000000002772EA0  cvtsi2sd   xmm0,r11
0000000002772EA5  add        ebp,10h
0000000002772EA8  movsd      mmword ptr [rsp+20h],xmm0
0000000002772EAE  mov        r10,qword ptr [rsp+20h]
0000000002772EB3  add        r10,r11
0000000002772EB6  cvtsi2sd   xmm0,r10
0000000002772EBB  movsd      mmword ptr [rsp+20h],xmm0
0000000002772EC1  mov        r11,qword ptr [rsp+20h]
0000000002772EC6  add        r11,r10
0000000002772EC9  cvtsi2sd   xmm0,r11
0000000002772ECE  movsd      mmword ptr [rsp+20h],xmm0
0000000002772ED4  mov        r10,qword ptr [rsp+20h]
0000000002772ED9  add        r10,r11
[...]
0000000002772FC0  cvtsi2sd   xmm0,r10
0000000002772FC5  movsd      mmword ptr [rsp+20h],xmm0
0000000002772FCB  mov        r11,qword ptr [rsp+20h]
0000000002772FD0  add        r11,r10
0000000002772FD3  cmp        ebp,r9d
0000000002772FD6  jl         0000000002772EA0

It actually unrolled the loop 16 times (which appears not be helping in the case), but otherwise the code generated is pretty similar to what we saw on the CLR. Of course, in HotSpot Double.doubleToRawIntBits() is also an intrinsic because in Java the only alternative would be to write it in native code and the JNI transition would add significant overhead in this case.

Almost four years ago the first feature request was entered into the boo issue tracker.

There should be a way to extend the parser so it could recognize custom measurement unit literals such as 1kg and 2cm.

Since then boo has improved a lot but with no solution for BOO-1 in the horizon.

I think I'm getting close to solve it in a interesting way using PEGs implemented as a graph of expression objects.

PEGs are very likable. Conceptually simple and composable.

Take the PEG that recognizes integer expressions involving the + and * operators:

grammar <- spaces addition eof
addition <- term ("+" spaces term)*
term <- factor ("*" spaces factor)*
factor <- [0-9]+ spaces
spaces <- (' ' / '\t')*
eof <- !.

where:

() means grouping.

* means zero or more matches.

+ means one or more matches.

/ is the prioritized choice operator. If the first expression succeeds, the whole expression succeeds. If the first expression fails, it backtracks and evaluates the second expression.

! is the not predicate operator which succeeds if its operand fails. It never consumes any input.

. matches any input.

The grammar can be translated to boo very simply using the peg macro from Boo.Pegs:

import Boo.Pegs

peg:
    grammar = spaces, addition, eof
    addition = term, --("+", spaces, term)
    term = factor, --("*", spaces, factor)
    factor = ++[0-9], spaces
    spaces = --(' ' / '\t')
    eof = not any()
    
assert grammar.Match(PegContext("  6*6 + 6 "))

I had to be a little creative in mapping the PEG operators to valid boo expressions because as it must be clear by now boo doesn't allow the introduction of completely new syntax and that's what the fuss is all about here.

I actually like the way it looks.

Implementing something more useful such as expression evaluation on top of that requires a few semantic actions operating a stack:

import Boo.Pegs
import System.Collections.Generic

stack = Stack[of int]()
push = stack.Push
pop = stack.Pop

peg:
    grammar = spaces, addition, eof
    addition = term, --("+", spaces, term, { push(pop() + pop()) })
    term = factor, --("*", spaces, factor, { push(pop() * pop()) })
    factor = ++[0-9], { push(int.Parse($text)) }, spaces
    spaces = --(' ' / '\t')
    eof = not any()
    
assert grammar.Match(PegContext("  6*6 + 6 "))
assert 42 == pop()

Semantic actions are just closures that get executed as matching succeeds. $text returns the text matched so far by the current rule.

Beautiful.

The underlying implementation based on a graph of expression objects really shines when one considers what it takes to extend the grammar above with support for hexadecimal literals:

peg:
    // rebind
    factor.Expression = hex_number / factor.Expression
    hex_number = "0x", ++hex_digit, { push(int.Parse($text[2:], NumberStyles.HexNumber)) }, spaces
    hex_digit = [0-9, a-f, A-F]
assert grammar.Match(PegContext("  0xa*2 + 11*0x02"))
assert 42 == pop()

It's not yet clear how this extensibility mechanism will be exposed at the boo language level but the simplicity at the peg level is encouraging.

One last feature worth pointing out is the ability to match based on a previously matched rule. For instance, the closing tag of a xml element must match the name in the starting tag:

import Boo.Pegs

peg:
    element = '<', tag, '>', content, '', @tag, '>'
    tag = ++(a-z)
    content = --(element / text)
    text = not "<", any()
    
assert element.Match(PegContext("<foo><bar>Hello</bar></foo>"))

I've found the idea for the last match operator @ first described in this article. Great idea.

I've been also greatly inspired by conversations I've had with Massi who's exploring similar territory and Jb during the last Mono Meeting in Barcelona and with Cedric over a beer in Paris. I think Massi will be pleased to know that I haven't given any thoughts to performance leaving all the fun to him.

Extensible parsing. Soon in a boo compiler close to you.

I've completed the 10th design of my 11 design campaign, which resulted in 2 skins, styleshout-stylevantage and styleshout-citrusisland, which can be seen on demo.mojoportal.com. Of course since its a public demo site, someone may come along and change it, but you can change it back or checkout the different skins by (key icon) Administration Menu > Site Settings.

I've completed the 9th design of my 11 design campaign, styleshout-brightsideoflife, which can be seen on demo.mojoportal.com. Of course since its a public demo site, someone may come along and change it, but you can change it back or checkout the different skins by (key icon) Administration Menu > Site Settings.

This will be in svn trunk later today for developers and will be in the next release of mojoPortal for everyone else, hopefully available sometime this weekend.

I've now completed 11 skins based on 8 designs of my 10 design campaign. The newest is snop-plasticboy-reflection, which can currently be seen on demo.mojoportal.com. Of course since its a public demo site, someone may come along and change it, but you can change it back or checkout the different skins by (key icon) Administration Menu > Site Settings.

A few of the designs I've made alternate versions of which accounts for 11 skins so far. I really only need 2 more designs to meet my original goal but I've decided to do 1 extra design. The 3 remaining designs will actually result in 6 more skins. These are the designs I've chosen to complete this skinning campaign:

styleshout-brightsideoflife
styleshout-citrusisland and  styleshout-stylevantage (which I consider as variations on the same design)
styleshout-refresh, styleshout-envision, and styleshout-coolwater, which are all variations on the same design in my opinion.

Hopefully, I can complete all of these by sometime this weekend and then ship a new release with all these new skins.

• View the release notes
  
• Banshee 1.0 Beta 1 Source (bz2)
  
• openSUSE 10.3 users can
  
• Foresight 2.0 users: Use PackageKit or Conary to install banshee-1

• Fullscreen video playback (go to Now Playing and press f or hit the Fullscreen button)
• Extensions can be enabled and disabled in the new Mange Extensions tab within your Preferences.
• Banshee can be scripted using Boo
• Improved gstreamer error handling (for missing files, codecs, etc)
• A bug with play counts, introduced in Alpha 3, has been fixed
• Writing metadata to file was not working in the Alphas, is fixed
• Issues with the play queue should all be resolved
• Limiting smart playlists by file size or duration works
• Shuffle and repeat are automatically disabled while playing Last.fm

Default Smart Playlists

Tamir Khason published an interesting approach at hosting standalone Silverlight applications.

His solution is a Windows.Forms application that hosts a Windows.Forms.WebControl and inside the WebControl he hosts Silverlight.

Unlike my proposal for standalone Silverlight Applications that is currently Moonlight-specific (and currently limited to Linux/X11) this approach works on Windows with .NET and with Linux using Mono and Moonlight:

Left side: .NET hosting WebControl and Silverlight on Windows; Right side: Mono hosting WebControl and Moonlight running on Linux.

In addition to hosting the WebControl for hosting Silverlight, a thread is running to dispatch http requests locally using HttpListener. HttpListener is an embeddable HTTP server that is part of the class libraries, and exposes a very limited API. You can host ASP.NET with HttpListener by doing the bindings by hand, or you could use our Mono.WebServer library (part of our XSP/mod_mono distribution) to allow your applications to have a fully hosted ASP.NET server.

Mono.WebServer is what iFolder uses to embed the ASP.NET server to expose SOAP-based WebServices to clients.

Of course, this currently does not work on MacOS X as we do have no implementation of WebControl for Windows.Forms on OSX, something that a contributor might want to look into.

You can get the source for the sample from Tamir's page.

We are looking for consultants to work on a six to nine month project at Novell to write a prototype for a Visual Studio addin in C# or C++ that will connect Visual Studio and its debugging infrastructure to a remote Linux machine running Mono and the Mono Debugger.

If you are interested in working with us in this project, you must have good C# and C++ skills, experience with networking and protocol design, knowledge of COM and assembly language programming are pluses.

We are looking to bring two consultants for the duration of this project. If you are interested, please click this link and attach your resume, pointers to some existing projects of yours and so on.

I've completed the 7th skin of my 10 skin campaign, andreasviklund-02, which can be seen on demo.mojoportal.com. Of course since its a public demo site, someone may come along and change it, but you can change it back or checkout the different skins by (key icon) Administration Menu > Site Settings.

Actually I made 2 skins from this design and named the second one andreasviklund-02-alt1. The interesting thing about this skin is that it has 2 levels of tabbed menus and then a vertical menu for any pages deeper than that. In my alt1 version I took it a step further and used 3 levels of tabs and then a vertical menu to show anything deeper. This screen shot shows the alt version with 3 levels of tabs and a vertical menu for pages below that.

This will be available in svn trunk later today for developers and will be in the next release of mojoPortal for everyone else.

A couple of fixes.

Changes:

• Remapped exceptions with explicit remapping code now call suppressFillInStackTrace (to make sure the proper stack trace is captured).
• Fixed memory mapped file bug (mapping at a non-zero file offset would fail).
• Fixed .NET type name mangling for nested types that contain a dot in their name (which the C# 3.0 compiler generates for some private helper types).
• Fixed java.io.File.getCanonicalPath() to swallow System.NotSupportedException (thrown when the path contains a colon, other than the one following the drive letter).
• Fixed bug in deserialization of double arrays.

Binaries available here: ikvmbin-0.36.0.12.zip
Sources (+ binaries):ikvm-0.36.0.12.zip

Most of my personal projects are built on top of ASP.NET, Mono and Lighttpd. One of the benefits of keeping them all running on the same stack (as opposed to mixing Python, Mono and PHP together) is that I don't need to maintain different infrastructure bits to keep them all up and running. Two key pieces that keep it easy to dive back into the the side-project whenever I have some (spurious) free time are my NAnt scripts and my push scripts.

NAnt
I use my NAnt script for a bit more than just building my web projects, more often than not I use it to build, deploy and test everything related to the site. My projects are typically laid out like:

• bin/ Built DLLs, not in Subversion
• configs/ Web.config files per-development machine
• libraries/ External libraries, such as Memcached.Client.dll, etc.
• schemas/ Files containing the SQL for rebuilding my database
• site/ Fully built web project, including Web.config and .aspx files
• sources/ Actual code, .aspx.cs and web folder (htdocs/ containing styles, javascript, etc)

Executing "nant run" will build the entire project and construct the full version of the web application in the site/ and finally fire up xsp2 on localhost for testing. The following NAnt file is what I've been carrying from project to project.

<?xml version="1.0"?>
<project name="MyProject" default="library" basedir=".">
  <property name="debug" value="true" overwrite="false" />
  <property name="project.name" value="MyProject"/>
  <property name="project.version" value="1.0.0"/>
  <property name="library" value="MyProject.dll"/>
  <tstamp property="build.date" pattern="yyyyMMdd" verbose="true" />
 
  <!-- These are being definde for posterity's sake, as they may change in the future -->
  <property name="bin_dir" value="bin"/>
  <property name="src_dir" value="sources"/>
  <property name="lib_dir" value="libraries"/>
  <property name="site_dir" value="site"/>
  <property name="deps_dir" value="deps/"/>
  <property name="contrib_dir" value="contrib/"/>
 
  <!-- xsp specific properties -->
  <property name="xsp_port" value="8088"/>
  <property name="xsp_root" value="${site_dir}"/>
  <property name="xsp_apps" value="/:../${bin_dir}"/>
  <!--//  End properties -->
 
  <target name="library" description="Basic MyProject build task" >
    <echo message="Building ${project.name}-${project.version}"/>    
    <csc target="library" output="${bin_dir}/${library}">
      <sources>
        <include name="${src_dir}/**.cs"/>
      </sources>
      <references>
        <include name="Npgsql.dll"/>
      </references>   
    </csc>  
  </target>
  <target name="clean" description="Clean up MyProject">
    <echo message="Cleaning ${project.name}"/>
    <delete>
      <fileset>
        <include name="${bin_dir}/**.dll"/>
        <include name="${site_dir}/**.dll"/>
        <include name="${site_dir}/**.aspx"/>
      </fileset>
    </delete>
  </target>
  <target name="site" description="Populate the site/ directory">
    <copy todir="${site_dir}">
      <fileset basedir="${src_dir}/htdocs">
        <include name="**"/>
      </fileset>
    </copy>
    <copy todir="${site_dir}/contrib">
      <fileset basedir="${contrib_dir}">
        <include name="**"/>
      </fileset>
    </copy>
    <copy todir="${bin_dir}">
      <fileset basedir="${lib_dir}">
        <include name="*"/>
      </fileset>
    </copy> 
      <!-- Copying bin/ into the site/ directory sucks, but xsp2 is buggy
        with the applications parameter
      -->
    <copy todir="${site_dir}">
      <fileset>
        <include name="bin/*.dll"/>
      </fileset>
    </copy> 
  </target>       
  <target name="run" description="Runs the xsp2 web server on the port cited above" depends="library, site">
    <echo message="Starting the xsp2 web server running on port ${xsp_port}"/>
    <exec program="xsp2" failonerror="true" commandline="--port ${xsp_port} --root ${xsp_root}"/>
  </target>
</project>

The Push Script
Since I usually build and deploy on the same machine, I use a simple script called "push.sh" to handle rsyncing data from the development part of my machine into the live directories.

#!/bin/bash
###############################
##      Push script variables
export NANT='/usr/bin/nant'
export STAGE=`hostname`
export SOURCE='site/'
export LIVE_TARGET='/serv/www/domains/myproject.com/htdocs/'
export BETA_TARGET='/serv/www/domains/beta.myproject.com/htdocs/'
export TARGET=$BETA_TARGET
###############################
 
###############################
##      Internal functions
function output {
        echo "===> $1"
}
function build { 
        ${NANT}|> && ${NANT}|> site
}
###############################
 
###############################
##      Build the site first
output "Building the site..."
build
if [ $? -ne 0 ]; then
        output "Looks like there was an error building! abort!"
        exit 1
fi
 
###############################
##      Start actual pushing
if [ "${1}" = 'live' ]; then 
        output " ** PUSHING THE LIVE SITE ***"
        export TARGET=$LIVE_TARGET
else
        output "Pushing the beta site"
fi
 
output "Using Web.config-${STAGE}"
output "Pushing to: ${TARGET}"
&n