January 28th, 2003
S#.NET Tech Preview Release Information:

It is January the 27th and the S#.NET tech-preview is not quite ready (sigh). It is, however, very close to ready -- see comments at the end if you're impatient to learn exactly when it will be out.

Given that I made a very hard date for this, I feel that I should tell you where it is at and give some sense of both why it is not ready now and how much more time is required to make it ready for release.

Basically the S#.NET release depends on the following elements:

a) The S#.AOS core system to provide the standard toolset, compilers, etc.

b) The [AOS Enabler] Net.Reflection.DLL/[dotnet assembly -- C# and IL]. It provides all [MS.NET] services for reflection, dynamic dispatch, default AOS classes, and dynamic module loading.

c) The [AOS IL to NET IL Cross Compiler Bridge] Net.Reflection.Bridge.DLL/[dotnet assembly -- C++]. It provides the bridge from S#.AOS to the S#.NET “Net.Reflection.DLL”. This piece will almost certainly go away – by virtue of being incorporated into the IL code within “Net.Reflection.DLL”.

d) The [AOS IL to NET IL Cross Compiler/JIT] AOS_Net_Bridge.DLL/[aos module]. It is written entirely in S#.AOS, and contains a full code analyzer, inliner, and code generator that converts AOS.IL to NET.IL and emits DotNet Assemblies.

e) Code samples and corresponding documentation materials.

The logistics:

1) I am leaving tomorrow for vacation with my wife in Hawaii, from the 28th through February 4th. I will periodically be online for this or that thing but basically will be incognito – unless there is a firestorm that needs quenching ;-(.

2) When I return, VSLive begins on the 12th of February, or thereabouts. I want to have the S#.NET release in place by VSLive or very close to it. [VSLive is the Visual Studio.NET Live Conference/Tradeshow in San Francisco].

The status:

(a) Modifications needed for S#.NET services were essentially completed in October.

(b) General architecture and mechanisms were completed in June 2000. Original design was recoded and cleaned up for v1.X of .NET from October to January. QA involving regression testing and performance work, as well as design review for future requirements was done at the start of January. Based on my dissatisfaction with the performance issues, and the realization that there was only one solution (native jitting), and the recognition that if I provided native jitting I would also have to provide a compatible pure-IL solution, I set about to recode significant portions of the mechanisms. Specifically, I revised the dynamic dispatch mechanism, the module-literal forms, the calling mechanism, and the ABIA/ThisSelector/Other-MOP facilities. I just completed that work yesterday – with the exception of a couple of hours of polishing.

(c) The Net.Reflection.Bridge was originally built in 1999-2000 and was not updated until fall of 2002. At that time I ended up re-writing it to incorporate extensive changes in the S# facilities. By the time (b) was initially ready at the start of January 2003, I had realized that (c) could be entirely eliminated because almost all its functionality had been generalized in (b) – assuming I provided a little glue and a couple of extra events and and a cross-resolver interface. I will almost certainly eliminate this piece before releasing S#.NET. It mainly exists now as a legacy piece. It was written in C++ because C# does not provide a mechanism to generate a DLL with external (non-dotnet) entrypoints; which are more desirable than COM components because a std dll works just fine with drag-install and COM components [which is the only way to expose C# to non-dotnet code] do not. However, for relatively minor work I can write external entry-points in IL (or C++) and merge that code into the (b) C# assembly build by hand.

(d) This [pure S# source code] piece was initially completed and demonstrated in June of 2000 at the Microsoft PDC [programmer developer] Conference in Florida. I essentially recoded the whole thing in the fall of 2002 in parallel with updating the base S#.AOS release. This allowed me to incorporate all the stabilized S# semantics and to provide general solutions to various DotNet edge case issues that I had been able to ignore for the PDC 2000 demonstration versions. This piece depends entirely on the code model and facilities in (a). As noted above, I chose to significantly restructure (a) and its semantics in the latter half of January 2003. I just completed that work today, which means I have not had time to go and revise (d) accordingly.

Where things are at [summary]:

Basically, all the DotNet side of things are done except for eliminating the (c) C++ bridge and replacing it with an IL variant in (b) – this should take at most a day or two.

On the S# [AOS.IL to Net.IL cross compiler/jitter] there is still significant work to be done.

1. The reflection code within the bridge needs to be modified to make use of the cleaner/more-efficient semantics offered by (b) [vis-a-vis the functionality provided by the [to-be] elided (c)]. This work could take as little as a day, or as many as two or three days.

2. The module (literals and IL) generator needs revision to incorporate the altered semantics for representation/encoding, call-invocation, and runtime-reflection querying. This also includes revising the various aspects of the logic analyzer to support the emitters by pre-computing certain information during the analysis passes. This work is tedious and therefore it may take as little as a few days or it may take a couple of weeks depending on how much refactoring I feel is required.

Why did I switch the plan at the end of the process and blow the committed/hard release date of January 28th, 2003?

To answer this on one level is very hard. I clearly thought I would make the Jan 28th schedule since I planned a vacation with my wife in Hawaii – a plan which intentionally included buffer time for the December holidays and business issues (unrelated to SmallScript) which I had to take care of.

The simple and direct answer is that I have been waiting to release S#.NET for a very long time and I wanted to initial release to withstand the inevitable problems without being dismissed. I was unhappy with the performance issues of Microsoft’s .NET Jitter for use in hi-performance dynamic dispatching scenarios – the how and why of this are much deeper discussion. I chose to incorporate my own scaled-back-jitter [borrowing it from S#.AOS] for x86 code within .NET. I was also deeply perturbed by what I considered flawed design constraints in DotNet that I had been putting up with to get the job done, which eventually I decided to aggressively circumvent – a decision which has made me mentally much happier but was the other contributing action that has resulted in this release delay.

That initial release depended on most of the S#.AOS design and infrastructure being in place and relative stable – that took a lot longer than I anticipated looking back at 2000/2001. I have been talking about S# and .NET for a long time now. When something has been anticipated for a long time, and discussed frequently in public venues, the expectation levels and standards by which it will be analyzed or assessed are also high.

I know that the #1 thing that both Smalltalkers looking at .NET and Java/CX/VB people looking at S# will do is ask the question is it “viable” as a usable solution. The first step in answering that question is invariably to test the range of features and the performance standards. On the Smalltalk side, people want to see the feature range [with reasonable performance and great ease of use/productivity]. On the Microsoft/NET side the belief is that manifest statically typed languages “rule” and scripting and dynamic languages are second class citizens – which is often demonstrated through performance/speed tests, security/type-analysis mechanisms, etc.

So, by early January everything was pretty much in place for release and it was now time for me to take on aggressive testing of my S# compiler mechanisms against all the code/classes combinations in the v1.1. Release of .NET and to look intensely over the machine code being generated by Microsoft’s v1.1 IL to x86 jitter facilities.

Needless to say, I was very unhappy with what I saw. I grew unhappier as things went on and I knew that what I had been putting off doing needed to be thought through now.

Specifically, I had always known that using IL to craft up dynamic dispatch implementations under .NET would not perform as well native Smalltalk/S# AOS jitters. Microsoft people and I had already discussed this issue at some length back in 1999-2000. We all understood and agreed that once demonstrable “right-answers” existed, the IL and .NET facilities could be upgraded – where eventually the political push became one involving ROTOR.

When I originally designed the mechanisms in 1999, there were some additional features in .NET which were dropped due to design/ship date requirements and some political/managerial communication snafu. I.e., I was the only person using a specific feature and the powers that be, in the schedule crunch, cut the feature because we had some communication snafu about the feature being “critical”.

The PDC 2000 demonstrations used the “cut” feature. So, following the PDC 2000, I had to come up with a still slower workaround mechanism that would still be fully verifiable and provide all the dynamic language and S# semantics flexibility. Which skipping a lot of other factors, led me to thread-local-variables and nested CALLI dispatching. I had been assured from early .NET pre-releases that TLS access was optimized for speed [but my January 2003 tests showed that it was anything but – at least compared to my own jitters for S#.AOS].

I simply could not stand the performance, the extra hoops, the combinatoric explosion of pre-computed classes and methods I had to generate, or the x86 code the .NET IL jitter was generating for the revised/compromise mechanism. I bit the bullet and started to take seriously the idea of dropping in my own jitter and re-doing the architecture. It was a very late date in the game to be going back into R&D mode – but the pain/anguish of the work and performance numbers was killing me.

After a week of R&D, careful thought, etc I realized that it would ultimately be a simpler design with much better maintenance, evolution, and performance gains to solve the problem now. My requirements for producing a verifiable fully featured S#.NET version remained unchanged. What was about to change was what happened within the internals of the Net.Reflection binding and dispatch architecture. Changes which would require me to revise significant portions of (d) [the AOS.IL to NET.IL cross-jitter and NET module/assembly emitter].

I chose to extract portions of my own x86 jitter and port them to C# and integrate them into .NET. This required that I write some magic code in IL to flexibly and reliably support certain “strange behavior” of the .NET runtime with regard the object layouts and GC security.

It took some time to make the port (I had to provide a pretty complete x86 compiler in C#). I had to run some exhaustive/extensive tests to assure myself that the .NET runtime integrity would be preserved by my jitter and the mechanisms by which I injected code into the runtime.

Once I had that work done I was then able to throw out gobs and gobs of yucky C# and IL code that had to be pre-generated to handle the various explosive combinatorics cases required for dynamic multi-method selector namespace dispatch [with other meta-goodies thrown in for good measure].

Now I was able to dynamically generate the dispatcher mechanisms [which are generally a very sparse matrix of the previous static compilation combinatoric explosion]. The size of the Net.Reflection.DLL dropped by a couple hundred kb, etc.

Even better, my new .NET dispatchers were acceptably close to the performance of my own AOS.IL jitters in S#.AOS. Translated, this means that it is now faster than any interpreted Smalltalk, and within a couple of cycles [of AOS performance] on a per dispatch decisions point basis. The difference between S#.NET dynamic dispatch/call performance C#.NET static dispatch performance is very reasonable [roughly within 6-15 cycles] per call. It could be directly cycle-for-cycle competitive if my S#.AOS jitter mechanisms were incorporated into Microsoft’s .NET jitters.

A week more of work and I had resolved all the design issues for providing both my own jitter for a hi-performance .NET x86 version, and a pure-IL solution that would work [albeit quite a bit slower by my performance requirements] on any .NET implementation [like the PocketPC]. This latter part played a big role in the requirement to revise the code generator/analyzer mechanisms in (d) – which is really where most of the schedule delay impact has been introduced.

At this point I now have fully working internal stuff that meets my requirements for performance, reflection/meta-operation semantics functionality, and DotNet security, verifiability, and pure IL deployment [with x86 etc hi-perf options]. Which means that the design is fully stable and there are no impediments or issues to be resolved, just nitty gritty tedious work to back-propagate the changes.

So, there you have the long explanation of where things are at and how they got there. You can make the call as to when S#.NET will be released [feel free to start a betting pool].

As for me, I am going on vacation in Hawaii on the island of Kauai for a week, and when I get back I will dig back into the S#.NET release with the goal to get it out the door [on or before 2/18] within two weeks of my return. I.e., certainly before my birthday on February 21st <g>.

P.S., I have lots of gripe/complaints to make about various items in C# and or .NET's CLR to help feed the mobs who seem to feed like sharks on this stuff. For me, they are just annoying things that I have had to work [very hard] to get around, and which S#.NET and its really nice facilities should serve to keep you happily insulated from. At some point or other I ought to document the explicit issues and design constraints together in a paper for release to both Microsoft and other interested parties -- but that will definitely come much later :).

Aloha,

-- Dave S. [www.smallscript.org]