Donnerstag, Januar 28, 2010

Factor @ Heilbronn University

It was an experiment -- and it went much better than I had imagined: I used Factor (a concatenative programming language) as the subject of study in a project week at Heilbronn University in a course called "Software Engineering of Complex Systems" (SECS). Maybe we are the first university in the world, where concatenative languages in general and Factor in specific are used and studied. Factor is the most mature concatenative programming language around. Its creator, Slava Pestov, and some few developers have done an excellent job.

Why concatenative programming? Why Factor?

Over the years I experimented with a lot of different languages and approaches. I ran experiments using Python, Scheme and also Prolog in my course. It turned out that I found myself mainly teaching how to program in Python, Scheme or Prolog (which still is something valuable for the students) instead of covering my main issue of concern: mastering complexity. In another approach I used XML as a tool for lightweight modeling to explore and study some techniques. The approach is innovative and still worth to be developed further but I wasn't satisfied.

My goal in the course "Software Engineering of Complex Systems" is to present and discuss practical techniques to conquer complexity in software systems. I did and still do a lot of research in this area. To make a long story short, I have come to the conclusion that Language-Driven Software Engineering (LDSE) is a very powerful and promising approach to conquer complexity. It's more than creating and using Domain Specific Languages (DSLs). It's consistently designing and creating languages throughout all levels and layers of a software implementation.

During my research I stumbled across Joy and Factor and learned about the concatenative paradigm. A series of excellent articles written by Manfred von Thun, creator of Joy, taught me the theory and fundamentals of concatenative languages. Factor, Slava Pestov's implementation of a practical concatenative language, turned out to be the best showcase I could think of: Factor is almost self-contained and extends itself by creating vocabularies of words using other vocabularies of words. Programs in Factor are written the very same style. Factor is seeing LDSE in action. I realized that it's the concatenative paradigm which enforces you to design software from a language-driven point of view.

How did the course go?

First I discussed the issue of complexity in software systems. Before I used Factor in the project week, I introduced the concatenative paradigm in the course. I presented it's mathematical foundation and used a pattern-driven approach to define the semantics of concatenative words as syntactic transformations. Finally, we defined a simplified grammar of Factor which we extended to cover pattern matching and template instantiation. All this served the purpose to smoothly prepare the ground for Factor. We also reflected and discussed a lot about what complexity is about and how it can be managed.

During the project week, my students (about 30 persons) worked with Factor four out of five days almost 8 hours per day. For my students it was full contact combat with an almost unknown programming paradigm and an exotic language. But they did really well. On day 1, the students worked through the introductory material supplied with Factor. On day 2, we studied types and object-orientation in Factor. On day 3, parsing and macros were studied in Factor. For these two days, the students worked with tutorials and worked their way through a number of exercise, which required them to write tiny programs in Factor to pass unit tests. On day 4, we worked on a topic unrelated to Factor. On day 5, two students thoroughly presented their project work, a real-world application in Factor, they had done in another course in the previous semester. We concluded the week with discussing and reflecting Factor's capabilities and the power of the concatenative paradigm in general.

Before I forget to mentioned it: Tim, research assistant in the software engineering department and PhD student, created the tutorials and the exercises and helped out a lot in class. Without him, the course wouldn't have been possible!

The students enjoyed the week very much. The evaluation of the course shows that they liked getting a new and different viewpoint on software development, object-orientation, parsing etc. They definitely realized and experienced that Factor helps them becoming better software engineers although Java is their main language.

Isn't Facor, aren't concatenative languages too esoteric to be useful?

Yes and no. There is no question that Factor is a niche language no one in industry shows interest for (besides Google, so far ;-). There might be some companies out there which use Forth and might be open for "concatenative thinking". However, even though the concatenative paradigm is almost unknown, concatenative languages are functional languages and functional languages are gaining in popularity. There's little doubt that learning functional programming broadens your scope and complements a student's skill set.

The fun part is that the concatenative approach to functional programming is much more simpler than the lambda calculus, which is traditionally taught. The math is simple and no intellectual barrier and formal transformations are easy to understand since there are no variable bindings and nesting scopes. Key concepts are stripped to their bare minimum. Did you ever try to explain the idea of continuations in Scheme? You might spend a good amount of time explaining continuations and running exercises. It's not unlikely that some students still don't get it. Continuations seem to be an extremely complex thing and appear to be somewhat mystical. In Factor, and in concatenative languages in general, continuations are a triviality! In principle, it's a snapshot of the data and the call stack. No big deal, since you juggle around with both stacks all the time. Are generic functions a specialty of CLOS. They come out naturally in a pattern-based approach to define concatenative words.

But my point goes beyond that. The way you create abstractions and refactor your programs in a concatenative language enforces you to continuously reflect about your design decisions. You have an enormous freedom of how you shape and constrain the design space of options at hand. It lets you think about words and vocabularies of words. It is thinking about creating and using languages. It combines software engineering and software programming in a way I haven't experienced in any other paradigm. That's why I introduced Factor in my course: You will start to engineer software, you'll explore new ways of creating abstractions and design frameworks.

Factor itself is an excellent case study for this approach. Factor starts from a relatively small kernel (which I -- admittedly -- haven't cleanly dissected, yet) and then consequently adds feature by feature with using Factor to extend Factor. A neat concatenative kernel turns itself into a powerful piece of software using a language-driven approach right from the start. Slava Pestov proves that this approach does result in a fast, interactive and highly reflective language. For me, Factor is a masterpiece of software engineering! It's definitely worth studying it!


What I experienced over the last two semesters is that some students become deeply attracted by Factor. Even if not, almost all students sense that there is a new world worth entering that takes them to a new level of understanding. It broadens their scope and skill set. Eventually, they'll leave the concatenative path for doing their Java/C# assignments in other courses or when they do some programming for a living. Still, I'm convinced that concatenative programming has an impact that lasts.

Do I sound too enthusiastic? Possibly, but I prefer to teach things I'm enthusiastic about! I'm still a student regarding the concatenative paradigm myself, I'm learning a lot each and every day about this paradigm. And one is for sure: I will continue to use Factor in the next semesters.

Update (2010-01-29) I received quite some requests to publish the Factor material we produced for the project week. The material is in German. Comments, ideas, corrections and improvements are welcome!

Day 1 - Intro: Getting started (Factor docu), Q&As

Day 2 - Object-Orientation: Intro, Tutorial, Q&As

Day 3 - Parsing and Macros: Intro, Tutorial, Q&As

Day 4 - Unrelated Topic:

Day 5 - Real-World Application in Factor: Presentation, Report, Sources (thanks to Andreas Maier and Marcel Steinle)

By the way, Daniel Ehrenberg indicated that Heilbronn University is not the first using Factor in a course. That's great to hear. Factor starts spreading!

In case you are interested in our research on concatenative languages, there is a paper available: "Concatenative Programming: An Overlooked Paradigm in Functional Programming".


Daniel Farina hat gesagt…

Would you mind posting your materials (if possible, including exercises/unit tests) that you taught with in the first few days of the class? I have mostly viewed Factor's implementation, but am curious about your treatment of getting acquainted with the language.

Anonym hat gesagt…

Any chance that you are willing to share the tutorials?
Reddit would be grateful

Benjamin Schweizer hat gesagt…


Mats hat gesagt…


dh hat gesagt…

Thanks for the interest. I updated the posting. You'll find all the material at the very end of this posting.

Let me know if you find the material useful!

Doug Coleman hat gesagt…

Integers are no longer sequences in the latest version of Factor.

In sokoban/ui/ui.factor you have to change

[ board-seq>> length ]
[ board-seq>> length iota ]

I wish you could control the game through the keyboard. See the use of set-gestures in extra/tetris/tetris.

Otherwise very cool!

Anonym hat gesagt…

The blog is in English but the posted materials are in German? Not exactly as helpful as I had hoped for. :(

dh hat gesagt…

I know and I'm sorry for that!

John Cowan hat gesagt…

Note that Forth is definitely the most mature concatenative language (language standardization, multiple implementations, etc.) Postscript, though somewhat specialized, is also a concatenative language that's been around for a very long time.

dh hat gesagt…

Sorry to disagree, John. In a concatenative language concatenation of words correspond to function composition -- this is the functional kernel. Related to this are quotations and the notion of calling a quotation. The semantics of dealing with the data passed from one function to another and the handling of quotations can be very well captured by the notion of a stack. Though, in principle, you do not need to talk about stacks.

In contrast, Forth was designed with some stacks in mind as the underlying execution architecture. The correspondence of words to functions wasn't the guiding principle in the language design as e.g. the IF word indicates (no quotations).

I prefer to refer to Forth as a stack-based language with postfix syntax and Factor as a concatenative language. They share stack semantics but their design principles differ to some degree.

I might write a more detailed blog post about this. But this is the line of argumentation.

dh hat gesagt…

By the way: There was a complaint about the blog post being in English and the course material in German.

When I wrote this posting I didn't anticipate that it would generate any interest in the course material at all. But I found it worth spreading the word what we are doing at Heilbronn University. English seemed to be the appropriate language for that.

Sam hat gesagt…

As Daniel pointed out, we've been using Factor in Télécom ParisTech since 2007 in my "Non-classical paradigms and languages" class, along with Scheme, Erlang, and Haskell. In the 2009 edition, I skipped Erlang and doubled the time spent on Factor instead, by using or reimplementing the Erlang messaging concepts in Factor (along with the parallism and the distribution aspects).

The students loved it, and one of them is still using Factor for his personal projects.

dh hat gesagt…

German speakers only: Note that for Day 5 there's now also the report from my students available. This might be a very good starting point for anyone interested in programming a real-world application in Factor. Enjoy! And a lot of thanks to Andreas Maier and Marcel Steinle! Good work!

dh hat gesagt…

Sounds cool, Sam! Factor really is some fun and highly attractive for students.

Malcolm hat gesagt…

I appreciate it's a lot to ask but I would love to see the materials translated into English. I'd like to get into Factor but I'm having trouble finding a middle ground between the very easy tutorial stuff, and the stuff written by Factor gurus like Slava who write very concise but not hugely newbie-friendly code. Anyway, glad to hear your teaching experiment was a success.

marry hat gesagt…

Blogs are so informative where we get lots of information on any topic. Nice job keep it up!!

Photography Dissertation

saurabh hat gesagt…

Humble request. Please translate the materials to English. It would be of great help for the newcomers.

Dmitry Ponyatov hat gesagt…

Why do you not just use SmallTalk? OOP in SmallTalk is in its primary form, and OOP is the best paradigm for system complexity control. You don't need to spend money using Squeak implementation. For your goals, you maybe should define some object abstraction library to directly manipulate your learning domain, and distribute tuned Squeak image among your students.

dh hat gesagt…

You are right, Smalltalk is an amazing language as well, and is an alternative to consider. I love the functional nature of concatenative languages. See, e.g. my implementation of a concatenative kernel