When designing the core Ioke libraries, I’ve tried to factor things out into different places, as much as possible. This have given rise to a pretty complicated object hierarchy, so I thought I’d describe it a bit closer here. This should make it a bit easier to understand how lookup of methods work, and where you should add functionality for the right effect. In this blog post I will go through most of the kinds in Ioke, notice how they fit together and also what kind of methods they contain.

OK, so let us start at the top. Even here we run into a small problem, of course - since Ioke is not single inheritance. One object can mimic more than one object, or none. There is no absolute requirement that everything is part of the same inheritance chain. We will get back to this in a while, but at first we can ignore that, and imagine that the hierarchy is simple.

At the top of the Ioke hierarchy is the object called Base. This object has no mimics, and contains only the most essential methods. These are methods that allow the introspection and modification of cells, such as “cell”, “cell=”, “cell?”, “cellNames”, “cells” and so on. It also includes “documentation” and “documentation=”. It includes the assignment method “=”. And finally it contains “mimic” that allow you to create new instances. This method is basically the only way of allocating new objects in Ioke, which means it is pretty important. If you need the functionality but without mimicking Base, you can either copy the method itself, or mimic BaseBehavior (we’ll get back to this in a minute.)

OK, so Base is the absolute base of everything in Ioke. But Base is not the equivalent of the Object found in most other languages - in most cases you won’t get the expected behavior if you mimic directly from Base. The object that is supposed to be used as the origin of new objects is called Origin. But before we go there, we need to talk about Ground.

Ioke actually overloads the term ground to mean two slightly different things. First, there is an object called Ground, and secondly, the ground of an evaluation is basically the default context. The default ground at the top level is the object Ground. Now, Origin mimics Ground. This means that if you add something to the top level, that will be in the inheritance chain of all objects. This is the closest you will get to a global scope in Ioke.

The Ground object mimics different things depending on which implementation you are running. If you’re on ikj, Ground will mimic both IokeGround and JavaGround, where JavaGround is the object where all Java integration machinery resides. On ikc, Ground currently only mimics IokeGround, but I expect to add CLRGround sooner or later. IokeGround is the place where all so called global names in the system is defined. This is where you find the cells for “Text”, “Symbol”, “Number”, “Range” and also “false”, “true” and “nil”. Yes, that is right, these names are not constants - they are just cells on IokeGround.

IokeGround mimics Base - so that is where the connection to the base of the hierarchy actually is made. But IokeGround also mimics and object called DefaultBehavior. As the name implies, this is where most of the Ioke behavior resides. DefaultBehavior in itself is basically a mixin, only providing these methods. You should not mimic DefaultBehavior in itself, if possible.

(Incidentally, that is the difference between an object that is supposed to be a mixin, and a regular object. In Ioke there is no real difference at the structure level - it is only about intent.)

DefaultBehavior doesn’t contain many methods in itself, but instead mixes in several more focused objects, such as “DefaultBehavior Aspects”, “DefaultBehavior Assignment”, “DefaultBehavior Boolean”, “DefaultBehavior Case”, “DefaultBehavior Conditions”, “DefaultBehavior Definitions”, “DefaultBehavior FlowControl”, “DefaultBehavior Internal”, “DefaultBehavior Literals” and “DefaultBehavior Reflection”.

If you want to make the intent of a mixin very clear, you mimic the Mixins object. This object include the bare necessary things to add new cells and introspect on them. “Mixins Comparing” and “Mixins Enumerable” are two examples of such mixins.

And that is basically the full story. Any other object you will find in the system is almost certainly derived from Origin, and the hierarchy under Origin is pretty shallow and flat, at least for now. I’ve considered making the intent of different collections a bit more specific, in the manner of Smalltalk - but this hasn’t happened yet.

1 Comment | By Ola Bini | In: ioke | tags: ioke, object hiearchy, programming language design. | #

One of the more advanced features in Ioke is the ability to work with first class messages. At the end of the day, you are manipulating the AST directly by doing this, which means that you can do pretty much anything you want. The manipulation of message chains is the main way of working with macros in Ioke, so understanding what you can do with them is pretty important.

The documentation surrounding these pieces is spread all over the place, so I thought I’d take a look at messages and the way you construct and modify them.

Messages

The first step in working with message chains is to actually understand the Message. Message is the core data structure in Ioke, and it has some native properties that define the full structure of the Ioke AST. There are four pieces of the structure that is central to messages, and a few more that is less interesting. So let us look at the core structure. It is actually extremely simple. These are the things that makes a Message:

• Name - all messages have a name. From the perspective of Ioke, this is a symbol. It will never be nil, but it can be empty.
• Arguments - a list of messages, zero or more.
• Prev - a pointer to the previous message in the chain, or nil if there is no previous.
• Next - a pointer to the next message in the chain, or nil if there is no next message.

A message can also wrap a value. In that case the message will always return that value, and no real evaluation will happen. This can be used to insert any kind of value into a message chain that will later be evaluated. This is called wrapping.

A message chain is just a collection of messages, linked through their Prev and Next pointers.

The arguments to a message are represented as a list of messages. This make sense if you think about it for a few seconds.

OK, now you know what the Ioke AST looks like. It isn’t harder than that. Now, if you actually want to start working with messages, there are several messages that Message can receive, that allow you to work with them. The simpler ones (that I won’t explain closer) is “name”, “name=”, “arguments”, “arguments=”, “next”, “next=”, “prev”, “prev=”.

There are a few more interesting ones that merit some explanation. First, “last”. This message will just return the last message in the message chain. It is the equivalent of following the next pointer until you come to the end.

It’s important to keep in mind that Message is a mutable structure, which means you need to be careful to not change things that will give you unexpected changes. For example, if someone sends in a message, you shouldn’t generally actually modify that without copying it. Now, if you only want to copy a message without copying recursively the next pointer, you can just mimic it. Otherwise you use the method “deepCopy” which will actually copy both the next pointer and the arguments recursively.

Now, if you want to add new arguments to a message, you can use “appendArgument”. This method is aliased as “<<”. It will also return the receiver, so you can add several arguments by linking calls to appendArgument/<<. If you want to add a message at the beginning of the argument list, you instead use >>.

One of the more annoying things is that once you set the next pointer, you generally need to make sure to set the previous pointer of the next value too, unless you are setting it to nil. The same thing is true when setting the prev pointer. So, in the cases when you want to link two messages, you shouldn’t set these specifically, but instead use the “->” method. This allow you to link two Ioke messages. For example “msg1 -> msg2″ will actually set the next pointer on msg1 and the prev pointer on msg2. If you do “msg1 -> nil” it will set the next pointer to nil.

And that’s basically it. If you need to actually evaluate the messages, you can either use “sendTo” or “evaluateOn”. The main difference here is that sendTo will actually not evaluate the message chain. It will only evaluate the message that is the receiver of the call. The evaluateOn method will follow the message chain and evaluate it fully, based on the context arguments given to it.

Oh, one last thing. To create new messages from scratch, there are a few different ways. First of all, you can wrap a value like this: “Message wrap(42)”. That will return a new message that wraps the number 42.

You can create a message chain from a piece of text by doing ‘Message fromText(”one two three”)’. This will return a message chain with three messages, linked together.

Finally, you can create a new message chain by using the from-method. You use it like this: “Message from(one two(three) four)”. What is returned is the message chain that is the argument. If you think about it for a few seconds, you can probably guess how to implement this using an Ioke macro.

Quoting

Now that we understand messages and message chains, let us take a look at how to create new chains in a flexible way.

First of all, all of the above methods are all very useful and nice, but they tend to be a bit verbose. Coming from a Lisp background I felt inclined to put the quoting characters to good use for this. So, first of all, the single quote (’) does the same thing as “Message from”. The back quote (`) does the same thing as “Message wrap”. So, to wrap the number 42, you can just do `42. In this case you don’t need parenthesis, since the back quote is an operator. To create a new message chain, use the single quote: ‘(foo bar(x) baz).

We almost have everything we need, except that we need some convenient ways of actually putting things into these message chains without having to put them together by hand.

Say for example we have a variable “blah” that contains an unknown message. We want to create a message “one” that is followed by the message in the variable “blah”. And then finally we want to add two messages “bax” and “baz” after it. We could do it like this: x = ‘one. x -> blah. x last -> ‘(bax baz). All in all, that is not too bad, but we can do better. This is done using the splice-quote operator, which is just two single quotes after each other. Using that it would look like this: ”(one `blah bax baz). In this case, the back quote inside of the splice-quote call will actually be evaluated in the current context and then have the result be spliced into the message chain being created. Now, only use the back quote if you are sure you can modify it. If you want to copy blah before inserting it, use the single quote again, instead of the back quote: ”(one ‘blah bax baz)

All in all, this is really all you need, and you can take a look at the core libraries and see how they are used. A typical example is the comprehensions library, and also the destructuring macros. In general, creating these message chains on the fly is the most useful inside of syntax macros.

I am planning to add a new feature to Ioke, that allow you to do tree rewriting for manipulating chains in different ways. This will be a feature built on top of the primitives described here, and these features will continue to be the main way of working with message chains for a long time.

No Comments | By Ola Bini | In: ioke | tags: ioke, macros, message chains, programming language design, syntax. | #

One of the nicer things with Ioke is that the syntax is highly flexible. This means that features that are generally considered part of the parsing step is not so in Ioke. There are still limitations on what you can take over, of course.

Another reason you can do much with Ioke’s syntax is that all operators are just method calls. This means you can override them in subclasses, which means these syntax changes can be totally localized.

A third way you can change things is by changing things at a global level, but only temporarily. This is possible since any cell in Ioke can act as a dynamic cell (or special) - by using the “let” method. This means you can make some very interesting changes to the syntax.

Understanding these three techniques make it possible to very easily create internal DSLs in Ioke, that feel like they are external. A fourth way of achieving this can be to massage message chains after the fact, to transform them into a different structure. You can do this directly, by working with the first class message structures - or if you don’t have to extravagant needs, you can interject specific behavior into the operator tables.

This post will give a quick introduction to these techniques, but it can’t really cover them all in full.

Flexible and polymorphic syntax elements

The things that in Ioke look like syntax elements but are mostly just message sends allow you to change the behavior locally of some things. Some examples of things where this is possible is in the creation of literals (such as numbers, texts and regexps), regular cell assignment (with =), and the creation of literal lists and dicts.

OK, to make this concrete, let me show a few examples. To start with, take the literals. The way literals work is that they will actually be translated into message sends. So when the parser see a number, it will generate a message send to “internal:createNumber”, and insert that into the message chain. This means you can override and change this behavior, which is something I do with my parser combinators example. As an extremely small example, take this code:

Parser(
  "foo" | "bar" | 42
)

This example creates a new parser that will parse either the literal string foo, the literal string bar, or the number 42. But how can we implement this? The method “|” is not even implemented for Texts in Ioke, and they definitely doesn’t return anything useful for the parser. We don’t want to override it to return the right thing either - it wouldn’t be a general solution - and what if someone wanted a parser that only matched one literal string? It’s clear that we need to hook into the handling of literals. (There is an alternative, we will talk about that in the final piece).

At this stage it might help to take a look at the canonical AST structure of the above code. It would look like this: internal:createText(”foo”) |(internal:createText(”bar”)) |(internal:createNumber(”42″)). With this structure, it should be more obvious how we can implement everything. The code for the above would look like this:

BaseParser = Origin mimic do(
  | = method(other,
    OrParser with(context: context, first: self, second: other)
  )
)

TextParser = BaseParser mimic
NumberParser = BaseParser mimic

ParserContext = Origin mimic do(
  internal:createText   = method(raw,
    TextParser with(context: self, text:   super(raw)))
  internal:createNumber = method(raw,
    NumberParser with(context: self, number: super(raw)))
)

Parser = dmacro(
  [code]
  context = ParserContext mimic
  code evaluateOn(context, context)
  context
)

The interesting pieces are in ParserContext. Inside it we override internal:createText and internal:createNumber to return parsers for their corresponding type. Notice how we call out to “super” to get the actual literal result. We then evaluate the argument to the Parser-method in the context of a newly created parser context. The only thing missing in the above code is the OrParser, and the actual matching pieces.

The other ways of hijacking syntax generally depend on executing code in a specific context, like the above.

I mentioned that “=”, [] and {} are overridable. Say that you for example like the syntax of blocks in Smalltalk, and want to use that within an internal DSL. That is actually extremely easy:

Smalltalk = Origin mimic do(
  [] = cell(:fn)
)

Smalltalk do(
  x = ["hello world" println]
  x call
  x call
)

Here we just assign [] to be the same as the method “fn” within the Smalltalk object. The same thing can be done for other operators, if wanted.

Using let to override syntax

As mentioned above, you can use the let method to override syntax (or any method really) for a specific bounded time. Lets say we want to do the above operation (Smalltalk blocks) for the duration of some execution. We can do it like this:

let(DefaultBehavior Literals cell("[]"), cell(:fn),
  x = ["hello world" println]
  x call
  x call
)

This will override the cell specified in the first argument with the value in the second argument - and then restore it at the end of the let-method. This is really not recommended for something like the [] method - as it will cause all kinds of problems in the internal implementations of methods. But you can definitely do it.

Transforming message chains

There are basically two techniques here. The first one is simply to add or remove or update the operator table
so that you can add operators that weren’t there before, or change the way they behave. This is not restricted to things with weird characters in them - indeed, in Ioke anything that appears in the operator tables count as an operator. A specific example of this is “return”. It can act as a unary operator, meaning it is possible to give return an argument without having parenthesis surrounding that argument.

To find out more information you can take a look at the documentation for Message OperatorTable here: http://ioke.org/dok/kinds/Message/OperatorTable.html.

The other way of transforming message chains is to actually take them apart and put them together again. I am planning a blog post dedicated to how to work with this, but I’ll take a quick peek at how to do it now. Lets take the earlier Parser example and see an alternative way of creating the text and number parsers without actually overriding the literals creation.

reformatCodeForParsing = method(code,
  ourCode = 'nil
  head = ourCode
  code each(msg,
    case(msg name,
      :internal:createText, ourCode -> ''createTextParser('msg),
      :internal:createNumber, ourCode -> ''createNumberParser('msg),
      :"|", ourCode -> ''withOrParser('(reformatCodeForParsing(msg arguments[0]))),
      else, ourCode -> msg mimic
    )
    ourCode = ourCode last
  )
  head
)

Parser = dmacro(
  [code]
  reformatCodeForParsing(code) evaluateOn(Ground, Ground)
)

This code is actually a bit annoying, but what it does is quite useful. What it does is that it will take the code above (”foo” | “bar” | 42) and restructure that into (createTextParser(”foo”) withOrParser(createTextParser(”bar”)) withOrParser(createNumberParser(42))).

The only thing that is a bit inscrutable is the way new message chains are put together using quoting of different kinds. I’m going to go into this with more depth in the next blog post. I am also working on a tree rewriting approach for making these kind of transformations much more idiomatic and readable.

So, this post have been a small introduction to several things you can do with Ioke to tweak its syntax. There is much more behind all these features, of course, and they all come from the fact that Ioke tries to unify and simplify all concepts as much as possible.

No Comments | By Ola Bini | In: ioke | tags: dsls, internal dsls, ioke, syntax. | #

Ioke is a very object oriented language. The amount of concepts in it is also quite small. I’ve tried to factor the implementation into smaller pieces, which are then composed into the full object hierarchy. One of the side effects of this is that it is possible to make objects that can’t be handled generically, since they don’t have all those things you expect objects to have. There is no requirement on Ioke objects to actually have a core set of methods. Things can be totally blank. Of course, a completely blank object is not very useful in itself…

So what kind of things become problematic then? Well, lets first talk about blank slates. In Ioke it’s pretty simple to create one - you just create an object, and then call the method “removeAllMimics!”. What you have after that point is something that is totally empty (unless if you added cells to the object itself before hand).

Now say that you want to add back a mimic to that object. Or say you want to add a new cell. As it happens, “=” is just a method on the Base kind. To add a new mimic, you generally use either “mimic!” or “prependMimic!”. These are both methods on ReflectionBehavior. So none of these methods are available.

Another example is mixins. In Ioke, a Mixin is nothing other than a regular object, except that it doesn’t have Base or DefaultBehavior in its mimic chain. Instead, the root of all mixins is the kind Mixin. Now, Mixin actually defines a copy of “=” and a few other things from Base. But it doesn’t allow you to add new mimics to mixins for example. This is a bit annoying - but since you have access to “=” you can actually fix it, by coping the “mimic!” method from ReflectionBehavior.

There are other circumstances where you want to be able to handle any object whatsoever in a uniform manner, introspect on a few things and so on. That wasn’t possible.

I’ve been considering these problems for a while, and finally decided that the pragmatic way of solving it is to add some methods that can be used to take a part and modify objects from the outside. I call the namespace that gives this functionality Reflector. Now, remember that you shouldn’t use the Reflector unless you’re certain that your objects need to be handled in that specific way. It’s more verbose, and not as object oriented as the methods that exist on regular objects. But it does allow you to do certain things on all objects in the system.

The rules are simple - all methods on Reflector are based on other methods in Base or ReflectionBehavior. They have the same name, except that they all start with “other:”. They all take an initial argument that is the object to work on. So, say we have an object “bs” that is a blank slate. We want to add a cell to it. We can do that simply like this:

Reflector other:cell(bs, :foo) = 42

Here we set the cell “foo” on the blank slated object to the value 42.

The problem of adding a new mimic to a mixin is easily solved too. Say we have an object ExtraEnum that we want to mix in to Enumerable. We can do it like this:

Reflector other:mimic!(Mixins Enumerable, ExtraEnum)

And so on. You can see all the methods available on Reflector in the doks.

As mentioned, the Reflector is a pretty specialized tool, but it’s totally indispensable when you need it. I will pretty soon sit down and rewrite DokGen to use it, since DokGen is a typical example of something that really need to be able to handle all objects in a generic way.

No Comments | By Ola Bini | In: ioke | tags: introspection, ioke, programming language design, programming languages, reflection. | #

On June 10th, I will give a talk about Ioke at the Chicago ACM. This is very exciting, and I hope to see many languages people there. It’s bound to be an interesting presentation, I think.

The talk is going to be held at Roosevelt University. You can find more information here: http://www.chicagoacm.org/.

No Comments | By Ola Bini | In: ioke | tags: acm, chicago, ioke, presentations. | #

It is less than a month to RailsWayCon in Berlin, so I thought I’d mention it here. This look like it will be a very nice conference. The dates are May 25 to May 27, in Berlin, Germany.

I will do two presentations and one keynote there. The presentations will be “JRuby Internals” and “Ioke for Ruby developers”. The keynote is called “Present and future of programming languages” and will feature my typical kind of ranting about programming languages.

Anyway. Hope to see you in Berlin! You can find more information here: http://it-republik.de/conferences/railswaycon.

1 Comment | By Ola Bini | In: blogging, ioke | tags: berlin, conference, conferences, ioke, jruby, keynote, presentations, railswaycon. | #

I will hold a presentation about Ioke in London, Thursday 14th. This will be held at Skills Matters location. You can find more information, and sign up, here: http://skillsmatter.com/event/java-jee/ljc-meetup-346.

No Comments | By Ola Bini | In: ioke | tags: geek night, ioke, london, presentations. | #

The last two weeks I’ve been furiously coding away to be able to do this. And I’m finally at the goal. I am very happy to announce that the first release of Ioke for the CLR is finished. It runs on both Mono and .NET.

Ioke is a language that is designed to be as expressive as possible. It is a dynamic language targeted at several virtual machines. It’s been designed from scratch to be a highly flexible general purpose language. It is a prototype-based programming language that is inspired by Io, Smalltalk, Lisp and Ruby.

Homepage: http://ioke.org
Download: http://ioke.org/download.html
Programming guide: http://ioke.org/wiki/index.php/Guide
Wiki: http://ioke.org/wiki

Ioke E ikc is the first release of the ikc Ioke machine. The ikc machine is implemented in C# and F# and run’s on Mono and .NET. It includes all the features of Ioke E ikj, except for Java integration. Integration with .NET types will come in another release.

Features:

• Expressiveness first
• Strong, dynamic typing
• Prototype based object orientation
• Homoiconic language
• Simple syntax
• Powerful macro facilities
• Condition system
• Aspects
• Runs on both the JVM and the CLR
• Developed using TDD
• Documentation system that combines documentation with specs

There are several interesting pieces in ikc. Among them I can mention a new regular expression engine (called NRegex), a port of many parts of gnu.math, providing arbitrary precision math, and also an implementation of BigDecimal in C#.

4 Comments | By Ola Bini | In: ioke | tags: .net, c#, f#, ikc, ioke, release. | #

Tomorrow (Tuesday) evening, at 6pm, I will give a presentation about Ioke, at the ThoughtWorks office in Bangalore. This is an open event. More information and registration here: http://www.thoughtworker.com/events/geeknight-ola-bini.

No Comments | By Ola Bini | In: ioke | tags: geeknight, ioke, presentations. | #

After a slightly longer development cycle then the last time, I am finally happy to announce that Ioke E has been released.

Ioke is a language that is designed to be as expressive as possible. It is a dynamic language targeted at the Java Virtual Machine. It’s been designed from scratch to be a highly flexible general purpose language. It is a prototype-based programming language that is inspired by Io, Smalltalk, Lisp and Ruby.

Homepage: http://ioke.org
Download: http://ioke.org/download.html
Programming guide: http://ioke.org/wiki/index.php/Guide
Wiki: http://ioke.org/wiki

Ioke E is the third release of Ioke. It includes many new features from Ioke S, among them full support for Java integration (including implementing interfaces and extending classes), many new methods in the core libraries, IOpt, TextScanner, BlankSlate, better support for regexp interpolation and escapes in regexps, support for Unicode analysis and much more.

Ioke E also includes a large amount of bug fixes.

Features:

• Expressiveness first
• Strong, dynamic typing
• Prototype based object orientation
• Homoiconic language
• Simple syntax
• Powerful macro facilities
• Condition system
• Aspects
• Java integration
• Developed using TDD
• Documentation system that combines documentation with specs
• Wedded to the JVM

The many things added in Ioke E could not have been done without the support of several contributors. I would like to call out and thank:
T W <twellman@gmail.com>
Sam Aaron <samaaron@gmail.com>
Carlos Villela <cv@lixo.org>
Brian Guthrie <btguthrie@gmail.com>
Martin Elwin <elvvin@gmail.com>
Felipe Rodrigues de Almeida <felipero@gmail.com>
Wes Oldenbeuving <narnach@gmail.com>
Martin Dobmeier <martin.dobmeier@googlemail.com>
Victor Hugo Borja <vic.borja@gmail.com>
Bragi Ragnarson <bragi@ragnarson.com>
Petrik de Heus

5 Comments | By Ola Bini | In: ioke | tags: ioke, ioke e, programming language design, programming languages, release. | #