"--start-group", "--end-group", aternatively "-(" "-)"

See the man page for LD for more info

Unfortunately this argument comes with the following little warning:

Using this option has a significant performance cost. It is best to use it only when there are unavoidable circular references between two or more archives.

The particular case I was dealing with was the havok version 6 libraries on linux. I was advised that there may be way to link them but hadn’t been worked out. In order to test this theory I hacked together a little bit of ruby to show library dependencies. I feel this may be of some use to someone else so I present it here:

require 'find'

$symbs = {}
$archives = []

class Archive
    attr_reader :name, :symbs, :depends, :undef
    def initialize name
        @name = name
        @symbs = {}
        @undef = {}
        @depends = Hash.new(0)
    end

    def add_definition loc, name
        @symbs[name] = loc
        $symbs[name] = self
    end

    def add_undefined name
    @undef[name] = 0
    end

    def resolve_undefined
        @undef.each do |k,v|
            archive = $symbs[k]
           if not archive or archive == self then
                next
            end
            @undef[k] = archive
            @depends[archive] = @depends[archive] + 1
        end
    end
end

Find.find('.') do |path|
    next if not /.a$/.match(path)

    io = IO.popen("nm #{path}")
    archive = Archive.new(path)
    $archives << archive
    while (str = io.gets) do
        case str
        when /\w+\.a/
            archive = Archive.new(str)
            puts "archive: #{str}"
        when /[A-F0-9]+ [A-Za-z] \w+/
            parts = str.split
            archive.add_definition parts[0], parts[2]
        when /\s+U\s+\w+/
            archive.add_undefined str.split[1]
        end
   end
   puts "processed #{path}, #{archive.symbs.count} defined, #{archive.undef.count} undefined"
end

$archives.each do |archive|
    archive.resolve_undefined
end

$archives.sort! {|a,b| a.depends.count <=> b.depends.count }

puts "dependency list:"
$archives.each do |archive|
    puts "#{archive.name}: #{archive.depends.count}"
    archive.depends.each do |depend,count|
        puts "    #{depend.name}: #{count}"
    end
end

The final solution was to simply remove the object files from the static libraries and put them into a single combined library:

rm *.o
find . -name \*.a -exec ar x {} \;
ar rcs libHavok.a *.o
ranlib libHavok.a
rm *.o

First of all you have your basic binary tree representation. I previously showed a simple binary tree example that demonstrates how binary trees are easily manipulated in ruby. Every binary tree node holds a value and links off to a right and left nodes that may also hold values. A leaf node is a node with dead links (left and right are both set to nil).

In the Huffman tree the leaves contain a single symbol while interior nodes summarize their sub-tree. I have simplified the idea so that all nodes wrap what is called an association list (assoc list). This is an old lisp idiom of a list containing name value pairs in the form ((name value) (name value)). To find a value in an association list you just walk the list looking at the first element of each sub-list. A reverse association list (rassoc list) takes the same form as an assoc list with the name and value pair swapped. For very small sets of data the performance of assoc lists is better than deeper structures such as a hash or tree. Ruby provides a few array APIs to deal with assoc and rassoc lists. I always store a rassoc list with the (occurance count, symbol) in a leaf and a sum of all contained lists in interior nodes. None of this information is important in understanding how Huffman encoding works but I use this concept below so I thought I should at least call it out.

class Node
    # binary tree representation
    attr_accessor :value, :left, :right
    def initialize(value=nil, left=nil, right=nil)
        @value, @left, @right = value, left, right
    end
    def leaf?
        return @left == nil && @right == nil
    end
end

By default the node is initialized to be empty with no links. A simple query method ‘leaf?’ is added for some syntactic sugar below when dealing with the tree.

The first step in building the Huffman tree is to build a set of occurrences for the data you want to encode. Some algorithms are able to this dynamically for streams of data but in this case I walk the entire input text first to find all symbols. In this implementation a symbol is a single character. Keep in mind that it is possible to use words or other unique symbols instead of single characters which could result in better compression.

def occurrences(text)
    # return an ordered array of [occurrences, character] ascending order
    occ = Hash.new(0)
    text.scan /./m do |b| occ[b] += 1 end 
    return occ.keys.map {|k| [occ[k], k]}.sort
end

The hash table occ is initialized with a default value of zero which causes all new entries to start at zero. Every character of the text is scanned and the characters are counted using the hash table. The final stage is to create an array of [count, character] entries in sorted order. Map executes the block over each key looking up the value of the key and outputting a single rassoc element per hash entry.

The final stage in building the tree is to convert the list into a tree representation. The occurrence entries are converted into leaf nodes and put into a queue in ascending order. Two entries are removed from the queue merged onto an interior node and placed onto a second queue. The interior node sums the rassoc list from both nodes that are consumed. If there are no nodes in the primary queue a node will be removed from the secondary queue. The final node remaining is the root of the tree.

def huffman(occlist)
    # build a huffman encoding binary tree from the occurrence list
    if occlist.empty? then
        puts "warning: no occurrences provided to build huffman tree"
        return nil
    end

    # create the initial queue with leaves, trees contain assoc lists
    leaves = occlist.map {|entry| Node.new([entry]) }
    interior = []
    deq = lambda {
        if (leaves.length > 0) then leaves.delete_at 0 
        else interior.delete_at 0 end
    }

    # create interior nodes
    while leaves.length + interior.length > 1
        l, r = deq.call, deq.call
        node = Node.new(l.value + r.value, l, r)
        interior << node
    end

    deq.call
end

Interior nodes are only formed in the second portion of the Huffman build function. It will only consume leaf nodes or or other interior nodes that represent the edge of the tree. In this way the tree is built bottom up and balanced.

Next up is a bit of a utility function that takes a Huffman tree and a symbol and converts it into an array of 0s and 1s that represent the binary encoding of the symbol. Walking the tree is represented by a 0 or 1 for the left and right sub-trees respectively.

def bits(ht, sym)
    # given a tree and symbol return an array of encoded bits  
    def bits_internal(ht, sym, enc)
        return enc, nil if not ht
        return enc, ht.value if ht.leaf?

        if ht.left.value.rassoc(sym) != nil
            enc << 0
            bits_internal(ht.left, sym, enc)
        else
            enc << 1
            bits_internal(ht.right, sym, enc)
        end
    end

    if ht.value.rassoc(sym) == nil
        puts "warning: no binary encoding for: '#{sym}'"
        return []
    end

    enc, value = bits_internal(ht, sym, [])
    if value[0][1] != sym 
        puts "warning: binary encoding: #{enc.inspect}, value: #{value.inspect} does not match '#{sym}'"
        return []    
    else
        return enc
    end
end

Bits makes use of the internally stored assoc lists to figure out which sub-tree should be visited. There are more efficient ways to implement this but this method is fine for example purposes. Bits() uses an internal method bitsinternal() which is implemented in a recursive style similar to visit() from the previous binary tree code. Bits does a bit of error checking on the result from bitsinternal() before returning the encoding value back to the caller.

Similar to bits_internal() above decode() will visit the tree using the binary representation in an array and choose the left or right sub-trees. Each time decode calls itself it uses a slice of the original array. If you are familiar with lisp code this method of traversal will look very familiar. When we have arrived at a leaf we know that we have found the symbol and it is returned.

def decode(ht, bits)
    if ht.leaf? 
        return ht.value[0][1]
    end

    case bits[0]
        when 0 
            decode(ht.left, bits[1..bits.length-1])
        when 1 
            decode(ht.right, bits[1..bits.length-1])
    end
end

Now for a simple test of the Huffman encoding procedures above. The standard Lorem Ipsum text will be used. The history of this randomized Latin text is very interesting, it is extracted from an early text on ethics by Cicero which was written in 45 BC.

text = <<TEXT
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Sed est nulla, suscipit vel, tempus sit amet, viverra sit amet, dui. Nunc ultrices, purus vulputate luctus sodales, mauris augue elementum diam, in ornare neque nisi pharetra lectus. In hac habitasse platea dictumst. Phasellus justo turpis, laoreet id, semper at, convallis a, nisi. Duis iaculis erat et mauris. Donec a arcu. Ut sed risus vel mi mollis vehicula. Aenean laoreet, lorem dapibus aliquam ultrices, ante velit vestibulum sem, vel molestie arcu elit sit amet nunc. Vivamus venenatis placerat dui. Mauris porttitor varius velit. 
TEXT

To measure the effectiveness of the compression I will sum the bits of the raw text assuming each character were to take a modern standard byte (8bits).

# build a huffman encoding tree
ht = huffman(occurances(text))

# measure the compression
bitsum = 0
for c in text.scan /./m
    bitsum += bits(ht, c).length
end

orig = text.length
new = bitsum / 8.0
printf("original %s bytes, huffman encoded: %s bytes, ratio: %.2f%%", 
orig, new, new/orig*100)

And what do we get for our efforts on this humble piece of ruby code:

original 598 bytes, huffman encoded: 374.25 bytes, ratio: 62.58%

~63% compression ratio. Not too shabby!

Click here to view the sample source code

Here’s your basic binary tree node representation, it holds a value and connects to a left and right node:

class Node
    # binary tree representation
    attr_accessor :value, :left, :right
    def initialize(value=nil, left=nil, right=nil)
        @value, @left, @right = value, left, right
    end
end

Insert receives a value and a block which is responsible for doing the comparison. Comparison would normally be done with the Ruby <=> operator (this is the case later when the chunkybacon is inserted into the tree). The block passed to insert receives two elements and should yield a -1 for less, 1 for greater or 0 for equal. This function is implemented in terms of itself, notable for it’s lack of any balancing.

def insert(node, v, &block)
    # binary tree insert without balancing, 
    # block performs the comparison operation
    return Node.new(v) if not node
    case block[v, node.value]
        when -1 
            node.left = insert(node.left, v, &block)
        when 1 
            node.right = insert(node.right, v, &block)
    end
    return node
end

Visit receives the order that the nodes should be visited as well as a block that will act as a visitor of the stored values. This function is also implemented in terms of itself.

def visit(n, order=:preorder, &block)
    # visit nodes in a binary tree, order can be determinied
    # block performs visit action
    return false unless (n != nil)

    case order 
        when :preorder 
            yield n.value
            visit(n.left, order, &block)
            visit(n.right, order, &block)
        when :inorder
            visit(n.left, order, &block)
            yield n.value
            visit(n.right, order, &block)
        when :postorder
            visit(n.left, order, &block)
            visit(n.right, order, &block)
            yield n.value
    end
end

And here is a simple example of inserting the string ‘chunkybacon’ into the binary tree. The result of visiting this tree using :inorder traversal is the string ‘abchknouy’.

if $0 == __FILE__
    # a simple test case
    root = nil
    "chunkybacon".scan(/./m) {|c| root = insert(root, c) {|a,b| a<=>b}}
    visit(root, :inorder) {|v| print v}
end

You may like it you will see, try it, try it and leave a comment for me.

There are certain scenarios where dynamic binding is particularly useful. It allows the caller to directly control the callee without using any intermediate objects, instead only using the underlying call chain. Certain forms of generic programming become very easy with dynamic binding.

Consider the following as an illustration of implicit dynamic binding:

def print_list(a_list)
  puts a_list.join(separator or ', ')  
end

print_list([1, 2, 3]) # -> 1, 2, 3
separator = ' '
print_list([1, 2, 3]) # -> 1 2 3

The simple example above also illustrates how implicit dynamic binding in a programming language can really start to dirty up a perfectly good code base. An unfortunate misspelling would be hard to debug in such a language. Additionally various rules about encapsulation are broken along the way.

The dynamic binding style still does have it uses however, especially in a language that supports functional abstraction. In ruby it is possible to define your own function objects and pass them around easily.

my_lambda = lambda {|x| puts x + 10 }
my_lambda.call(1) # -> 11
my_lambda.call(2) # -> 12

The lambda method in Kernel#lambda returns a new Proc that wraps the code in the block ‘{}’.

In some cases it could be useful to define a dynamically bound lambda that defines actor behavior in a generic way.

def execute_all(actor_list, proc)
  actor_list.each {|actor| actor.execute(proc) }
end
execute_all(lambda {play_anim(anim_family, 'walk', speed) }

The details of the lambda are open to the interpretation of the actor at the time the code runs. I’m not of the opinion that this is the best way to write this code but it certainly wins in brevity and in being fairly generic. There are better examples of using dynamic binding in the common lisp system.

To contrast dynamic binding here is an example of lexical binding. The lambda defined here binds a_var at the point the lambda is written, not where it is run (hence lexical scoping). This create a lexical closure which closes over the definition of a_var to keep it around later. The value returned below is 1 although at the time the code runs there is a binding for a_var that changes it’s meaning to 2.

def foo(p)
   a_var = 2
   puts p.call
end
a_var = 1
foo(lambda { a_var }) # -> 1

If ruby used dynamic binding by default the output of the code snipped above would be 2. To resolve a dynamically bound variable you walk up the chain of current activation records to find the nearest variable binding. In ruby the lexical variable binding captures a lot, including the current ‘self’, block iterator, and locals.

Now that I’ve covered all that I can talk about ruby tools for working with the bindings and different ways to program with a more dynamic mode.

In ruby it is possible to access the binding from code. The kernel method binding offers the caller access to an opaque Binding object. Currently the only way to use this binding object that I have found is via eval().

def foo()
  a = 1
  return binding
end

b = foo()
eval('puts a', b) # -> 1

The binding is manually captured here and passed into eval. This is a form of explicit dynamic binding. The only drawback of this method is that it doesn’t work for forms that have already been evaluated.

Explicit run-time variable binding is a pretty common software pattern. The alternative that is often used in C++ or Java is to create a special context object which is passed as a parameter to a method. The context is a way of explicitly controlling the binding. You create a custom mapping object (a hash for example) and pass it around to create your own variable binding. Here’s an example:

def foo(ctx, proc)
    puts proc.call() # -> 1
    ctx[:x] = 2
    puts proc.call() # -> 2
end

ctx = {:x=>1}
foo(ctx, lambda {|ctx| ctx[:x] })

Wouldn’t it be nice if we could encapsulate this pattern without requiring the extra parameter, dictionary mapping and extra indirection of having to make all variable access through the mapping?

I haven’t looked at the C code changes that would be required to ruby to do this but I think it’s possible to roll this pattern of coding directly into the ruby built-in library.

The additions are Binding#from_hash(aHash), Proc#call_with_binding(aBinding).

def let(k, &block)
  block.call_with_binding(Binding.from_hash(k))
end

let('a_var'=>1) {
  puts a_var
} # -> 1

dynamic_bound = lambda {
  puts a_var
}
a_var = 1
dynamic_bound.call_with_binding(binding) # -> 1

a_var = 2
dynamic_bound.call_with_binding(binding) # -> 2

The above code is a hypothetical approach. One that might be interesting to implement. There is one more alternative to lexical binding in ruby that I discovered after I had initially written this post. It relies directly or indirectly on the usage of instance_eval. Here’s a simple example:

class Foo
  def initialize(a) @a = a end
end

f = Foo.new(1)
f.class.send(:define_method, :foo, lambda { puts @a })
f.foo # -> 1

f2 = Foo.new(2)
f2.foo # -> 2

The method :define_method defines the proc in terms of Foo. The @a binds to the meaning of @a in the context of an instance of Foo.

A less permanent way to do this is to use instance_eval directly:

f.instance_eval { @a = @a + 10 }

Evaluate ruby with the press of a button

You can see the source of this tip from the vim wiki. I find the last bit of code the best of them:

function! Ruby_eval_vsplit() range
  let src = tempname()
  let dst = "Ruby Output"
  " put current buffer's content in a temp file
  silent execute ": " . a:firstline . "," . a:lastline . "w " . src
  " open the preview window
  silent execute ":pedit! " . dst
  " change to preview window
  wincmd P
  " set options
  setlocal buftype=nofile
  setlocal noswapfile
  setlocal syntax=none
  setlocal bufhidden=delete
  " replace current buffer with ruby's output
  silent execute ":%! ruby " . src . " 2>&1 "
  " change back to the source buffer
  wincmd p
endfunction

vmap <silent> <F7> :call Ruby_eval_vsplit()<cr>
nmap <silent> <F7> mzggVG<F7>`z
imap <silent> <F7> <ESC><F7>a
map <silent> <S-F7> <C-W>l:bw<cr>
imap <silent> <S-F7> <ESC><S-F7>a

Select some ruby code and hit F7 or hit F7 to evaluate the entire buffer. A new split is automatically opened or it uses an existing split if one is available.

Create a usable scratch buffer

Now if you combine that with the scratch.vim plugin you have a nice little 1-2 punch combination of ruby prototyping bliss. Nub tip: download scratch.vim put into $VIMDIR\vimfiles\plugins\ restart your editor and type :Scratch then type :set ft=ruby to get ruby syntax highlighting in the scratch buffer.

I may be wrong but I think this stuff comes with the emacs ruby mode for free (scratch buffer is a built-in emacs feature).

I’m going to have to wait until after we ship TR before I do any of the real work that I want to get done on RubyCube.

The DSL that I’m using is pretty small right now, which lends itself to easier understanding and explanation.

Ruby code tarball can be downloaded here.

A FSM is defined in a ruby script file without any class definition using the following rules:

1. All attributes are specified in a method named ‘setup’. Setup takes an argument hash which may contain arguments that can be used to setup a FSM entity.
2. All states are specified by defining top level methods with the name ‘while<statename>’. These methods are called in a tick update while that state is active.
3. All transitions are initiated by saying start :statename

def whileidle

target_nearest

start :chasing if @target

end

World#load uses the Object#instance_eval method to evaluate the contents of the FSM definition in terms of a FSM base instance. The FSM class defines some methods that can be called within the FSM definition to control state transitions and to access common methods.

Practically speak this is a very simple way to enable easy extension of a system. It provides an easier path to refactor the individual state machines and lowers the barrier to entry for writing state machine code. Additionally, it would be nice to include certain types of behavior from modules so that state machine authors could have easy access to some of the methods currently defined in creature FSM. This would include methods like target_nearest.  You could also mix in native C modules to get access to things like a path finding grid.

Ideas for playing with the code:

1. Make creatures pick up the ‘ammo’ entities
2. Make creatures compete for the ‘fark’ entities
3. Add a new term to the FSM for state transitions. The method names should be named ‘whenchangingfrom<statename>, whenchangingto<statename>, whenchangingfrom<statename>to_<statename>’ and should be executed when those state change patterns are matched.
4. Allow entities to die

A great article about some of the different types of DSL creation and the mindset from one of the coders at 37signals:

http://weblog.jamisbuck.org/2006/4/20/writing-domain-specific-languages

Careful of using missing_method, it bit me in the arse a number of times:

http://redhanded.hobix.com/inspect/theBestOfMethod_missing.html

It’s really hard to track down bugs related to methodmissing, any code you write inside an object implementing this needs to be carefully checked. It can also lead to blowing your stack when an error in methodmissing recalls method_missing.. and on and on.

This little gem is nice to keep in your back pocket when trying to raise a descriptive exception:

http://blog.nicksieger.com/articles/2006/09/06/rubys-exception-hierarchy

The sources in the distribution and on trunk can be built for Mac OS X for anyone who would like to experiment on that platform. Window start up time on my dual G5 is a bit slow for my tastes and I haven’t gotten in to optimize it yet.

I would like to thank basaah for his great suggestions and feedback on the 0.1.1 release.

I feel the code structure is straining a bit now that I’m starting to add in new objects.  It’s not terribly hard to add new functionality but the code base is definitely much larger than it was when I first threw this together as a proof of concept.

I’m going to do a dot release to document sprites and flush out the functionality. I’d love to get more feedback from anyone using RubyCube as to what you would like to see to make it more useful.

Since documentation is not yet ready for sprites the best way to see the API is in the new test/testsprite.rb driver.

• Download available here
• Updated docs available here

First problem I ran into was:

objc: failed objc_getClass(NSObject) for GLUTApplication->isa->isa objc: please link appropriate classes in your program Trace/BPT trap

This was pretty easily fixed by making sure to pass ‘-framework Foundation’ to the linker.

The next issue I’ve run into and I don’t have a good solution for so far is the GLUT startup performance on OS X. According to Sampler most of the time spent is in glutInit. When I drill down into glutInit I see that it is spending all it’s time in HIDUpdateDeviceList which it calls once during __glutGetNumberOfMouseButtons and again in __glutLoadPrefs. This descends down into a long chain of methods that appear to build an internal dictionary of devices (connected HID devices?). It appears this process is painfully slow, at least on my sample configuration.

I want the graphics window to pop up immediately (if at all possible) and so I’m going to have to spend more time figuring out the internals of what is really going on here. I’m sure it’s possible to speed it up but at what cost of my personal time? I’m not yet ready to write a specific windowing layer for the Mac.