Using dlopen() for fast engine prototyping...

I recently became tired of the cycle of hacking on the renderer, starting the whole engine, checking where and how it's failing, exiting the engine and the pointless time wasted during this process. Typically it's only a specific area I'm working on, thus perhaps one or two C source code files are actually changed, so it doesn't make sense to reload the entire engine and all media assets.

I started working on a smaller project for rapid-fire prototyping, μEngine. Currently it's a pretty basic project. It provides the basic framework for detecting that a shared object has changed, unloading the current version from memory, loading the new library, resolving the required symbols, and continuing the "rendering" loop (which actually doesn't perform any OpenGL rendering yet.)

Here's an example of the engine running:

~/uengine/src (master) $ make && ./uengine 
make: Nothing to be done for `all'.
... .libs/libuengine.so
0xdeadbeef
0xdeadbeef
0xdeadbeef
0xdeadbeef
... .libs/libuengine.so
0xcafebabe
0xcafebabe
0xcafebabe
0xcafebabe
... .libs/libuengine.so
0x0
$ 

While the engine was running, I edited libuengine.c, which is compiled by Automake into a shared object:

$ sed -i 's/0xdeadbeef/0xcafebabe/' libuengine.c && make
  CC     libuengine.lo
  CCLD   libuengine.la

Now I want to terminate the program (and I'm too lazy to have written a signal handler yet) so μEngine's main-loop is setup to terminate when dlsym_main returns a zero value.

$ sed -i 's/0xcafebabe/0x0/' libuengine.c && make 
  CC     libuengine.lo
  CCLD   libuengine.la

μEngine also links with libengine (which provides all of the math/geometry/etc functionality for Trinity), libdecl (which provides access to id Software style declaration files), and liblwo2 (which, well, loads LWO2 model files.)

This little experiment is far from complete. I still need to setup an OpenGL and GLX context, handle X11 input, and keep track of any ARB fp/vp or GLSL programs loaded. GPU programs will be reloaded in much the same way as the shared object itself.

There is of course a few frames delay between invocation of make and the new library being loaded, because should stat fail to obtain the file modification time (because it's still being written out) μEngine just continues with the old shared library until stat succeeds. The modification time is checked every "frame."

I think this should help with faster prototyping, since μEngine has the ability to access all of the assets and libengine library functions that Trinity can, but without the requirement of loading all the assets for a full world map into memory at start up.

More BSP compiler hacking...

I've started working on my engine's BSP compiler (based on Q3Map2/XMAP2) looking to add proper shader parsing and areaportal detection. Currently the shaders are all parsed by libdecl which avoids duplication of code between the engine and utilities.

The engine does not yet use portal based rendering (which is why you can see light clipping through the wall in the images.) The BSP compiler has portal debugging enabled, which outputs an extra surface with a translucent shader for areaportal visualization and debugging.

This is a very simple demo map, however you can already see several issues. The BSP compiler needs to carve the polygons into discrete areas (e.g. the floor polygon should not span two areas.) There are still a few areaportal bugs in the BSP compiler, and of course the engine must be updated to use areaportal based rendering.

 Image showing light leaking through the wall/areaportal.

Image showing the triangles rendered; the floor and walls should be carved at the areaportal boundary.








Who knows, maybe I'll even get this finished before id Tech 4 is (hopefully) released under the GNU General Public License.

Expression parsing and evaluation...

Something which has been annoying for me for a while was the lack of proper expression parsing in libdecl, my engine's declaration parsing library.

This library is responsible for parsing the sound and table/material declarations as defined by id Software, and making them available to the renderer (or other backend: sound engine, etc) in easy to read structures guaranteed to have sane and correct values. libdecl is the component which will complain about your syntax, the backend doesn't have to care about any of this. Really makes the code look a lot nicer. :-)

I was debating the best way to parse expressions, because they can either be very simple, or quite complicated. I realized that I could not rely on line terminators, because it's very common to see an expression span multiple lines with all kinds of tabs and white-space in the file... Of course the lexer ignores white-space, but keeps track of the line count.

After a bit of thinking about this, I decided to rip out the current shunting yard algorithm from the renderer, move it into libdecl, clean it up and rework it quite a bit. Now, it does the following:

1. Get a token from the lexer,
2. Check this token matches our expected rules,
1. the token is an operator or operand, or
2. the token is a table name. (Detected from 1st-pass.)
3. If our checks pass, go ahead with the normal algorithm, otherwise,
4. Push the token back to the lexer, and return the completed expression in Postfix notation.

Here's a hypothetical example that I wrote with more comments than necessary...

---

/* "material" keyword is optional when inside the material directory */
material textures/screenBlur
{
        sort postProcess

        /* first stage */
        {
                if glslPrograms != 0 &&
                        !isMultiplayer  /* don't blur the screen in multi-player */
                map textures/blur       /* any image format supported */
        }

        /* ... */
}

---

Obviously simply parsing the "if" expression until the end of the line would fail, furthermore, "if" expressions may be written as if ... or if (...), so matching on parentheses won't work either.

It turns out the easiest thing to do is first a pre-pass which only looks at tables (so that we know "foobarTable" is valid), then a secondary pass which looks at everything (sounds, tables, materials.)

In this example "glslPrograms" and "isMultiplayer" are built-in variables which the parser correctly detects as operands. Therefore we'll only stop parsing after reading the "map" token, realize it's not a valid operator or operand, push it back to the lexer, and return the expression (after some processing.)

This solution seems to be quite elegant, despite the 2-pass algorithm, and handles all the cases correctly.

The next step is to get rid of static stack allocation. Some expressions are very long, so the stack is setup for 64 elements, but this is ridiculous for "time * 0.001" (for example.)

Holiday boredom...

I don't have any plans for the Christmas holidays, and Helsinki is at just the right temperature to make doing anything outside really difficult. Some of the snow has melted and frozen into ice, making it feel a lot like walking on broken glass: physically and emotionally; I stopped counting the number of unintentional acrobatics after about the 5th or 6th time sliding in some way.

I might implement zlib compression for network traffic in my engine, which should only be a few dozen lines... or something a bit more interesting; stencil shadow volumes could still use more optimization, or some SIMD stuff...

There are at least a few dozen minor X server issues discovered by Coverity, but that's getting dangerously close to Nokia work over the holidays. The majority of the issues are error paths anyway, nothing really critical.

In the more being lazy and doing nothing area, Avatar looks like it could be an interesting movie...

Building a better flashlight...

I started looking into the projective light issues in XreaL and my forked engine again. It turned out the problem was setting up the light rotation via both the quaternion and light projection vectors.

The flashlight was working, but would point in the opposite direction you were looking. Right was left and left was right. Of course, it seems simple in hindsight, but these kind of bugs always annoy me until I find out what was going wrong.

Boring screenshot included for the fun of it...