Codea Craft: Again.

I’m setting out to learn, again, about Codea’s ‘Craft’ capability, its 3D voxel-based facilities. Today, I’ll talk about my reasons, ambitions, and look at some code.

Art is never finished, only abandoned.
– Leonardo da Vinci (attributed)

Tuesday Morning

I’m bone-tired. I’m tired of the Dungeon program. I’m tired of the pandemic. I’m tired of political lunacy. I’m tired of being old. Look up Madeline Kahn, in Blazing Saddles, singing “I’m Tired”. That’s how I feel, except that I’m better looking.

I write these articles for a number of reasons, the first of which, I suppose, is that it keeps me off the streets. I like to try to make the computer do fun things, I like to write code, I like to think about it, I like to see how to make it better, and I like telling my readers, if any, about it.

It has been a long time since I did anything with Codea’s 3D capability, “Craft”. I did a few articles right about the time the pandemic became public knowledge, and then went idle for a while, and then veered off into Asteroids. That was a lot of fun, by the way, and it was more in my wheelhouse, 2D graphics.

I’ll index those past articles in the codea-craft tab, but I’m essentially starting over. I have a few ambitions for this effort:

1. Entertain myself with something a bit challenging;
2. Derive some understanding of the Codea 3D stuff;
3. Perhaps write some kind of 3D game;
4. Create some documentation that’s actually useful for Craft.

I’m not sure about #4 there. I would like to try a sort of parallel development of two threads:

• How to learn Codea Craft if you learn things the way I do;
• A more linear kind of documentation for Codea Craft: a tiny book, if you will.

At this moment, I’m not sure how to do any of those things. I plan to sort of muddle through, as before.

One more thing. There are quite a few Craft examples built into Codea, 17 at this writing. Let’s list them for future reference:

1. Learn Craft
2. AR Face
3. 3D Asset View
4. Lights
5. Planet 3D
6. Froggy
7. Stacker
8. Voxel Terrain
9. Voxel Editor
10. Voxel Player 11, Block Library
11. World Generator
12. UI
13. Cameras
14. Touches
15. AR
16. Voxel Editor AR

These examples are all rather good, in that they show off Codea’s capability well. They are also generally large and I don’t find them easy to learn from.

Here’s an example of “AR Face” running:

There are about 340 lines of code (and comment) in the “AR Face” program. Here’s the “scene” code for that program:

FaceScene = class()

function FaceScene:init()
    -- Create a new craft scene
    self.scene = craft.scene()

    -- Setup lighting
    self.scene.ambientColor = color(63, 63, 63, 255)
    self.sunLight = self.scene.sun:get(craft.light)
    self.sunLight.intensity = 0.7
    self.scene.sun.rotation = quat.eulerAngles(80, 0, 0)

    -- Setup bloom post processing effect
    self.cameraComponent = self.scene.camera:get(craft.camera)
    self.cameraComponent.hdr = true
    self.cameraComponent.colorTextureEnabled = true
    self.bloom = craft.bloomEffect()
    self.cameraComponent:addPostEffect(self.bloom)
    self.bloom.threshold = 1.5
    self.bloom.intensity = 1.2
    self.bloom.softThreshold = 0.4
    self.bloom.iterations = 8

    -- Keep a list of detected faces
    self.faces = {}


    -- The current tracking state
    self.trackingState =
    {
        [AR_NOT_AVAILABLE] = "Not Available",
        [AR_LIMITED] = "Limited",
        [AR_NORMAL] = "Normal"
    }

    -- Create some materials to apply to faces
    self:createMaterials()

    self:runAR()
end

Now, frankly, I find that daunting, not least because the camera notions of hdr, colorTextureEnabled, bloomEffect, and so on, are not documented anywhere that I can find.

Now there are certainly things we can learn from examples like this, and you can see in the video that there are sliders and buttons on the screen that let us change things, and we can certainly trace those down in the code and see how to manipulate things. But even that isn’t very helpful.

In the video, I slid the “Intensity” slider of the glowing eyes demo. Here’s the complete code for that mode of the demo:

            name = "Glowing Eyes",
            material = craft.material(asset.builtin.Materials.Basic),
            setup = function(face, material)
                face.entity.material = face.blankMaterial
                face.leftEye.material = material
                face.rightEye.material = material

                -- Combine color and intensity for HDR effects
                Intensity = 5
                parameter.color("Color", color(255, 50, 0, 0), function(c)
                    local d = vec3(Color.r/255, Color.g/255, Color.b/255) * Intensity
                    material.diffuse = d
                end)

                parameter.number("Intensity", 0.0, 100, 5, function(i)
                    local d = vec3(Color.r/255, Color.g/255, Color.b/255) * Intensity
                    material.diffuse = d
                end)
            end

We can sort of get the drift here. We have a material, which I think is semi-documented somewhere. Elsewhere, in the comments, we can learn that the eyes in this program are modeled as spheres, and we can guess that they are somehow positioned behind the face.

What is material? Digging through the Craft documentation begins to make it more clear. At least it answers some questions. I’m not clear why it blooms like it does, but I did see the word bloom somewhere in the program.

Personally, I hate this way of learning, needing to dig through large programs to find small bits of information. I’m not at all sure that I can do better, but I think I’ll try, at least for a while.

Wednesday 0800

Here begins a long slog through the “Learning Craft” demo. I manage to drag through three of its five tabs of examples. It’s quite a slog. If you proceed further, please ping me on twitter or email or slack and tell me how better to deal with this sort of thing.

A legal answer is “do this privately and give us the essence”. So is “do something else, this is dead boring”. Ideal answers would help me with this particular quest.

My summary of feelings and frustration yesterday caused me to stop programming and writing with what you see above. Today I have two things in mind.

First, and this is already done, I’ve gone back and tagged all my previous Codea Craft articles as codea-craft-old. That will let us find them if we want to, and we quite likely will. They do include some key learnings.

Second, I’m going to begin anew to understand Codea Craft, as if I remembered nothing about it. That’s not quite true, but since I stopped playing with it as the pandemic hit, it’s fair to say that I remember almost no details, though I probably have a general understanding of how things are done. Since my purpose here includes–possibly–creating some sensible documentation for Codea Craft, starting fresh makes some sense.

I’m going to begin with the “Learn Craft” project that comes with Codea.

Learn Craft

The Learn Craft demo has a clever arrangement of five small demo programs that you can tick through one at a time. What I’ll do here is tick through them, looking at the code for the demo and seeing what we can learn from it.

The first demo is called “Scene”. It displays a black screen and some text in the console, which reads:

Where is everything? This is a blank scene, the starting point for a 3D project.
To setup an initial blank scene:
Create an instance of craft.scene in setup()
The scene then needs to be updated and drawn each frame to work.
Theres no need to clear the background as scene will do it for you.
2D drawing can be done after scene:draw() as normal.”)

The code that does this null drawing is this:

function setup()
    -- Create a new craft scene
    scene = craft.scene()
    scene.sky.active = false
end

function cleanup()
    if viewer then
        touches.removeHandler(viewer)
        viewer = nil
    end
    scene = nil
    collectgarbage()
end

function update(dt)
    -- Update the scene (physics, transforms etc)
    scene:update(dt)
end

-- Called automatically by codea 
function draw()
    update(DeltaTime)

    -- Draw the scene
    scene:draw()
  
    -- 2D drawing goes here  
    drawStepName()	   
end

The function cleanup above has absolutely nothing to do with Codea Craft. It is there to handle the clever code that switches between demos, each of which is kept in its own tab. The drawSceneName shows the current demo name. See the second picture below.

In further examples, I’ll try to remove from our view all the code that has nothing to do with what we’re doing.

What have we learned?

We need to create a scene. (Remember all those times someone told you “Don’t create a scene”? Well, they were wrong. That goes like this:

function setup()
    -- Create a new craft scene
    scene = craft.scene()
    scene.sky.active = false
end

Except that right away we’re like “What is this sky.active stuff about? No one has mentioned that. I’ll try setting it true. I see no difference. So that’s useless. I think the “correct” code is more like this:

function setup()
    scene = craft.scene()
end

We are told that we have to update each frame. I’ve made a copy of this program, because I intend to edit it down until it confesses what’s needed and what isn’t.

I move cleanup down to the bottom where it won’t confuse anyone ever again.

Then I look at draw and move it upward:

-- Called automatically by codea 
function draw()
    update(DeltaTime)
    
    -- Draw the scene
    scene:draw()
    
    -- 2D drawing goes here  
    drawStepName()	   
end

Let’s seriously consider the need for a comment saying “draw the scene” right ahead of scene:draw(). If I were going to put tutorial comments here, I’d do this:

-- Called automatically by codea 
function draw()
    -- first we update the scene, then draw it
    update(DeltaTime)
    scene:draw()
    
    -- 2D drawing goes here  
    drawStepName()	   
end

Of course on most days I wouldn’t include comments like that. Just when we’re doing an early tutorial thing. Then we see this:

function update(dt)
    -- Update the scene (physics, transforms etc)
    scene:update(dt)
end

If this is a tutorial, comments here are missing. I’d do this:

-- Not called automatically. 
-- Call from draw, passing DeltaTime
function update(deltaTime)
    -- Update the scene (physics, transforms etc)
    scene:update(deltaTime)
end

It seems to me that it’s important that we realize that update is not called automatically, but that we’re required to call it from draw. It’s also important that it be called with DeltaTime. Why it doesn’t just use DeltaTime internally is not given to us to know.

We wind up with this:

function setup()
    scene = craft.scene()
end

-- Called automatically by codea 
function draw()
    -- first we update the scene, then draw it
    update(DeltaTime)
    scene:draw()
    
    -- 2D drawing goes here  
    -- this has nothing to do with Codea Craft
    drawStepName()	   
end

-- Not called automatically. 
-- Call from draw, passing DeltaTime
function update(deltaTime)
    -- Update the scene (physics, transforms etc)
    scene:update(deltaTime)
end

I think this tells the story better. Let’s move on to the next demo. It displays a sort of robot thing, and there is a rotate slider. When you slide the slider, the robot rotates.

The text in the console reads:

Entities are flexible objects used for displaying 3D models, simulating physics and more
To create an entity use - myEntiy = scene:entity()
To attach a 3D model, use myEntity.model = craft.model(…)
Use the x, y, z and position properties to move entities around
Use the eulerAngles and rotation properties to rotate an entity

And the code, as provided:

function setup()
    -- Create a new craft scene
    scene = craft.scene()
    scene.sky.active = false
    createGround(-1.125)

    -- Create a new entity
    myEntity = scene:entity()

    -- Set its model for drawing
    myEntity.model = craft.model(asset.builtin.Blocky_Characters.Robot)

    -- Adjust position and scale
    myEntity.y = -1
    myEntity.z = 0
    myEntity.scale = vec3(1,1,1) / 8

    -- Move camera back a little
    scene.camera.z = -4

    parameter.number("Rotate", 0, 360, 180)
end

-- Creates the ground using a box model and applies a simple textured material
function createGround(y)
    local ground = scene:entity()
    ground.model = craft.model.cube(vec3(4,0.125,4))
    ground.material = craft.material(asset.builtin.Materials.Specular)
    ground.material.map = readImage(asset.builtin.Blocks.Dirt)
    ground.material.specular = color(0, 0, 0, 255)
    ground.material.offsetRepeat = vec4(0,0,5,5)
    ground.y = y
    return ground
end

function update(dt)
    -- Rotate the entity
    myEntity.eulerAngles = vec3(0, Rotate, 0)

    -- Update the scene (physics, transforms etc)
    scene:update(dt)
end

-- Called automatically by codea
function draw()
    update(DeltaTime)

    -- Draw the scene
    scene:draw()

    -- 2D drawing goes here
    drawStepName()
end

The point of interest in update is that we rotate the robot by saying

    myEntity.eulerAngles = vec3(0, Rotate, 0)

This tells us something important. Two things in fact. First, the parameter Rotate goes from 0 to 360, so it’s in degrees, not radians. More important is that the angle is placed in the Y coordinate of the input vector.

That tells us that in Codea’s 3D model, Y is upward. Some folks would expect Z to be upward. I imagine this decision is not uncommon and relates to the fact that on a 2D screen, Y is upward. Anyway, for those of us who think Z is up, we need to reset our brains.

I think we’ll find that X goes from left to right and Z probably goes front to back, larger Z being further away.

We can test that.

    myEntity.z = 5 -- formerly 0

Sure enough, he moves back. It looks like he’s well above the ground, but I think it’s just that the platform is small. Let’s see how that’s created.

-- Creates the ground using a box model and applies a simple textured material
function createGround(y)
    local ground = scene:entity()
    ground.model = craft.model.cube(vec3(4,0.125,4))
    ground.material = craft.material(asset.builtin.Materials.Specular)
    ground.material.map = readImage(asset.builtin.Blocks.Dirt)
    ground.material.specular = color(0, 0, 0, 255)
    ground.material.offsetRepeat = vec4(0,0,5,5)
    ground.y = y
    return ground
end

The ground is only 4x4 and I set him back 5. Let’s make the ground larger, 12x12. The picture looks better:

We’d like to get a better view but that’s coming up in a later example. But let’s see what we can glean from what we have here.

There’s a reference to camera in setup. Let’s see what happens if we change that.

If I set it to zero, I see nothing but the back of the platform. Since the Robot’s Z is zero, my viewpoint is inside him. Setting camera.z to -1, I get a very close up view of the Robot:

It’s tempting to mess with the camera’s x and y, isn’t it?

Let’s do. I’ll add two parameters.

    parameter.number("X", -5,5,0)
    parameter.number("Y", -5,5, 0)

And I’ll use them in update:

function update(dt)
    -- Position the camera
    scene.camera.x = X
    scene.camera.y = Y
    
    -- Rotate the entity
    myEntity.eulerAngles = vec3(0, Rotate, 0)

    -- Update the scene (physics, transforms etc)
    scene:update(dt)
end

That gives a very nice effect:

I also notice that the Robot is a bit above ground. I have no real knowledge of his size or center or the like. We may have to figure out ways to learn things about these models.

What else do we see in this program? Well, there’s a lot. Let’s see how things are created. Looking further at ground:

-- Creates the ground using a box model and applies a simple textured material
function createGround(y)
    local ground = scene:entity()
    ground.model = craft.model.cube(vec3(12,0.125,12))
    ground.material = craft.material(asset.builtin.Materials.Specular)
    ground.material.map = readImage(asset.builtin.Blocks.Dirt)
    ground.material.specular = color(0, 0, 0, 255)
    ground.material.offsetRepeat = vec4(0,0,5,5)
    ground.y = y
    return ground
end

What can we work out here? First, what can we get from the Codea documentation?

That’s a bit general, but right below we find this, about the cube.

So that’s telling me that cube is really a rectangular parallelepiped, which I suppose they thought was too long a name. What can we learn about materials?

We find this on material:

This type represents a surface material. Materials are used to control the physical appearance of 3D objects, such as models, sky boxes and voxels.

Set the material property on a craft.renderer component to apply a given material to it. Materials are built out of modular shaders with a set of dynamic properties.

Each named property has a different effect on the material’s appearance. For instance, the map property can be set with an image or asset string to change the surface appearance of the Standard material. Other properties can change the normals, roughness, metalness and reflectiveness of materials.

This is both more and less than we wanted to know. We’re left with doing this by rote, or we can experiment a bit. I choose to experiment. Let’s see what happens if we use another material rather than Specular, whatever that is.

Tapping on the material in the code line:

    ground.material = craft.material(asset.builtin.Materials.Specular)

Brings up this picture, which is actually enlightening:

It looks like Standard has a sort of soft highlighting, Specular a harder highlighting. Basic is interesting, being all white. I think I’ll try that one to see what we get.

I see no difference in the ground. Perhaps we’d see an effect if we had any lighting, or if we could change our viewing angle. Let’s try that again later.

I decide to see what options there are for the map attribute. Tapping that gives me this picture:

I pick Brick Red and the picture changes to this:

Let’s see what happens if we change this:

    ground.material.offsetRepeat = vec4(0,0,5,5)

There’s no documentation for this, other than another example that sets it. I think I’ll change it to 1,1:

Well. That spread the texture over our whole big cube. Let’s instead give it 12,12, the size of the cube.

When I try 36,36, I get this:

So, given that they called this offsetRepeat, I’m guessing that the first two values are numbers between zero and one, offsetting the texture by that fraction, and that the last two are the number of times the texture is to be repeated across and up-down. (Some outside experience makes me think that the repeats will be in U and V, the commonly used coordinates for materials, since X, Y (and Z) are geometric, while materials can be at any angle. The horizontal coordinate on a rectangular texture is called U and the vertical is called V. We can check this by setting our two values differently.

    ground.material.offsetRepeat = vec4(0,0, 36,6)

I expect these to be small across and big front to back. And they are:

So we’re building up an understanding of materials, which is combined from meager documentation, speculation, experimentation, and past experience. Not really great. Let me try to write up my understanding so far.

Materials and Textures

A Codea Craft entity has surfaces. They are actually made up of triangles, which can create approximations to any shape if you use enough of them.

The surface of an entity can be divided up into one (or perhaps more) “materials”, and a given material can have a “map”, which is a rectangular graphical picture often called a “texture”. Viewing the texture in its normal display mode, the horizontal coordinate is called U and the vertical is called V.

Materials have many properties, as shown in this example, presented without further explanation:

local e = scene:entity()
e.model = craft.cube(vec3(1,1,1))

-- Load the standard material (physically based rendering)
local m = craft.material(asset.builtin.Materials.Standard)
e.material = m

-- Surface color
m.diffuse = color(255, 255, 255)
-- Opacity (0.0 fully transparent, 1.0 fully opaque)
m.opacity = 1.0
-- Surface color texture map
m.map = "Surfaces:Basic Bricks Color"
-- Texture map offset and repeat (tiling)
m.offsetRepeat = vec4(0.0, 0.0, 3.0, 3.0)
-- Normal map for small scale surface details
m.normalMap = "Surfaces:Basic Bricks Normal"
-- How intense the normal map effect is in tangent space (also used for flipping)
m.normalScale = vec2(1, 1)
-- How rough the material is
m.roughness = 1.0
-- A texture that controls the roughness
m.roughnessMap = "Surfaces:Basic Bricks Roughness"
-- How metallic the material is
m.metalness = 1.0
-- A texture that controls how metallic the surface is
m.metalnessMap = nil
-- The environment map (a CubeTexture), specular illumination
m.envMap = nil
-- The ambient occlusion map, used for self-occluding shadows
m.aoMap = "Surfaces:Basic Bricks AO"
-- How intense the aoMap is
m.aoMapIntensity = 2.5
-- The displacement map which modifies vertex positions based on normals
m.displacementMap = nil
-- Base offset of the displacement map
m.displacementBias = 0
-- Scale of the displacement map
m.displacementScale = 1

The above example is fascinating. I am tempted to build a little program that uses that. But not now, we’re still learning some basics.

When a map is assigned to a material, Codea displays that part of the entity with that graphic picture on it. Depending on the part of the entity the map covers, its alignment, and other aspects of the model, the graphic may appear rotated, scaled, warped, and so on.

Depending on the material and other attributes, the map can appear to be shiny, rough, or metallic. The material attributes can give an appearance of roughness, or even cast local shadows as if the material has a 3D aspect.

I Gotta Try Some of This

I can’t resist that example. I’m going to past some of that stuff into our robot demo.

-- Creates the ground using a box model and applies a simple textured material
function createGround(y)
    local ground = scene:entity()
    ground.model = craft.model.cube(vec3(12,0.125,12))
    ground.material = craft.material(asset.builtin.Materials.Specular)
    ground.material.map = readImage(asset.builtin.Surfaces.Basic_Bricks_Color)
    ground.material.normalMap = readImage(asset.builtin.Surfaces.Basic_Bricks_Normal)
    ground.material.normalScale = vec2(1,1)
    ground.material.roughnessMap = readImage(asset.builtin.Surfaces.Basic_Bricks_Roughness)
    ground.material.roughness = 1
    ground.material.specular = color(0, 0, 0, 255)
    ground.material.offsetRepeat = vec4(0,0, 36,6)
    ground.y = y
    return ground
end

That gives me this pic:

Let me turn off the normal and roughness, see if that makes a difference. I see no difference. We may have to wait for this until we get some lighting.

We’ve been at this for two hours, that’s enough pain for now. Let’s sum up.

Summary

I’ve spent two hours reading the first three Learn Craft tabs and trying to absorb what’s there and what seems to be implied. Basically I get that:

There is a “scene”, containing all the “entities” that we want to draw. The entities have coordinates in X, Y, and Z, and can be rotated around any axis, to any angle. An entity has a “model” and other attributes

Models can apparently be provided as units, such as our robot, which came in as a unit, with an interesting shape and fully painted. And there are some standard models, including cube, which makes a rectangular object of any dimensions. That model has no particular color or pattern, which are provided by “materials” and “maps”.

Materials seem to be a basic signal to Codea about how to display the model. Sample materials vary from quite plain, to one with what looks like a sharp highlight, to one with a softer highlight. I believe without evidence that the effect of those depends on lighting, which I know exists but don’t know anything about it.

The documentation suggests that the material can have roughness, metallic character, a normal map (which outside experience tells me will control where highlights appear as we view the surface) and other aspects. We have not yet seen those in action.

As for me, I find this very tiring, trying to figure out what parts of the provided code are important and why they are there, and digging through the documentation to find out what is going on and what the underlying theory is.

YMMV, but I’m done for the morning.