Projection Mapping on 3D Surfaces

What we do and who we are?

For the past years we have been working with different projects that usually involve

• Graphic design
• Programming
• Projection mapping
• Electronics

.. as well as creating art installations and

• VJ-ing
• Curating
• Organizing workshops
• And audiovisual events

Irina Spicaka

Irina holds BA and MA in Visual Communication and Audiovisual Arts, has studied at the Art Academy of Latvia and Brera Academy in Milan, Italy. In her MA thesis, she was focusing on projection mapping, generative art and audiovisual composition.

Irina works as a graphic designer, curator as well as journalist in fields related to art, business and technology.

Krisjanis Rijnieks

For the past few years, Krisjanis has been exploring different media, programming languages and uses of technology and finally has arrived to a point where he can see how to finally complete his MA studies.

He is a student at the New Media MA programme of the Aalto University of Art, Design and Architecture in Helsinki and also has a background in painting.

CodeBark and CC4AV

We run a small digital media studio CodeBark and are also developing the CC4AV (Creative Coding for Live Audio and Visuals) platform that works as a placeholder for workshops and other activities that we have done in Helsinki (Pixelache pfatform), Riga (Skaņu mežs festival) and Helsingor (Click festival) so far.

We are also collaborating with

• School of Machines
• Fab Lab Berlin
• Betahaus Berlin

Projection Mapping

Multiple Meaning of Projection Mapping

The term mapping highlights the role of space in the context of visual art as this term includes a function – real time projections adjusted to spatial objects.

Projection mapping or real time projections adjusted to spatial objects is a recently created set of electronic audio visual media methods which allows to surpass standard-size projections multiplying the picture, as well as to adjust the picture for different surfaces.

A brief history

While projection mapping has recently exploded into the consciousness of artists and advertisers everywhere, the history of projection mapping dates back longer than you may imagine.

If you try Googling for “Projection Mapping” you won’t find anything older than 3 years. That is because projection mapping’s older, academic name is “Spatial Augmented Reality”. The field is also known as “video mapping”, but projection mapping seems to be winning out in the United States.

For the purposes of this history, I’m only including work that considered projection onto an arbitrarily complex surfaces. Projection onto flat and cylindrical/spherical surfaces has a much older history and goes back to the invention of cinema.

Inventions chronologically

Why It Has Been Invented?

The reason why the projection mapping technique (as we know it today) was invented and developed further was a problem when there was no place for a screen at an event venue. The projector could also be put in a place somewhere in the room where the beam of light breaks partially on the ceiling, partially on one wall and plus on some other surface, thus being absolutely useless. There was no appropriate software at that time yet and specialists of the field worked out – if gamers had various advanced possibilities for using a room in 3D environment then VJs could use it dividing the room by walls and thus providing a more in-depth overview – getting beyond the classic rectangular screen.

How can it be categorized

1. Fixed Surface Projection Mapping
   Projection mapping on buildings or objects that are unlikely to change or, so to say, carved in stone. This allows much more predictable workflow for content creation.
2. Variable Surface Projection Mapping
   Projection mapping on previously unknown surfaces. This limits the content creation to elements of subjective choice. One might prepare videos for different types of surfaces (quads, triangles, circles) or generative visuals with adjustable parameters to fit many scenarios.
3. Custom Surface Projection Mapping
   The type of projection mapping that we will do in this workshop. It allows the most flexibility as you are the architect of your own surface and the author of your own visuals.

Back to the first type it should be noted that when making projections for certain surfaces, the work is first done in 3D and animation software — classical animation methods are applied, such as lightning and special effects, narrative. The most classic example is projection mapping on architectural facades used at light festivals. The buildings are modeled first (or ready-made, modeled prototypes of them are used) and the story is created taking into consideration the design of the particular building. The process is connected with very accurate work where each mistake (let it be no more than a millimeter) both, in animation process or in the projection on the particular surface can be seen as a bias which looks like a defect. Therefore this process cannot be associated with real time performance.

Where projection mapping can be used?

• Stage Design involved in Vj-ing (inluding Light Mapping )
• Audiovisual Performance
• Art Installation
• Scenography and Motion Art
• Large Scale Projections on Architecture
• Window Display and Store Design

Tools

Decent software for creating media content and projection mapping.

Sketching tools

3D Modeling tools

Projection Mapping

Visual programming languages

Text based programming languages

Game Engines

Frame-based animation

Protocols

Software that can bridge visual output from one software to another in real-time.

Tips and Tricks

Best practices from experience.

1 Test it first

Before becomming too artistic, test if the technology works as intended.

2 Backup working versions

If your solution works, save it somewhere safe so when you continue to add changes, you can always switch back to the version that actually did work.

3 Know your surface

The more precise you will know the projection surface, the better. Especially when mapping outdoors on buildings.

4 Sketching helps

Make sketches and storyboards before doing it for real. Going through the process that way initially will help you overcome difficulties before you encounter them in the production process.

5 Always have your own adaptors

And cables. It might save you as well as others. Always have an extra cable and adaptor. It can make you the hero of the day. Or night.

6 No screensavers

Switch off screensavers and sleep for both, power and battery modes, on your laptop.

7 Cold corners

Disable hot corners or Exposé as it ruins the magic when a spectator notices what operating system are you using or that you have the same mess on the desktop as you have.

8 No instant messaging

Disable Skype or any other kind of messenger or service that tends to irritate you when least necessary.

Introduction to Blender

Creating a 3D surface for projection mapping in Blender.

Requirements

• A computer
• A 3-button mouse
• Keyboard with numeric keypad (optional)

Tasks

• Learn Blender basics
• Create your own 3D object
• Export it for assembly in real life

Install Blender

1. Download Blender from blender.org/downloads
2. Install it on your computer
3. Open it

Briefly about Blender

• Blender is a free and open source 3D modelling suite. It supports modeling, rigging, animation, simulation, rendering, compositing and motion tracking, even video editing and game creation. Advanced users can code in Python to customize the application.
• The first version of Blender has been released in the summer of 2000
• Blender is cross-platform and runs equally well on Linux, Windows and Mac computers

3D Computer Graphics

• Three-dimensional representation of coordinate values in 3D space
• Based on Cartesian coordinate system usually
• Uses a process called rendering to represent 3D objects on a 2D screen

3D Objects in 3D Software

• Consists of vertices
• Vertices form faces
• Faces form meshes
• Meshes form objects

Vertices

• Vertices are Cartesian coordinates of a point in space
• Each vertex has a X, Y and Z coordinate
• (1.2, -4, 2.5) - this is a vertex as the computer sees it

Faces

• Are supposed to be flat surfaces most of the time
• A face is formed by no less than 3 vertices
• The simplest form of a face is a triangle

Meshes

• A mesh consists of one or more faces
• It can be a open or closed 3D object
• One or more meshes form objects in Blender

Normals

• Each face has a normal
• A vector that is usually perpendicular to the face and pointing outwards the object
• Is used by the rendering engine to calculate lightning

Blender Shortcut and Cheatsheet

A couple of images that have been made to make the beginning of your Blender modeling life easier.

More about Blender

A couple of links to websites that can help you get confortable with Blender

• blender.org
  Blender main site tutorials. All the basics are there.
• blendtuts.com
  All kinds of Blender tutorials from the Blender user community.
• YouTube
  A lot of questions about Blender can be answered by searching in YouTube.

Exporting for assembly

To make a 3D object out of a 3D model we have to export it as a 2D surface

Install Export Paper Model plugin

1. Download Export Paper Model plugin for Blender:
   github.com/addam/Export-Paper-Model-from-Blender
2. Open Blender User Preferences by hitting Ctrl + Alt + U while in Blender
3. Go to Addons tab
4. Click on Install from File button and select the downloaded io_export_paper_model.py file
5. Enable the plugin by checking the box next to the plugin name in the Addons section

Prepare the Model for Exporting

1. Make sure that you use the correct units (metric) in Scene properties
2. Make sure that the object has scale 1 along all axes
3. If not, go to Object › Apply › Scale to make the current size of the object to be the default scale of it
4. Triangulate all faces by entering edit mode (hit Tab), selecting all faces (hit A) and going to Mesh › Faces › Triangulate Faces or hitting Ctrl + T

Export Paper Model

1. Select the object
2. Go to File › Export › Paper Model (.svg)
3. Adjust properties (detailed explanation in next slide)
4. Chose destination folder and file name
5. Hit the Export Paper Model button

Export Paper Model Properties

• Scale
  For precise output use scale 1:1 by entering 1 in the field. If you will attempt to export a cube that is 20cm wide, high and deep, also the exported faces will be 20cm wide and high.
• Page Size
  Page size that will be used for laying out the unwrapped faces of the model. It should be bigger than the final drawing or the output will be scaled down to fit the page.
• Unfolder DPI
  Dots Per Inch or resolution that will be used by the plugin to generate the drawing. Set it to 72 as it is the default screen resolution.
• Other properties
  You can skip other properties as they depend on subjective taste. Experiment with them and adjust if needed.

Introduction to Quartz Composer

Generative Animation in Quartz Composer

Requirements

• Mac OS X computer
• XCode, Apple Developer Tools
• with Quartz Composer

How to get XCode

1. Apple App Store
   Search for the latest version of XCode
2. Apple Developer Portal
   Sign in, go to Mac › Downloads

Graphics Tools for XCode

As they are not a part of main XCode installation for some time, you have to download them separately from Apple Developer Portal: developer.apple.com

1. Sign in, go to Mac › Downloads
2. Search for Graphics Tools

Tasks

• Learn Quartz Composer basics
• Create your own patches
• Use them in your projection mapping project

Briefly about Quartz Composer

• Quartz Qomposser node-based visual programming language that works by connecting patches and setting parameters.
• A set of patches can be combined in a macro patch. Macro patches can be placed in various hierarchies and can be edited.
• In Quartz Composer you can combine graphical and non-graphical technologies - Quartz 2D, Core Image, Core Video, OpenGL, QuickTime, MIDI, System Services and XML.

Patches

There are 3 types of patches:

1. Consumer
   Usually patches that render something on the screen
2. Processor
   Usually effects or other data processors
3. External
   MIDI controllers, keyboards, mice

Connections

1. Outputs connect to inputs
2. Output has to be of the same datatype as the input
3. You can connect an output to multiple inputs,
   but not vice versa

Layers

Consumer patches are organized in layers. They are executed one after another according to the assigned layer number.

Help on Individual Patches

You can get help on individual patches by reading instructions in the Library panel.

More about Quartz Composer

• quartzcomposer.com
  Unofficial Quartz Composer user community website
• Quartz Composer User Guide
  with love from Apple
• Learning Quartz Composer
  by Graham Robinson and Surya Buchwald

Quartz Composer Step-by-step

Get used to the environment and learn

Quartz Composer Advanced

Advanced Quartz Composer patches

Math patch

Executes a mathematical operation that involves 2 variables. Used for simple math operations like:

• (+) Addition
• (-) Substraction
• (*) Multiplication
• (/) Division
• (%) Modulo or remainder of division
• (Pow) Power, a number squared or cubed
• (Min) and (Max)
  for deciding which of the two variables is lesser or greater

Mathematical Expression

Used for defining more complex mathematical equations (e.g. round(a*b/c)*rand()) than it is possible with the Math object. Select the patch and:

• Go to Inspector › Settings to define the equation
• Each letter that you use in the equation is automatically assigned as an input of the patch
• There is one single output that is called Result
• You can use more predefined mathematical and programmatical functions like ceil(), floor(), rand(), log(), etc.

Iterator

• A replicator, used as a macro patch - that means that you have to double click an instance of it and place your logic inside.
• A QC way of making for, while or other type of programmatical loop functionality.

Iterator Variables

• Inside the Iterator patch you would use the Iterator Variables patch to get the number of assigned iterations and current iteration index.
• You can publish more inputs from within the Iterator patch to use the parameters along with Iterator Variables.

JavaScript

• If not enough with the Mathematical Expression patch.
• Can operate arrays (structures), strings as well as numbers.
• More input and output variable type possibilities than with the Mathematical Expression patch.
• Is quite heavy on the processor.

GLSL and Open CL patches

If you know what is GLSL and Open CL, you can use your knowledge in Quartz Composer through these patches.

Kineme QC patches

You can find a lot of custom Quartz Composer patches in the Kineme Quartz Composer community website: kineme.net

Projection Mapping with MadMapper

We have come so far

Summary

We are going to use our computers and available projectors to projection map a surface that has been prepared for this workshop.

A surface?

The surface for projection mapping has it's story on it's own. It has been created in Blender and then made real with the help of a lasercutter.

How is this going to work?

1. Split into groups.
2. Download and install software.
3. Create image or animation source.
4. Have fun with projection mapping!

Software

We are going to use MadMapper during this workshop. Demo version is free, but you won't be able to save your projects with it.

Use LPMT or VPT instead if you like free software.

MadMapper Cheatsheet

A couple of images that have been made to make the beginning of your mad mapping life easier.