GMU:Tutorials/Networking/Controlling Processing with The Captury

Introduction

Processing is usually used for the electronic arts, new media art and visual design.
If you want to have a creative work in processing with Captury, you need to receive the OSC messages from the Captury and then use the data as the variable.
This includes OSC with Processing and an example of controlling the video in Processing with Captury.
Follow this Step by Step tutorial for using PROCESSING to play the VIDEO with CAPTURY!

Tips Before Beginning

• In this tutorial, you will see how to use a single value getting from the Captury to control processing. For further processed data, referencing the tutorial by Qianqian Li: Measuring Motion with Processing

• Before the coding in Processing, make sure that the following libraries have been installed: Sketch > Import Library > Add Library
  • oscP5: An Open Sound Control (OSC) implementation
  • Video: GStreamer-based video library for Processing

• Prepare the video with a high resolution that can be showed on the video wall well. Because the change of the play speed, the video need a high frame rate to play fluently even with low speed. Here is the information about the video used here as the example:
  • Format: .mov (H.264 Codec)
  • Resolution: 1920*1280 px
  • Length: 10s
  • Frame Rate: 60 FPS
  • Frame: 0~599

Use The Data From The Captury In Processing

Data sending and receiving between the Captury and Processing over a network connection require OSC (Open Sound Control) protocol. The following is the anatomy of OSC in Processing.

PART 01 – Connect the Captury and Processing

• To build a connection between the Captury and Processing, you need to define port number. This could basically any number, and using a 4- or 5-digit number to stay out of the range of most common ports.

int localport = 12000;
int remoteport = 1065;  //Port number the Captury used to send messages

• Start oscP5, listening for incoming messages at local port.

osc = new OscP5(this, localport);

• Remote is a NetAddress, which includes an IP address and a port number. Here it used as parameter to sending OSC messages back to the Capury.

remote = new NetAddress("kosmos.medien.uni-weimar.de", remoteport);

PART 02 – Receive data from the Captury

• Inform Processing the name of the skeleton and the bone in use.

String skeleton1 = "LiQianqian";  //Skeleton name same with the Captury
String bone1 = "Root";  //The bone in use

动捕系统截图圈出名称

• Incoming OSC messages are forwarded to the oscEvent method and this function runs every time you receive data. Print the Address Pattern and the Typetag of the received OSC messages in the console.
  • Address Pattern is used to differentiate between different messages you are sending on one port. Here to differentiate between different bones and skeletons.
  • Typetag is used to define what the data type is. For the data of position x, y, z receiving from the Captury, they are all float 32 type and will be showed in console as 'f' ('i' for int 32, 's' for OSC-string, 'b' for OSC-blob, etc.)

void oscEvent(OscMessage msg) {
  if(debug) {
    print("### received an osc message.");
    print(" addrpattern: "+msg.addrPattern());
    println(" typetag: "+msg.typetag());
  }
}

本地接收信息框截图

• To make sure that all the bones in use are available and refresh them every second or so. And

void refreshSubscriptions() {
  subscribeBone(skeleton1, bone1);
}

if (frameCount % 10 == 0) {
refreshSubscriptions();
}

void subscribeBone(String skeletonId, String bone) {
  OscMessage msg = new OscMessage("/subscribe/" + skeletonId + "/blender/" + bone + "/vector");  //Create OSC message
  msg.add(50.0);  //Add a float to the OSC message
  msg.add(0.0);
  msg.add(100.0);
  osc.send(msg, remote);  //Send the message
}

动捕系统接收信息截图

PART 03 – Pass data to the object

• plug a specific bone

void plugBone(Object target, String skeleton, String bone) {
  String path =  "/" + skeleton + "/blender/" + bone + "/vector";
  osc.plug(target, "update" + bone, path);
}

• objects in space

Sphere sphere1;

sphere1 = new Sphere(1);

• pass position of Asha's root bone to the first sphere object

plugBone(sphere1, skeleton1, bone1);

PART 04 – Use the data as a variable

• position of the sphere

float x, y, z;

• update the position of the root node // (this function will be plugged to OSC)

public void updateRoot(float x, float y, float z) {
  this.x = map(x, 0, 100, -width/2, width/2);
  this.y = map(y, -100, 100, -height/2, height/2);
  this.z = map(z, 0, 100, 0, depth);
}

Play The Video By Processing

PART 01 –

Let's Enjoy The Example File

Thanks for your attention to this tutorial, and feel free to debug and improve it.

• Here is the complete code for this tutorial uploading on GitHub: Complete Code of Wool's World
  
• Example File including the movie is shared on Dropbox: Example File of Wool's World