GMU:Tutorials/Networking/Controlling Processing with The Captury: Difference between revisions

From Medien Wiki
No edit summary
No edit summary
Line 28: Line 28:
int remoteport = 1065;  //Port number the Captury used to send messages
int remoteport = 1065;  //Port number the Captury used to send messages
</source>
</source>
[[Image:remotePort.png|thumb|left|256px|Port for the Captury  to send messages]]
<br style="clear:both">


*Start oscP5, listening for incoming messages at local port.
*Start oscP5, listening for incoming messages at local port.
Line 42: Line 44:
*Inform Processing the name of the skeleton and the bone in use.  
*Inform Processing the name of the skeleton and the bone in use.  
<source lang="Java" line start= "25">
<source lang="Java" line start= "25">
String skeleton1 = "LiQianqian";  //Skeleton name same with the Captury
String skeleton1 = "LS";  //Skeleton name same with the Captury
String bone1 = "Root";  //The bone in use  
String bone1 = "Root";  //The bone in use  
</source>
</source>
动捕系统截图圈出名称
动捕系统截图圈出名称



Revision as of 22:41, 28 July 2016

Introduction

Processing is usually used for the electronic arts, new media art and visual design.
If you want to have a creative work in processing with Captury, you need to receive the OSC messages from the Captury and then use the data as the variable.
This includes OSC with Processing and an example of controlling the video in Processing with Captury.
Follow this Step by Step tutorial for using PROCESSING to play the VIDEO with CAPTURY!

Tips Before Beginning

  • In this tutorial, you will see how to use a single value getting from the Captury to control processing. For further processed data, referencing the tutorial by Qianqian Li: Measuring Motion with Processing
  • Before the coding in Processing, make sure that the following libraries have been installed: Sketch > Import Library > Add Library
    • oscP5: An Open Sound Control (OSC) implementation
    • Video: GStreamer-based video library for Processing
  • Prepare the video with a high resolution that can be showed on the video wall well. Because the change of the play speed, the video need a high frame rate to play fluently even with low speed. Here is the information about the video used here as the example:
    • Format: .mov (H.264 Codec)
    • Resolution: 1920*1280 px
    • Length: 10s
    • Frame Rate: 60 FPS
    • Frame: 0~599

Use The Data From The Captury In Processing

Data sending and receiving between the Captury and Processing over a network connection require OSC (Open Sound Control) protocol. The following is the anatomy of OSC in Processing.

PART 01 – Connect the Captury and Processing

  • To build a connection between the Captury and Processing, you need to define port number. This could basically any number, and using a 4- or 5-digit number to stay out of the range of most common ports.
int localport = 12000;
int remoteport = 1065;  //Port number the Captury used to send messages
Port for the Captury to send messages


  • Start oscP5, listening for incoming messages at local port.
osc = new OscP5(this, localport);
  • Remote is a NetAddress, which includes an IP address and a port number. Here it used as parameter to sending OSC messages back to the Capury.
remote = new NetAddress("kosmos.medien.uni-weimar.de", remoteport);

PART 02 – Receive data from the Captury

  • Inform Processing the name of the skeleton and the bone in use.
String skeleton1 = "LS";  //Skeleton name same with the Captury
String bone1 = "Root";  //The bone in use

动捕系统截图圈出名称

  • Incoming OSC messages are forwarded to the oscEvent method and this function runs every time you receive data. Print the Address Pattern and the Typetag of the received OSC messages in the console.
    • Address Pattern is used to differentiate between different messages you are sending on one port. Here to differentiate between different bones and skeletons.
    • Typetag is used to define what the data type is. For the data of position x, y, z receiving from the Captury, they are all float 32 type and will be showed in console as 'f' ('i' for int 32, 's' for OSC-string, 'b' for OSC-blob, etc.)
void oscEvent(OscMessage msg) {
  if(debug) {
    print("### received an osc message.");
    print(" addrpattern: "+msg.addrPattern());
    println(" typetag: "+msg.typetag());
  }
}

本地接收信息框截图

  • To make sure that all the bones in use are available and refresh them every second or so. And send the message back to the Captury.
void refreshSubscriptions() {
  subscribeBone(skeleton1, bone1);
}
if (frameCount % 10 == 0) {
refreshSubscriptions();
}
void subscribeBone(String skeletonId, String bone) {
  OscMessage msg = new OscMessage("/subscribe/" + skeletonId + "/blender/" + bone + "/vector");  //Create OSC message
  msg.add(50.0);  //Add a float to the OSC message
  msg.add(0.0);
  msg.add(100.0);
  osc.send(msg, remote);  //Send the message
}

动捕系统接收信息截图

PART 03 – Pass data to the object

  • Define a function to plug a specific bone to a object.
void plugBone(Object target, String skeleton, String bone) {
  String path =  "/" + skeleton + "/blender/" + bone + "/vector";
  osc.plug(target, "update" + bone, path);
}
  • Declare and construct a object (sphere) from the class Sphere, which is the receiver of the data.
Sphere sphere1;
sphere1 = new Sphere(1);
sphere1.draw();
  • Use plugBone function to pass position data of LiQianqian's Root bone to the sphere object
plugBone(sphere1, skeleton1, bone1);

PART 04 – Use the data as a variable

  • In the class Sphere, define the float variable for the position of the sphere.
float x, y, z;
  • Update the position of the root node. This function will be plugged to OSC.
public void updateRoot(float x, float y, float z) {
  this.x = map(x, 0, 100, -width/2, width/2);
  this.y = map(y, -100, 100, -height/2, height/2);
  this.z = map(z, 0, 100, 0, depth);
}

Then you can have a sphere moving follow the data from the Captury! For further application, you can use x, y, z to control whatever you want in Processing!

Control The Video By Processing

Since we can successfully use the data from the capture, it can be creatively to use the position of the object to control whatever you want in Processing!
The following is an example of using a single value (position x) to play or reverse the video and change the play speed of the video.

PART 01 – Play a video in Processing

  • Basic code to play a video in Processing.
import processing.video.*;  //Import libraries
Movie myMovie;  //Declare a object as Movie
void setup() {
  size(1920, 1080);  //or use 'fullScreen(P3D);'
  frameRate(60);
  background(0);
  myMovie = new Movie(this, "Chinese Knot_Final.mov");  //Load a movie, normally followed with 'play()' or 'loop()'
}
  • Draw the movie by the class object (Sphere)
void draw() {
    image(myMovie, 0, 0, width, height);  //Position to put the movie
}
  • Called every time a new frame is available to read.
void movieEvent(Movie m) {
  m.read();
}

PART 02 – Play or reverse the video and change the play speed

  • Using position x as the condition. If user stand at the left edge, the speed will be slowest. The righter user stand, the faster the movie play.
  • Using position y as the condition. If user stand close to the platform, the movie will play as usual, if else the movie will reverse.
int play;  //Declare a value to judge play or reverse the video
  • Get the total number of the movie frames.
  int getLength() {
      return int(myMovie.duration() * myMovie.frameRate);
  }
  • Get current frame.
  int getFrame() {    
      return ceil(myMovie.time() * 60) - 1;
  }
  • Pay attention that if the movie play to the end, it must be reversed. And if the movie reverse to the beginning, it must play as usual.
    if(y > 200 && getFrame() < (getLength() - 1)){
      play = 1;
      myMovie.play();
    }
    else {
      play = -1;
      myMovie.play();
    }

    if(getFrame() < 1) myMovie.play();

    float newSpeed = map(x, 0, 150, 0*play, 2*play);
    myMovie.speed(newSpeed);

Let's Enjoy The Example File

Thanks for your attention to this tutorial, and feel free to debug and improve it.