GMU:Tutorials/Networking/Controlling Processing with The Captury: Difference between revisions

From Medien Wiki
No edit summary
No edit summary
 
(28 intermediate revisions by 2 users not shown)
Line 1: Line 1:
!! See functioning Processing [[:File:skeletonTrackingWithCaptury.pde|sketch]] provided by Felix
== Introduction ==
== Introduction ==
Processing is usually used for the electronic arts, new media art and visual design.<br>
Processing is usually used for the electronic arts, new media art and visual design.<br>
Line 22: Line 25:
Data sending and receiving between the Captury and Processing over a network connection require OSC (Open Sound Control) protocol. The following is the anatomy of OSC in Processing.<br>
Data sending and receiving between the Captury and Processing over a network connection require OSC (Open Sound Control) protocol. The following is the anatomy of OSC in Processing.<br>
<br>
<br>
*To build a connection between the Captury and Processing, the listening port and the port of the remote location address need to be same. This could basically any number, and using a 4- or 5-digit number to stay out of the range of most common ports.
'''PART 01 – Connect the Captury and Processing'''
*To build a connection between the Captury and Processing, you need to define port number. This could basically any number, and using a 4- or 5-digit number to stay out of the range of most common ports.
<source lang="Java" line start= "18">
<source lang="Java" line start= "18">
int localport = 12000;
int localport = 12000;
int remoteport = 1065;  //Port number the Captury used to send messages
int remoteport = 1065;  //Port number the Captury used to send messages
</source>
</source>
[[Image:remotePort.png|thumb|left|Port for the Captury  to Send Messages]]
<br style="clear:both">


*Start oscP5, listening for incoming messages at local port.
*Start oscP5, listening for incoming messages at local port.
Line 37: Line 43:
remote = new NetAddress("kosmos.medien.uni-weimar.de", remoteport);
remote = new NetAddress("kosmos.medien.uni-weimar.de", remoteport);
</source>
</source>
<br>
 
*To receive a value from the Captury, it's necessary to inform Processing the name of the skeleton and the bone.  
'''PART 02 – Receive data from the Captury'''
*Inform Processing the name of the skeleton and the bone in use. Remember to rename your own skeleton to tell the difference from the other.
<source lang="Java" line start= "25">
<source lang="Java" line start= "25">
String skeleton1 = "LiQianqian";  //Skeleton name same with the Captury
String skeleton1 = "LS";  //Skeleton name same with the Captury
String bone1 = "Root";  //The bone you want to track
String bone1 = "Root";  //The bone in use
</source>
</source>
<br>
[[Image:SkeletonBoneName.png|thumb|left|Name of Skeleton and Bone]]
*To make sure all the bones you want to track always can be called确保更新
<br style="clear:both">
<source lang="Java" line start= "53">
 
refreshSubscriptions();
*Incoming OSC messages are forwarded to the oscEvent method and this function runs every time you receive data. Print the Address Pattern and the Typetag of the received OSC messages in the console.
**Address Pattern is used to differentiate between different messages you are sending on one port. Here to differentiate between different bones and skeletons.
**Typetag is used to define what the data type is. For the data of position x, y, z receiving from the Captury, they are all float 32 type and will be showed in console as 'f' ('i' for int 32, 's' for OSC-string, 'b' for OSC-blob, etc.)
<source lang="Java" line start= "78">
void oscEvent(OscMessage msg) {
  if(debug) {
    print("### received an osc message.");
    print(" addrpattern: "+msg.addrPattern());
    println(" typetag: "+msg.typetag());
  }
}
</source>
</source>
[[Image:DataReceive.png|thumb|left|Data Received in Processing]]
<br style="clear:both">
*To make sure that all the bones in use are available and refresh them every second or so. And send the message back to the Captury.
<source lang="Java" line start= "58">
<source lang="Java" line start= "58">
void refreshSubscriptions() {
void refreshSubscriptions() {
Line 53: Line 74:
}
}
</source>
</source>
<source lang="Java" line start= "71">
<source lang="Java" line start= "71">
if (frameCount % 10 == 0) {
if (frameCount % 10 == 0) {
Line 58: Line 80:
}
}
</source>
</source>
*plug a specific bone
 
<source lang="Java" line start= "103">
void subscribeBone(String skeletonId, String bone) {
  OscMessage msg = new OscMessage("/subscribe/" + skeletonId + "/blender/" + bone + "/vector");  //Create OSC message
  msg.add(50.0);  //Add a float to the OSC message
  msg.add(0.0);
  msg.add(100.0);
  osc.send(msg, remote);  //Send the message
}
</source>
[[Image:DataReceive_Capture.png|thumb|left|Data Received in the Captury]]
<br style="clear:both">
 
'''PART 03 – Pass data to the object'''
*Define a function to plug a specific bone to a object.
<source lang="Java" line start= "88">
<source lang="Java" line start= "88">
void plugBone(Object target, String skeleton, String bone) {
void plugBone(Object target, String skeleton, String bone) {
Line 65: Line 101:
}
}
</source>
</source>
*capture all OSC events
 
<source lang="Java" line start= "78">
*Declare and construct a object (sphere) from the class Sphere, which is the receiver of the data.
void oscEvent(OscMessage msg) {
  if(debug) {
    print("### received an osc message.");
    print(" addrpattern: "+msg.addrPattern());
    println(" typetag: "+msg.typetag());
  }
}
</source>
*objects in space
<source lang="Java" line start= "22">
<source lang="Java" line start= "22">
Sphere sphere1;
Sphere sphere1;
Line 82: Line 109:
sphere1 = new Sphere(1);
sphere1 = new Sphere(1);
</source>
</source>
*pass position of Asha's root bone to the first sphere object
<source lang="Java" line start= "68">
sphere1.draw();
</source>
 
*Use plugBone function to pass position data of LS's Root bone to the sphere object
<source lang="Java" line start= "50">
<source lang="Java" line start= "50">
plugBone(sphere1, skeleton1, bone1);
plugBone(sphere1, skeleton1, bone1);
</source>
</source>


<source lang="Java">
'''PART 04 – Use the data as a variable'''
class Sphere {
*In the class Sphere, define the float variable for the position of the sphere.
  // position of the sphere
<source lang="Java" line start= "3">
  float x, y, z;
float x, y, z;
  Sphere( color c) {
</source>
    this.c = c;
 
  }
*Update the position of the root node. This function will be plugged to OSC.
  // draw the sphere
<source lang="Java" line start= "50">
  void draw() {
public void updateRoot(float x, float y, float z) {
 
  this.x = map(x, 0, 100, -width/2, width/2);
  }
  this.y = map(y, -100, 100, -height/2, height/2);
  // update the position of the root node  
  this.z = map(z, 0, 100, 0, depth);
  // (this function will be plugged to OSC
}
  public void updateRoot(float x, float y, float z) {
</source>
    this.x = map(x, 0, 100, -width/2, width/2);
 
    this.y = map(y, -100, 100, -height/2, height/2);
Then you can have a sphere moving follow the data from the Captury! For further application, you can use x, y, z to control whatever you want in Processing!
    this.z = map(z, 0, 100, 0, depth);}}   
 
== Control The Video By Processing ==
Since we can successfully use the data from the capture, it can be creatively to use the position of the object to control whatever you want in Processing!  <br>
The following is an example of using position values to play or reverse the video and change the play speed of the video.<br>
This video is about a 3D Chinese Knot tied from loose to tight. The 3D animation is made by 3DS MAX.<br>
<br>
'''PART 01 – Play a video in Processing''' <br>
*Basic code to play a video in Processing.
<source lang="Java" line start= "12">
import processing.video.*;  //Import libraries
</source>
<source lang="Java" line start= "16">
Movie myMovie;  //Declare a object as Movie
</source>
 
<source lang="Java" line start= "31">
void setup() {
  size(1920, 1080);  //or use 'fullScreen(P3D);'
  frameRate(60);
  background(0);
  myMovie = new Movie(this, "Chinese Knot_Final.mov");  //Load a movie, normally followed with 'play()' or 'loop()'
}
</source>
*Draw the movie by the class object (Sphere)
<source lang="Java" line start= "22">
void draw() {
    image(myMovie, 0, 0, width, height); //Position to put the movie
}
</source>
*Called every time a new frame is available to read.
<source lang="Java" line start= "62">
void movieEvent(Movie m) {
  m.read();
}
</source>
</source>


<source lang="Java">
'''PART 02 – Play or reverse the video and change the play speed''' <br>
*First thing to know is about the relationship between coordinate axis in the Captury and the Performance Platform. Here is the diagram:
[[Image:CapturyAxis.png|thumb|left|Coordinate Axis of the Performance Platform]]
[[Image:CapturyAxis2.png|thumb|left|Diagram]]
<br style="clear:both">
 
*Using position x as the condition. If user stand in the center of the platform, the speed will be slowest. The farther away from the center user stand, the faster the movie play.
*Using position y as the condition. If user stand at the side close to the Video Wall, the movie will play as usual, if else the movie will reverse.
<source lang="Java" line start= "4">
int play;  //Declare a value to judge play or reverse the video
</source>


*Get the total number of the movie frames.
<source lang="Java" line start= "7">
  int getLength() {
      return int(myMovie.duration() * myMovie.frameRate);
  }
</source>
</source>


*Get current frame.
<source lang="Java" line start= "11">
  int getFrame() {   
      return ceil(myMovie.time() * 60) - 1;
  }
</source>


*Pay attention that if the movie play to the end, it must be reversed. And if the movie reverse to the beginning, it must play as usual.
<source lang="Java" line start= "26">
    if(x < -50 && getFrame() < (getLength() - 1)){
      play = 1;
      myMovie.play();
    }
    else {
      play = -1;
      myMovie.play();
    }


== Play The Video By Processing ==
    float newSpeed = map(abs(y)/10, 0, 150, 0*play, 2*play);  //abs() to have absolute position value
    myMovie.speed(newSpeed);
</source>


'''STEP 01 – ''' <br>
== Let's Enjoy The Example File ==
Thanks for your attention to this tutorial, and feel free to debug and improve it.
*Here is the complete code for this tutorial uploading on GitHub: [https://github.com/Yunbaby1028/WoolsWorld.git Complete Code of Wool's World]<br>
*Example File including the movie is shared on Dropbox: [https://www.dropbox.com/sh/f8jcuw03nekoelz/AABeKONtJAI1vfNK4tWSZ3dTa?dl=0 Example File of Wool's World]


<br style="clear:both">
== Demonstration ==
{{#ev:youtube|https://www.youtube.com/watch?v=dufgDEGYieY|1000|left|Demonstration Video of Controlling the Video in Processing with Captury|frame}}
[https://www.youtube.com/watch?v=dufgDEGYieY Demonstration Video of Controlling the Video in Processing with Captury]

Latest revision as of 15:23, 21 June 2017

!! See functioning Processing sketch provided by Felix


Introduction

Processing is usually used for the electronic arts, new media art and visual design.
If you want to have a creative work in processing with Captury, you need to receive the OSC messages from the Captury and then use the data as the variable.
This includes OSC with Processing and an example of controlling the video in Processing with Captury.
Follow this Step by Step tutorial for using PROCESSING to play the VIDEO with CAPTURY!

Tips Before Beginning

  • In this tutorial, you will see how to use a single value getting from the Captury to control processing. For further processed data, referencing the tutorial by Qianqian Li: Measuring Motion with Processing
  • Before the coding in Processing, make sure that the following libraries have been installed: Sketch > Import Library > Add Library
    • oscP5: An Open Sound Control (OSC) implementation
    • Video: GStreamer-based video library for Processing
  • Prepare the video with a high resolution that can be showed on the video wall well. Because the change of the play speed, the video need a high frame rate to play fluently even with low speed. Here is the information about the video used here as the example:
    • Format: .mov (H.264 Codec)
    • Resolution: 1920*1280 px
    • Length: 10s
    • Frame Rate: 60 FPS
    • Frame: 0~599

Use The Data From The Captury In Processing

Data sending and receiving between the Captury and Processing over a network connection require OSC (Open Sound Control) protocol. The following is the anatomy of OSC in Processing.

PART 01 – Connect the Captury and Processing

  • To build a connection between the Captury and Processing, you need to define port number. This could basically any number, and using a 4- or 5-digit number to stay out of the range of most common ports.
int localport = 12000;
int remoteport = 1065;  //Port number the Captury used to send messages
Port for the Captury to Send Messages


  • Start oscP5, listening for incoming messages at local port.
osc = new OscP5(this, localport);
  • Remote is a NetAddress, which includes an IP address and a port number. Here it used as parameter to sending OSC messages back to the Capury.
remote = new NetAddress("kosmos.medien.uni-weimar.de", remoteport);

PART 02 – Receive data from the Captury

  • Inform Processing the name of the skeleton and the bone in use. Remember to rename your own skeleton to tell the difference from the other.
String skeleton1 = "LS";  //Skeleton name same with the Captury
String bone1 = "Root";  //The bone in use
Name of Skeleton and Bone


  • Incoming OSC messages are forwarded to the oscEvent method and this function runs every time you receive data. Print the Address Pattern and the Typetag of the received OSC messages in the console.
    • Address Pattern is used to differentiate between different messages you are sending on one port. Here to differentiate between different bones and skeletons.
    • Typetag is used to define what the data type is. For the data of position x, y, z receiving from the Captury, they are all float 32 type and will be showed in console as 'f' ('i' for int 32, 's' for OSC-string, 'b' for OSC-blob, etc.)
void oscEvent(OscMessage msg) {
  if(debug) {
    print("### received an osc message.");
    print(" addrpattern: "+msg.addrPattern());
    println(" typetag: "+msg.typetag());
  }
}
Data Received in Processing


  • To make sure that all the bones in use are available and refresh them every second or so. And send the message back to the Captury.
void refreshSubscriptions() {
  subscribeBone(skeleton1, bone1);
}
if (frameCount % 10 == 0) {
refreshSubscriptions();
}
void subscribeBone(String skeletonId, String bone) {
  OscMessage msg = new OscMessage("/subscribe/" + skeletonId + "/blender/" + bone + "/vector");  //Create OSC message
  msg.add(50.0);  //Add a float to the OSC message
  msg.add(0.0);
  msg.add(100.0);
  osc.send(msg, remote);  //Send the message
}
Data Received in the Captury


PART 03 – Pass data to the object

  • Define a function to plug a specific bone to a object.
void plugBone(Object target, String skeleton, String bone) {
  String path =  "/" + skeleton + "/blender/" + bone + "/vector";
  osc.plug(target, "update" + bone, path);
}
  • Declare and construct a object (sphere) from the class Sphere, which is the receiver of the data.
Sphere sphere1;
sphere1 = new Sphere(1);
sphere1.draw();
  • Use plugBone function to pass position data of LS's Root bone to the sphere object
plugBone(sphere1, skeleton1, bone1);

PART 04 – Use the data as a variable

  • In the class Sphere, define the float variable for the position of the sphere.
float x, y, z;
  • Update the position of the root node. This function will be plugged to OSC.
public void updateRoot(float x, float y, float z) {
  this.x = map(x, 0, 100, -width/2, width/2);
  this.y = map(y, -100, 100, -height/2, height/2);
  this.z = map(z, 0, 100, 0, depth);
}

Then you can have a sphere moving follow the data from the Captury! For further application, you can use x, y, z to control whatever you want in Processing!

Control The Video By Processing

Since we can successfully use the data from the capture, it can be creatively to use the position of the object to control whatever you want in Processing!
The following is an example of using position values to play or reverse the video and change the play speed of the video.
This video is about a 3D Chinese Knot tied from loose to tight. The 3D animation is made by 3DS MAX.

PART 01 – Play a video in Processing

  • Basic code to play a video in Processing.
import processing.video.*;  //Import libraries
Movie myMovie;  //Declare a object as Movie
void setup() {
  size(1920, 1080);  //or use 'fullScreen(P3D);'
  frameRate(60);
  background(0);
  myMovie = new Movie(this, "Chinese Knot_Final.mov");  //Load a movie, normally followed with 'play()' or 'loop()'
}
  • Draw the movie by the class object (Sphere)
void draw() {
    image(myMovie, 0, 0, width, height);  //Position to put the movie
}
  • Called every time a new frame is available to read.
void movieEvent(Movie m) {
  m.read();
}

PART 02 – Play or reverse the video and change the play speed

  • First thing to know is about the relationship between coordinate axis in the Captury and the Performance Platform. Here is the diagram:
Coordinate Axis of the Performance Platform
Diagram


  • Using position x as the condition. If user stand in the center of the platform, the speed will be slowest. The farther away from the center user stand, the faster the movie play.
  • Using position y as the condition. If user stand at the side close to the Video Wall, the movie will play as usual, if else the movie will reverse.
int play;  //Declare a value to judge play or reverse the video
  • Get the total number of the movie frames.
  int getLength() {
      return int(myMovie.duration() * myMovie.frameRate);
  }
  • Get current frame.
  int getFrame() {    
      return ceil(myMovie.time() * 60) - 1;
  }
  • Pay attention that if the movie play to the end, it must be reversed. And if the movie reverse to the beginning, it must play as usual.
    if(x < -50 && getFrame() < (getLength() - 1)){
      play = 1;
      myMovie.play();
    }
    else {
      play = -1;
      myMovie.play();
    }

    float newSpeed = map(abs(y)/10, 0, 150, 0*play, 2*play);  //abs() to have absolute position value
    myMovie.speed(newSpeed);

Let's Enjoy The Example File

Thanks for your attention to this tutorial, and feel free to debug and improve it.

Demonstration

Demonstration Video of Controlling the Video in Processing with Captury

Demonstration Video of Controlling the Video in Processing with Captury