Courses

CS5263-ProgrammingProject#2-2013

CS5253 Programming Project 2

Video Streaming Using RTSP and RTP to Your Android Phone

The Code

In this project you will implement a streaming video server and client that communicate using the Real-Time Streaming Protocol (RTSP) and send data using the Real-time Transfer Protocol (RTP). Your task is to implement the RTSP protocol in the client and implement the RTP packetization in the server.

We will provide you code that implements the RTSP protocol in the server, the RTP de-packetization in the client, and takes care of displaying the transmitted video. You do not need to touch this code. The partially finished java files can be found here

Notice that the RTP/RTSP client was developed for desktop computer. You will need to port the software to Android, and do a live demonsration in the class.

Classes

There are 4 classes in the assignment.

Client: This class implements the client and the user interface which you use to send RTSP commands and which is used to display the video. Below is what the interface looks like. You will need to implement the actions that are taken when the buttons are pressed.

Server: This class implements the server which responds to the RTSP requests and streams back the video. The RTSP interaction is already implemented and the server calls routines in the RTPpacket class to packetize the video data. You do not need to modify this class.

RTPpacket: This class is used to handle the RTP packets. It has separate routines for handling the received packets at the client side which is given and you do not need to modify it. You will need to complete the first constructor of this class to implement RTP-packetization of the video data. The second constructor is used by the client to de-packetize the data. You do not need to modify that.

VideoStream: This class is used to read video data from the file on disk. You do not need to modify this class.

Running the Code

After completing the code, you can run it as follows.

First, start the server with the command

java Server server_port

where server_port is the port your server listens to for incoming RTSP connections. The standard RTSP port is 554, but you will need to choose a port number greater than 1024.

Then, start the client with the command

java Client server_name server_port video_file

where server_host is the name of the machine where the server is running, server_port is the port the server is listening on, and video_file is the name of the file you want to request (we have provided one example file movie.Mjpeg, which can be found in the tar file). The file format is described in the Appendix.

The client opens a connection to the server and pops up a window like this:

cs5262-project1-fig1

You can send RTSP commands to the server by pressing the buttons. A normal RTSP interaction goes as follows.

  1. Client sends SETUP. This command is used to set up the session and transport parameters.
  2. Client sends PLAY. This starts the playback.
  3. Client may send PAUSE if it wants to pause during playback.
  4. Client sends TEARDOWN. This terminates the session and closes the connection.

The server alway replies to all the messages the client sends. The reply codes are roughly the same as in HTTP. The code 200 means that the request was successful. In this project you do not need to implement any other reply codes. For more information about RTSP, please see RFC 2326.


1. Client

Your first task is to implement the RTSP on the client side. To do this, you need to complete the functions that are called when the user clicks on the buttons in the user interface. For each button in the interface there is a handler function in the code. You will need to implement the following actions in each handler function.

When the client starts, it also opens the RTSP socket to the server. Use this socket for sending all RTSP requests.

SETUP

  • Create a socket for receiving RTP data and set the timeout on the socket to 5 milliseconds.
  • Send SETUP request to server. You will need to insert the Transport header in which you specify the port for the RTP data socket you just created.
  • Read reply from server and parse the Session header in the response to get the session ID.

PLAY

  • Send PLAY request. You must insert the Session header and use the session ID returned in the SETUP response. You must not put the Transport header in this request.
  • Read server's response.

PAUSE

  • Send PAUSE request. You must insert the Session header and use the session ID returned in the SETUP response. You must not put the Transport header in this request.
  • Read server's response.

TEARDOWN

  • Send TEARDOWN request. You must insert the Session header and use the session ID returned in the SETUP response. You must not put the Transport header in this request.
  • Read server's response.

Note: You must insert the CSeq header in every request you send. The value of the CSeq header is a number which is incremented by one for each request you send.

Example

Here is a sample interaction between the client and server. The client's requests are marked with C: and server's replies with S:. In this project both the client and the server are very simple. They do not have sophisticated parsing routines and they expect the header fields to be in the order you see below. That is, in a request, the first header is CSeq, and the second one is either Transport (for SETUP) or Session (for all other requests). In the reply, CSeq is again the first and Session is the second.

C: SETUP movie.Mjpeg RTSP/1.0
C: CSeq: 1
C: Transport: RTP/UDP; client_port= 25000
S: RTSP/1.0 200 OK
S: CSeq: 1
S: Session: 123456
C: PLAY movie.Mjpeg RTSP/1.0
C: CSeq: 2
C: Session: 123456
S: RTSP/1.0 200 OK
S: CSeq: 2
S: Session: 123456
C: PAUSE movie.Mjpeg RTSP/1.0
C: CSeq: 3
C: Session: 123456
S: RTSP/1.0 200 OK
S: CSeq: 3
S: Session: 123456
C: PLAY movie.Mjpeg RTSP/1.0
C: CSeq: 4
C: Session: 123456
S: RTSP/1.0 200 OK
S: CSeq: 4
S: Session: 123456
C: TEARDOWN movie.Mjpeg RTSP/1.0
C: CSeq: 5
C: Session: 123456
S: RTSP/1.0 200 OK
S: CSeq: 5
S: Session: 123456

Client State

One of the key differences between HTTP and RTSP is that in RTSP each session has a state. In this project you will need to keep the client's state up-to-date. Client changes state when it receives a reply from the server according to the following state diagram.

cs5262-project1-fig2

 


2. Server

On the server you will need to implement the packetization of the video data into RTP packets. For this you will need to create the packet, set the fields in the packet header, and copy the payload (i.e., one video frame) into the packet.

When the server receives the PLAY-request from the client, it starts a timer which is triggered every 100 ms. At these times the server will read one video frame from the file and send it to the client. The server creates an RTPpacket-object which is the RTP-encapsulation of the video frame.

The server calls the first constructor of the class RTPpacket to perform the encapsulation. Your task is to write this function. You will need to do the following: (the letters in parenthesis refer to the fields in the RTP packet format below)

  1. Set the RTP-version field (V). You must set this to 2.
  2. Set padding (P), extension (X), number of contributing sources (CC), and marker (M) fields. These are all set to zero in this project.
  3. Set payload type field (PT). In this project we use MJPEG and the type for that is 26.
  4. Set the sequence number. The server gives this the sequence number as the Framenb argument to the constructor.
  5. Set the timestamp. The server gives this number as the Time argument to the constructor.
  6. Set the source identifier (SSRC). This field identifies the server. You can pick any integer value you like.

Because we have no other contributing sources (field CC == 0), the CSRC-field does not exist. The length of the packet header is therefore 12 bytes, or the first three lines from the diagram below.

cs5262-project1-fig3 

You must fill in the header in the array header of the RTPpacket-class. You will also need to copy the payload (given as argument data) to the variable payload. The length of the payload is given in the argumentdata_length.

The above diagram is in the network byte order (also known as big-endian). The Java Virtual Machine uses the same byte order so you do not need to transform your packet header into the network byte order.

For more details on RTP, please see RFC 1889.

Twiddling the Bits

Here are some examples on how to set and check individual bits or groups of bits. Note that in the RTP packet header format smaller bit-numbers refer to higher order bits, that is, bit number 0 of a byte is 2^7 and bit number 7 is 1 (or 2^0). In the examples below, the bit numbers refer to the numbers in the above diagram.

Because the header-field of the RTPpacket class is an array of type byte, you will need to set the header one byte at a time, that is in groups of 8 bits. The first byte has bits 0-7, the second byte has bits 8-15, and so on. In Java an int is 32 bits or 4 bytes.

To set bit number n in variable mybyte of type byte:

mybyte = mybyte | 1 << (7 - n);

To set bits n and n + 1 to the value of foo in variable mybyte:

mybyte = mybyte | foo << (7 - n);

Note that foo must have a value that can be expressed with 2 bits, that is, 0, 1, 2, or 3.

To copy a 16-bit integer foo into 2 bytes, b1 and b2:

b1 = foo >> 8;
b2 = foo & 0xFF;

After this, b1 will have the 8 high-order bits of foo and b2 will have the 8 low-order bits of foo.

You can copy a 32-bit integer into 4 bytes in a similar way.

Bit Example

Suppose we want to fill in the first byte of the RTP packet header with the following values:

  • V = 2
  • P = 0
  • X = 0
  • CC = 3

In binary this would be represented as

 1 0  | 0  | 0  | 0 0 1 1
V=2   P   X   CC = 3
2^7 . . . . . . . . 2^0

Bonus Points

Students are encouraged to turn in a short report to address the following questions. The report must be in PDF format, and turned in along with the source code of your projects. Each question is worth 1 point. 

  • Instead of the normal server given to you, use the class called FunkyServer.class, i.e., run it with java FunkyServer server_port. What do you see at the client? Explain what happens, why, and fix it.
  • Calculate statistics about the session. You will need to calculate RTP packet loss rate, video data rate (in bits or bytes per second), and any other interesting statistics you can think of.
  • The user interface on the client has 4 buttons for the 4 actions. If you compare this to a standard media player, such as RealPlayer or Windows Media Player, you can see that they have only 3 buttons for the same actions, namely, PLAY, PAUSE, and STOP (roughly corresponding to TEARDOWN). There is no SETUP-button available to the user. Given that SETUP is mandatory in an RTSP-interaction, how would you implement that in a media player? When does the client send the SETUP? Come up with a solution and implement it. Is it appropriate to send TEARDOWN when user clicks on the stop-button?
  • Currently the client and server only implement the minimum necessary RTSP interactions and PAUSE. Implement the method DESCRIBE which is used to pass information about the media stream. When the server receives a DESCRIBE-request, it sends back a session description file which tells the client what kinds of streams are in the session and what encodings are used.

Appendix

We use a proprietary MJPEG (Motion JPEG) format.

In this project, the server streams a video which has been encoded into a proprietary MJPEG file format. This format stores the video as concatenated JPEG-encoded images, with each image being preceded by a 5-Byte header which indicates the bit size of the image. The server parses the bitstream of the MJPEG file to extract the JPEG images on the fly. The server sends the images to the client at periodic intervals. The client then displays the individual JPEG images as they arrive from the server.

Submission

Please submit a tar file (<last_name>.tar.gz) of your source directory (including the directory). Please email the TA ( This email address is being protected from spambots. You need JavaScript enabled to view it. ) your file by 3:30 p.m. on Monday. In the same email, you may include an optional PDF file addressing the bonus question (please see above). Students will set up a demo time with the TA. A students gets 10 points if the smartphone demo is successful, and gets 1 points for correctly answering each bonus question. 

 

 

CS5262-ProgrammingProject#1-2011

CS5252 Programming Project 1

Video Streaming Using RTSP and RTP

The Code

In this project you will implement a streaming video server and client that communicate using the Real-Time Streaming Protocol (RTSP) and send data using the Real-time Transfer Protocol (RTP). Your task is to implement the RTSP protocol in the client and implement the RTP packetization in the server.

We will provide you code that implements the RTSP protocol in the server, the RTP de-packetization in the client, and takes care of displaying the transmitted video. You do not need to touch this code. The partially finished java files can be found here

Classes

There are 4 classes in the assignment.

Client: This class implements the client and the user interface which you use to send RTSP commands and which is used to display the video. Below is what the interface looks like. You will need to implement the actions that are taken when the buttons are pressed.

Server: This class implements the server which responds to the RTSP requests and streams back the video. The RTSP interaction is already implemented and the server calls routines in the RTPpacket class to packetize the video data. You do not need to modify this class.

RTPpacket: This class is used to handle the RTP packets. It has separate routines for handling the received packets at the client side which is given and you do not need to modify it. You will need to complete the first constructor of this class to implement RTP-packetization of the video data. The second constructor is used by the client to de-packetize the data. You do not need to modify that.

VideoStream: This class is used to read video data from the file on disk. You do not need to modify this class.

Running the Code

After completing the code, you can run it as follows.

First, start the server with the command

java Server server_port

where server_port is the port your server listens to for incoming RTSP connections. The standard RTSP port is 554, but you will need to choose a port number greater than 1024.

Then, start the client with the command

java Client server_name server_port video_file

where server_host is the name of the machine where the server is running, server_port is the port the server is listening on, and video_file is the name of the file you want to request (we have provided one example file movie.Mjpeg, which can be found in the tar file). The file format is described in the Appendix.

The client opens a connection to the server and pops up a window like this:

cs5262-project1-fig1

You can send RTSP commands to the server by pressing the buttons. A normal RTSP interaction goes as follows.

  1. Client sends SETUP. This command is used to set up the session and transport parameters.
  2. Client sends PLAY. This starts the playback.
  3. Client may send PAUSE if it wants to pause during playback.
  4. Client sends TEARDOWN. This terminates the session and closes the connection.

The server alway replies to all the messages the client sends. The reply codes are roughly the same as in HTTP. The code 200 means that the request was successful. In this project you do not need to implement any other reply codes. For more information about RTSP, please see RFC 2326.


1. Client

Your first task is to implement the RTSP on the client side. To do this, you need to complete the functions that are called when the user clicks on the buttons in the user interface. For each button in the interface there is a handler function in the code. You will need to implement the following actions in each handler function.

When the client starts, it also opens the RTSP socket to the server. Use this socket for sending all RTSP requests.

SETUP

  • Create a socket for receiving RTP data and set the timeout on the socket to 5 milliseconds.
  • Send SETUP request to server. You will need to insert the Transport header in which you specify the port for the RTP data socket you just created.
  • Read reply from server and parse the Session header in the response to get the session ID.

PLAY

  • Send PLAY request. You must insert the Session header and use the session ID returned in the SETUP response. You must not put the Transport header in this request.
  • Read server's response.

PAUSE

  • Send PAUSE request. You must insert the Session header and use the session ID returned in the SETUP response. You must not put the Transport header in this request.
  • Read server's response.

TEARDOWN

  • Send TEARDOWN request. You must insert the Session header and use the session ID returned in the SETUP response. You must not put the Transport header in this request.
  • Read server's response.

Note: You must insert the CSeq header in every request you send. The value of the CSeq header is a number which is incremented by one for each request you send.

Example

Here is a sample interaction between the client and server. The client's requests are marked with C: and server's replies with S:. In this project both the client and the server are very simple. They do not have sophisticated parsing routines and they expect the header fields to be in the order you see below. That is, in a request, the first header is CSeq, and the second one is either Transport (for SETUP) or Session (for all other requests). In the reply, CSeq is again the first and Session is the second.

C: SETUP movie.Mjpeg RTSP/1.0
C: CSeq: 1
C: Transport: RTP/UDP; client_port= 25000
S: RTSP/1.0 200 OK
S: CSeq: 1
S: Session: 123456
C: PLAY movie.Mjpeg RTSP/1.0
C: CSeq: 2
C: Session: 123456
S: RTSP/1.0 200 OK
S: CSeq: 2
S: Session: 123456
C: PAUSE movie.Mjpeg RTSP/1.0
C: CSeq: 3
C: Session: 123456
S: RTSP/1.0 200 OK
S: CSeq: 3
S: Session: 123456
C: PLAY movie.Mjpeg RTSP/1.0
C: CSeq: 4
C: Session: 123456
S: RTSP/1.0 200 OK
S: CSeq: 4
S: Session: 123456
C: TEARDOWN movie.Mjpeg RTSP/1.0
C: CSeq: 5
C: Session: 123456
S: RTSP/1.0 200 OK
S: CSeq: 5
S: Session: 123456

Client State

One of the key differences between HTTP and RTSP is that in RTSP each session has a state. In this project you will need to keep the client's state up-to-date. Client changes state when it receives a reply from the server according to the following state diagram.

cs5262-project1-fig2

 


2. Server

On the server you will need to implement the packetization of the video data into RTP packets. For this you will need to create the packet, set the fields in the packet header, and copy the payload (i.e., one video frame) into the packet.

When the server receives the PLAY-request from the client, it starts a timer which is triggered every 100 ms. At these times the server will read one video frame from the file and send it to the client. The server creates an RTPpacket-object which is the RTP-encapsulation of the video frame.

The server calls the first constructor of the class RTPpacket to perform the encapsulation. Your task is to write this function. You will need to do the following: (the letters in parenthesis refer to the fields in the RTP packet format below)

  1. Set the RTP-version field (V). You must set this to 2.
  2. Set padding (P), extension (X), number of contributing sources (CC), and marker (M) fields. These are all set to zero in this project.
  3. Set payload type field (PT). In this project we use MJPEG and the type for that is 26.
  4. Set the sequence number. The server gives this the sequence number as the Framenb argument to the constructor.
  5. Set the timestamp. The server gives this number as the Time argument to the constructor.
  6. Set the source identifier (SSRC). This field identifies the server. You can pick any integer value you like.

Because we have no other contributing sources (field CC == 0), the CSRC-field does not exist. The length of the packet header is therefore 12 bytes, or the first three lines from the diagram below.

cs5262-project1-fig3 

You must fill in the header in the array header of the RTPpacket-class. You will also need to copy the payload (given as argument data) to the variable payload. The length of the payload is given in the argumentdata_length.

The above diagram is in the network byte order (also known as big-endian). The Java Virtual Machine uses the same byte order so you do not need to transform your packet header into the network byte order.

For more details on RTP, please see RFC 1889.

Twiddling the Bits

Here are some examples on how to set and check individual bits or groups of bits. Note that in the RTP packet header format smaller bit-numbers refer to higher order bits, that is, bit number 0 of a byte is 2^7 and bit number 7 is 1 (or 2^0). In the examples below, the bit numbers refer to the numbers in the above diagram.

Because the header-field of the RTPpacket class is an array of type byte, you will need to set the header one byte at a time, that is in groups of 8 bits. The first byte has bits 0-7, the second byte has bits 8-15, and so on. In Java an int is 32 bits or 4 bytes.

To set bit number n in variable mybyte of type byte:

mybyte = mybyte | 1 << (7 - n);

To set bits n and n + 1 to the value of foo in variable mybyte:

mybyte = mybyte | foo << (7 - n);

Note that foo must have a value that can be expressed with 2 bits, that is, 0, 1, 2, or 3.

To copy a 16-bit integer foo into 2 bytes, b1 and b2:

b1 = foo >> 8;
b2 = foo & 0xFF;

After this, b1 will have the 8 high-order bits of foo and b2 will have the 8 low-order bits of foo.

You can copy a 32-bit integer into 4 bytes in a similar way.

Bit Example

Suppose we want to fill in the first byte of the RTP packet header with the following values:

  • V = 2
  • P = 0
  • X = 0
  • CC = 3

In binary this would be represented as

 1 0  | 0  | 0  | 0 0 1 1
V=2   P   X   CC = 3
2^7 . . . . . . . . 2^0

Bonus Points

Students are encouraged to turn in a short report to address the following questions. The report must be in PDF format, and turned in along with the source code of your projects. Each question is worth 1.5 point. 

  • Instead of the normal server given to you, use the class called FunkyServer.class, i.e., run it with java FunkyServer server_port. What do you see at the client? Explain what happens, why, and fix it.
  • Calculate statistics about the session. You will need to calculate RTP packet loss rate, video data rate (in bits or bytes per second), and any other interesting statistics you can think of.
  • The user interface on the client has 4 buttons for the 4 actions. If you compare this to a standard media player, such as RealPlayer or Windows Media Player, you can see that they have only 3 buttons for the same actions, namely, PLAY, PAUSE, and STOP (roughly corresponding to TEARDOWN). There is no SETUP-button available to the user. Given that SETUP is mandatory in an RTSP-interaction, how would you implement that in a media player? When does the client send the SETUP? Come up with a solution and implement it. Is it appropriate to send TEARDOWN when user clicks on the stop-button?
  • Currently the client and server only implement the minimum necessary RTSP interactions and PAUSE. Implement the method DESCRIBE which is used to pass information about the media stream. When the server receives a DESCRIBE-request, it sends back a session description file which tells the client what kinds of streams are in the session and what encodings are used.

Appendix

We use a proprietary MJPEG (Motion JPEG) format.

In this project, the server streams a video which has been encoded into a proprietary MJPEG file format. This format stores the video as concatenated JPEG-encoded images, with each image being preceded by a 5-Byte header which indicates the bit size of the image. The server parses the bitstream of the MJPEG file to extract the JPEG images on the fly. The server sends the images to the client at periodic intervals. The client then displays the individual JPEG images as they arrive from the server.

Submission

Please submit a tar file (<last_name>.tar.gz) of your source directory (including the directory). Please email the instructor your file by 10:00 a.m. on Friday. In the same email, you may include an optional PDF file addressing the bonus question (please see above). Students will do a demo on the instructor's laptop (schedule will be announced at a later time). A students gets 10 points if the demo is successful, and gets 1.5 points for correctly answering each bonus question. 

 

 

CS5262-Project-2011

Each student is responsible to propose and execute a term project. For a larger-scale project, two or three students may form a group, but each of them must work on a different part of the whole project, e.g., a student may work on adaptive streaming server while the other one works on the decoder on a smartphones.


Kinds of projects:

  • Brave new idea: Develop a new idea to solve a challenging research problem in multimedia networking and systems. You must present your innovative idea. Simulation and/or experimental results shall be presented. Hopefully, you will get a publishable technical report (and A+) by the end of the semester.
  • Quantitative and/or qualitative comparisons of multiple published solutions: Implement existing algorithms, techniques, or systems. You need to conduct simulations or experiments to evaluate the solutions. Lessons learned from the simulations or experiments must be discussed.
  • Survey a well-studied problem and its solutions: Write a survey paper to discuss an interesting research problem in multimedia research. You need to discuss the directions of future research related to that research problem.

Potential project topics, please feel free to suggest new topics:

  • Location-based bandwidth estimation system. Please see the instructor if you want to know more. 
  • Generic remote rendering system for online games and (network) latency compensation techniques. Please see the instructor if you like to know more. 
  • Optimized multimedia content transfer on Android smartphones. Please see the instructor if you want to know more. 
  • 3D Virtual Reality System. Please see the instructor if you like to know more. 
  • Intuitive human-computer interaction among multiple smartphones and tablets enabled by WiFi Direct. Please see the instructor if you like to know more. 
  • Port opensvc, an open-source H.264/SVC decoder to Android 2.3+ using NDK. Understand the limitations of opensvc. Optimize opensvc in terms of time and space complexities.
  • Modify the open-source Darwin streaming server to support H.264/SVC streams. Partial implementation (without verification) is available upon request.
  • Implementation and evaluation of a media gateway (a WiFi router that provides simple QoS for multimedia streams). See http://dl.acm.org/citation.cfm?id=1496061 and http://dl.acm.org/citation.cfm?id=1989245 for examples, and propose enhancements on top of their works.
  • Develop an SVC streaming system that leverages stream re-writing feature of H.264/SVC standard. Stream re-writing efficiently converts an scalable H.264/SVC stream into a regular H.264/AVC stream. This allows common video players (such as VLC) and most smartphones to decode SVC streams. The H.264/SVC standard documents the details of how to implement this. Port your code to a WiFi gateway (e.g., an OpenWRT compatible access point) and conduct real experiments on smartphones to show the benefit (if any) of scalable video coding.
  • Haptics-based Multimedia Communications. Multimedia applications that interact with biometric sensors (e.g., e-touch, smell, ...). See http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5228689 for an overview. For more examples, check the work being done at University of Ottawa, MCRLab.
  • Stereoscope and/or high definition video encoding and streaming: survey and comparison of different models
  • Survey existing open-source SIP video phones. Set up a SIP server and several open-source SIP video phones among smartphones and/or laptops. Qualitatively and quantitatively compare different implementations, and identify potential research problems.
  • Efficient implementation of network coding schemes in distributed media streaming systems.
  • Video streaming and broadcasting over 3G, WiFi, or WiMAX networks to mobile devices, such as Android smartphones.

Executions:

The main objective of the term project is to show the students how to conduct systems research in multimedia area. The instructor will walk you through well-defined steps to complete the project. These steps are explained below.

  1. Literature survey: We first need to know what have been done before, more importantly, what haven't been done in the literature. In this phase, a student searches and reads papers published in reputable journals and conferences. You are excepted to classify all existing solutions into meaningful groups, and give high-level description on representative solutions.
  2. Problem statement: After knowing what hasn't been done, the student should formally define the research problem he/she plans to address in the term project. The student should clearly define the system models and mathematical notations.
  3. Problem solution: Each student proposes a new, better solution to address the research problem. Often, your solution is inspired by existing one studied in the literature survey; you may also borrow (algorithmic) ideas from other areas. The student should analyze the solution from the aspects of time and space complexities, and argue that the solution is (reasonably) scalable.
  4. Simulations/Experiments:A student must evaluate the proposed solution using simulations and/or experiments. Real experiments are much more convincing than simulations, because simulator often make assumptions to speed up implementation. While real experiments are highly encouraged, simulations are also useful when the test scenarios are too complex to be set up in a real testbed.

Deliverables:

In this course, students are required to turn in their reports three times. The reports must be written in Latex and this template. In the first month, each student will pick a research topic, and write a proposal consisting of literature survey and problem statement. If a students needs to borrow equipments (such as wireless routers and smartphones) from the instructor, he/she should explain why the equipments are needed (e.g., experimental setups) and identify the suitable make/model in the presentation. Each student is given 15 min to present their proposal in front of the class. Students are then given 1.5 month to work on their problem solutions and conduct some simulations/experiments. They add new materials to their proposal, and give a 15 min presentation to report the progress. Next, students work on their projects for another 1.5 month, and present and (optional) demo their term projects in the week before the final exam. We don't have final exams, and the final technical reports are due one week after the last lecture. In summary, each students will turn in three incremental technical reports, give three 15-min talks, and optionally demo their system in the final presentation. All these deliverable will be graded; demos are highly encouraged and incurs considerable bonus points.

To ensure the course covers all aspects of multimedia networking, each student will give a 1-hr presentation about one or two papers relevant to his/her project topic. The presentation will be graded. Students should recommend good and recent papers to present. If that's not possible, the instructor will assign papers to individual students after the proposal presentation.

CS5263-ProgrammingProject#2-2012

CS5253 Programming Project 2

Video Streaming Using RTSP and RTP to Your Android Phone

The Code

In this project you will implement a streaming video server and client that communicate using the Real-Time Streaming Protocol (RTSP) and send data using the Real-time Transfer Protocol (RTP). Your task is to implement the RTSP protocol in the client and implement the RTP packetization in the server.

We will provide you code that implements the RTSP protocol in the server, the RTP de-packetization in the client, and takes care of displaying the transmitted video. You do not need to touch this code. The partially finished java files can be found here

Notice that the RTP/RTSP client was developed for desktop computer. You will need to port the software to Android, and do a live demonsration in the class.

Classes

There are 4 classes in the assignment.

Client: This class implements the client and the user interface which you use to send RTSP commands and which is used to display the video. Below is what the interface looks like. You will need to implement the actions that are taken when the buttons are pressed.

Server: This class implements the server which responds to the RTSP requests and streams back the video. The RTSP interaction is already implemented and the server calls routines in the RTPpacket class to packetize the video data. You do not need to modify this class.

RTPpacket: This class is used to handle the RTP packets. It has separate routines for handling the received packets at the client side which is given and you do not need to modify it. You will need to complete the first constructor of this class to implement RTP-packetization of the video data. The second constructor is used by the client to de-packetize the data. You do not need to modify that.

VideoStream: This class is used to read video data from the file on disk. You do not need to modify this class.

Running the Code

After completing the code, you can run it as follows.

First, start the server with the command

java Server server_port

where server_port is the port your server listens to for incoming RTSP connections. The standard RTSP port is 554, but you will need to choose a port number greater than 1024.

Then, start the client with the command

java Client server_name server_port video_file

where server_host is the name of the machine where the server is running, server_port is the port the server is listening on, and video_file is the name of the file you want to request (we have provided one example file movie.Mjpeg, which can be found in the tar file). The file format is described in the Appendix.

The client opens a connection to the server and pops up a window like this:

cs5262-project1-fig1

You can send RTSP commands to the server by pressing the buttons. A normal RTSP interaction goes as follows.

  1. Client sends SETUP. This command is used to set up the session and transport parameters.
  2. Client sends PLAY. This starts the playback.
  3. Client may send PAUSE if it wants to pause during playback.
  4. Client sends TEARDOWN. This terminates the session and closes the connection.

The server alway replies to all the messages the client sends. The reply codes are roughly the same as in HTTP. The code 200 means that the request was successful. In this project you do not need to implement any other reply codes. For more information about RTSP, please see RFC 2326.


1. Client

Your first task is to implement the RTSP on the client side. To do this, you need to complete the functions that are called when the user clicks on the buttons in the user interface. For each button in the interface there is a handler function in the code. You will need to implement the following actions in each handler function.

When the client starts, it also opens the RTSP socket to the server. Use this socket for sending all RTSP requests.

SETUP

  • Create a socket for receiving RTP data and set the timeout on the socket to 5 milliseconds.
  • Send SETUP request to server. You will need to insert the Transport header in which you specify the port for the RTP data socket you just created.
  • Read reply from server and parse the Session header in the response to get the session ID.

PLAY

  • Send PLAY request. You must insert the Session header and use the session ID returned in the SETUP response. You must not put the Transport header in this request.
  • Read server's response.

PAUSE

  • Send PAUSE request. You must insert the Session header and use the session ID returned in the SETUP response. You must not put the Transport header in this request.
  • Read server's response.

TEARDOWN

  • Send TEARDOWN request. You must insert the Session header and use the session ID returned in the SETUP response. You must not put the Transport header in this request.
  • Read server's response.

Note: You must insert the CSeq header in every request you send. The value of the CSeq header is a number which is incremented by one for each request you send.

Example

Here is a sample interaction between the client and server. The client's requests are marked with C: and server's replies with S:. In this project both the client and the server are very simple. They do not have sophisticated parsing routines and they expect the header fields to be in the order you see below. That is, in a request, the first header is CSeq, and the second one is either Transport (for SETUP) or Session (for all other requests). In the reply, CSeq is again the first and Session is the second.

C: SETUP movie.Mjpeg RTSP/1.0
C: CSeq: 1
C: Transport: RTP/UDP; client_port= 25000
S: RTSP/1.0 200 OK
S: CSeq: 1
S: Session: 123456
C: PLAY movie.Mjpeg RTSP/1.0
C: CSeq: 2
C: Session: 123456
S: RTSP/1.0 200 OK
S: CSeq: 2
S: Session: 123456
C: PAUSE movie.Mjpeg RTSP/1.0
C: CSeq: 3
C: Session: 123456
S: RTSP/1.0 200 OK
S: CSeq: 3
S: Session: 123456
C: PLAY movie.Mjpeg RTSP/1.0
C: CSeq: 4
C: Session: 123456
S: RTSP/1.0 200 OK
S: CSeq: 4
S: Session: 123456
C: TEARDOWN movie.Mjpeg RTSP/1.0
C: CSeq: 5
C: Session: 123456
S: RTSP/1.0 200 OK
S: CSeq: 5
S: Session: 123456

Client State

One of the key differences between HTTP and RTSP is that in RTSP each session has a state. In this project you will need to keep the client's state up-to-date. Client changes state when it receives a reply from the server according to the following state diagram.

cs5262-project1-fig2

 


2. Server

On the server you will need to implement the packetization of the video data into RTP packets. For this you will need to create the packet, set the fields in the packet header, and copy the payload (i.e., one video frame) into the packet.

When the server receives the PLAY-request from the client, it starts a timer which is triggered every 100 ms. At these times the server will read one video frame from the file and send it to the client. The server creates an RTPpacket-object which is the RTP-encapsulation of the video frame.

The server calls the first constructor of the class RTPpacket to perform the encapsulation. Your task is to write this function. You will need to do the following: (the letters in parenthesis refer to the fields in the RTP packet format below)

  1. Set the RTP-version field (V). You must set this to 2.
  2. Set padding (P), extension (X), number of contributing sources (CC), and marker (M) fields. These are all set to zero in this project.
  3. Set payload type field (PT). In this project we use MJPEG and the type for that is 26.
  4. Set the sequence number. The server gives this the sequence number as the Framenb argument to the constructor.
  5. Set the timestamp. The server gives this number as the Time argument to the constructor.
  6. Set the source identifier (SSRC). This field identifies the server. You can pick any integer value you like.

Because we have no other contributing sources (field CC == 0), the CSRC-field does not exist. The length of the packet header is therefore 12 bytes, or the first three lines from the diagram below.

cs5262-project1-fig3 

You must fill in the header in the array header of the RTPpacket-class. You will also need to copy the payload (given as argument data) to the variable payload. The length of the payload is given in the argumentdata_length.

The above diagram is in the network byte order (also known as big-endian). The Java Virtual Machine uses the same byte order so you do not need to transform your packet header into the network byte order.

For more details on RTP, please see RFC 1889.

Twiddling the Bits

Here are some examples on how to set and check individual bits or groups of bits. Note that in the RTP packet header format smaller bit-numbers refer to higher order bits, that is, bit number 0 of a byte is 2^7 and bit number 7 is 1 (or 2^0). In the examples below, the bit numbers refer to the numbers in the above diagram.

Because the header-field of the RTPpacket class is an array of type byte, you will need to set the header one byte at a time, that is in groups of 8 bits. The first byte has bits 0-7, the second byte has bits 8-15, and so on. In Java an int is 32 bits or 4 bytes.

To set bit number n in variable mybyte of type byte:

mybyte = mybyte | 1 << (7 - n);

To set bits n and n + 1 to the value of foo in variable mybyte:

mybyte = mybyte | foo << (7 - n);

Note that foo must have a value that can be expressed with 2 bits, that is, 0, 1, 2, or 3.

To copy a 16-bit integer foo into 2 bytes, b1 and b2:

b1 = foo >> 8;
b2 = foo & 0xFF;

After this, b1 will have the 8 high-order bits of foo and b2 will have the 8 low-order bits of foo.

You can copy a 32-bit integer into 4 bytes in a similar way.

Bit Example

Suppose we want to fill in the first byte of the RTP packet header with the following values:

  • V = 2
  • P = 0
  • X = 0
  • CC = 3

In binary this would be represented as

 1 0  | 0  | 0  | 0 0 1 1
V=2   P   X   CC = 3
2^7 . . . . . . . . 2^0

Bonus Points

Students are encouraged to turn in a short report to address the following questions. The report must be in PDF format, and turned in along with the source code of your projects. Each question is worth 1 point. 

  • Instead of the normal server given to you, use the class called FunkyServer.class, i.e., run it with java FunkyServer server_port. What do you see at the client? Explain what happens, why, and fix it.
  • Calculate statistics about the session. You will need to calculate RTP packet loss rate, video data rate (in bits or bytes per second), and any other interesting statistics you can think of.
  • The user interface on the client has 4 buttons for the 4 actions. If you compare this to a standard media player, such as RealPlayer or Windows Media Player, you can see that they have only 3 buttons for the same actions, namely, PLAY, PAUSE, and STOP (roughly corresponding to TEARDOWN). There is no SETUP-button available to the user. Given that SETUP is mandatory in an RTSP-interaction, how would you implement that in a media player? When does the client send the SETUP? Come up with a solution and implement it. Is it appropriate to send TEARDOWN when user clicks on the stop-button?
  • Currently the client and server only implement the minimum necessary RTSP interactions and PAUSE. Implement the method DESCRIBE which is used to pass information about the media stream. When the server receives a DESCRIBE-request, it sends back a session description file which tells the client what kinds of streams are in the session and what encodings are used.

Appendix

We use a proprietary MJPEG (Motion JPEG) format.

In this project, the server streams a video which has been encoded into a proprietary MJPEG file format. This format stores the video as concatenated JPEG-encoded images, with each image being preceded by a 5-Byte header which indicates the bit size of the image. The server parses the bitstream of the MJPEG file to extract the JPEG images on the fly. The server sends the images to the client at periodic intervals. The client then displays the individual JPEG images as they arrive from the server.

Submission

Please submit a tar file (<last_name>.tar.gz) of your source directory (including the directory). Please email the instructor your file by 10:00 a.m. on Monday. In the same email, you may include an optional PDF file addressing the bonus question (please see above). Students will do a demo on the instructor's laptop (schedule will be announced at a later time). A students gets 10 points if the smartphone demo is successful, and gets 1 points for correctly answering each bonus question. 

 

 

CS5263-Project-2012

Each student is responsible to propose and execute a term project. For a larger-scale project, two or three students may form a group, but each of them must work on a different part of the whole project, e.g., a student may work on adaptive streaming server while the other one works on the decoder on a smartphones. Students are encouraged to construct real testbeds using Android phones. Android 4.0 phones and other computing devices will be provided to the students on a as-needed basis.


Kinds of projects:

  • Brave new idea: Develop a new idea to solve a challenging research problem on wireless multimedia research. You must present your innovative idea. Simulation and/or experimental results shall be presented. Hopefully, you will get a publishable technical report (and A+) by the end of the semester.
  • Quantitative and/or qualitative comparisons of multiple published solutions: Implement existing algorithms, techniques, or systems. You need to conduct simulations or experiments to evaluate the solutions. Lessons learned from the simulations or experiments must be discussed.
  • Survey a well-studied problem and its solutions: Write a survey paper to discuss an interesting research problem in wireless multimedia research. You need to discuss the directions of future research related to that research problem.

Potential project topics, please feel free to suggest new topics:

  • Stereoscope video encoding and streaming, e.g., solving the view-switching problem of multiview videos on mobile devices.
  • Video streaming and broadcasting over heterogeneous networks to mobile devices.
  • Hepatics-based Multimedia Communications. Multimedia applications that interact with biometric sensors (e.g., e-touch, smell, ...). For more examples, check the work being done at University of Ottawa, MCRLab.
  • Develop an SVC streaming system that leverages stream re-writing feature of H.264/SVC standard. Stream re-writing efficiently converts an scalable H.264/SVC stream into a regular H.264/AVC stream. This allows common video players (such as VLC) and most smartphones to decode SVC streams. The H.264/SVC standard documents the details of how to implement this. Port your code to a WiFi gateway (e.g., an OpenWRT compatible access point) and conduct real experiments on smartphones to show the benefit (if any) of scalable video coding.
  • Modify the open-source Darwin streaming server to support H.264/SVC streams. Partial implementation is available upon request.
  • Understand the limitations of opensvc, an open-source H.264/SVC decoder. Optimize the opensvc on Android in terms of time and space complexities.
  • 3D Virtual Reality System.
  • Optimized multimedia content transfer on Android smartphones.
  • Generic remote rendering system for online games and (network) latency compensation techniques.
  • Location-based bandwidth estimation system.
  • Implementation and evaluation of a media gateway (a WiFi router that provides simple QoS for multimedia streams). See http://dl.acm.org/citation.cfm?id=1496061 and http://dl.acm.org/citation.cfm?id=1989245 for examples, and propose enhancements on top of their works.
  • Efficient implementation of network coding schemes in distributed media streaming systems.

Executions:

The main objective of the term project is to show the students how to conduct systems research in wireless multimedia. The instructor will walk you through the well-defined steps to complete the project. These steps are explained below.

  1. Literature survey: We first need to know what have been done before, more importantly, what haven't been done in the literature. In this phase, a student searches and reads papers published in reputable journals and conferences. You are excepted to classify all existing solutions into meaningful groups, and give high-level description on representative solutions.
  2. Problem statement: After knowing what hasn't been done, the student should formally define the research problem he/she plans to address in the term project. The student should clearly define the system models and mathematical notations.
  3. Problem solution: Each student proposes a new, better solution to address the research problem. Often, your solution is inspired by existing one studied in the literature survey; you may also borrow (algorithmic) ideas from other areas. The student should analyze the solution from the aspects of time and space complexities, and argue that the solution is (reasonably) scalable.
  4. Simulations/Experiments:A student must evaluate the proposed solution using simulations and/or experiments. Real experiments are much more convincing than simulations, because simulator often make assumptions to speed up implementation. While real experiments are highly encouraged, simulations are also useful when the test scenarios are too complex to be set up in a real testbed.

Deliverables:

In this course, students are required to turn in their reports three times. The reports must be written in Latex and this template. In the first month, each student will pick a research topic, and write a proposal consisting of literature survey and problem statement. Each student is given 15 min to present their proposal in front of the class. If a students needs to borrow equipments (such as wireless routers and smartphones) from the instructor, he/she should explain why the equipments are needed (e.g., experimental setups) and identify the suitable make/model in the presentation. Students are then given 1.5 month to work on their problem solutions and conduct some simulations/experiments. They add new materials to their proposal, and give a 15 min presentation to report the progress. Next, students work on their projects for another 1.5 month, and present and (optionally) demo their term projects in the week before the final exam. We don't have final exams, and the final technical reports are due one week after the last lecture. In summary, each students will turn in three incremental technical reports, give three talks, and optionally demo their systems. All these deliverables will be graded; demos are highly encouraged. Moreover, each student will give a 1-hr presentation about one or two papers relevant to his/her project topic. The presentation will be graded. Students should recommend good and recent papers to present. If that's not possible, the instructor will assign papers to individual students after the proposal presentation.