How to Transmit Raspberry Pi Video Frames to PC

Remote monitoring of Raspberry Pi captured video is an intriguing task. For convenient viewing, you might want to transmit the video frames from the Raspberry Pi to your PC. This not only eliminates the hassle of equipping the Raspberry Pi with a display but also allows real-time monitoring of the camera’s images when needed. Therefore, in this tutorial, TechSparks will guide you through the process of using the UDP protocol to efficiently transmit video frames captured on the Raspberry Pi to the PC.

Table of Contents

Network Protocol for Video Frame Transmission

Raspberry Pi and PC may be located in different physical locations. To establish a communication channel between them, we need to use network protocols for data transmission over the network. Common approaches include TCP and UDP.

TCP (Transmission Control Protocol): It is a connection-oriented protocol that ensures the reliability and order of data by establishing a connection at both ends of communication. If data packets are lost or damaged during transmission, TCP is responsible for retransmitting them.

UDP (User Datagram Protocol): It is a connectionless protocol designed to send data packets to the destination without establishing a connection. It does not guarantee the reliability of data and does not require connection establishment.

Obviously, TCP is a more reliable protocol and is more frequently used in network communication. However, in this project, our goal is to achieve the transmission of Raspberry Pi video frames, and UDP may be a better choice because our focus should be on transmission speed rather than data integrity.

In the Python Socket code, we can see the characteristics of UDP clearly. For the sender, we first create a UDP socket object:

				
					server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
				
			

Here, socket.AF_INET indicates the use of the IPv4 address family, and socket.SOCK_DGRAM indicates the creation of a UDP socket.

Next, we use the connect function to specify the address of the UDP sender, i.e., HOST and PORT:

				
					server.connect((HOST, PORT))
				
			

It’s worth noting that, unlike TCP, the connect function in UDP does not actually establish a connection. It only binds the socket to the specified target address. Therefore, in UDP, the connect function returns almost instantly without waiting for a successful connection.

Afterward, we can directly call the send function on the socket object to send data.

Optimizing Image Transmission

Although the UDP protocol is faster, according to the protocol specifications, the size limit for a single data packet is 65507 bytes (excluding the header). However, the size of image frames captured by the Raspberry Pi far exceeds this limit, ultimately causing incomplete transmission of data packets. TechSparks provides two solutions here:

Split Transmission: Divide the image into multiple parts for transmission, and the receiving end uses a loop to receive multiple times. However, this requires us to add additional custom rules, and the transmission speed will be affected.

Compression Encoding: Encode and compress the image before sending, and then decode it at the receiving end. Although this may reduce the image quality, it does not affect the transmission speed and is therefore recommended.

In OpenCV, the imencode and imdecode methods can be used for image encoding and decoding. Specifically:

imencode function prototype: cv2.imencode(ext, img[, params]) → retval, buf

  • ext (str): File extension specifying the storage format, such as .jpg.
  • img: Image data to be encoded.
  • params (optional): Specified encoding flags that can be used to adjust compression quality and other parameters.

imdecode function prototype: cv2.imdecode(buf, flags) → retval

  • buf: Encoded image data.
  • flags: Specifies the type of image reading, which can be used to specify settings such as depth and channels.

Setup Raspberry Pi Program

By combining socket objects and encoding/decoding functions, we can write the code for the sender. However, it’s crucial to pay special attention to data format issues during this process. Socket communication involves sending and receiving data in the form of byte streams or byte arrays. When sending data, it needs to be sent in byte stream format, while on the receiving end, the received byte stream needs to be converted into the appropriate data type.

Typically, the image obtained from the camera is a numpy.ndarray type with dimensions 480X640X3. After encoding using the imencode function, we get a numpy.ndarray object with dimensions ?X1. Through testing, we found that this object can be sent directly. On the receiving end, we get a byte array. It’s important to note that this array needs to be converted into a numpy.ndarray type before undergoing imdecode processing.

In the actual code, we first send the length of the encoded bytes. This is done to allow the receiving end to perform a simple check, and it enables the receiving end to determine whether the program should be closed. Similarly, we can customize a single byte to represent a shutdown message.

				
					import cv2
import numpy
import socket
import struct
HOST = '192.168.1.122'
PORT = 9999
# Create a UDP socket
server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server.connect((HOST, PORT))  # Connect to the target address
print('Now starting to send frames...')
# Get the camera device
capture = cv2.VideoCapture(0)
try:
    while True:
        success, frame = capture.read()
        while not success and frame is None:
            success, frame = capture.read()  # Get a video frame
        # Encode the image
        result, imgencode = cv2.imencode('.jpg', frame, [cv2.IMWRITE_JPEG_QUALITY, 50])
        # Send the byte length of the encoded image data
        server.sendall(struct.pack('i', len(imgencode.tobytes())))
        # Send the video frame data
        server.sendall(imgencode)
        print('Have sent one frame')
except Exception as e:
    print(e)
    # Send a close message
    server.sendall(struct.pack('B', 1))
    # Release camera resources
    capture.release()
    # Close the socket
    server.close()
				
			

Setup PC Program

When your computer acts as the receiving end, the transmitted image data is in the form of a byte stream. On the receiving side, you need to convert this byte stream into a numpy.ndarray object so that you can utilize OpenCV for image processing and display.

				
					import cv2
import numpy
import socket
import struct
HOST = '192.168.191.122'
PORT = 9999
buffSize = 65535
# Create a UDP socket
server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
# Bind the address and port
server.bind((HOST, PORT))
print('Now waiting for frames...')
while True:
    # First, receive the byte length
    data, address = server.recvfrom(buffSize)
    # If receiving a close message, stop the program
    if len(data) == 1 and data[0] == 1:
        server.close()
        cv2.destroyAllWindows()
        exit()
    # Perform a simple check; the length value is of type int and takes up four bytes
    if len(data) != 4:
        length = 0
    else:
        length = struct.unpack('i', data)[0]  # Length value
    # Receive encoded image data
    data, address = server.recvfrom(buffSize)
    # Perform a simple check
    if length != len(data):
        continue
    # Format conversion
    data = numpy.array(bytearray(data))
    # Decode the image
    imgdecode = cv2.imdecode(data, 1)
    print('Have received one frame')
    # Display the frame in a window
    cv2.imshow('frames', imgdecode)
    # Press "ESC" to exit
    if cv2.waitKey(1) & 0xFF == 27:
        break
# Close the socket and window
server.close()
cv2.destroyAllWindows()
				
			

Ending

As my Raspberry Pi is associated only with Python 2’s OpenCV, I use python2 UDP_Frame_Send.py to launch the sending program (ensuring the camera is connected properly). On the computer, I navigate to the program file directory in the Windows system console and enter python UDP_Frame_Recv.py to start the receiving program, and the window display is relatively smooth. However, I encountered some issues:

If you are having difficulties starting the camera on the Raspberry Pi, you may try restarting it.

Since the recvfrom function on the computer is blocking, when the program reaches this function, it will wait indefinitely until data arrives. Therefore, if there is an error on the sender side, the receiving program will not exit normally. For this reason, TechSparks suggests using the TCP & UDP Debug Assistant to simulate a termination signal, allowing the receiving program to exit.

You Might Be Interested

raspberry pi autostart
How to Auto Start Programs on Raspberry Pi

Automating program startup on the Raspberry Pi can be achieved through various methods. Editing the “/etc/rc.local” file or using desktop applications, while simpler, may not

raspberry pi
What is Raspberry Pi?

Raspberry Pi, a revolutionary single-board computer introduced by the Raspberry Pi Foundation, has become a global sensation, initially designed for educational purposes. With its integrated

Scroll to Top