Forwarding camera images from Carla simulator with MQTT

In the public sponsored research project MANNHEIM-FlexKI we are using the Carla simulator as a data generation / simulation system for the optimization of vulnerable road user detection (as a use case for general AI optimization for various systems).

In our setup, we are using Carla on a PC with a decent GPU to simulate the traffic and then send it to a Jetson (Nano/Orin/Xavier) or other systems to do the AI processing.

Some of our Carla systems run on Windows and we would like to share a few findings:

Carla Installation

• Setup of Carla 0.9.x on Windows with the Python interface can be a pain - you might need to find the right version of a specific DLL on the internet and try to add this so that it could be find. On one PC we could not get this to run, on the other it run out-of-the box, even without custom (re-) installation of the DLL
• Setup of Carla 0.10.x on Windows with Python interface worked like a charm. The Carla team revised the built and everything is integrated. It runs fine out of the box with the Python interface.

Forwarding images - Gstreamer

• Our initial idea was to use gstramer to forward the camera images retrieved by the Python interface. However, again, installation on Windows is not smooth and cumbersome, needs third party installations etc.
• So the next idea was to call the Python interface from a WSL instance (Linux running on Windows). Gstreamer installs fine. However, calling thy Carla Python interface from WSL to a Carla instance running on Windows results in a exception in the bowels of the interface. Dead end.

Forwarding images - MQTT

• Since we are using MQTT in other contexts as well, we went for sending the images through Paho MQTT libraries with a mosquitto server. If you go down that route, here is a recommendation:
• Many code examples on the net work by explicitly encoding/decoding the image in Base64 before sending with MQTT. In our config, we found that this is not necessary - we are sending the byte data through the APIs:

image = image_queue.get()
img = np.reshape(np.copy(image.raw_data), (image.height, image.width, 4)) 
paho_client.publish("carla-image", img.tobytes())        

And on the receiver side:

def on_message(client, userdata, msg):
    image = np.frombuffer(msg.payload, dtype=np.uint8).reshape(args.width, args.height, 4)
    
    cv2.imwrite('received_image.png', image)

# Create an MQTT client and attach the on_message callback
client = mqtt.Client(client_id="carla-img", callback_api_version=mqtt.CallbackAPIVersion.VERSION2)
client.on_message = on_message        

Let us know if this article was useful for your work with the great Carla simulator.