Fixing Lerobot-record DTS Out Of Order Error

Oct 10, 2025 by ADMIN 45 views

Fixing `lerobot-record` Error: DTS Out of Order, Non-Monotonically Increasing

Encountering the lerobot-record error "DTS out of order, non monotonically increasing" can be a frustrating experience when trying to record datasets for robotics projects. This error typically arises during the video encoding process, specifically when concatenating video files, and indicates that the timestamps of the video frames are not in the correct sequential order. Let's dive deep into understanding this issue and explore potential solutions to resolve it.

Understanding the Error

The error message DTS out of order, non monotonically increasing essentially means that the Decoding Timestamp (DTS) for video frames is not in the expected ascending order. Video codecs rely on timestamps to properly decode and display frames in the correct sequence. When the DTS values are jumbled, the encoding process fails, leading to the aforementioned error and halting the recording process. This issue is more likely to surface after the first episode because the concatenation process combines multiple video segments, increasing the chances of timestamp inconsistencies.

Common Causes

Several factors can contribute to this timestamp disorder:

Camera Synchronization Issues: When using multiple cameras, ensuring that their clocks are synchronized is crucial. Discrepancies in frame capture times can lead to DTS mismatches.
Frame Dropping: If frames are dropped during the recording process due to processing bottlenecks or hardware limitations, the remaining frames might have DTS values that are no longer sequential.
Encoding Problems: Issues within the video encoding library (in this case, likely related to SVT-AV1) or the encoding settings themselves can cause timestamp corruption.
Hardware Limitations: Insufficient hardware resources can lead to erratic frame capture and processing times, contributing to DTS errors.

Troubleshooting Steps

Now, let's explore several strategies to address this issue. These steps range from basic checks to more advanced configurations.

1. Verify Camera Configuration

First, ensure that all your cameras are correctly configured and functioning as expected. Double-check the camera indices or paths specified in your lerobot-record command:

--robot.cameras="{ \
    side: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30}, \
    gripper: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}, \
    top: {type: opencv, index_or_path: /dev/video6, width: 640, height: 480, fps: 30}}"

Make sure that each /dev/videoX corresponds to the correct camera and that no other process is interfering with the camera streams. You can use tools like v4l2-ctl to verify camera settings and capture test frames.

2. Reduce Frame Rate

High frame rates can exacerbate synchronization issues and processing bottlenecks. Try reducing the frame rate (fps) for each camera in your configuration. For example, lower it from 30 fps to 20 fps:

--robot.cameras="{ \
    side: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 20}, \
    gripper: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 20}, \
    top: {type: opencv, index_or_path: /dev/video6, width: 640, height: 480, fps: 20}}"

This reduces the amount of data being processed per second, potentially alleviating the DTS issue.

3. Optimize Video Encoding Settings

The SVT-AV1 encoder settings can significantly impact video encoding stability. While the provided logs show the encoder configurations, tweaking these settings might help. Consider the following:

Preset: Experiment with different preset values. A slower preset (e.g., preset 6 or 7) might provide better encoding accuracy at the cost of encoding speed.
Tune: If you're not particularly concerned with PSNR (Peak Signal-to-Noise Ratio), try a different tuning option, such as VMAF or SSIM.
GOP Size: Adjust the GOP (Group of Pictures) size. A larger GOP size might improve encoding efficiency but could also introduce latency. A smaller GOP size might be more robust against DTS errors.

Unfortunately, lerobot-record doesn't expose these SVT-AV1 settings directly via command-line arguments. You may need to modify the underlying code to adjust these parameters. This typically involves delving into the lerobot/datasets/video_utils.py and modifying the encoder configurations.

4. Investigate Hardware Bottlenecks

Ensure that your system meets the hardware requirements for real-time video encoding from multiple cameras. Monitor CPU and GPU utilization during recording. If either is consistently maxed out, it could indicate a bottleneck. Consider the following:

CPU: The SVT-AV1 encoder is CPU-intensive. Ensure your CPU has enough cores and processing power to handle the encoding load.
GPU: While the logs indicate that CUDA is enabled, ensure that the GPU is effectively utilized for any hardware-accelerated encoding tasks.
Storage: Verify that your storage device (SSD or HDD) has sufficient write speed to handle the video data. A slow storage device can cause frame drops and DTS issues.

5. Disable Video Encoding Temporarily

To isolate whether the issue stems from the video encoding process, try disabling video recording temporarily. If the lerobot-record script runs without errors when video recording is disabled, it strongly suggests that the problem lies within the video encoding pipeline. You can achieve this by commenting out or modifying the relevant sections of the lerobot_dataset.py that call the video encoding functions.

6. Check Camera Drivers and Firmware

Outdated or incompatible camera drivers can lead to unpredictable behavior. Ensure that you have the latest drivers installed for your cameras. Similarly, check if there are any firmware updates available for your cameras, as these updates often include bug fixes and performance improvements.

7. Implement Frame Buffering

Introducing a frame buffer can help smooth out inconsistencies in frame arrival times. Implement a buffer that stores frames temporarily before passing them to the encoder. This can help mitigate DTS errors caused by slight variations in frame capture intervals.

8. Review and Modify `video_utils.py`

Since the error originates from video_utils.py, examining the concatenate_video_files function is crucial. Pay close attention to how the video files are being concatenated and how timestamps are handled. You might need to add error handling or timestamp correction logic to this function.

Here's a snippet of the relevant code from the traceback:

File "/home/riccardo/lerobot3/src/lerobot/datasets/video_utils.py", line 467, in concatenate_video_files
    output_container.mux(packet)

9. Update `av` Library

The traceback indicates that the error occurs within the av library (a Pythonic binding for FFmpeg). Ensure that you have the latest version of the av library installed:

pip install --upgrade av

Newer versions often include bug fixes and performance improvements that could address the DTS issue.

10. Debugging with Verbose Logging

Add more verbose logging to the concatenate_video_files function and the surrounding code to gain deeper insights into the timestamp values and the muxing process. This can help pinpoint exactly where the DTS disorder is occurring.

Specific Code Modifications (Advanced)

If none of the above steps resolve the issue, you might need to resort to modifying the lerobot source code. Always back up your code before making any changes.

Timestamp Correction: Within video_utils.py, before muxing the packet, add a check to ensure that the DTS value is monotonically increasing. If it's not, adjust the DTS value to be greater than the previous DTS value.

def concatenate_video_files(input_files: List[str], output_file: str):
    output_container = av.open(output_file, mode='w')
    stream = None
    previous_dts = None

    for input_file in input_files:
        input_container = av.open(input_file, mode='r')
        for packet in input_container.demux(input_container.streams.video[0]):
            if stream is None:
                stream = output_container.add_stream(template=input_container.streams.video[0])

            if packet.dts is not None:
                if previous_dts is not None and packet.dts <= previous_dts:
                    packet.dts = previous_dts + 1  # Adjust DTS value
                previous_dts = packet.dts

            output_container.mux(packet)

    # Flush stream
    for packet in stream.encode():
        output_container.mux(packet)

    # Close the file
    output_container.close()

Error Handling: Add more robust error handling around the mux function to catch and log any exceptions that might be occurring.

Conclusion

The lerobot-record error "DTS out of order, non monotonically increasing" can be a complex issue to resolve, often requiring a combination of troubleshooting steps and code modifications. By systematically addressing potential causes such as camera synchronization, encoding settings, and hardware limitations, and by carefully examining the video_utils.py code, you can increase your chances of successfully recording your robotics datasets. Remember to always back up your code and test your changes thoroughly. Good luck, and happy recording!