Docker in Automotive Embedded Systems: Lessons from the Field (Part 1)

Hey there! I’m starting a new series where I’ll share the various challenges and solutions I’ve encountered while implementing Docker in automotive embedded systems. This series will cover real-world problems I’ve faced and the lessons learned from solving them. This first post is about the subtle interactions between Docker containers and mount points - and boy, did this one keep me up at night! 😅

🕵️‍♀️ The Mystery: “The Case of the Vanishing Files”

One day, something really weird happened in our automotive embedded system. After a reboot, I launched a Docker container and… all the files that were there before had just vanished! The container started up just fine, but all the files in the mounted directory had disappeared into thin air. Like ghosts! 👻

🔍 Root Cause Analysis: The Critical Relationship Between Docker Startup and Mount States

After some detective work, I found out this issue was deeply related to system boot sequence timing in embedded environments.

Here’s what was happening:

1. Our system starts the Docker daemon through init.d scripts
2. The system also mounts network storage via NFS during boot time
3. The crucial discovery: Depending on when Docker starts, containers can reference completely different mount states!
4. If Docker daemon starts first and storage gets mounted later, containers end up referencing empty directories or stale states
5. This makes containers think “there are no files here” after reboot

This might seem like a simple timing issue, but it’s actually a critical system architecture design problem. In automotive embedded environments, service dependencies and startup ordering are absolutely crucial.

💡 Temporary Fix: Mount Verification Before Docker Startup

To solve this, I added mount verification logic to the docker.init script:

# Storage volume mount status check
trials_mount=5
while [ $trials_mount -gt 0 ]; do
    dfoutput=$(df -PTh mount_path)
    echo "df output: $dfoutput" >/dev/kmsg
    
    if echo "$dfoutput" | grep -q "nfs"; then
        echo "NFS mount confirmed, starting Docker daemon" >/dev/kmsg
        break
    else
        echo "NFS not ready, waiting... (attempts left: $trials_mount)" >/dev/kmsg
        trials_mount=$((trials_mount-1))
        sleep 2
    fi
done

if [ $trials_mount -eq 0 ]; then
    echo "ERROR: NFS mount failed after multiple attempts" >/dev/kmsg
    exit 1
fi

# Now start Docker daemon
start-stop-daemon --start --quiet --pidfile $DOCKER_PIDFILE --exec $DOCKER_DAEMON -- $DOCKER_OPTS

🔧 The Real Solution: Proper Service Dependencies

While the above fix worked as a band-aid, the real solution was implementing proper service dependencies. In automotive systems, we need to guarantee that:

1. Network services are fully operational
2. Storage systems are mounted and verified
3. Only then should containerized services start

Here’s how I restructured the service dependencies:

# In our systemd service file
[Unit]
Description=Docker Application Container Engine
After=network-online.target storage-mount.target
Wants=network-online.target storage-mount.target
Requires=storage-mount.target

[Service]
Type=notify
ExecStart=/usr/bin/dockerd
Restart=on-failure

📝 Key Takeaways

1. Boot sequence matters: In embedded systems, the order of service startup can make or break your application.

2. Mount states are tricky: Docker containers “remember” the mount state from when they were created/started.

3. Always verify dependencies: Don’t assume external resources are ready just because your service started.

4. Automotive = reliability: In automotive applications, these kinds of race conditions can be catastrophic.

🚀 What’s Next?

In the next part of this series, I’ll talk about another fun challenge: handling container updates in automotive systems where downtime isn’t really an option. Spoiler alert: it involves some creative juggling with container orchestration!

Have you run into similar timing issues with Docker in embedded systems? I’d love to hear about your experiences and how you solved them!

This was definitely one of those “why is this so hard?” moments that taught me a lot about the intricacies of system integration. The automotive world doesn’t forgive timing issues!