How to Run AI Inference in Containers on a NavEdge Edge Server

Step by step

1. Check the accelerator

   If the edge server has an NVIDIA GPU, install the NVIDIA Container Toolkit and confirm containers can see it.

   docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
   

2. Describe the service

   Pin the image version, give it the GPU, mount the models read-only and pass camera URLs as configuration, not in the image.

   services:
     inference:
       image: registry.example.com/vision/detector:3.2.0
       restart: unless-stopped
       environment:
         CAMERAS: rtsp://analytics:${CAM_PW}@10.1.30.101/stream2,rtsp://analytics:${CAM_PW}@10.1.30.102/stream2
         EVENTS_URL: http://10.1.99.30:8080/events
       volumes: ["./models:/models:ro"]
       deploy:
         resources:
           reservations:
             devices: [{ driver: nvidia, count: 1, capabilities: [gpu] }]
   

3. Use a read-only camera account

   Create a dedicated camera user with view-only rights for the analytics service, and store its password in .env, not in Git.

4. Start and watch

   Start the stack and watch latency, frames per second and GPU load.

   docker compose up -d
   docker compose logs -f inference
   nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 5
   

5. Send events, not video

   Forward only events and metadata to the VMS, NVR or business system; keep raw video on site.

6. Update models safely

   Treat a new model like new software: test it on one site, compare accuracy, then roll out with a new image tag and keep the previous tag for rollback.

Commands are generic, industry-standard examples. Exact syntax and menu names vary by model and firmware; check your device's manual on the Documentation page.