https://mainstreetautonomy.com/resources/ Recent content in Videos, datasets, and reports on Main Street Autonomy Hugo en-us https://mainstreetautonomy.com/resources/autonomous-lawnmower-operating-with-perception-based-localization/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/autonomous-lawnmower-operating-with-perception-based-localization/ MSA co-developed a fully autonomous lawnmower shown operating with perception-based localization. Lidar, camera, IMU, and wheel encoder data are used to build a map of the perimeter of the mow site. The mower then fills in the mow site with rows driven to cm-level accuracy, leaving no "mow-hawks" of uncut grass. Calibration Anywhere is used to calibrate sensors, and Pose Engine is used for localization. https://mainstreetautonomy.com/resources/calibration-anywhere-software-introduction/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/calibration-anywhere-software-introduction/ Main Street Autonomy's Calibration Anywhere software calibrates perception sensors, like lidars, radars, cameras, IMUs, and GPS units, using only sensor data captured during motion. The process is easy! Move the robot near some static structure, store the sensor data, and run the calibration. In maybe 10 minutes, you'll get extrinsics: 6-dof pose for all sensors (lidars, radars, cameras, IMUs, and GPS units), plus a base_link kinematic frame reference. Plus intrinsics: openCV-compatible lens models for cameras (including depth), rolling shutter times, wheel encoder gain, and lidar and depth intrinsics. Plus time offsets for all sensors relative to each other. The calibration is repeatible, reliable, and can happen anywhere. You don't need checkerboards, targets, or engineers involved in the calibration process. The benefits are significant: improve your sensor data quality, enable straightforward sensor fusion, and unblock your perception team. Don't spend your time wrestling with calibration, get MSA to take care of it for you. https://mainstreetautonomy.com/resources/camera-localization-indy-autonomous-racecar-texas-motor-speedway/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/camera-localization-indy-autonomous-racecar-texas-motor-speedway/ Camera localization of the racecar is shown alongside current feature observations and the pre-built map of the Texas Motor Speedway, an asphalt 1.5-mile quad oval track in Fort Worth, TX. The map was pre-built using camera data from the autonomous indy car. Localization to the map happens using current camera observations, which are shown in the images and in a local view on the bottom of the screen. Localization occurs at 50-100Hz with <10ms latency, with accuracy of approximately 1-2cm and <1mRadian. https://mainstreetautonomy.com/resources/colorized-lidar-figure-8-calibration-motion-parking-lot/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/colorized-lidar-figure-8-calibration-motion-parking-lot/ A colorized lidar sensor fusion visualization (where each point is a lidar range measurement painted with the associated pixel color) shows how the operator of this robot rapidly executes the figure-8 calibration motion for Calibration Anywhere. The visualization demonstrates perfect fusion even during high-speed turns and fast motion. Blue pixels are outside the shared field-of-view of the lidar and camera. https://mainstreetautonomy.com/resources/colorized-lidar-figure-8-calibration-motion-offroad/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/colorized-lidar-figure-8-calibration-motion-offroad/ A colorized lidar sensor fusion visualization (where each point is a lidar range measurement painted with the associated pixel color) shows how the operator of this robot rapidly executes the figure-8 calibration motion for Calibration Anywhere. The visualization demonstrates perfect fusion even during bumpy offroad operation. Blue pixels are outside the shared field-of-view of the lidar and camera. https://mainstreetautonomy.com/resources/feature-map-and-track-figure-8-calibration-motion-warehouse/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/feature-map-and-track-figure-8-calibration-motion-warehouse/ A colorized lidar sensor fusion visualization (where each point is a lidar range measurement painted with the associated pixel color) shows how the operator of this robot rapidly executes the figure-8 calibration motion for Calibration Anywhere. The visualization demonstrates perfect fusion even during bumpy offroad operation. Blue pixels are outside the shared field-of-view of the lidar and camera. https://mainstreetautonomy.com/resources/feature-map-and-track-figure-8-calibration-motion-nvidia-carter-warehouse/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/feature-map-and-track-figure-8-calibration-motion-nvidia-carter-warehouse/ A colorized lidar sensor fusion visualization (where each point is a lidar range measurement painted with the associated pixel color) shows how the nVIDIA Carter robot moves in a figure-8 calibration motion for Calibration Anywhere. The visualization demonstrates perfect sensor fusion in the indoor warehouse environment. https://mainstreetautonomy.com/resources/ihmc-nadia-humanoid-robot-playing-ping-pong-calibrated-by-msa/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/ihmc-nadia-humanoid-robot-playing-ping-pong-calibrated-by-msa/ Nadia is an advanced, highly mobile humanoid robot developed by the Florida Institute for Human and Machine Cognition (IHMC) in collaboration with Boardwalk Robotics. Main Street Autonomy provided critical camera calibrations using Calibration Anywhere software. https://mainstreetautonomy.com/resources/lidar-camera-sensor-fusion-on-road/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/lidar-camera-sensor-fusion-on-road/ Lidar and camera sensor fusion: colorized lidar on the left, where each point is a lidar range measurement, painted the appropriate pixel color from the camera images. The right shows lidar-in-camera, where lidar range measurements are projected into the camera images. Note the cameras used on this vehicle have wide fisheye lenses. Additionally, all four cameras are rolling shutter. https://mainstreetautonomy.com/resources/lidar-localization-and-lidar-scan-map-of-lawrenceville-pa/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/lidar-localization-and-lidar-scan-map-of-lawrenceville-pa/ This visualization shows a lidar scan map built from a robot moving on the sidewalk in Lawrenceville, PA. The robot has a lidar, IMU, and wheel encoders. The breadcrumbs show the robot trajectory, determined by MSA Pose Engine perception-based localization software. Lidar scans captured from the robot are coregistered to the robot's pose, motion compensated, and shown in a world frame. The quality and coherence of the resulting lidar scan map is due to the accurate sensor calibration and accurate localization of the robot. Calibration or localization inaccuracies would appear in the fused lidar clouds. https://mainstreetautonomy.com/resources/lidar-scan-map-parking-garage/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/lidar-scan-map-parking-garage/ This visualization shows a lidar scan map built from a vehicle traversing the Bakery Square Parking Garage in Pittsburgh, PA. The sensor system is comprised of an Ouster lidar and an IMU mounted on the roof of a car. The green breadcrumbs show the vehicle trajectory, determined by MSA Pose Engine perception-based localization software. Lidar scans captured from the vehicle are coregistered to the vehicle pose, motion compensated, and shown in a world frame. The quality and coherence of the resulting lidar scan map is due to the accurate sensor calibration and accurate localization of the vehicle. Calibration or localization inaccuracies would appear in the fused lidar clouds. https://mainstreetautonomy.com/resources/lidar-in-camera-sensor-fusion-on-road/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/lidar-in-camera-sensor-fusion-on-road/ Lidar-in-camera sensor fusion, where lidar range measurements are projected into the camera images. Note this camera is rolling shutter with a wide fisheye lens and is generating images at 30Hz. The lidar is an Ouster OS0 capturing data at 10Hz. https://mainstreetautonomy.com/resources/multi-lidar-multi-camera-sensor-fusion-offroad/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/multi-lidar-multi-camera-sensor-fusion-offroad/ Main Street Autonomy can simplify the challenges your robotics team is facing. MSA's Calibration Anywhere software generated the 6dof pose, camera lens and lidar beam intrinsics, and time offsets for the 3D lidars, four RGB rolling-shutter cameras, wheel encoders, and GPS antenna. With an excellent calibration, sensor fusion, perception, mapping, and localization are easier. https://mainstreetautonomy.com/resources/multi-robot-outdoor-lidar-localization/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/multi-robot-outdoor-lidar-localization/ One robot maps the Allegheny Cemetery (blue/green), a second maps the nearby Lawrenceville PA area (orange/purple). The robots are both using Ouster lidars; neither is using GPS/GNSS. The visualization demonstrates causal localization. Map merging is shown after the maps are built. https://mainstreetautonomy.com/resources/msa-sensor-calibration-nvidia-perceptor-workflow/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/msa-sensor-calibration-nvidia-perceptor-workflow/ Workflow for calibrating sensors for nVIDIA Isaac Perceptor. https://mainstreetautonomy.com/resources/sensor-extrinsics-for-marble-robot/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/sensor-extrinsics-for-marble-robot/ Sensor extrinsics are illustrated on a robot wireframe: two lidars, four cameras, IMU, and GNSS. The robot kinematic frame is shown between the two drive wheels. This robot was built by Marble Robot, Inc (acquired by Caterpillar in 2020). https://mainstreetautonomy.com/resources/sensor-extrinsics-for-nvidia-carter-robot/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/sensor-extrinsics-for-nvidia-carter-robot/ Sensor extrinsics are illustrated on a robot wireframe: two lidars, four cameras, IMU, and GNSS. The robot kinematic frame is shown between the two drive wheels. This robot was built by Marble Robot, Inc (acquired by Caterpillar in 2020). https://mainstreetautonomy.com/resources/sensor-fusion-figure-8-calibration-motion-warehouse/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/sensor-fusion-figure-8-calibration-motion-warehouse/ A colorized lidar sensor fusion visualization (where each point is a lidar range measurement painted with the associated pixel color) shows the figure-8 calibration motion of this robot in a warehouse. https://mainstreetautonomy.com/resources/sensor-fusion-figure-8-calibration-motion-nvidia-carter-warehouse/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/sensor-fusion-figure-8-calibration-motion-nvidia-carter-warehouse/ A colorized lidar sensor fusion visualization (where each point is a lidar range measurement painted with the associated pixel color) shows the how the nVIDIA Carter robot moves in a figure-8 calibration motion for Calibration Anywhere. https://mainstreetautonomy.com/resources/sensor-fusion-visualization-outside-inside-warehouse/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/sensor-fusion-visualization-outside-inside-warehouse/ The visualization shows colorized lidar: each point shown is a lidar range measurement that is painted with the color of the appropriate pixel captured by the cameras. The lidar and cameras are not time synchronized and are capturing data at different rates (10Hz and 30Hz, respectively). To generate this output, the lidar and camera sensors must be perfectly calibrated for the extrinsic pose of the sensors, the intrinsic corrections for the camera lens, and the time offsets between the lidar and cameras. https://mainstreetautonomy.com/resources/visual-localization-and-mapbuilding-outside-inside-warehouse/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/visual-localization-and-mapbuilding-outside-inside-warehouse/ The visualization shows the various tracked feature points and map of the vehicle's trajectory. Pose Engine visual localization is generating the pose estimates shown here, using camera images fused with IMU and wheel encoders. https://mainstreetautonomy.com/resources/visual-localization-multistory-parking-garage/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/visual-localization-multistory-parking-garage/ This visualization follows the path of an autonomous vehicle building a map of the Bakery Square Parking Garage in Pittsburgh, PA. The localization shown here uses the output from four cameras and an IMU mounted on the roof of a car. No GPS/GNSS data are used. Pose Engine is shown generating 6DoF pose estimates. The localization algorithm consumes 3-4 cores of an Nvidia Jetson processor. https://mainstreetautonomy.com/resources/visual-localization-multirotor-drone-cameras-only/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/visual-localization-multirotor-drone-cameras-only/ Data capture from a small quadcopter drone with two GoPro cameras attached. No IMU, no lidar, no GPS/GNSS, and no other sensors were used. The video shows how we use image features for both visual odometry, to build a map of the environment, and to localize within the map. You can see the blue crosses in the video identifying various features; cross size scales with image patch size. The map view on the bottom shows keyframes being created and connected in a covisibility graph, along with the point cloud of the local sparse map currently being tracked in the imagery. The keyframes in the local map have their little camera pairs drawn above them. https://mainstreetautonomy.com/resources/visual-localization-farm-tractor/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/visual-localization-farm-tractor/ This robot is a hybrid autonomous/human piloted farm tractor with four cameras and an IMU. The robot has no lidar and no GPS/GNSS. The video shows how we use image features for visual odometry, to build a map of the environment, and to localize within the map. You can see the blue crosses in the video identifying various features; cross size scales with image patch size. Note how the four camera feeds have independently-controlled white balance and lots of image artifacts: dirt on the lenses, glare and reflections on all four cameras, and a bouncing seat in the FOV. Our software handles these without delaying the pose estimate latency or requiring massive compute. The map view on the bottom shows keyframes being created and connected in a covisibility graph, along with the point cloud of the local sparse map currently being tracked in the images. The keyframes in the local map have little camera icons indicating the pose of the sensors at that moment. https://mainstreetautonomy.com/resources/what-is-a-camera-calibration/ Mon, 01 Jan 0001 00:00:00 +0000 https://mainstreetautonomy.com/resources/what-is-a-camera-calibration/ Cameras map points in the world into pixels, calibration tells us how the mapping works. Pixel coordinates are used to find a ray in 3D space, which can be used to measure the world.