2.1.

Hardware design of garbage salvage system

The hardware used in the garbage salvage system in this article mainly includes STM32, Arduino UNO, Raspberry Pi, monocular camera, motor, lidar sensor, and four-servo robotic arm. The STM32 development board is the bottom control board of the salvage ship. Lidar is mainly used for map construction, obstacle detection and location of unmanned ships on available maps. The system uses a 5G wireless router to build a local area network. The portable computer serves as the host of the ROS system. Sensors such as lidar and monocular cameras are connected to the Raspberry Pi. Use ROS distributed communication to transfer all sensor data to the local area network. The PC host computer is used to calculate and analyze the collected environmental information data and the attitude information of the unmanned ship. The hardware design framework of the garbage salvage system is shown in Figure 2.

Figure 2.

Garbage salvage system hardware design.

2.2.

Software design of garbage salvage system

The garbage salvage system in this article is built on the basis of the Robot Operating System (ROS)² and the Ubuntu system. ROS is an open-source robot operating system, which provides class operating system services and is easy to develop. Both the PC host and Raspberry Pi slave in this system are installed with Ubuntu 16.04 and kinetic version ROS. Path planning and target detection algorithms are both carried in the ROS system. The two-dimensional grid map is created using the Gmapping-SLAM algorithm, which uses environmental information data scanned by lidar, IMU attitude information, and odometer information, then combining with the path planning A* algorithm, and calling the move_base package of ROS to realize the autonomous cruise of the garbage salvage ship. The garbage detection algorithm uses the YOLO v4 model to identify and locate the target in each frame of the video. The video is returned by the monocular camera in front of the ship.

Algorithms such as map construction, path planning, and target detection all require a lot of calculations. Therefore, this system uses the ROS distributed communication processing framework³ to transmit the video data, lidar data, and mileage data collected by the main controller to the host computer for processing.

The basic process of target detection is shown in Figure 3: collect garbage picture data to make a data set; perform preprocessing operations such as segmentation, deduplication, and rotation correction on the training set; then train the model and tune it on a workstation with GPU-accelerated computing; finally, deploy the trained model to the ROS host, and perform real-time online detection of the information stream collected by the camera.

Figure 3.

Basic flow chart of target detection.

The garbage detection module in this article uses the YOLO v4 network⁴, and the structure of the network is shown in Figure 4. This network includes CSPDarknet53 backbone network, SPP, PANet and HEAD.

Figure 4.

YOLO v4 network structure.

The YOLO v4 backbone network is based on the YOLO v3 network⁵ with the addition of the CSP⁴ structure. And the Mish activation function is added to the backbone network to replace the Relu activation function. It is used to reduce the calculation amount of the algorithm while ensuring the accuracy of the detection. The Mish activation function is shown in equation (1):

The Dropblock used in Yolo v4 is a regularization method to alleviate overfitting. Dropblock is very flexible and can customize different combinations. The probability of discarding can be modified in various stages of training, and it performs better in the convolutional network.

The SPP⁴ module is applied to the last layer of the backbone network feature extraction, which significantly increases the receptive field, can better separate the context features, and the network operation speed remains almost unchanged. The PANet structure performs feature fusion on the input information, which can accurately preserve the spatial structure⁶. The extracted feature information is input to the Head layer, and the results of the prediction layer are combined to make the final prediction.

The loss function of the YOLO v4 algorithm is consists of three parts: regression box prediction loss L_CIoU, confidence loss L_Conf, and classification loss L_cls. The YOLO v4 loss function loss_total algorithm is shown in Table 1.

Table 1.

Loss function algorithm.

2.3.

Autonomous cruise

This system uses RPLIDAR A3 lidar to complete the map building fusion and obstacle avoidance functions. RPLIDAR A3 has a high-speed laser ranging and sampling capability of 16000 times per second and a measurement range of 25 meters in radius. It has a good ability to anti-sunlight interference and is suitable for constructing outdoor SLAM maps. First, gather the coordinate transformation and topic publishing nodes in the startup file of Gmapping. Then start the lidar node connected with the slave Raspberry Pi and the Gmapping mapping node through the master control machine. After, the unmanned ship uses the surrounding environment information data collected by the lidar to map the navigation environment. Finally, save the unmanned ship navigation environment map in the working directory through the map_saver node, and prepare the established raster map for subsequent use.

First, run the navigate navigation node, use the Rviz visualization tool in ROS to give the robot’s initial position, and add the topic that needs to display the target point. Then, run the multipoint navigation program to enable the patrol_nav node. Use the A* algorithm to find the shortest path and use it as the global path to make the base_local_planner package get the local path^7-9. Finally, calculate the required real-time speed and angle, and publish this information as a control signal to the hardware platform. Automatically run the set node file after booting and automatically cruise by importing the manually selected target point.

2.4.

Robotic arm design

The robot arm mainboard uses Arduino UNO, an Arduino development board based on ATmega328P. It has 14 digital input/output pins, of which 6 can be used for PWM output. It is used to control the up and down, left and right, front and rear, and grabbing actions of the four servos, and can complete the task of collecting small target garbage. It contains the functions required by the microcontroller. Connect it to the USB interface of the central controller for driving. The PWM output waveform controls the steering gear. The high-level duration of the waveform corresponds to the angle of rotation, achieving a rotation angle of 0-180°.

3.1.

Experimental setup

The target detection algorithm in this system is built using the Darknet deep learning framework, and the programming languages are C++ and Python. The initial value of the learning rate (LR) in the garbage detection algorithm is 0.001, the attenuation is set to 0.0005, and the maximum number of iterations is 10,000. The model is deployed in the operating system of Linux Ubuntu 16.04. The GPU configuration is GTX3060. The environment of the computing platform is CUDA 10.1 and cuDNN 7.6.4. The CPU frequency is 2.90GHz, and the memory is 16GB.

3.2.

Experiment procedure

First, the roscore node manager in the host computer is enabled, and at the same time we run the chassis communication node, lidar node and Gmapping mapping program to create a two-dimensional grid map and save it. Then, we run the navigation program and target detection program to complete the task of garbage identification and positioning. Finally, the obtained relative position of the garbage is sent to the robotic arm through the serial port to salvage the trash.

3.3.

Dataset

According to the classification statistics of floating garbage on the water surface, plastic garbage accounted for about 41%, foam and wood product garbage accounted for about 19% and 15%, respectively¹⁰. Therefore, the system collects five types of common garbage on the water through the Internet and offline: plastic, glass bottles, cardboard, metal garbage, and foam snack boxes. We selected 5000 pictures as the data set, divided the data set into the training set and test set evenly according to the ratio of 4:1, and labelled the data set with the LabelImg labelling tool.

3.4.

Experimental results

The real-time detection effect diagram of garbage salvage is shown in Figure 6.

Figure 6.

Real-time detection effect diagram.

The system detects the five types of garbage collected, and their accuracy, precision and recall rates are shown in Table 2. The information collected by the camera is detected through the trained model, the real-time detection speed can reach 45fps, and the average recognition accuracy is as high as 88.8%. The detected information is sent to the robotic arm through the serial port, and the robotic arm salvages the garbage, which can realize the function of the overall garbage salvage system.

Table 2.

Comparison table of detection performance of various types of garbage.