A Framework for Real-Time Autonomous Robotic Sorting and Segregation of Nuclear Waste: Modelling, Identification, and Control of Dexter Robot

1. Introduction

Nuclear reactors are often preferred over fossil fuels for efficient and sustainable large-scale power generation without refueling for a long time. Developing nations worldwide consider nuclear energy as one of the important energy sources for meeting the growing energy demands [1]. Even though nuclear energy production contributes towards clean energy, disposal of nuclear wastes efficiently and safely is challenging[2]. In addition to that, efficient methods for decommissioning and clean-up of aging nuclear reactors, for example, Sellafield(UK) [3], and technology developments for handling nuclear disasters, for example, Fukushima (Japan) [4], are in need. The Nuclear Decommissioning Authority (NDA) oversees many legacy facilities in the UK that are in the process of being decommissioned and with rising nuclear wastes the scale of the problem is enormous [5]. Safe, effective and optimized waste management is a significant part of these long-term nuclear maintenance or decommissioning tasks. The radioactive wastes in nuclear power plants are commonly classified as High-Level Waste (HLW), Intermediate Level Waste (ILW), Low-Level Waste (LLW) and Very Low Level Waste (VLLW) according to the nature of material, level of radioactivity and the heat produced [6]. In order to facilitate a waste management strategy, these wastes needed to be sorted and segregated for optimal packing. Currently, these materials are handled by trained operators wearing personal protective equipment (PPE) [7] or by remotely controlled robots [8]. The human-led nuclear waste handling tasks are challenging due to proximity to radioactive wastes, limited efficiency, and high time consumption. In order to address these challenges, methodologies using robotic technologies have emerged to find ways to perform safe and efficient handling of nuclear materials.

Several attempts have been made earlier using robots [9,10,11,12], but robotic sorting and segregation of nuclear wastes are seldom reported in the literature. The Joint European Torus (JET) machine employs a human-in-loop approach for teleoperating the robotic boom and a dual-arm robotic system inside the JET Torus [13]. A robotic arm mounted on a mobile robot, Hydro-Lek, for nuclear decommissioning tasks is presented in [14,15]. Hydro-Lek is a seven degrees-of-freedom (DoF) hydraulic manipulator with a continuous jaw-rotation mechanism designed to grasp objects and cut ropes. The Oak Ridge National Laboratory’s Advanced Servomanipulator System and Dual-Arm Remote Sawing Manipulator System are task-specific robotic systems designed for specific nuclear decommissioning applications [16]. A modular robotic system for Reactor Core Detector Removal was presented in [17] for positioning, extracting, transporting, cutting, and coiling tasks. As robotics and artificial intelligence based computer vision technologies [18,19,20] have advanced through industrial automation, it is now clear that autonomous robotics can also play a key role in nuclear waste sort and segregation applications. [21,22] presented industrial non-nuclear waste sorting robotic methodologies using KUKA arms and a 4-DoF fast Parallel robot. Conventional robotic mechanisms and motion planning approaches work well for structured or semi-structured environments [23]. However, in harsh environment, the precarious nature of the physical tasks involved and regulatory constraints impose several other challenges in terms of deploying commercial robots for nuclear clean-up. A recent survey on challenges for future robotic sorters suggested the requirement for a highly dexterous dual-arm robotic systems for efficient sorting of unstructured and heap of objects [24]. Even though there is advancement in image-based detection of waste objects and environmental monitoring using computer vision and AI, developing a new autonomous dexterous robotic system specifically for sorting and segregation of nuclear waste is challenging.

Robotic remote operation technologies under human supervision are needed to perform hazardous decommissioning and sorting activities. For faster development and to overcome the entry barriers in a highly regulated nuclear domain, the authors solve the problem by building a novel solution to further automate and induce intelligence to the existing commercial Dexter^TM teleoperation system by Veolia Nuclear Solutions (VNS). A new methodology for switching from teleoperation to autonomous operation is presented based on rigorous system identification and dynamic modelling. This paradigm shift allows a quicker and more efficient introduction of shared autonomy into the previously manual tasks. The Dexter^TM manipulator comprises cable driven joints, complex gear assembly and parallel linkages. Identifying the dynamics of such a system is not straightforward and requires, precise modelling and system identifications. Dynamical model parameter identification of the Dexter^TM manipulator through excitation trajectories is one of the major contribution of this paper. The complex friction model and joint elasticity of cable-driven joints are also considered to obtain a comprehensive dynamic model through constrained optimization. This enables the Dexter^TM to perform sort and segregation tasks autonomously. The new framework presented realise object identification, classification, tracking, robotic grasping, sorting and packaging of nuclear wastes according to the radiation levels. For more complex grabs and tangled wastes, the integrated system would also have the ability to call on the expertise of a skilled operator to take over the task for manual operation; the remote Dexter^TM robot will replicate the operator’s movements on the local Dexter^TM. By sharing the tasks between human and computer intelligence, this approach offers a step change in safety and efficiency through limited intervention by operators for complex tasks. This methodology will enable accelerated adoption of automation and autonomy in extreme environments without rigorous changes in highly regulated sectors.

This paper first introduces the Dexter^TM robotic teleoperation system, and the sort and segregation experimental setup in Section II. In Section III, an outline of the proposed sort and segregation framework is presented. Section IV elaborates on the modeling, system identification, and control of the Dexter^TM for autonomous operation. The sensor integration and autonomous procedures for the radiation scanning, sorting, segregating, and packing of the wastes are described in Section V. Section VI covers experimental verification and validation, and finally, the concluding inferences are stated in Section VII.

2. Preliminaries and System Overview

This section presents relevant preliminary details that define the baseline architecture and building blocks of the new framework covered in Section III. Initially, the Dexter^TM, a commercial haptic-enabled and radiation-hardened teleoperated robotic platform, is introduced. Later, an overview of the experimental setup is provided, which accomplishes autonomous nuclear sort and segregation activities.

2.1. Dexter^TM teleoperation system

Dexter^TM, in Figure 1, is an exemplar of a collection of telemanipulation capabilities purposed for extremely hazardous environments, notably extreme radiation [13]. The Dexter^TM system is primarily developed for Human-in-Loop operations using kinematically identical local and remote arms, the former in a safe operations centre and the later in a hazardous operational environment. The Mascot (the predecessor to Dexter^TM) started life as an indispensable part of the Joint European Torus (JET) nuclear fusion program’s remote handling operation and it supports the nuclear fuel debris retrieval from a stricken reactor at the Fukushima Daiichi nuclear power plant in Japan. The capabilities of VNS’s Dexter^TM include a system that is able to repair, maintain and/or re-tool a larger complex mechanical system, including the ability to respond to unforeseeable upset conditions. In essence, this high-value system provides an insurance policy guaranteeing that the system can be kept in service despite not being accessible to humans. The human operator performs the tasks with the local manipulator, which the remote manipulator replicates in real-time in the remote location. The remote manipulator can be positioned up to $8km$ from the local manipulator using only data cables and there is no mechanical connection between the two systems. One of the key feature of the Dexter^TM arm is that the arm can be considered as a straightforward replication of the operator’s arms. The human operator can prepare for tasks and carry them out almost exactly as if they were intervening in the environment directly. Along with the extreme smoothness, responsiveness, and sensitive force feedback of the arms coupled with camera feedback creates a sense of “Telexistence” for the operator.

A single arm of Dexter^TM system comprises all revolute joints to exhibit 6-DoF motion. The first three joints are driven by a gear system and last three joints are cable driven in addition to cable driven two finger end-effector. The electrical actuators responsible for each joint motions are placed together at the base of the Dexter^TM in a specialised configuration with in a protective casing. This is beneficial for reliable operations in a nuclear environment where radiations and high temperatures will effect the electrical systems. In order to achieve heavy load carrying capacity, a specialised parallelogram mechanism is also provided between joint one and three. For transforming this torque controlled Dexter^TM teleoperation system to autonomous system, where currently the remote arm joint positions follow local arm joint positions, an accurate kinematic and dynamical model has to be obtained. The dynamical model identification of Dexter^TM robot is challenging considering the above mentioned configurations and specialised actuation mechanisms. Further, the frictions in joints and the elasticity of cables bring additional challenges in model identification.

2.2. Sort and Segregation Experimental Setup Overview

The experimental setup comprising Dexter^TM local and remote arms is shown in Figure 2(a). The remote system is isolated to perform complete sort and segregation operations safely similar to a nuclear environment. As shown in Figure 2b, sensors and sorting tables are placed within the worspace of remote arm for performing autonomous operations. In order to obtain the 3D information of the object, two industrial grade SICK sensors are placed facing towards the robot workspace in an eye-to-hand configuration. Initially, the sorting table is loaded with all the waste objects that need sorting. Once a waste object is picked up and its characteristics identified, it will be deposited in ILW, LLW, or recycling trays, as shown in Figure 2(b). Since the robot base is stationary, a trolley mechanism manually moves the corresponding trays into the robot’s workspace after characterizing the waste item.

3. Outline of Proposed Framework

The new framework presented in this paper aims to advance the autonomy and automation of Dexter^TM system and integrate it with advanced characterization and tracking technologies to identify, classify, transfer and package waste items commonly found in the nuclear industry. Figure 3 shows a top-level system process flow for a nuclear sort-and-segregation task using the Dexter^TM system, which comprises various control modules, as shown in Figure 4. The low level control module consists of Dexter^TM kinematic and derived dynamic models, and position controllers. Using ROS framework, various submodules and Task Groups (TG) are interconnected for task transitions, sensor interfaces, safety, teleoperation and Digital Twin in the middle-level control. The GUI enables an interface to control various control tasks in middle-level control and also act as an information display system for the operator to make decisions. The system is designed such that high priority is given for system and operation safety of robot through process control. The process control is also responsible for identification of various failure modes, such as object slip and drop, and failure to generate feasible trajectory to desired pose. The failure modes will generate corresponding flags in the GUI for the operator to intervene and take necessary actions.

4. Robot Modelling, Identification and Control

This section initially describes Dexter^TM’s kinematic model and an analytical formulation of dynamic equations, including friction and joint elasticity components. Next, a system identification procedure is presented for obtaining reliable dynamic parameters. Finally, a joint space position controller is implemented using a pre-computed feed-forward torque controller.

An open kinematic chain of Dexter^TM joint frames, shown in of Figure 5, is considered for obtaining the minimal kinematic model of the robot. The calculated Denavit–Hartenberg (DH) notation based kinematic parameterization of the Dexter^TM is provided in Table 1, where $a_i$, $d_i$ and $q_i$ are corresponding link length, offsets and joint angles of Dexter^TM as shown in Figure 5.

4.1. Dexter^TM Kinematic and Dynamic Model

The motion of the 6-DoF Dexter^TM manipulator can be described by the general rigid body dynamics as in (1)

$$\mathit{\tau }=\mathit{M}\left(\mathit{q}\right)\ddot{\mathit{q}}+\mathit{C}(\mathit{q},\dot{\mathit{q}})\dot{\mathit{q}}+\mathit{G}\left(\mathit{q}\right)+{\mathit{\tau }}_f+{\mathit{\tau }}_e,$$

(1)

where $\mathit{q}$, $\dot{\mathit{q}}$, $\ddot{\mathit{q}}$, $\in {\mathbb{R}}^6$ represent the joint position, velocity, and acceleration vectors respectively. $\mathit{M}\left(\mathit{q}\right)\in {\mathbb{R}}^{6\times 6}$ represent the mass/inertia matrix containing respective mass $m_i$ and inertia tensors $I_i$. $\mathit{C}(\mathit{q},\dot{\mathit{q}})\in {\mathbb{R}}^{6\times 6}$ matrix contains corresponding Coriolis and Centrifugal forces/torques. The terms $\mathit{G}\left(\mathit{q}\right)$ and $\mathit{\tau }\in {\mathbb{R}}^6$ are the gravitational force and input torque vectors respectively. In order to incorporate the dynamical effects due to friction and joint elasticity of Dexter^TM mechanisms, two additional terms ${\mathit{\tau }}_f$ and ${\mathit{\tau }}_e$ are incorporated into the dynamical model. The joint friction term ${\mathit{\tau }}_f\in {\mathbb{R}}^6$ is modeled as

$${\mathit{\tau }}_f=\left(\left({\mathit{F}}_c+({\mathit{F}}_s-{\mathit{F}}_c)e^{-|\dot{\mathit{q}}|/{\dot{\mathit{q}}}_s}\right)+{\mathit{F}}_v\dot{\mathit{q}}\right),$$

(2)

where ${\mathit{F}}_s$, ${\mathit{F}}_v$, ${\mathit{F}}_c$, and ${\dot{\mathit{q}}}_s$ represents static, viscous, Coulomb and Striebeck constants respectively [25]. Similarly, the elasticity of the joints are modeled as

$${\mathit{\tau }}_e=\mathit{K}\mathit{q},$$

(3)

where $\mathit{K}\in {\mathbb{R}}^{6\times 6}$ is a diagonal stiffness matrix.

The identification of constant dynamical parameters for the Dexter^TM includes additional terms from the friction and elasticity models. These additional parameters are required for the accurate position of Dexter^TM through torque control, considering the complex gear systems and cable-driven actuation methods.

4.2. Dynamic Model Parameter Identification

The Dynamical parameter identification [26] is a crucial part in the process of transforming purely teleoperated Dexter^TM robot to an autonomous system. The direct identification of model parameters are not straightforward considering the advanced Dexter^TM robot mechanisms. Therefore, data collected by actuating the Dexter^TM manipulator using excitation trajectories are used to identify the dynamical model parameters through convex optimization. An outline of the parameter identification procedure is shown in Figure 6. From the generic dynamic model of the Dexter^TM robot (1), the unknown parameters that needs to be identified for $i^{th}$ link are mass $m_i$, Center-of-Mass (CoM) $l_i=[l_x,l_y,l_z]$, inertia parameters ($I_{ixx},I_{ixy},I_{ixz},I_{iyy},I_{iyz},I_{izz}$), friction constants ($F_{is},F_{ic},F_{iv}$) and stiffness constant $K_i$. In total 84 parameters need to be identified for a 6-DoF Dexter^TM manipulator.

For simplicity, (1) is represented in the linear form [27] by

$$\mathit{\tau }=\mathit{H}(\mathit{q},\dot{\mathit{q}},\ddot{\mathit{q}})\mathit{\beta },$$

(4)

where $\mathit{H}\in {\mathbb{R}}^{6\times 84}$ represents a regression matrix and $\mathit{\beta }\in {\mathbb{R}}^{84}$ represents the unknown dynamic parameters vector. From the Dexter^TM robot’s physical configuration its understandable that not all dynamical parameters will contribute to the dynamic model and becomes zero. Hence, a minimum set of identifiable parameters is obtained from (4) using numerical QR decomposition method [28]. The dynamic equation with reduced number of parameters ($r=57$) can be represented as

$$\mathit{\tau }={\mathit{H}}_r(\mathit{q},\dot{\mathit{q}},\ddot{\mathit{q}}){\mathit{\beta }}_r,$$

(5)

where ${\mathit{H}}_r$ and ${\mathit{\beta }}_r$ are reduced regression matrix and vector of identifiable dynamic parameter set. From an excitation trajectory $q_i\left(t\right)$, where $t=t_1,t_2,\cdots ,t_k$ time step, and using the collected joint position, velocity, acceleration and torque values, the linear equation in (5) can be represented as

$$\mathit{\tau }=\left[\begin{array}{c}\mathit{\tau }\left(t_1\right)\\ \mathit{\tau }\left(t_2\right)\\ ⋮\\ \mathit{\tau }\left(t_k\right)\end{array}\right]={\mathit{H}}_e{\mathit{\beta }}_r,$$

(6)

where ${\mathit{H}}_e$ is an extended regression matrix

$${\mathit{H}}_e=\left[\begin{array}{c}{\mathit{H}}_r(\mathit{q}\left(t_1\right),\dot{\mathit{q}}\left(t_1\right),\ddot{\mathit{q}}\left(t_1\right))\\ {\mathit{H}}_r(\mathit{q}\left(t_2\right),\dot{\mathit{q}}\left(t_2\right),\ddot{\mathit{q}}\left(t_2\right))\\ ⋮\\ {\mathit{H}}_r(\mathit{q}\left(t_k\right),\dot{\mathit{q}}\left(t_k\right),\ddot{\mathit{q}}\left(t_k\right))\end{array}\right].$$

(7)

4.2.1. Identification by Minimization

The dynamical parameter estimation is posed as a minimization problem between measured torque vector ${\mathit{\tau }}_{\mathit{m}}$ and predicted torque vector $\mathit{\tau }$ in (6) using an excitation trajectory. The parameter identification is performed by solving the following constrained optimization problem

$$\begin{array}{cc}\hfill \underset{{\mathit{\beta }}_r}{min}& \left|\right|{\mathit{\tau }}_m-{\mathit{H}}_e{\mathit{\beta }}_r{\left|\right|}_2{\mathit{\beta }}_r\in \mathcal{D}\hfill \\ \hfill \mathrm{subject}\mathrm{to}& \left\{\begin{array}{c}\mathit{h}\left({\mathit{\beta }}_r\right),\hfill \end{array}\right.\hfill \end{array}$$

(8)

where $\mathit{h}\left({\mathit{\beta }}_r\right)$ represents the constraints and $\mathcal{D}$ denotes search space. In order to obtain a physically feasible parameter set following physical consistency constraints, $\mathit{h}\left({\mathit{\beta }}_r\right)$, are defined.

• The mass of the links must be positive definite: $m>0$,
• The inertia tensors must be positive definite $I>0$,
• Eigenvalues of the inertia tensors (${\sigma }_x,{\sigma }_y,{\sigma }_z$) must satisfy triangle inequality conditions ${\sigma }_x+{\sigma }_y>{\sigma }_z,{\sigma }_x+{\sigma }_z>{\sigma }_y,{\sigma }_y+{\sigma }_z>{\sigma }_x$, as in [29],
• The mass center should remain in its convex hull that is $m{l}^{lb}<l_i$ and $m{l}^{ub}>l_i$[30], where $l^{lb}$ and $l^{ub}$ represents lower and upper bounds of the $l_i$ respectively,
• The stiffness of the spring is positive definite: $K>0$

4.2.2. Optimal Excitation Trajectory Generation

It is important to ensure that the excitation trajectories provided to the Dexter^TM must induce all the modeled dynamical effects for accurate parameter identification. Therefore, finite Fourier Series based excitation trajectories are used to obtain variable amplitudes and frequencies. Using Fourier Series functions the angular position trajectories of $i^{th}$ joint can be written as

$$\begin{array}{cc}\hfill q_i\left(t\right)& =\sum _{n=1}^N{\displaystyle \frac{a_{n,i}}{w_{f}n}}sin\left(w_{f}nt\right)-{\displaystyle \frac{b_{n,i}}{w_{f}n}}cos\left(w_{f}nt\right)+q_{0,i},\hfill \end{array}$$

(9)

where $w_f$, N and $q_{0,i}$ are frequency, the number of the harmonics and initial joint angles respectively. $a_{n,i}$ and $b_{n,i}$ are trigonometric term constants and $2N+1$ parameters have to be identified per joint. Since the joint angle’s range and workspace of Dexter^TM manipulator is limited, another constrained nonlinear optimization is performed in order to generate sufficiently rich trajectories. Minimizing the condition number of the regression matrix ${\mathit{H}}_e$ [31] is defined as an objective function towards this with joint and pose constraints:

$$\begin{array}{cc}\hfill argmin& cond\left({\mathit{H}}_e\right)\hfill \\ \hfill \mathrm{subject}\mathrm{to}& \left\{\begin{array}{c}{\mathit{q}}_{min}\le \mathit{q}\le {\mathit{q}}_{max},\hfill \\ {\dot{\mathit{q}}}_{min}\le \dot{\mathit{q}}\le {\dot{\mathit{q}}}_{max},\hfill \\ {\ddot{\mathit{q}}}_{min}\le \ddot{\mathit{q}}\le {\ddot{\mathit{q}}}_{max},\hfill \\ {\mathit{s}}_{min}\le \mathit{s}\le {\mathit{s}}_{max},\hfill \end{array}\right.\hfill \end{array}$$

(10)

where $\mathit{s}$ represent task space end-effector positions.

4.3. Position Control System Design of Dexter^TM

Advanced control methods are required for highly non-linear robots like Dexter^TM for accurate and efficient trajectory execution. The computed torque control method with linearization for compensating the robot dynamics is one of the common nonlinear control methods [32]. Instead of altering the Dexter^TM’s internal low-level torque control architectures, an offline calculated feed-forward computed torque controller is utilized in this research.

Based on the desired joint trajectories (${\mathit{q}}_d,{\dot{\mathit{q}}}_d,{\ddot{\mathit{q}}}_d$), predicted joints torques are obtained offline using (1). The controller then applies these predicted joints torques to Dexter^TM joints together with a Proportional–Integral–Derivative (PID) control as shown in Figure 7. The joint control command $\mathit{u}$ for the Dexter^TM system can be written as:

$$\begin{array}{ccc}\hfill \mathit{u}& =& \mathit{M}\left({\mathit{q}}_d\right){\ddot{\mathit{q}}}_d+\mathit{C}({\mathit{q}}_d,{\dot{\mathit{q}}}_d){\dot{\mathit{q}}}_d+\mathit{G}\left({\mathit{q}}_d\right)+{\mathit{\tau }}_f\left({\dot{\mathit{q}}}_d\right)\hfill \\ & & +{\mathit{\tau }}_e\left({\mathit{q}}_d\right)+K_p\mathit{e}+K_d\dot{\mathit{e}}+K_i\int \mathit{e},\hfill \end{array}$$

(11)

where joint error $\mathit{e}$ is defined as ${\mathit{q}}_d-\mathit{q}$ and $K_p,K_d,K_i$ are PID gains.

4.4. Motion Planning and Trajectory Generation

For generating optimal motion trajectories on Dexter^TM robot in-order to reach desired positions, Open Motion Planning Library’s (OMPL’s) Rapidly Exploring Random Tree (RRT*) algorithm is utilised through ROS [33]. The implemented motion planner considers joint limits, singularities and self-collisions of Dexter^TM manipulator. In order to avoid collisions with objects in the Dexter^TM workspace, a 3D occupancy grid map (Octomap) of the environment is also generated and included in the planner. The real-time point cloud input of the environment is obtained from depth cameras. Figure 8 presents the point cloud of the environment generated on the test setup described in Section II(B) for optimal trajectory generation with collision avoidance.

5. Autonomous Object Grasping and Packing

Autonomous detection of objects from a pile, its characterisation and packing using a robotic system requires efficient algorithms and their integration into a single framework. The autonomous framework developed in this research (Figure 3) around Dexter^TM system utilizes various AI based methodologies to perform nuclear sort and segregation tasks in a single pipeline. The methodologies used for identifying the objects, how and where to grasp the objects, material characterisation, radiation scanning of the objects and optimal packing are presented in this section.

5.1. Object Detection, Material Characterization and Grasp Point Generation

The vision system of the autonomous Dexter^TM consists of two stationary SICK Visionary-S RGBD sensors attached to the sensor frames as shown in Figure 2(b). These industrial sensors are traditionally used for verifying box packing and are ideal for identifying a large number of items simultaneously with high frequency of operation. Two SICK sensors are carefully placed in the Dexter^TM workspace such that a single point cloud of the whole scene can be generated from two RGBD images, which almost completely describes the scene in 3D. The next step in object detection and classification involves using efficient AI-based image processing algorithms.

In order to improve performance and to avoid integration complexities in terms of system requirements and dependencies, segmentation, classification and material characterization is implemented together using the SoftGroup [34] AI model. The 3D instance segmentation model, SoftGroup, also provided the flexibility for isolating point cloud of the target object from the robot gripper to perform efficient single object characterization rather than only from piles. Around 130 different objects commonly seen in nuclear waste sites were considered, and over 400 scans of these objects were carried out to formulate the training dataset for the SoftGroup model. Each scan was created by both SICK cameras taking an RGBD image simultaneously, which was then processed into a single point cloud of the whole scene. Along with the scans from a pile of $20-40$ objects on the sorting table, objects held by Dexter^TM above the sorting table were also collected at different orientations. In the preprocessing stage, ground truth segmentation masks and classification tags consisting of object category, material, mass, volume and density were manually annotated. For material categorisation of the objects, the object category and material classifications are merged to encode both geometric/semantic and material information. For example, instead of having “steel” and “aluminium” as material classes and “rod” and “metal scrap” as category classes, “steel rod”, “steel scrap”, “aluminium rod” and “aluminium scrap” are considered as separate independent classes. In this way, a single SoftGroup model performs all segmentation and classification tasks for the whole system, significantly reducing the system’s complexity and running time, without sacrificing the model’s prediction accuracy.

Once the objects in the pile are detected and identified, the next step is to identify feasible grasp poses [35] on the objects in order to perform grasping by the Dexter^TM. An object grasp pose generator pipeline is developed using Contact-GraspNet [36], which takes the segmented and classified point cloud as input.

5.2. Geometry Characterisation and Mass Estimation

Objects’ geometric information, such as length, width, height, surface area, and volume, is another important characteristic required for optimal packing. This information is crucial, along with the object type and name in nuclear applications for keeping a catalog of objects in a packed container, especially if radioactive. The geometric characterisation is performed on objects in a pile and also after the objects held by the gripper for revalidation. A sequence of operations is performed using the point cloud generated by two cameras, which outputs a watertight mesh of the object under consideration for geometric characterisation. Generally, with two RGBD cameras, the complete capture of all the sides of the object is not possible, and the corresponding mesh is unreliable for accurate characterisation. In the first step, the combined 3D point cloud obtained from two cameras is down sampled and an interpolation is performed to fill the vacant space on the object point cloud. Next, a simple mesh (dense 3D geometry) is created from the interpolated point cloud using a surface reconstruction method called Ball pivoting using Open3D [37]. In order to calculate the volume of objects, a watertight mesh consisting of a continuous and closed surface is needed. Using the manifold method in Open3d, the watertight meshes are generated, and finally, the geometric information of the object is obtained.

On top of identifying the object mass from camera input, a direct mass estimation model using Dexter^TMrobot is also developed for better accuracy. For this, 25 different objects with known masses and the torque value of Dexter^TM joint-1 to lift the object are considered. After the object is grasped, the Dexter^TM is moved to a specific predefined configuration to estimate the mass. Using the ground truth mass and corresponding torque values, a fourth-order function is fitted using the linear least square curve fitting method Figure 9. Using this mapping, the object’s mass m is obtained using the following function:

$$m=0.0002{\tau }_1^4-0.0013{\tau }_1^3-0.0154{\tau }_1^2+0.3462{\tau }_1-0.6673,$$

(12)

where ${\tau }_1$ is the corresponding measured torque values to lift the object.

5.3. Radiological Surveying and Decision

In a nuclear sort and segregation application, one of the crucial tasks is the efficient detection of object’s radioactivity levels. In this research, real radioactive objects were not available for testing and therefore a radiological surveying pipeline is designed where random activity levels are assigned to segmented items in the point cloud and radiation measurements are simulated on a mock up radiation detector held by the Dexter^TM robot. For performing radiation scanning, a 2D end-effector trajectory is generated at a fixed height from the tray where objects that need to be picked are placed. Although the trajectory planner aims to scan the whole tray, it is focused more on the object location inputs from the point cloud and segmentation outputs to reduce scan time requirements. The Dexter^TM performs slow and detailed scanning movements near the objects and finally, if calculated confidence for waste sentencing is low, random scanning is performed over the tray in order to obtain a more accurate object activity result.

The radiation detector demonstrated here is Createc’s N-Visage Recon. As this is a gamma detector, the alpha and beta content of waste items is determined though application of radionuclide fingerprints in this example. Use of an alpha/beta monitor was explored and the positional accuracy of the Dexter^TM demonstrated surface monitoring with this type of monitor to be possible for simple flat surfaces though challenging for more complex surfaces. The output activity for each waste item is obtained through calculation in Createc’s N-Visage algorithm [38], where measurement uncertainty and Minimum Detectable Activity (MDA) is determined through a Monte Carlo based random sampling method in the algorithm and application of the Currie Equation [39]. For sorting the object into one of the ILW, LLW, or recycling trays by the Dexter^TM, this activity model is used.

5.4. Bin Packing

Efficient bin packing is another challenge that is faced in the nuclear industry. The strategy of dropping objects randomly into the bin often results in instability of heterogeneous objects inside the bin, uneven weight distribution and inefficient space utilization. An efficient container packing algorithm, Tetris, is utilized for autonomous bin packing using Dexter^TM robot[40]. The algorithm calculates the quasi-optimal position and orientation to place the picked objects in the destination container. Initially, using the 2D Tetris algorithm the bottom layer of the bin is filled using a target function such that it minimizes the total unoccupied bottom surface area of the container. The target function places the objects as close as possible to this bottom wall and also minimize the holes in the bottom surface area of the container. Next, based on the shape of the object in hand and the landscape of the existing objects in the container, another target function executes Tetris in the z-direction and tries to maximize the number of objects packed and simultaneously minimize the gap volume of the container. With this target function, it would try to put each object onto the lowest region as possible in the container, but in the meantime, also try to minimise the void volume underneath the object that would be created after putting it, as this volume would not be recovered.

6. Experimental Evaluation

This section presents the experimental evaluation of the proposed sort-and-segregation framework, outlined in Section III, using Dexter^TM manipulator. The experimental setup with cameras and sorting tables is shown in Figure 2. The initial part of this section presents the results of Dexter^TM dynamical model parameter identification. For verifying its effectiveness, the tracking performance is studied by implementing a position controller. Next, the results of sort and segregation subsystems, presented in Section V, are provided. Finally, the integrated system performance is evaluated through successful nuclear sort-and-segregation experiments on similar objects from the nuclear industry.

6.1. Parameter Identification and Model Validation

The input excitation trajectories for dynamical model parameter identification of the Dexter^TM manipulator are generated by solving (10) using convex optimization, as explained in Section IV(B). The generated reference trajectories induces the modeled dynamical effects, friction and elasticity, of the Dexter^TM manipulator. The corresponding joint positions are shown in Figure 10.

Using these reference trajectories, the Dexter^TM joints are actuated using the position controller shown in Figure 7 where an approximate of dynamical parameters were used from Dexter^TM CAD model. In order to find the exact dynamical model parameters, initially, the executed joint torques and angular motion information ($\mathit{q},\dot{\mathit{q}},\ddot{\mathit{q}}$) are collected and stacked across each time step as the training data set. Using this data, a suitable set of dynamic parameters is estimated through convex optimization using (8). The designed position controller (11) is tested along with the updated dynamical parameters to accomplish Dexter^TM autonomous control as shown in Figure 11. Although there are minimal deviations during the motion due to the nonlinearity caused by its cable-driven nature and the unmodeled dynamic effects, the controller showed good performance that enabled the desired target to be achieved.

Table 2 compares the torque tracking error and their deviation for all joints for the generated training and test trajectory. Even though joint-2 and joint-3 errors are slightly higher, overall percentage error and standard deviations are minimal, resulting in less than $0.7cm$ end-effector mean positional error through the feed-forward nonlinear control. These results confirm the accuracy of the dynamical model parameters identified.

6.2. Sort and Segregation Subsystems

The results of sort and segregation operation performed using experimental setup, explained in Section II(B), in a real environment are provided below. Integrating individual subsystems led to expected full system performance, which successfully sorted and segregated various objects from a heap.

6.2.1. 3D Instant Segmentation and Classification

The pipe-line for sort and segregation process starts by generating two point clouds of the environment using two SICK sensors. Initially, the two point clouds were processed into a single point cloud of the whole scene using conventional Multiview Stereo (MVS) reconstruction method [41] and performed a denoising. The denoising is required to overcome the intrinsic device limitation of the SICK sensors, especially when measuring highly reflective surfaces. The developed SoftGroup model takes the combined point cloud input and performed segmentation and classification of each object agnostically from the pile and object in Dexter’s hand. An example of such a pair of acquired RGB-D images for a scene, merged single point cloud and the result of the SoftGroup model-based pipeline are shown in Figure 12. It can be seen that most objects were cleanly segmented, and the safety hat in the gripper was correctly classified (other objects were also classified by the model but not illustrated in the figure for brevity). A second level of object classification, while Dexter^TM holds the object and rotates it to three predefined orientations, is also performed by extracting its isolated point cloud. The SoftGroup model predicts the probabilistic vector of the target object belonging to each candidate category. The pipeline then reports the most likely category of the target object as the category with the highest probability averaged across the predictions. The target object’s point cloud was extracted from each scan orientation’s using the segmentation mask predicted by the SoftGroup model, and then merged together to obtain a complete point cloud of the object. Figure 13 shows the complete extracted point cloud of a safety hat, and also the overall categorical probability histogram of it as averaged across the detailed scans’ model predictions. A sample of the objects used and their category are provided in Table 3.

6.2.2. Radialogical Surveying

After scanning the pile and detecting objects, radiological surveying is conducted. Initially, Dexter^TM grasps the radiation detector and performs the scanning operation according to the scanning trajectory calculated offline. The radiation scanning trajectory is generated based on the segmented object positions in the point cloud of the pile. Figure 14 demonstrates the generated end-effector trajectories, which perform slow scanning near objects and a final random scanning if confidence in radiation measurement and subsequent waste sentencing route is determined to be low. In the absence of real radioactive waste items, random radioactivity levels are assigned to objects for the experiments conducted. In experiments, the implemented model creates simulated detector measurements and uses these to create a radioactivity map superimposed on the point cloud while the trajectory is being executed. Item radiation activity is then matched with associated detected objects. The video (https://youtu.be/v2hlDxKDGcM) demonstrates a radiation-scanning experiment post object detection.

6.2.3. Grasp Pose Generation

The core grasp-pose candidate generation method used in this research is Graspnet (Section 5.1). On top of Graspnet, a heuristic system is implemented on the generated grasp candidates to create higher quality and deterministic/reproducible grasp poses in cluttered piles and extreme cases such as the small lego block. This grasp generation system prioritizes grasp poses of the targeting objects having larger top surface areas and higher up in a pile. This strategy helped to pick the objects without much disturbance to the pile and identify objects that were hidden in the previous scan. It also prioritizes grasp poses closer to the geometric centre of each object, as such grasp poses would lead to a lower chance for the object to rotate, tip over, or slip out from the gripper during grasping and after being picked up. Figure 15 shows the top 3 highest priority grasp poses generated by the algorithm for two different clutter piles.

6.2.4. Object Characterisation

The pipeline for object geometry and physical characterization was based on conversion of point cloud of the object into a watertight mesh body, as explained in Section 5.2. Figure 16 shows two examples of the extracted point clouds for a wellington boot on the table and a Plastic hose held by Dexter^TM. By using Open3D’s inbuilt functionality, approximate geometric information of the object from the watertight mesh is obtained. On comparing the accuracy against the ground truth manual measurements of over 50 objects, closer results were observed except for objects with complex shapes, in particular, those with internal hollowness or folded structures. The estimated masses of objects were calculated using the estimated volume of object from Open3D, the estimated object material from category classification and the object category density values. In addition, an estimated mass of the object held by the Dexter^TM gripper is also calculated based on the torque feedback of the motors. The torque-based measurements were more closer to the mass of ground truth objects. Hence, the vision-based mass information is used as a redundant information for second validation.

6.3. Bin Packing

Four different types of “human benchmarks” packing algorithms are analysed and tested in simulation to validate the efficacy of the developed Tetris algorithm. The benchmarks are 1) Center Algorithm: randomly oriented object dropped at the center of the container, 2) Random Algorithm: randomly oriented object dropped at a random position of the container, 3) Lower Bound Algorithm [42], and 4) Orthogonal Low algorithm [43]. Each algorithm runs ten times for the same pile of objects. Table 4 presents the average values of the performance of the algorithms for comparison.

The unoccupied volume (voidage) of the container is used as a scoring method for comparison and is calculated as the volume of voidage = volume of container - the total volume of objects. The packed container volume in the experiments is $43.261L$. Although the Tetris algorithm is slower than the other algorithms, the optimal packing with the minimum voidage rate is achieved. The result of bin packing using the developed Tetris algorithm in a real experiment is shown in Figure 17. In this experiment, a total of 40 items are placed into the container with a total volume of $43,261L$ and voidage $19.221L$. A $44\%$ container voidage is obtained in this experiment with an average pose generation run time of $0.23959sec$.

6.4. Full System Demonstration

The designed sort and segregation subsystems and the Dexter^TM control systems are integrated into a full system demonstrator, as shown in Figure 18. It consists of a Dexter^TM manipulator, two SICK Visionary-S cameras, one sorting tray, LLW, ILW, and recyclable trays. The full system provides a continuous sequential operation of sub-tasks, as shown in Figure 19. The examples provided in Figure 19 show the complete execution of the integrated system, from picking to dropping for various objects.

Multiple experiments are performed and the efficacy of the developed system and framework was observed through successful sorting and segregation of the objects used. The result showcases the successful and efficient segregation of objects into LLW, ILW, and recyclable trays. The database created contains the objects utilized, their properties and parameters Table 5) and a summary of each container (Table 6). The video (https://youtu.be/iIb8am8WkWs) presents a complete sort and segregation system demonstration for a pile of objects. The video shows that the designed system can successfully sort and segregate nuclear waste. It is also shown in the video that the process is completed by switching to teleoperation mode as a result of the manipulation error due to a huge object in the pile that cannot get into the scanning position.

7. Conclusion

Efficient and scalable robotic methodologies are required to overcome the entry barriers for autonomous systems in challenging environments like the nuclear industry. An autonomous robotic sort and segregation framework for nuclear waster handling was presented by introducing autonomy into Dexter^TM, a purely teleoperated nuclear robot. The framework relies on dynamical model parameter identification of Dexter^TM through Fourier series based excitation trajectories for autonomous operations and integration of methodologies for object identification, classification, characterisation and radiological surveying through ROS. For more complex grabs and tangled waste, the integrated system can call on the expertise of a skilled operator to take over the task through teleoperation. The effectiveness of this framework was verified by performing several sort and segregation operations safely on a mock-up nuclear waste test-bed using similar waste materials found in the nuclear industry. A new database of the objects packed into different bins and their properties is also generated and catalogued for future referencing. The results presented in this paper show a paradigm shift in enabling the adoption of autonomy in addition to automation in the regulated sectors.