2.1

Image representation

Our method is based on the image representation of the sensor sequence³. Wang et al.⁴ encoded the joint trajectory image into an image through three different spatial perspectives. Liu et al.⁵ proposed a combination of bone visualization methods and jointly trained these methods on multiple streams.

2.2

Few-shot classification

A matching network¹ created a new type of method which can use attention algorithm to calculate the classifier of the nearest neighbor that can perform calculus and a prototypical network⁶ enlarges it by making the definition of prototypes as the average value of embedded support examples for each class. Meta-LSTM⁷ updates the model by using LSTM, making its model’s parameters to be the hidden states. MAML⁸ only concentrates on the initial parameters’ initial values and simply uses SGD. Reptile⁹ uses only first-order gradients.

2.3

Graph neural network

Gori et al.¹⁰ firstly proposed the graph neural network, which acts like a trainable cyclic message transfer whose fixed points can adjust differently. Li et al.¹¹ further proposed a method using gated loop units and advanced optimization techniques. Kipf and Welling¹² groundbreakingly applied semi-supervised learning of graph structure data, which is scalable. They^{2, 13} have applied the GNNs to few-shot learning and labeled the framework on the node level.

3.1

Problem definition

We consider the one-shot action recognition problem as a few-shot metric learning problem. First, we encode the sequence of actions. Convert the signal level to an image representation. The input in our example is a signal matrix S ∈ ℝ^N×M where each row vector represents a discrete 1-dimensional signal and each column vector represents a sample of all sensors at one specific time step. The matrix is transformed to an RGB image I∈{0,…,255}^H×W×3 by normalizing the signal length M to W and the range of the signals to H. The identity of each signal is encoded in the color channel. So our dataset consist of of N training images I_1,…,N with labels y_i ∈ {1,…,C}. We consider input-output pairs (T_iY_i)_i drawn from a distribution P of partially-labeled image collections

and Y = (y₁,…,y_t)∈{1,K}^t for arbitrary values of s,r,t and K. s is the number of labeled samples, r is the number of unlabeled samples and t is the number of samples to classification. K is the number of classes. 𝒫_l (ℝ^N) denotes a class-specific image distribution over ℝ^N. Given a training set {(T_i,Y_i)_i}_i≤L, we consider the standard supervised learning objective , using the model specified in Section 5 and is 𝒱 a standard regularization objective.

When r = 0, t = 1 and s = qK, there is a single image in the collection with unknown label. In addition, if each label occurs exactly q times, this setting is called q -shot, K -way learning. And in our work, we only focus on one-shot problem.

3.2

Representation

Our method is based on discernible image representation. Therefore, we propose a novel and compact signal level representation. Multivariable signals or higher-level feature sequences are recombined into a 3-channel image. Each row of the result image corresponds to one joint, and each channel corresponds to one sample in the sequence. The color channels, red, green and blue, represent respectively the signals’ x-, y- and z-values. The resulting images are normalized to the range of 0-1.

3.3

Feature extraction

The feature extraction module of our model mainly used a Resnet network-based structure. We consider that after the joint training is implemented, the feature extraction module can be strongly combined with few-shot classifier module. We used Resnet18 and Resw (the network optimized by our approach) for feature extraction.

On one hand, we apply a 1×1 convolution layer before the 2 layers of 3×3 convolution in every main stage of Resnet18, i.e., the residual block of Resnet. With this optimization, the nonlinearity of the model can be increased before each residual calculation module in the network is down-sampled, and the results show that the effectiveness of feature extraction is improved. On the other hand, we also postponed the down-sampling of the residual module by adjusting the stride of the two-layer convolution.

After the last feature layer, we use a two-layer perceptron to transform the features into the embedding size. The embedding is refined by the metric learning approach.

3.4

Metric network

The goal of few-shot learning is to propagate label information from labeled samples to unlabeled query images. This propagation of information can be formalized as a posterior inference over a graphical model determined by the input images and labels. The input 𝒯 contains a collection of labeled and unlabeled images. This information dissemination can be formalized as a posteriori reasoning of the graphical model determined by the input image and the label. We associate 𝒯 with a fully-connected graph G_𝒯 = (V,E) where nodes v_a ∈ V correspond to the images present in 𝒯 (both labeled and unlabeled).

In the graph neural network part, given an input signal F ∈ℝ ^V×d on the vertices of a weighted graph G. We apply a family M of graph intrinsic linear operators that act locally on this signal. And the adjacency operator where (AF)_i :=∑_j~i w_i,jF_j with i~j if (i,j) ∈ E and w_i,j is associated weight. A GNN layer Gc(·) receives as input a signal and produces as

where , are trainable parameters and ρ(∙) is a point-wise non-linearity. Based on this basic formula, we did some exploration. Particularly, inspired by message-passing algorithms, we consider a Multilayer Perceptron stacked after the absolute difference between two vector nodes.

where ψ is a symmetric function parametrized with a neural network, etc. And ψ is a metric, which is learned by nonlinear combination of the absolute difference between the individual features of the two nodes. Then, we normalize the trainable adjacency to a random kernel by using a softmax along each row. By adding the edge feature kernel A^(k)into the generator family M. and applying (2), the update rule of the node feature is obtained.

Adjacency learning is especially important in the following applications: the input set is considered to have a certain geometric structure, but the metric is unknown a priori, as in our case. In a general graph, the depth of the network is selected in the order of the diameter of the graph, so that all nodes obtain information from the entire graph. However, in our context, since the graphics are closely connected, depth is simply interpreted as giving the model greater expressive power.

3.5

Composition of initial node features

The input set 𝒯 is mapped into node features, as shown below. For signal images x_i∈𝒯 with known label l_i, the one-hot encoding of the label is concatenated with the embedding features of the image at the input of the GNN.

where φ is the Resnet embedding network and is a one-hot encoding of the label.

5.1

Datasets

NTU RGB+D 120: The NTU RGB+D 120¹⁴ dataset is a wide-ranging action recognition dataset involving RGB+D mode data and skeleton sequences data. The dataset consists of 114,480 sequences containing 120 action classes from 106 subjects in 155 different views. This article uses the one-shot standard given by the author of the dataset. The categories of these standards are divided into two parts: training set and test set. The categories contained in the training set and the test set do not overlap. There are 100 categories for training and 20 categories for testing. A1, A7, A13, A19, A25, A31, A37, A43, A49, A55, A61, A67, A73, A79, A85, A91, A97, A103, A109, A115 are previously unseen.

b) UTD-MHAD: The UTD-MHAD¹⁵ is a dataset containing 27 actions of 8 individuals performing 4 repetitions each. RGB-D image source data, skeleton sequences and inertial data are involved. The RGB-D camera is placed frontal to the demonstrating person. This article used 23 classes for training and other classes for testing. Besides the 23/4 training/testing setting, we also testing the setting of 19/8 and 15/12 training/testing setting. Other works even applied the setting of 11/16 and 7/20, we believe our three setting is Forcefully and credibly.

5.2

Results

On the NTU RGB+D 120 one-shot dataset we compare against APSR¹⁴ and SL-DML³. Table 1 shows the results with a training set size of 100 action classes and a test set size of previously unseen 20 action classes. Our method (using our own feature extraction backbone Resw) is 1.2% higher than the best method (SL-DML) before. Table 2 shows results for an increasing amount of training classes (100 training classes and 20 test classes are considered as the standard protocol). And Table 2 shows the results of using different feature extraction networks to affect the classification effect. We use Resnet18 and our own feature extraction network Resw optimized based on Resnet. It can be seen that using our backbone, we can achieve better results than regular Resnet18. All our results are presented in percentage. The best results are bolded. By the results, we can see that our method performs better when the train classes is small, our method has a stronger robustness.

Table 1.

One-shot results on NTU-RGB+D 120 one-shot.

Table 2.

One-shot results with different train classes on NTU-RGB+D 120 one shot.

On the UTD-MHAD dataset, we conducted experiments based on two criteria. And we compare our results with the SL-DML on different training testing ratio, the results are shown in Table 3. On UTD-MHAD dataset we only use our backbone Resw, and all the results below are based on it.

Table 3.

One-shot results with different train classes on UTD-MHAD.