S. No. | Operator name | Standards |

1 | GatherElements | ONNX |

2 | ScatterElements | ONNX |

3 | GatherND | ONNX, TensorFlow, mxnet |

4 | ScatterND | ONNX, TensorFlow, mxnet |

5 | Gather | ONNX, TensorFlow Caffe2 |

6 | Take | Mxnet, Numpy |

#### GatherElements

**ONNX v11 API:**

*Tensor*Output = GatherElements(

*Tensor*Data,

*Tensor*Index,

*int*axis = 0) This is an ONNX specific operator that gathers individual elements along the specified axis of a given tensor based on the index values. It forms an inverse operator pair with ScatterElements. The

*axis*input is optional and has a default value of 0 (outermost or the slowest changing dimension). Here, output tensor has the same shape as the index tensor and the output at each location is determined by the following way: out[i][j][k] = input[index[i][j][k]][j][k] if axis = 0, out[i][j][k] = input[i][index[i][j][k]][k] if axis = 1, out[i][j][k] = input[i][j][index[i][j][k]] if axis = 2 The data is gathered from the input at the same location as the output coordinate, but the axis coordinate is replaced with the corresponding index value. This is further illustrated with some examples below. The colors indicate the correspondence between the index value and the data value.

**Example 1:**One exciting feature of this operator is that it accepts negative values for axis and index values. Axis has an accepted range of [-r, r-1], where r indicates the input tensor’s rank (or dimensionality), and the negative value simply means wrapping around the tensor dimensions. For example, axis = -1 indicates the innermost dimension, i.e., (r-1)th dim of the tensor. Similarly, the index value can range between [-s, s-1], where s is the size of the Data tensor along the axis, and the negative values simply wrap around the axis, as illustrated in the example below.

**Example 2:**

#### ScatterElements

**ONNX v11 API**:

*Tensor*Output = ScatterElements(

*Tensor*Data,

*Tensor*Indices,

*Tensor*Updates,

*int*axis = 0) As mentioned before, this operator is the inverse of the GatherElements operator. ScatterElements takes three inputs: data, indices, and updates. It also takes an optional input axis with default value of 0 (outermost or the slowest changing dimension). To obtain the output, firstly a copy of the data is created so that both output and input tensors have the same shape. Then values specified by

*updates*are updated at index positions specified by indices. There should be an index specified for each update value, and thus these tensors have the same shape. For instance, output[indices[i][j][k]][j][k] = updates[i][j][k] if axis = 0, output[i][indices[i][j][k]][k] = updates[i][j][k] if axis = 1, output[i][j][indices[i][j][k]] = updates[i][j][k] if axis = 2, Thus, the given updates are scattered to the given locations of the output tensor and the remaining locations contain the original input. Here, there is a one-to-one mapping between indices tensor and updates tensor and this is indicated with the colors in the example below.

**Example:**

#### GatherND

GatherND is a generalized version of the GatherElements operator described earlier. This operator gathers slices of data from the specified indices and stores to output. This operator is defined in various frameworks including ONNX, TensorFlow and MxNet.**ONNX v11 API**:

*Tensor*Output = GatherND(

*Tensor*Data,

*Tensor*Indices) Unlike the previous GatherElements operator, each tensor’s rank and shape could be different, and is as given below.

**Data:**Tensor of rank r >= 1**Indices:**Tensor of rank q >= 1**Output:**Tensor of rank, o = q + r - m - 1; m = indices_shape[-1]

- If r - m = 0 - Element is gathered at each index tuple
- If r - m = 1 - Lines are gathered at each index tuple
- If r - m = 2 - Sheets are gathered at each index tuple
- If r - m = 3 - Cuboids are gathered at each index tuple

**Example 1:**

- data (3D): [[[1,2],[3,4]],[[5,6],[7,8]]]
- Indices (3D): [[[0,0]],[[1,0]]]
- output (3D) = [[[1,2]],[[5, 6]]]

**Example 2:**

- data (2D): [[1,2],[3,4]]
- Indices (2D): [[-2,0]],[[1,1]]
- output (1D) = [1, 4]

**TensorFlow 2.1.0 API:**tf.gather_nd(params, indices, batch_dims=0, name=None) TensorFlow gather_nd works the same way as ONNX gather_nd. Here,

*params*is similar to the

*data*tensor and the operator indexes into the first (r-m) dimensions of the params tensor of rank r, based on each m-dimensional index tuple in indices tensor. One crucial difference between TensorFlow 2.1.0 API and ONNX v11 API is the argument

*batch_dims*, which specifies the leading number of outermost dimensions of params (data) and indices tensor. Batch_dims = b means the gather operation starts from (b+1)th dim of the data, and thus the output dimensions equations should be adjusted as: o = q + r - m - b - 1, since we skip over the outermost b dimensions. (

**Note**: batch_dims is supported in ONNX v12 Operator Set onwards) There are other minor differences also, including that negative indices are not allowed. In case of out of bound indices on CPU, an error is returned, and on GPU, a 0 is stored in the corresponding output value.

**MxNet v1.6 API:**mxnet.ndarray.gather_nd(data=None, indices=None, out=None, name=None, **kwargs) MxNet version is similar but differs from ONNX and TensorFlow as the index coordinates are picked along the outermost dimension, also called the slowest changing dimension(SCD). If we consider example 2 above, index tensor is accessed along SCD to obtain coordinates. Hence, we get (0,1) and (0,1) as coordinates instead of (0,0) and (1,1). So, output becomes [data(0,1), data(0,1)] that is [2,2]. Like TensorFlow, negative indices are not supported in Mxnet as well. One point of interest is that in all three frameworks, the first element of a tuple always refers to the outermost dimension and then moves inwards. The differences of GatherND operator across various frameworks are captured in Table 1.

#### ScatterND

This operator is a generalized version of Scatter Elements operator and it updates given slices of data into the output at specified indices. This operator is present in mxnet, ONNX and TensorFlow.**ONNX V11 API:**Tensor output = scatter_nd(

*Tensor*Data,

*Tensor*Indices,

*Tensor*Updates) The output of scatter ND is obtained by creating a copy of

*data*and updating the values from

*updates*at output locations specified by

*indices*. Similar to gather_nd, the tensor dimensions are as below:

**Data:**Tensor of rank r >= 1,**Indices:**Tensor of rank q >=1**Updates:**Tensor of rank q + r - m - 1; m = indices shape [-1]**Output:**Tensor of rank r, with the same shape as data tensor

**Scatter_nd element (r-m = 0)**

**Scatter_nd sheet (r - m = 2)**A numerical example is given below to illustrate the operation further.

**Example:**The index order and data layout are the same as in Gather ND. Also, indices must not contain duplicate entries, that is having more than one update for the same location is not allowed.

**TensorFlow 2.1.0 API:**tf.scatter_nd(indices, updates, shape, name=None) In TensorFlow scatter. nd, the updates are scattered to output locations specified by indices without making a copy of data. Hence there is no data argument here. The locations that are not specified by indices are filled with zeroes. Also, unlike ONNX Scatter ND, indices are allowed to contain duplicates and if indices contain duplicates, then their updates are accumulated (summed), as illustrated below.

**Example 1 (element):**Updates (2x2x2) Indices (2x2x2x2) Output (3x2)

**Example 2 (sheet):**Updates (2x2x2x2) Indices (1x2x2) Output (2x2x4)

**MxNet v1.6 API**: scatter_nd(data, indices, shape) MxNet scatter. ND works similarly as TensorFlow scatter_nd but the coordinates are picked along outermost dimension (SCD) instead of the innermost dimension (FCD). Also, MxNet does not support duplicate indices. The output will be non-deterministic if the indices contain duplicate entries because updates will be overwritten. The following table compares GatherND and ScatterND across various frameworks.

| Gather ND | Scatter ND | ||||

| ONNX | TF | MxNet | ONNX | TF | MxNet |

Negative indices | Supported | Not Supported | Not Supported | Supported | Not supported | Not supported |

Duplicate indices | Allowed | Allowed | Allowed | Error | Accumulate | Non-deterministic result |

Locations not covered by indices | NA | NA | NA | Same as data tensor | Zero | Zero |

Out of bound indices | Error | Error/Zero | Error | Error | Zero | Error |

Table 1: Comparison of Gather ND/ Scatter ND across frameworks

#### Gather/Take

The Gather operator gathers/takes entries along axis based on index locations. This operator is present in ONNX, Caffe2, TensorFlow and is very much similar to the Take operator in Numpy and mxnet.**ONNX v11 API:**

*Tensor*output = Gather (Tensor Data, Tensor Indices,

*int*axis =0) This operator can also gather various slices from the data, but unlike GatherND, the indices are applied along the specified axis only. Here too, the output is a collection of such slices and the output rank is given by o = r + q - 1, where r is the rank of input and q is the rank of indices. In general, the output is calculated as below: Let k = indices[i_{0}, ..., i_{q-1}]

- output[i_{0}, ..., i_{q-1}, j_{0}, ..., j_{r-2}] = input[
**k**, j_{0}, ..., j_{r-2}] if axis = 0, - output[i_{0}, ..., i_{q-1}, j_{0}, ..., j_{r-2}] = input[ j_{0},
**k****,**..., j_{r-2}] if axis = 1, - output[i_{0}, ..., i_{q-1}, j_{0}, ..., j_{r-2}] = input[ j_{0}, j_{1},
**k****,**..., j_{r-2}] if axis = 2 and so on.

**TensorFlow v2.1.0 API:**tf.gather(params, indices, validate_indices=None, name=None, axis=None, batch_dims=0) The TensorFlow version is similar to the ONNX version but with certain significant differences.

- The axis input is a tensor and not a scalar parameter. Thus, it is possible to apply different axis values within a single Gather operation.
- Like the TensorFlow GatherND, this operator supports a parameter called batch_dims, which specifies the leading number of outermost dimensions (of data and indices). The output tensor shape in this case is given by: output_shape = params_shape[:axis] + indices_shape[batch_dims:] + params_shape[axis + 1:]

**Caffe2 API**:

*Tensor*output = Gather (

*Tensor*data,

*Tensor*indices) The caffe2 version is similar to the ONNX version, but unlike the ONNX version, it does not allow gathering along an arbitrary axis. Here the gather operation is always along axis = 0, i.e., the outermost dimension.

**Numpy API**: numpy.take(a, indices, axis=None, out=None, mode='raise')[source]¶ As mentioned before, the numpy take operator is functionally similar to the ONNX gather operator. It can be thought of as a superset of the ONNX gather, as it has additional features. The most notable feature is that it has mode input which allows controlling the behavior of the operator when indices are either negative or out of bound. The supported modes are:

- ‘raise’ – raise an error (default)
- ‘wrap’ – wrap around
- ‘clip’ – clip to the range

**Mxnet v1.6 API**: mxnet.ndarray.take (a=None, indices=None, axis=_Null, mode=_Null, out=None, name=None, **kwargs) The mxnet take operator is similar to the numpy take operator mentioned above and also supports various modes in case of out of bounds indices. The table below highlights the differences and similarities in Gather operator between different standards.

| ONNX Gather | TF Gather | Caffe2 Gather | Numpy Take | Mxnet Take |

If axis is not specified | Defaults to the outermost axis | Defaults to the first non-batch dimension | NA, axis is always outermost axis | Input is flattened | Defaults to the outermost axis |

Negative axis | Supported | Supported | NA | Supported | supported |

Negative indices | Supported (always wrap around) | Not Supported | Not Supported | Supported with different modes (raise, wrap, clip) | Supported with different modes (raise, wrap, clip) |

Out of bound indices | Error | Zero in GPU/Error in CPU | Error | Supported with different modes (raise, wrap, clip) | Supported with different modes (raise, wrap, clip) |

Table 2: Comparison of Gather/Take across frameworks

#### Conclusion

Gather-Scatter operators are used in deep learning applications for various indexing operations. The backpropagation along a gather layer is implemented using the corresponding scatter operator and vice-versa. We have described several of these Gather/Scatter operators in detail here. As can be seen, there are several similarities among these operators across various frameworks. For example, an index tuple of {a, b, c} means coordinate a is along the outermost dimension, b along the second outermost dimension, and c along the innermost dimension, for all the operators and frameworks. Similarly, axis 0 refers to the outermost dimension in all frameworks. However, there are considerable functional and API level differences among these operators and frameworks. We summarized the functionalities of these operators with visual examples and compared and contrasted them across various frameworks. We hope this helps other deep learning programmers and researchers to understand better and use these operators.#### References

- https://github.com/onnx/onnx/blob/master/docs/Operators.md#GatherElements
- https://github.com/onnx/onnx/blob/master/docs/Operators.md#ScatterElements
- https://github.com/onnx/onnx/blob/master/docs/Operators.md#GatherND
- https://www.tensorflow.org/api_docs/python/tf/gather_nd
- http://beta.mxnet.io/r/api/mx.nd.gather.nd.html

**Further Reading**