Point Transformers

Prayash Phuyal

078bme031.prayash@pcampus.edu.np

Copyright © AnK Pvt. Ltd. All Rights Reserved.

1

10/16/2025

Why Transformers?

• They were created to overcome the limitations of Recurrent Neural Networks in processing sequential data.
• They handle long range data more effectively and enable parallel processing .
• A mechanism called self-attention is used to model the sequence of different words
• RNNs lose information over long sequence (vanishing gradient problem) . Transformer eliminates this problem as self attention mechanism connects every word to other word.

​

• Transformer works between tokenization and detokenization.
• Tokenization means converting words into numbers(token Ids). This is done by tokenizer using the tokenizer dictionnary.
• The token ids are converted to vector which is done in the embedding stage
• Token Embedding is a high dimensional space where each token is mapped to unique position called vector.
• Similar words are near each other in the high dimensional space.
• These embeddings are used to calculate the position of every token in the sequence of tokens. This is called positional embeddings.

​

​

• Then, the self attention layer adds weight to every word relative to every other word.
• If there are many self attention layers, then it is called multi headed self attention.
• In the feed forward network, it predicts the next word by assigning scores for every word. The prediction is called logit. The scores together form a vector of logits.
• A vector of logits is a list of scores that the model gives to each possible next word in a sentence.
• The scores are converted to probabilities in softmax output.
• Once the vector of probability is obtained, then it gets converted to token ids and into words. Also called detokenization.

Attention is all you need

• Transformer was first introduced in 2017 in the paper “Attention is All You Need” by a team of researchers at google dispensing recurrance and convolutions.
• It achieved 28.4 BLEU(Bilingual Evaluation Understudy) on the WMT(Workshop on Machine Translation) and after training for 3.5 days on 8 P100 GPUs, the score increased to 41.8
• The transformer solves the problem of sequential computation at the cost of reduced effective resolution due to averaging attention weighted positions.

​

​

• Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence aligned RNNs or convolution.
• The transformer follows the overall architecture using stacked self-attention and pointwise, fully connected layers for both encoder and decode.
• Encoder and Decoder stacks consist of N=6 identical layers and each layer has two sub layers.
• The output of each sublayer in encoder is LayerNorm(x + sublayer(x)), where sublayer(x) is the function implemented by sublayer itself.
• Output of dimension d_model = 512 was employed.

​

​

• An attention function can be described as mapping a set of query and a set of key-value pairs to an output.
• Two types of attention is used: Scaled-Dot product Attention and Multi-Head Attention.

• The above formula is used to calculate the self attention
• d_k refers to the queries and keys of dimension.
• Q, K and V are matrices.
• Softmax function is used on the overall dot product of query with all keys.
• Additive attendion is also commonly used which computes a compatibility function using feed-forward network with single hidden layer.

​

• The above formula is used to calculate multihead attention
• dv is the value of dimension.
• h is employed 8 parallel attention layers/heads
• d_v = d_k = d_model/h = 64
• The total computational cost is similar to that of the single-head attention with full dimensionality.

• In addition to sub layers, each layer contains fully connected to feed forward network which is applied to each position separately and identically.
• It consists of two linear transformation with ReLU actvation in between.
• Different parameters are used from layer to layer.
• dimensionality of input and output is d_model = 512 and inner layer has dimensionality d_ff = 2048

• To make use of the order of sequence, some information about relative position of tokens is injected.

​

​

​

​

​

​

• Sine and cosine of different frequencies are used. Above, pos is the position and i is the dimension.

https://arxiv.org/pdf/2012.09164

• Highly expressive point transformer layer for point cloud processing invariant of permutation and cardinality.

​

• Point Transformer networks for classification and dense prediction on point clouds
• 3D point clouds are processed using either projection based, voxel based or point based methods.
• In projection based networks, 3d point clouds are projected into various image planes. Then 2d CNN extract feature representations.
• In voxel based networks, quantization in the voxel grids is performed. Good accuracy but might lose geometric detail.

​

• Point based networks ingest point clouds directly as sets embedded in continuous space

​

• Different approaches connecting point set into graph are listed below:
• Dynamic Graph Convolutional Neural Network(DGCNN)
• Pointweb
• Edge-Conditioned Convolution(ECC)
• SuperPoint Graphs(SPG)
• Kernel Correlation Network(KCNet)
• Guided Aggregation Convolution Network( GACNet)
• Hierarchical Point Embedding and Information Network
• Deep Graph Convolution Network

​

• Methods based on continuous convolutions with no quantization:
• Permutation invariant Convolutional Neural Network
• SpiderCNN
• SphericalCNN
• Kernel Point Convolution (KPConv)
• PointCNN

​

• Transformers and self-attention networks can match or even outperform convolutional networks on sequences and 2D images.

• Applying self-attention locally enables scalability to large scenes with millions of points.
• Self attention operators can be classified into two types: scalar attention and vector attention.

​

​

• The standard scalar dot product attention layer is represented by the above equation. It computes the scalar product between the features transformed.

​

​

• The standard vector dot product attention layer is represented by the above equation. Attention weights are vectors that can modulate individual feature channels.

​

​

• Self-attention is a natural fit for point clouds because point clouds are essentially sets embedded irregularly in a metric space.
• Position Encoding allows the operator to adapt

to local structure of data.

• The encoding function θ is an MLP with two linear layers and one ReLU nonlinearity.
• PE is important for both the attention generation branch and the feature transformation branch.
• The transformer block integrates the self-attention layer, linear projections that can reduce dimensionality and accelerate processing, and a residual connection
• transition down for feature encoding and transition up for feature decoding

• Transition down model reduces the cardinality of the point set as required. eg: from N to N/4.
• Transition up maps features from downsampled input point set onto its superset.
• Three types of blocks are used by the network:
• Point Transformer Block
• Transition Block Down
• Transition Block Up

• x is the feature vector

of individual points.

• p is the 3d coordinate

of the corresponding

feature vector.

• y is the new feature

vector

• It has 2 linear layer and

1 PT layer.

• Linear layer reduces

dimension of feature to

boost computation speed.

• PT layer performs self

attention in local region.

• The PT layer is defined by the above function.
• There are two types of branches: Attention Generation Branch and Feature Transformation Branch.
• The result of two branch is combined using hadamard product.
• φ, ψ, α: Pointwise Feature Transformation Matrix (linear layers).
• ρ: this is a Sotmax Function used for normalization.
• X(i): the subset of the features around feature xi.
• xj: features inside the X(i) region.
• δ: the position embedding
• γ: an mlp with 2 linear layers and 1 ReLU Function.

Transition Block Down

• It reduces the feature numbers.
• It uses farthest point sampling

algorithm to reduce number of

points from p1 to p2.

• Uses KNN algorithm to group in

subset of points.

• Each layer goes through mlp

layer individually.

• Finally, max pooling within local

subset to obtain p2

Transition Up Block

• It is paired to Transition Down

Block in the encoder part.

• It upsamples features to p2.
• Each input feature goes through

a linear layer, Batch Normalization layer and a ReLU function.

• Interpolation is conducted to upsample features. (Trilinear interpolation is used)
• Add the interpolated result with the output of the paired TD block from the encoder.

DGCNN

• Dynamic Graph Convolutional Neural Network (DGCNN) is a neural network architecture designed for processing point clouds and 3D data by dynamically constructing and updating graphs in the feature space at each layer.
• Unlike CNNs which rely on fixed graphs, DGCNN adapts its graph structure during training to better capture both local and global geometric information.

Key Principles:

• The core principle of DGCNN is the edge convolution module.
• Some other principles are:
• Dynamic Graph Construction: At each layer, new k-nearest neighbor is calculated based on proximity of points.This allows points that are similar and distant to become neighbor in deeper space.
• Edge Conv : Edge conv operates on edges of dynamic graph.The edge features define the relationship between central point and its neighbors using using an asymmetric function.
• Aggregation : A symmetric aggregation function (e.g., max pooling) is applied to the edge features associated with each point to produce a new feature representation for that point.

Properties

• Permutation Invariance: Achieved using symmetric aggregation function. It means the output of model remains same even if the input is rearranged(permuted).
• Translation Invariance: Achieved using relative coordinates within EdgeConv module. It means moving all points in the input data by same fixed vector doesnot change the model’s fundamental output regarding object’s properties.
• State of the Art performance: DGCNN achieves high performance on ModelNet40 and S3DIS for object classification, part segmentation and semantic segmentation.

EdgeConv

• EdgeConv starts by dynamically constructing a k-nearest neighbor (k-NN) graph in the current feature space for each point.
• For each edge in the graph, a feature is computed using an asymmetric function that encorporates centre point’s feature and relative feature difference( xj - xi).
• A symmetric aggregation function, such as channel-wise max pooling, is applied to all edge features connected to a central point. This operation produces a single, fixed-size feature vector that robustly represents the local neighborhood of that point.
• Multiple EdgeConv layers are stacked, allowing the network to learn increasingly abstract and global features.
• In the final segmentation network, the concatenated features are processed

Thank You !!!