ADAPTIVE SCENE SEGEMENTATION USING RCE-NEURAL NETWORK

NTU-INDIA CONNECT

PROJECT GUIDE

Dr. Xie Ming

​

PRESENTED BY

Sri Siddarth Chakaravarthy P

(N2101776C)

​

PROBLEM STATEMENT

Problem Definition

• Colour scene image segmentation is a crucial task for autonomous vehicles for region classification. This can be used to detect different class of objects in a vehicle’s vicinity. In the past few decades researchers have introduced several colour image segmentation techniques for lane detection, ego vehicle detection, object avoidance, etc. In autonomous vehicles, these techniques have drawbacks when it comes to real-world applications and implementations these techniques do not achieve the optimal accuracy in a dynamic outdoor environment.

​

• Hence there is a need to solve the problem of colour classification in which colour classes are represented by both disjoint class distributions and non-separable classes whose distributions overlap in colour space.

​

• Autonomous vehicles have to be able to discern the differences between objects that have overlapping colour boundaries for example, speed breakers that do not have white stripes and have been submerged with the road, the vehicle needs to be able to perceive the region so that it could adjust its path plan accordingly.

​

• Restricted Coulomb Energy (RCE) neural network is one of the competitive learning networks that is able to classify input data, together with the Self-Organizing Map (SOM) and the Learning Vector Quantization(LVQ). In the RCE network, there is no need for setting the number of required neurons before learning because the RCE network dynamically creates new neurons to classify input data into suitable classes.

​

• This would allow autonomous vehicles to operate with adaptive learning, thereby learning from experiences, this would enable autonomous vehicles to be more flexible and optimal for real-world scene segmentation.

​

• In this project we aim to research the benefits and drawbacks of RCE Neural Network, we also aim to develop a model for adaptive scene segmentation using the RCE Neural Network architecture.

OBJECTIVE

The objective of this Research Project

• In this research project work we would be working on developing a RCE neural network model for scene segmentation with experience learning that can enable autonomous vehicles with adaptive learning.

​

• In this project we would like to completely explore RCE Neural Network and understand the network’s architecture and how it is different from other existing models. We also want to learn its benefits and drawbacks.

​

• As a practical implementation of the project, we would also like to explore the potential use of RCE-Neural Network for semantic segmentation of traffic environment by training the model on a public dataset (CARLA simulator) and testing it on real-time images.

​

• Autonomous vehicles are complex systems that rely on sensors for perception. The data that is captured from these sensors, is then used to understand the vehicle’s vicinity and aid in detection of objects ahead and distinguish between such detected objects.

​

• Cognitive Systems play a crucial role in autonomous vehicles, such systems are needed to perform several important functionalities such as lane detection, path planning, object avoidance and etc. The most important task is for these systems to be able to distinguish between objects ahead.

​

• This is where colour image segmentation comes into play, in the past few decades several colour image segmentation techniques have been discovered such as, edge detection, multiple thresholds, region growing, space clustering, etc.

​

• However, these approaches have not been effective for the problem of autonomous vehicles where we have to also consider the influence of colour fluctuation and additional effects by variable illuminations on colour objects. Hence, there is a need for novel image segmentation model that can overcome these problems and introduce dynamic instance segmentation that will aid towards object classification and detection in autonomous vehicles. 

INTRODUCTION

What is RCE-Neural Network, How is it different?

• The RCE neural network is a supervised learning scheme for estimating and classifying feature regions, which has three layers: the input layer, the prototype layer, and output layer. For the neural network architecture, there are full connections between each layer.

​

• Restricted Coulomb Energy(RCE) network is one of the competitive learning networks that can classify input data, together with the Self-Organizing Map (SOM) and the Learning Vector Quantization(LVQ).

​

• In the RCE network, there is no need for setting the number of required neurons before learning because the RCE network makes new neurons automatically to classify input data into correct categories.

​

• Restricted Coulomb energy (RCE) neural network can be used to remove image noise by creating clusters training patterns.

INTRODUCTION

What is RCE-Neural Network, How is it different?

• RCE neural network is a supervised pattern classifier used for the judgment of feature area. It provides a way of region modification that is middle among Parzen-window and K-nearest-neighbour. Parzen window uses fixed window size where KNN uses variable window size. Throughout the network training, the size of the hyper spherical window is in sync in orientation to the adjacent point of a diverse class in feature space.

​

• The most important advantage resides in the simplicity of the algorithm and its ability to modify the number of units. The decision regions are built with respect to a threshold associated to each decision units. The algorithm has only to decide whether a new point belongs to an existing class and, in this case, if it is correctly classified. Then the decision of creation of a new decision unit is taken according to the result of this comparison. A new decision unit defines a new hypersphere whose centre coordinates are chosen as the actual input prototype.

​

• The feature region of each class is enclosed by generated hyper spherical prototypes. It is not desirable to straight use RCE neural network for the supervised segmentation. The major disadvantage of RCE learning is the necessity of a whole sample set for every classes.

Output Layer

(Symbol Grounding)

Prototype Layer (Unsupervised Classification)

Input Layer

(Feature Vector)

RCE NEURAL NETWORK

Network Architecture

• The architecture of the RCE network contains two layers: A hidden layer and an output layer.

​

• The hidden layer is fully interconnected to all components of an input pattern (vector) x  Rn.

​

• The output layer consists of L units. The output layer is sparsely connected to the hidden layer; each hidden unit projects its output to one and only one output unit.

​

• The architecture of the RCE net is shown in Figure. Each unit in the output layer corresponds to a pattern category.

​

• The network assigns an input pattern to a category l if the output cell yl is activated in response to the input.

​

• The decision of the network is "unambiguous" if one and only one output unit is active upon the presentation of an input, otherwise, the decision is "ambiguous."

RCE NEURAL NETWORK

Network Architecture

The transfer characteristics of the jth hidden unit is given by�

�

where j  Rn is a parameter vector called "center", rj  R is a threshold or "radius," and D is some predefined distance metric between vectors j and x (e.g., Euclidean distance between real-valued vectors or Hamming distance between binary-valued vectors). Here, f is the threshold activation function given by

On the other hand, the transfer function of a unit in the output layer is the logical OR function. The jth hidden unit in the RCE net is associated with a hyperspherical region of the input space which defines the unit's region of influence. The location of this region is defined by the center j and its size is determined by the radius rj. According to Equation, any input pattern falling within the influence region of a hidden unit will cause this unit to fire. Thus, the hidden units define a collection of hyperspheres in the space of input patterns. Some of these hyperspheres may overlap. When a pattern falls within the regions of influence of several hidden units, they will all "fire" and switch on the output units they are connected to.

RCE NEURAL NETWORK

Neural Network Training

• Training the RCE net involves two mechanisms: Unit commitment and modification of hidden unit radii. Units may be committed to the hidden and output layers. When committed, units are interconnected so that they do not violate the RCE interconnectivity pattern described above.

​

• Initially, the network starts with no units. An arbitrary sample pattern x1 is selected from the training set and one hidden unit and one output unit are allocated. The allocated hidden unit center 1 is set equal to x1 and its radius r1 is set equal to a user-defined parameter rmax (rmax is the maximum size of the region of influence ever assigned to a hidden unit). This unit is made fully interconnected to the input pattern and projects its output z1 to the allocated output unit (OR gate). This output unit represents the category of the input x1. Next, we choose a second arbitrary example x2 and feed it to the current network. Here, one of three scenarios emerges. First, if x2 causes the output unit to fire and if x2 belongs to the category represented by this unit, then do nothing and continue training with a new input. In general, this scenario might occur at a point during training where the network has multiple hidden and output units representing various categories. In this case, if the input pattern causes only the output unit representing the correct category to fire, then do nothing and continue the training session with a new input. On the other hand, the correct output unit may fire along with one or more other output units. This indicates that the regions of influence of hidden units representing various categories overlap and that the present input pattern lies inside the overlap region. Here, proceed by reducing the threshold values (radii) of all active hidden units that are associated with categories other than the correct one until they become inactive.

RCE NEURAL NETWORK

Neural Network Training

• The second scenario involves the case when the input x2 happens to belong to the same category as x1 but does not cause the output unit to fire. Here, allocate a new hidden unit with center at 2 = x2 and radius rmax and connect the output z2 of this unit to the output unit. The general version of this scenario occurs when the network has multiple output units. Now, if the input pattern causes no output units (including the one representing the category of the input) to fire, then allocate a new hidden unit centered at the current input vector/pattern and assign it a radius r = min(rmax, dmin), where dmin is the distance from this new center to the nearest center of a hidden unit representing any category different from that of the current input pattern. The new allocated unit is connected to the output unit representing the category of the input pattern.

​

Note: that setting of r may cause one or more output units representing the wrong category to fire. This should not be a problem since the shrinking of the region of influence mechanism described under the first scenario will ultimately rectify the situation. If, under this scenario, some hidden units representing the wrong category fire, then the radii of such units are shrunk as described earlier under the first scenario.�

• Finally, the third scenario represents the case of an input with a new category that is not represented by the network. Here, as in the first step of the training procedure, a hidden unit centred at this input is allocated and its radius is set as in the second scenario. Also, a new output unit representing the new category is added which receives an input from the newly allocated hidden unit. Again, if existing hidden units become active under this scenario, then their radii are shrunk until they become inactive. The training phase continues (by cycling through the training set or by updating in response to a stream of examples) until no new units are allocated and the size of the regions of influence of all hidden units converges.

​

RCE NEURAL NETWORK

Exploring more about RCE Neural Network

• The RCE net is capable of developing proper separating boundaries for nonlinearly separable problems. The reader is referred to Figure 6.3.2 for a schematic representation of separating boundaries realized by the regions of influence for a nonlinearly separable two-class problem in two-dimensional pattern space.

​

• The RCE net can also handle the case where a single category is contained in several disjoint regions. In principle, an arbitrary degree of accuracy in the separating boundaries can be achieved if no restriction is placed on the size of the training set. Dynamic category learning is also possible with the RCE network.

​

• That is, new classes of patterns can be introduced at arbitrary points in training without always involving the need to retrain the network on all of its previously trained data. Note that, in its present form, the RCE network is not suitable for handling overlapping class regions. Here, the learning algorithm will tend to drive the radii of all hidden unit regions of influence to zero. It also leads to allocating a large number of hidden units approximately equal to the number of training examples coming from the regions of overlap.

​

• Several variations of the RCE network are possible. For example, one might employ mechanisms that allow the centers of the hyperspheres to drift to more optimal locations in the input space. A second variation would be to allow the hyperspheres to grow. These two mechanisms have been considered for more general hyperspherical classifiers than RCE classifiers (Batchelor, 1974). Modifications to the RCE network for handling overlapping class regions can be found in Reilly and Cooper (1990). Empirical examination of RCE classifiers appeared in Lee and Lippmann (1990) and Hudak (1992).

​

LITERATURE REVIEW

Survey of some existing research work in this domain

• The RCE algorithm is a well-known method used for classification task. It is based on incremental modification of a neural network structure. RCE neural network identified as a supervised pattern classifier used on behalf of the estimation of feature region.

​

• The RCE is one of the first incremental models of neural networks. In this model decision unit are characterized by their influence region, defined by a hypersphere around the unit, whose radius is equal to the threshold of the unit. The state space is then divided into zones, each dominated by different decision units. New units are created with an initial chosen radius if a presented template does not fit into one of the influence regions of the units associated with the correct class. On the other hand, radii associated to units' belongings to a wrong class but whose influence region include the presented pattern are lowered to avoid this situation.

​

• The most important advantage resides in the simplicity of the algorithm and its ability to modify the number of units. The decision regions are built with respect to a threshold associated to each decision units. The algorithm has only to decide whether a new point belongs to an existing class and, in this case, if it is correctly classified. Then the decision of creation of a new decision unit is taken according to the result of this comparison. A new decision unit defines a new hypersphere whose centre coordinates are chosen as the actual input prototype.

​

• The RCE neural network is a supervised learning scheme for estimating and classifying feature regions, which has three layers: the input layer, the prototype layer, and output layer. For the neural network architecture, there are full connections between each layer.

​

LITERATURE REVIEW

Some interesting findings

• A RCE Neural Network colour segmentation model for hand image segmentation in the gesture-based human-service robot interaction system. (work by Prof. Xie Ming). The study on skin colour distributions in different colour spaces indicates that skin colour cluster in a small region in a colour space. The RCE neural network characterizes the skin colour distribution region using skin colour prototypes together with their spherical influence fields during training stage, and identifies the skin regions in the colour image during running stage.

​

• A hands segmentation scheme based on human skin colour classification using the Restricted Coulomb Energy (RCE) neural network. An improved iteration strategy for the RCE neural network, with a reduction in the number of repetitive calculations, is utilized in our work. The experimental results show that our system is more accurate and less computational expensive than previous schemes.

​

• An approach that attempts to improve conventional quality function deployment (QFD) technique in terms of effective design knowledge handling in product concept development is proposed and described. For this purpose, a QFD-enabled product conceptualisation system was established. It consists of three cohesively interacting modules, namely, design knowledge elicitation module using laddering technique, design knowledge representation module using design knowledge hierarchy (DKH), and design knowledge organisation module using restricted Coulomb energy (RCE) neural network.

​

• A supervised fuzzy neural network called a fuzzy restricted Coulomb energy (fuzzy RCE) network for classification problems. In a fuzzy RCE, each hidden neuron is a fuzzy prototype which can be used to represent one or many training patterns. At the learning stage, the fuzzy membership functions of prototype neurons can be automatically adjusted according to the training data. 

​

​

LITERATURE REVIEW

Some interesting findings

• A restricted coulomb energy neural network (RCE-NN) with an improved learning algorithm and presents the hardware architecture design and VLSI implementation results. The learning algorithm of the existing RCE-NN applies an inefficient radius adjustment, such as learning all neurons at the same radius or reducing the radius excessively in the learning process. Moreover, since the reliability of eliminating unnecessary neurons is estimated without considering the activation region of each neuron, it is inaccurate and leaves unnecessary neurons extant.

​

• To overcome this problem, the proposed learning algorithm divides each neuron region in the learning process and measures the reliability with different factors for each region. In addition, it applies a process of gradual radius reduction by a pre-defined reduction rate. In performance evaluations using two datasets, RCE-NN with the proposed learning algorithm showed high recognition accuracy with fewer neurons compared to existing RCE-NNs. The proposed RCE-NN processor was implemented with 197.8K logic gates in 0.535 mm 2 using a 55 nm CMOS process and operated at the clock frequency of 150 MHz.

​

• A novel hybrid neural network structure for the classification of the electrocardiogram (ECG) beats. Two feature extraction methods: Fourier and wavelet analyses for ECG beat classification are comparatively investigated in eight-dimensional feature space. ECG features are determined by dynamic programming according to the divergence value. Classification performance, training time and the number of nodes of the multi-layer perceptron (MLP), restricted Coulomb energy (RCE) and a novel hybrid neural network are comparatively presented. In order to increase the classification performance and to decrease the number of nodes, the novel hybrid structure is trained by the genetic algorithms (GAs). 

​

FEATURE VECTOR OF COLOUR REGIONS

Feature Vector? What are the feature vector components for our colour regions?

• In image segmentation, a feature is an individual measurable property or characteristic of a phenomenon. Choosing informative, discriminating and independent features is a crucial element of effective algorithms in pattern recognition, classification and regression. Features are usually numeric, but structural features such as strings and graphs are used in syntactic pattern recognition. The concept of "feature" is related to that of explanatory variable used in statistical techniques such as linear regression.

​

• The features may represent one mere pixel or an entire image. The granularity depends on what someone is trying to learn or represent about the object. You could describe a 3-dimensional shape with a feature vector indicating its height, width, depth, etc. Here we have taken the image colour and number of vertices.

​

​

No. of Vertices

Shades of colour

RESULTS

Cognition Recognition

SUPERVISIED TRAINING

• Supervised learning is the types of machine learning in which machines are trained using well "labelled" training data, and on basis of that data, machines predict the output. The labelled data means some input data is already tagged with the correct output.

​

• In supervised learning, the training data provided to the machines work as the supervisor that teaches the machines to predict the output correctly. It applies the same concept as a student learns in the supervision of the teacher.

​

• Supervised learning is a process of providing input data as well as correct output data to the machine learning model. The aim of a supervised learning algorithm is to find a mapping function to map the input variable(x) with the output variable(y).

​

• In the real-world, supervised learning can be used for Risk Assessment, Image classification, Fraud Detection, spam filtering, etc.

​

UNSUPERVISIED CLASSIFICATION

• Unsupervised classification is where the outcomes (groupings of pixels with common characteristics) are based on the software analysis of an image without the user providing sample classes. The computer uses techniques to determine which pixels are related and groups them into classes.

​

• An unsupervised classifier does not compare pixels to be classified with any prior information. Rather, it examines a large number of unknown data points and divides them into classes based on properties inherent to the data themselves.

​

• Unsupervised image classification is based entirely on the automatic identification and assignment of image pixels to spectral groupings. It considers only spectral distance measures and involves minimum user interaction. This approach requires interpretation after classification. The user often specifies the number of classes to find and these classes are anonymous - i.e. it is up to the user to define their physical meaning.

​

• In general terms, an unsupervised classifier requires the following parameters to be specified by the user:
• Number of classes
• Number of bands
• Spectral distance or radius in spectral distance
• Spectral space distance parameters when merging clusters

​

​

Unsupervised classification begins with a spectral plot of the whole image, on which the required number of class centres are initiated. The goal of the unsupervised classification algorithm is to group the records into a set of classes, such that the members of a given class are similar to each other and distinct from the members of all the other classes

​

​

BENEFITS OF RCE

How can RCE help, what are its advantages?

• This classifier is like a k-nearest neighbor classifier in that it adapts rapidly over time, but it typically requires many fewer exemplar nodes than a nearest neighbor classifier. During adaptation, more nodes are recruited to generate more complex decision regions, and the size of hyper-spheres formed by existing nodes is modified. (uses spherical influence field for prototype generation)

​

• Theoretical analyses and experiments with RCE classifiers demonstrate that they can form complex decision regions rapidly. Experiments also demonstrated that they can be trained to solve Boolean mapping problems such as the symmetry and multiplexer problems more than an order of magnitude faster than back-propagation classifiers.

​

• Finally, classifiers such as the RCE classifier require less memory than k-nearest-neighbor classifiers but adapt classifier structure over time using simple adaptation rules that recruit new nodes to match the complexity of the classifier to that of the training data" 

CONCLUSION

What can be observed through this research project?

 Scene semantic segmentation is a challenging area of research, that has scope for future development, as we need models that have awareness of new class labels and distinguish between class labels with reduced computational costs and more accurate predictions. In this research project, existing semantic segmentation models have been reviewed and analysed for the problem statement of autonomous vehicles. A new model of colour prototype with spherical influence field is proposed to solve the problem of non-linear and non-separable colour distributions. The proposed model utilizes the concept of colour prototype to provide a novel adaptive scene segmentation algorithm built on the RCE (Restricted Coulomb Energy) Neural Network framework. This model segments colour regions in an image using colour prototype learning. The proposed algorithm is tested by various situations, the following were the observations:

The mode of colour prototype has more potential in acquiring features of colour objects, hence it becomes better to signify each class labels more accurately.

• The spherical influence field of colour prototype is proved to be better for describing arbitrary colour distributions.

​

• The representative colour extraction is effective in abstracting the essential colour features of colour objects.

​

• The segmentation algorithm can segment various colour regions in an image based on its output probabilities mode and fast response mode.

​

CONCLUSION

What can be observed through this research project?

• Using our introduced RCE Neural Network for classification of graphic shapes most of the image pixels pertaining to the graphic shapes can be separated from their backgrounds. Although few image pixels cannot be detected, they do not influence the resultant morphological characteristics.

​

• However, in this training method, the Euclidean distances between neurons and training pixels are evaluated iteratively. Originally, for each pixel, this calculation would be executed several times. The distance estimation process could be very time-consuming, and therefore we introduce a new iteration strategy for the RCE neural network training.

​

• Network Classification: Comparing with the network training process, network classification is much more straightforward. Given an image I, every pixel in this image will be supplied to the RCE neural network in a sequential manner. The pixels are graphic shape only if they fall into the range of one of neurons.

​

• The RCE neural network can efficiently describe graphic shapes by constructing several spherical neurons with different radius. This method can cluster contours in an enclosed environment with no background interference as we are using an RGB colour space which is prone to noise from background interference.

​

• In this project we have developed a RCE neural network based graphic shape detection from learning strategy for identifying contours in input image. Our future work may focus on developing a hybrid neural network based on RCE neural networks and other kinds of neural networks to solve more general classification problems such as scene segmentation.

​

REFERENCES

Journal:

1. Yin X, Guo D, & Xie M (2001). Hand image segmentation using colour and RCE neural network. Robotics and Autonomous Systems, 34(4), 235-250.
2. Sui C, Kwok N M, & Ren T (2011, May). A restricted coulomb energy (RCE) neural network system for hand image segmentation. In 2011 Canadian Conference on Computer and Robot Vision (pp. 270-277). IEEE.
3. Roan S M, Chiang C C, & Fu H C (1993, March). Fuzzy RCE neural network. In [Proceedings 1993] Second IEEE International Conference on Fuzzy Systems (pp. 629-634). IEEE.
4. Yan W, Khoo L P, & Chen C H (2005). A QFD-enabled product conceptualisation approach via design knowledge hierarchy and RCE neural network. Knowledge-Based Systems, 18(6), 279-293.
5. Cho J, Jung Y, Lee S, & Jung Y (2019). Vlsi implementation of restricted coulomb energy neural network with improved learning scheme. Electronics, 8(5), 563.
6. Yushkevich P, Joshi S, Pizer S M, Csernansky J G & Wang L E (2003, July). Feature selection for shape-based classification of biological objects. In Biennial International Conference on Information Processing in Medical Imaging (pp. 114-125). Springer, Berlin, Heidelberg.
7. Sampedro C, Rodriguez-Vazquez J, Rodriguez-Ramos A, Carrio A, & Campoy P (2019). Deep learning-based system for automatic recognition and diagnosis of electrical insulator strings. IEEE Access, 7, 101283-101308.
8. Dokur Z, & Ölmez T (2001). ECG beat classification by a novel hybrid neural network. Computer methods and programs in biomedicine, 66(2-3), 167-181.

​

​

​

REFERENCES

​

Weblinks:

• https://neuron.eng.wayne.edu/tarek/MITbook/chap6/6_3.html
• https://www.academia.edu/4340060/Classifying_seismic_signals_via_RCE_neural_network
• https://www.general-vision.com/pub3rdparty/3P_ZISC_noise_reduction.pdf
• https://pdfs.semanticscholar.org/a069/35aadf328dcbb59a3b7e53903ecbf384f8e4.pdf
• https://www.semanticscholar.org/paper/A-Restricted-Coulomb-Energy-(RCE)-Neural-Network-Sui-Kwok/466a30f5e6aa7405e8e5c9949377f9a142f70688

​

​

​

​

​

--------- THANK YOU ---------