What is a neural network?

A neural network is an information technology system that is based on the structure of the human brain and equips computers with artificial intelligence features. Neural networks are a central component and one of many methods of modern AI applications, for example chatbots such as ChatGPT.

There are different types of artificial neural networks, each of which offers different possibilities for information processing when it comes to deep learning. Research in this area has made huge leaps in recent years. Neural networks are therefore a key technology for teaching machines to think for themselves, enabling computers to solve problems independently and improve their skills. Neural networks are now part of multimodal systems that can combine text, images, audio and video.

How do neural networks work?

Neural networks are based on the structure of the human brain, which processes information via a network of neurons.

Artificial neural networks can be described as models consisting of at least two layers — an input layer and an output layer — and usually additional layers in between (hidden layers). Modern networks such as Convolutional Neural Networks (CNNs) or Transformer models often require many layers, even for simple tasks, because the depth contributes to their efficiency. Each layer of the network contains a large number of specialized artificial neurons.

Processing information in the neural network

Information processing in the neural network always follows the same process: Information in the form of patterns or signals is received by the input layer neurons which process it. Each neuron has a weight assigned to it so that neurons receive information of different levels of importance. The weight and the transfer function determine the input which is then forwarded.

In the next step, an activation function and threshold value calculate and weight the output value of the neuron. Depending on the value and weight of the information, additional neurons are linked and activated to a greater or lesser extent.

This linking and weighting creates an algorithm which generates a result for each input. With each training, the weighting and thus the algorithm are modified so that the network will deliver even more accurate and better results.

Example of neural network applications

Neural networks can be used for image recognition. Unlike humans, a computer cannot tell at a glance whether an image shows a person, plant or object. It has to examine the image for individual characteristics. The computer determines which characteristics are relevant using the implemented algorithm or through data analysis.

In each network layer, the system checks the input signals (i.e. the images) for specific criteria such as colors, corners and shapes. With each check, the computer becomes better at being able to determine what is shown in the image.

Initially, the results will be prone to error. When a neural network receives feedback from a human trainer and thus is able to adapt its algorithm, this is called machine learning. Deep learning does not require training from humans. In this case, the system learns from its own experience and improves according to the amount of image materials it has available to it.

Ideally, the final algorithm can accurately identify the content of images, regardless of whether they are black and white, the subject’s position, or the perspective from which the subject is viewed.

Types of neural networks

Different structures of neural networks are used depending on the learning method used and what the intended application is.

Perceptron

This is the simplest form of neural network and originally referred to a network formed from a single neuron which is modified by weightings and a threshold value. Now, the term “perceptron” is also used for single-pass feedforward neural networks.

Feedforward neural networks

These artificial neural networks can only process information in one direction. These can be single-layer networks (i.e. consisting only of input and output layers) or multi-layer networks with multiple hidden layers.

Recurrent neural networks

In recurrent neural networks, information can also go through feedback loops and thus return to previous layers. This feedback allows the system to develop a memory. Recurrent neural networks are used for purposes such as voice recognition, translation and handwriting recognition.

Convolutional neural networks

These networks are a subtype of multi-layer networks. They consist of a minimum of five layers. Pattern recognition is conducted on each layer, and the results from one layer are transferred to the next. This type of neural network is used for image recognition.

Learning methods

To properly establish connections in artificial neural networks so that they can perform tasks, the neural networks must first be trained. There are two basic methods:

Supervised learning

In supervised learning, specific outcomes are assigned to each input option. For instance, if the system is designed to recognize cat images, humans oversee the system’s categorization process and provide feedback on which images were correctly identified and which were not. This feedback adjusts the network’s weightings, leading to algorithm optimization.

Unsupervised learning

During unsupervised learning, the result for the task is not predefined. Instead, the system learns exclusively from the input information. Hebb’s rule and adaptive resonance theory are used in this learning method. Nowadays, practice focuses on algorithms such as Stochastic Gradient Descent (SGD).

Applications for neural networks

Neural networks are especially effective when there is a large amount of data to be evaluated and only limited systematic problem-solving knowledge. Classic application uses include text, image and voice recognition (i.e. situations in which computers examine data for specific characteristics in order to categorize it).

Neural networks such as convolutional neural networks (CNNs) enable computers to recognize content in images. This technology is used in medical image analysis or in automated quality control in industry. There, neural networks are sometimes used in control engineering, in which they monitor target values and automatically take countermeasures in the event of deviations or in which they independently specify target values based on their data evaluation.

Language models such as ChatGPT, which are based on neural networks, generate realistic-sounding texts, answer questions or analyze large amounts of text data.

Artificial neural networks can also be used for predictions and simulations (e.g. weather forecasts or medical diagnoses). Convolutional neural networks (CNNs), for example, enable computers to recognize content in images. This technology is used in medical image analysis, for example to identify tumors on X-ray images.

Developments in the area of unsupervised learning of artificial neural networks are in the process of massively expanding the application area and performance of the networks. One of the most prominent applications of self-learning neural networks is speech synthesis of voice assistants. Systems such as Alexa, Siri or Google Assistant use neural networks to convert spoken language into text and react to it. Transformer models such as GPT or BERT have revolutionized the quality of machine translations.

History and future outlook

In the past ten years, neural networks have moved into the public consciousness due to the discussions around artificial intelligence. However, the basic technology has already been around for many decades.

Talk of artificial neural networks can be dated back to the early 1940s. Back then, Warren McCulloch and Walter Pitts described a model that linked together elementary units and was based on the structure of the human brain. It was supposed to be able to calculate almost any arithmetic function. In 1949, Donald Hebb developed Hebb’s rule which we explained about earlier. It is still used in many neural networks today.

In 1960, a neural network was developed which had worldwide commercial use for echo filtering in analog telephones. After that, research in the field ground to a halt. One reason for this was that leading scientists had come to the conclusion that the neural network model could not solve important problems. Another was that effective machine learning requires large amounts of digital data which was not available at the time.

This only changed with the advent of big data. With the introduction of the backpropagation algorithm, the training of multi-layer networks became possible. This laid the foundation for modern deep learning models. The combination of enormous amounts of data and the computing power of modern graphics processing units (GPUs) led to a breakthrough in the 2010s. Frameworks such as TensorFlow and PyTorch made the development of neural networks more accessible.

Interest in artificial intelligence and neural networks resurfaced, and the victory of a CNN in the ImageNet competition in 2012 marked the beginning of modern deep learning. Since then, this technology has rapidly gained significance and is shaping nearly every field of computer science.

Since then, development in this field has continued at a rapid pace. As promising as the results are, neural networks are not the only technology to implement artificial intelligence in computers. They are only one option, even if they are often presented in the public debate as the only practicable way. Today, research goes beyond classical neural networks. The focus is on multimodal models that combine text, image and speech, and approaches to reduce energy consumption. At the same time, neural networks are being more and more integrated into everyday applications, from smartphones to intelligent household appliances.