What is transfer learning and how is it used to pre-train models for new tasks?

Transfer learning is an approach where a pre-trained model is applied to a new, related task. This method helps save time and resources while enhancing the per­for­mance of machine learning models. There are various strate­gies for adapting pre-trained models to fit a new task.

What is transfer learning?

Transfer learning is a method from machine learning where a trained model is optimized for a new, similar task. Rather than training a new model from scratch for a specific task, the existing knowledge is used. Through minor ad­just­ments, the pre-trained model is adapted to the new task, enabling it to handle different features. This approach saves time and resources, as it requires much smaller datasets for training, making it both more efficient and powerful.

How does transfer learning work?

Transfer learning involves taking a model that has already been fully trained for a specific task and applying it to a new, similar task. This method is es­pe­cial­ly effective when working with un­struc­tured data, such as images or videos. For example, a model trained to recognize images of cars can be adapted to identify trucks, as many features, such as wheels, doors, and overall shape, are shared between the two cat­e­gories.

Selecting a trained model

As a starting point, you need a pre-trained model, which is created by training on a large dataset with labeled examples. The model learns to recognize patterns and re­la­tion­ships in the data, allowing it to perform the intended task. In machine learning, this process involves layers that are in­ter­con­nect­ed and used to perform cal­cu­la­tions. The more layers a model has, the more complex patterns it can recognize.

In transfer learning, you choose a model that has already suc­cess­ful­ly completed this process. It’s important to closely examine the source task of the existing model. The more similar it is to the new task, the easier it will be to adapt the model for the new ap­pli­ca­tion.

Re­con­fig­ur­ing and training the model

The second step is to configure the pre-trained model for the new task. There are generally two practical methods for this, and you can choose the one that best suits your needs.

In the first method, the last layer of the trained output model is replaced. This layer, also called the output layer, acts as the final clas­si­fi­ca­tion unit, de­ter­min­ing whether a given input matches the trained pa­ra­me­ters. For example, this layer might decide whether an image rep­re­sents a car. In many cases, you can remove this layer and replace it with a new one that is tailored to your specific ap­pli­ca­tion. In our example, the new layer would be designed to identify trucks instead of cars.

Al­ter­na­tive­ly, with transfer learning, it’s possible to freeze the previous pa­ra­me­ters and add new layers instead. These new layers are specif­i­cal­ly designed to align with the new task and are in­te­grat­ed into the model. The adapted model is then trained with a much smaller dataset con­tain­ing the relevant examples. During this training, the model rec­og­nizes patterns and re­la­tion­ships while lever­ag­ing the knowledge gained from the original training.

Checking progress

The final step must be carried out in any case: You can only train the AI for the new task through con­sci­en­tious mon­i­tor­ing and, if necessary, ad­just­ments to the training material and possibly the new shifts. If the pa­ra­me­ters are adjusted during training, the accuracy will also increase and the model will learn to meet the new re­quire­ments.

What are the different strate­gies?

There are different strate­gies for the use of transfer learning. Which one is right for you depends primarily on the desired purpose. These are some ap­proach­es:

• Feature ex­trac­tion: In feature ex­trac­tion, you use the pre-trained model to extract basic features, such as textures, while the new layers are designed to recognize more specific features. This approach is useful when the source and target tasks have sig­nif­i­cant overlap.
• Inductive transfer learning: In this case, the source and target domains are the same, but the tasks differ. This method allows new functions to be trained quicker, as the model can leverage knowledge from the source task to improve learning in the target task.
• Trans­duc­tive transfer learning: In this strategy, the knowledge gained from the source task is trans­ferred directly to specific instances of the new task, for example in order to be able to classify them better. This approach is promising if the source and target tasks have com­par­a­tive­ly few sim­i­lar­i­ties.
• Un­su­per­vised transfer learning: Here, the source and target domains are similar, but the tasks differ. However, no labeled data is provided. Instead, the model learns the dif­fer­ences and sim­i­lar­i­ties of the unlabeled data, enabling it to gen­er­al­ize and make pre­dic­tions based on this in­for­ma­tion.
• Mul­ti­task­ing: In this approach, a model si­mul­ta­ne­ous­ly performs multiple tasks that are not identical but are related to each other. This enables shared knowledge.
• Pre­dic­tion: In this form of transfer learning, the model is supposed to fill in certain missing aspects of the data itself. For example, words within a sentence are predicted. The results are then improved through fine-tuning.
• Zero-shot and Few-shot: This is also a form of transfer learning in the field of gen­er­a­tive AI, in which knowledge from a source is to be trans­ferred to a target if there are only a few overlaps (Few-Shot) or no overlaps at all (Zero-Shot) between the two. The method is used when only very little training data is available.
• Dis­en­tan­gle­ment: For this approach, data is split into different factors. The model can then consider and ma­nip­u­late style and content sep­a­rate­ly, for example.

What areas of ap­pli­ca­tion does transfer learning have?

There are numerous potential areas of ap­pli­ca­tion for transfer learning. The method offers sig­nif­i­cant savings in cost, time, and resources, making it highly ad­van­ta­geous. The most important ap­pli­ca­tions to date include:

• image recog­ni­tion
• speech recog­ni­tion
• object lo­cal­iza­tion
• di­ag­nos­tics in the health­care sector

In the future, however, transfer learning is likely to be applied in many other areas as well.