How to use Python pickle to serialize and de­se­ri­al­ize objects

With Python pickle it is possible to serialize and later de­se­ri­al­ize objects. Numerous data types can be con­sid­ered for con­ver­sion. However, as malicious code can also be stored in a memory file, you should only convert files from trust­wor­thy sources into the original format.

What is Python pickle?

Python pickle may seem to have an unusual name at first, but if you take a closer look at how the module works and what it is used for, you will quickly un­der­stand why it was given this name. The module allows you to save objects (i.e. “preserve them”, hence “pickle”) in order to use them at a later time for another project. To do this, the objects are converted into a saveable format. This practice is called se­ri­al­iz­ing. Python pickle can also be used to de­se­ri­al­ize objects, i.e. to convert them back to their original format. Python pickle is par­tic­u­lar­ly useful if you want to use objects fre­quent­ly.

The object is converted into a byte stream, whereby all in­for­ma­tion is trans­ferred unchanged. In addition, Python pickle provides in­struc­tions for suc­cess­ful de­se­ri­al­iza­tion, through which the original structure can be re­con­struct­ed down to the smallest detail. Using Python pickle saves a lot of time, as once an object is created, it does not have to be recreated for each use. The format for saving is .pkl.

Which data types can be converted?

Python pickle can serialize the following data types:

• Boolean values: “true” and “false”, also “none”
• Integers and complex numbers
• Strings (normal and Unicode)
• Lists
• Sets
• Python tuples
• Di­rec­to­ries con­sist­ing ex­clu­sive­ly of cor­re­spond­ing objects
• Functions
• Python classes

What are the different methods?

There are four methods for working with Python pickle, which are provided within the module:

• pickle.dump(obj, file, protocol=None, *, fix_imports=True, buffer_callback=None): Is used for se­ri­al­iza­tion and creates a file with the desired result
• pickle.dumps(obj, protocol=None, *, fix_imports=True, buffer_callback=None): Is also used for se­ri­al­iza­tion, but returns a byte string
• pickle.load(file, *, fix_imports=True, encoding='ASCII', errors="strict", buffers=None): Is used for de­se­ri­al­iza­tion and reads the saved file for this purpose
• pickle.loads(bytes_object, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None): Is also used for de­se­ri­al­iza­tion, but operates with a byte string

To dif­fer­en­ti­ate between the methods, you can remember that the “s” in pickle.dumps and pickle.loads stands for “String.”

How to use Python pickle with an example

To better il­lus­trate how Python pickle works, we will work with a simple example. We will create a simple list con­tain­ing four colors. This is our code:

import pickle
colors = ['Blue', 'Red', 'Yellow', 'Orange']

Then we open a text file in the .pkl format and use pickle.dump() to save our list there. This is the code we use for this:

with open('colors_file.pkl', 'wb') as f:
	pickle.dump(colors, f)

The ab­bre­vi­a­tion wb instructs the system to open the file in binary form. This also outputs the contained data as a bytes object. The “colors” list is then saved in this file with dump(). Finally, the file is closed au­to­mat­i­cal­ly.

How to convert a saved file to its original format

If you now want to de­se­ri­al­ize a binary file again, use the Python method pickle.load(). With the following code, it is possible to convert the object back to its original format and initiate an output. We add the ab­bre­vi­a­tion rb, which stands for “read binary”.

with open('colors_file.pkl', 'rb') as f:
	colors_deserialized = pickle.load(f)
	print(colors_deserialized)

This gives us the following output:

['Blue', 'Red', 'Yellow', 'Orange']

How to serialize dic­tio­nary with Python pickle

You can also easily serialize more complex data types such as di­rec­to­ries with Python pickle and then convert them back to their original form. To do this, we first create a directory with the name “persons”. Here we store different data for different people:

import pickle
persons = {
	'Person 1': {
		'Name': "Maria", 'Age': 56, 'City': "Los Angeles"
	},
	'Person 2': {
		'Name': "Paul", 'Age': 66, 'City': "Los Angeles"
	},
	'Person 3': {
		'Name': "Lisa", 'Age': 22, 'City': "Chicago"
	},
	'Person 4': {
		'Name': "Lara", 'Age': 34, 'City': "San Francisco"
	}
}

In the following code, we create a new file, convert the data, and then test by con­vert­ing it back to serialize this directory:

with open("persons_dict.pkl", "wb") as f:
	pickle.dump(persons, f)
with open("persons_dict.pkl", "rb") as f:
	deserialized_dict = pickle.load(f)
	print(deserialized_dict)

The resulting output looks like this:

persons = {
    'Person 1': { 'Name': "Maria", 'Age': 56, 'City': "Los Angeles"},
    'Person 2': { 'Name': "Paul", 'Age': 66, 'City': "Los Angeles"},
    'Person 3': { 'Name': "Lisa", 'Age': 22, 'City': "Chicago"},
    'Person 4': { 'Name': "Lara", 'Age': 34, 'City': "San Francisco"}
}

You can now access the in­for­ma­tion as usual. We request the following output as an example:

# Define dictionary
deserialized_dict = {
    'Person 1': {'Name': "Maria", 'Age': 56, 'City': "Los Angeles"},
    'Person 2': {'Name': "Paul", 'Age': 66, 'City': "Los Angeles"},
    'Person 3': {'Name': "Lisa", 'Age': 22, 'City': "Chicago"},
    'Person 4': {'Name': "Lara", 'Age': 34, 'City': "San Francisco"}
}
# Print output
print(
    "The name of the third person is"
    + deserialized_dict["Person 3"]["Name"]
    + " and they are "
    + str(deserialized_dict["Person 3"]["Age"])
    + " years old."
)

This is what our output looks like:

The name of the third person is Lisa and she is 22 years old.

How to convert a class to a string

In the next example, we use Python pickle to save a class in a string. This class contains com­plete­ly different data types, but all of them can be taken into account. We create a class called “Ex­am­ple­Class” and then serialize it. The cor­re­spond­ing code is:

import pickle
class ExampleClass:
	def __init__(self):
		self.a_number = 17
		self.a_list = [5, 10, 15]
		self.a_tuple = (18, 19)
		self.a_string = "hello"
		self.a_dict = {"color": "blue", "number": 3}
example_object = ExampleClass()
serialized_object = pickle.dumps(example_object)
print(f"This is a serialized object:\n{serialized_object}\n")
example_object.a_dict = None
deserialized_object = pickle.loads(serialized_object)
print(f"This is a_dict from the deserialized object:\n{deserialized_object.a_dict}\n")

After se­ri­al­iz­ing the class and then con­vert­ing it back to its original format, we get this output:

This is a serialized object:
b'\x80\x03c__main__\nExampleClass\nq\x00)\x81q\x01.'
This is a_dict from the deserialized object:
{'color': 'blue', 'number': 3}

How to compress se­ri­al­ized objects

In principle, files saved with Python pickle are com­par­a­tive­ly compact. Nev­er­the­less, it is also possible and, in some cases, advisable to compress the memory files even more. This works, for example, with the free com­pres­sion program bzip2, which is part of the pro­gram­ming language’s standard library. In the following example, we create a string, serialize it and then apply the com­pres­sion program:

import pickle
import bz2
exampleString = """Almost heaven, West Virginia
Blue Ridge Mountains, Shenandoah River
Life is old there, older than the trees
Younger than the mountains, growin' like a breeze
Country roads, take me home
To the place I belong
West Virginia, mountain mama
Take me home, country roads."""
serialized = pickle.dumps(exampleString)
compressed = bz2.compress(serialized)

Security in­struc­tions for working with Python pickle

Even though Python pickle is a practical and effective method for con­vert­ing objects, there is one major drawback that you should be aware of when working with the module, which is that there is the pos­si­bil­i­ty of trans­fer­ring malicious code via se­ri­al­ized data. Although this is not a problem with your own data, you have to be careful when it comes to third-party files. Therefore, only ever de­se­ri­al­ize memory files when you know and trust the source!