Friday, August 16, 2024

Quick Code Examples Demonstrating Python Unpacking

Code snippets are used to demonstrate Python unpacking. Carefully selected code snippets can replace the many words used in an explanation.

It is assumed that the reader is familiar with Python lists, dictionaries and tuples.

Example 1.A: multiple assignment using a list

Code

a, b, c = [1, 2, 3]

print('Example 1.A')

print(f"a = {a}")
print(f"b = {b}")

print(f"c = {c}")

Output of Code

Example 1.A

a = 1
b = 2

c = 3

Example 1.B: multiple assignment using any iterable

Code

a, b, c = (1, 2, 3)

d, e, f = '456'

print('Example 1.B')

print(f"a = {a}")
print(f"b = {b}")

print(f"c = {c}")

print(f"d = {d}")

print(f"e = {e}")

print(f"f = {f}")

Output of Code

Example 1.B

a = 1
b = 2

c = 3

d = 4

e = 5

f = 6

Example 1.C: multiple assignment using a dictionary

Code

x, y = {'b': 2, 'c': 3}

print('Example 1.C')

print(f"x = {x}")
print(f"y = {y}")

Output of Code

Example 1.C

x = b
y = c

Single Asterisk (*)

Example 2.A: using single asterisk to unpack a list into the last assignment variable

Code

a, *b = [1, 2, 3]

print('Example 2.A')

print(f"a =  {a}")
print(f"b =  {b}")
print(f"b[0] = {b[0]}")
print(f"b[1] = {b[1]}")

Output of Code

Example 2.A

a =  1

b =  [2, 3]

b[0] = 2
b[1] = 3

Example 2.B: using single asterisk to unpack a list into the middle assignment variable

Code

a, *b, c = [1, 2, 3, 4]

print('Example 2.B')

print(f"a =  {a}")

print(f"b =  {b}")
print(f"b[0] = {b[0]}")
print(f"b[1] = {b[1]}")

print(f"c =  {c}")

Output of Code

Example 2.B

a =  1

b =  [2, 3]

b[0] = 2
b[1] = 3

c =  4

Example 3: using single asterisk for unpacking a list into positional arguments of a function (*args)

Code

def print_d(a, b, c):
    print(f"a =  {a}")
    print(f"b =  {b}")
    print(f"c =  {c}")

d = (1, 2, 3)

print('Example 3')

print_d(*d)

Output of Code

Example 3

a =  1
b =  2
c =  3

Example 4: multiple assignment using a dictionary

For the details on the items() method of a dictionary, click here.

Code

d = {'a': 1, 'b': 2, 'c': 3}

x, y, z = d.items()

print('Example 4')

print(f"x =  {x}")
print(f"x[0] =  {x[0]}")
print(f"x[1] =  {x[1]}")
print(f"y =  {y}")
print(f"z =  {z}")

Output of Code

Example 4

x =  ('a', 1)
x[0] =  a
x[1] =  1

y =  ('b', 2)
z =  ('c', 3)

Double Asterisk (**)

Example 5: double asterisk for unpacking the dictionary into keyword arguments that match the function parameters(**kwargs)

Code

def print_d(a, b, c):
    print(f"a =  {a}")
    print(f"b =  {b}")
    print(f"c =  {c}")

d = {'a': 1, 'b': 2, 'c': 3}

print('Example 5')

print_d(**d)

Output of Code

Example 5

a =  1
b =  2
c =  3

Example 6.A: double asterisk for unpacking dictionaries in order to merge them

Code

d_1 = {'a': 1, 'b': 2}
d_2 = {'c': 3, 'd': 4 }

merged_dict = {**d_1, **d_2}

print('Example 6.A')

print(f"merged_dict = {merged_dict}")

Output of Code

Example 6.A

merged_dict = {'a': 1, 'b': 2, 'c': 3, 'd': 4}

Example 6.B: double asterisk for unpacking dictionaries in order to merge them and override as needed

Code

d_1 = {'a': 1, 'b': 2}
d_2 = {'b': 99, 'c': 3, 'd': 4 }

merged_dict = {**d_1, **d_2}

print('Example 6.B')

print(f"merged_dict = {merged_dict}")

Output of Code

Example 6.B

merged_dict = {'a': 1, 'b': 99, 'c': 3, 'd': 4}

Note

The above is used to demonstrate dictionary unpacking. The more Pythonic way to merge dictionaries is to use the vertical bar (|) operator.

merged_dict = d_1 | d_2

Sunday, November 20, 2022

Parallelizing Nested For-Loops in Python

Introduction

Once upon a time Smith had to obtain new transportation. He had to first decide whether he was going to buy a car or truck. Once he decided to buy a car, he had to decide between the various brands of cars like Ford and GM. Luckly, once Smith decided that he wanted a car; Mary (Smith's wife) decided to help with the research. Consequently, the Ford and GM car models could be researched concurrently.

Generalized Problem Statement

The above story can be used to create a generalized problem statement. For example, Smith has to decide whether he wants a car or a truck before he conducts any other research. This can be generalized to say that all categories in level 1 (car, truck, ...) must be processed before any level 2 categories (Ford, GM, ...) can be addressed.

We can continue with this generalization by saying that all level 2 categories can be processed concurrently. Remember that Smith can research Ford cars while Mary researches GM cars. In other words, the research of Ford and GM cars can happen concurrently.

The above has created a hierarchy of categories 2 levels deep. There are already many good examples on the internet that demonstrate parallelizing a hierarchy 2 levels deep. One example of this is "How to reduce the time complexity of nested loops" by Leandro Proença. So, we are going to add a third level to the hierarchy to make the problem more realistic. The other additional twist is that all level 3 categories associated with a category level 2 can be processed in parallel once the level 2 category has been processed. Luckly for us, we can assume that the processing of each category is approximately the same.

Below is a pictoral of the category hierarchy that will be used in the proof of concept code (input_data.py). For a PDF version of the data click here. For a spreadsheet version of the data click here.

Once all the categories in all the hierarchies are processed, summary counts are computed.

The above problem is a little complicated. This is done on purpose. There are already many good examples on the internet that demonstrate parallelizing simple problems. My goal in this blog is to introduce some complexity in order to reflect a more real world problem.

For those who prefer to go to straight to the code, click here. For those who prefer a description of the problem as well as the architecture used to solve the problem, please continue reading.

Nested For-Loops

For those who prefer to go to straight to the code for nested for-loops, click here. For those who prefer a description as to the how and why of the code, please continue reading.

One of the simplest things to do is to go through the 3 levels of the hierarchy and process all the combinations via nested for-loops. Think of it as examining a data cube. Sometimes people can get carried away with nested for-loops. Below is a picture to emphasize this.

The nested for-loop portion of the code is below.

for category_lvl_1 in categories_lvl_1:

    try:

        categories_lvl_2 = categories_lvl_1_2[category_lvl_1]

        for category_lvl_2 in categories_lvl_2:

            common_processing.process_categories_non_lvl_1(
                2, category_lvl_1, category_lvl_2
            )

            try:

                categories_lvl_3 = categories_lvl_2_3[category_lvl_2]

                for category_lvl_3 in categories_lvl_3:

                    common_processing.process_categories_non_lvl_1(
                        3, category_lvl_2, category_lvl_3
                    )

            except KeyError:

                ...

        except KeyError:

            ...

Notice that the processing of the combination category N and (N + 1) is identical and is encapsulated by common_processing.process_categories_non_lvl_1(...). We will use this fact when we parallelize the processing. Also the processing of level 1 categories [process_categories_lvl_1()] as well as the computing of summary counts [summary_counts()] is included inn the common_processing module. In other words we have prepared the code for parallelization.

There are several issues with using nested for-loops. The use of them implies an ordering even though one may not exist. In our case there is no ordering because all level 2 categories can be processed in parallel.

The other issue is that nested for-loops are often the source of performance issues. In our case O(N³).

Also, if you notice the above code has try/except blocks because all the generated combinations do not necessarily exist. It would be better not to generate combinations that don't exist and in so doing avoid adding try/except code blocks.

Concurrently Process All Level 2 Categories

For those who prefer to go to straight to the code for concurrently processing all level 2 categories, click here. For those who prefer a description as to the how and why of the code, please continue reading.

The nature of the problem is such that it allows for the parallel processing of all level 2 categories. The ideal translation of this into code is as follows:

with Pool() as pool:
    pool.imap_unordered(process_category_lvl_2, get_level_2_categories())
    pool.close()
    pool.join()

All that is needed is to code up process_category_lvl_2() and get_level_2_categories(). The implementation of process_category_lvl_2() involves the following steps

Obtain the category level 1 associated with the input level 2 category.
Process the level 1, 2 category combination.
Process the level 2, 3 combinations for the input level 2 category.

The implementation of get_level_2_categories() is straight forward. In most cases it will involve a simple query to a database.

Consequently, the main function is simply

common_processing.process_categories_lvl_1()

with Pool() as pool:
    pool.imap_unordered(process_category_lvl_2, get_level_2_categories())
    pool.close()
    pool.join()

common_processing.summary_counts()

Notice how much simpler the above code is compared to the nested for-loops described previously.

The multiprocessing.pool.Pool() is being used as a throttling mechanism. We don't want to flood the system with a large number of concurrent processes each fighting for resources.

If you want to impress people with big words, you can say that you used the "Single Instruction, Multiple Data (SIMD)" design pattern. Another set of big words that you can use is "embarrassingly parallel".

Nested For-Loops was a Lookup Problem in Disguise

Notice how a nested for-loop was actually a lookup problem in disguise.

There is an initial lookup to get the set of level 2 categories.
There is another lookup to get the level 1 category associated with a level 2 category.
There is a final lookup to get all the level 3 categories associated with a level 2 category.

Further Parallelization

Let's try to take advantage of the fact that all level 3 categories associated with a completed level 2 category can be processed in parallel.

One approach is to pass the pool to the concurrent processes being executed. Unfortunately, this won't work because "the methods of a pool should only ever be used by the process which created it."

Another approach is to create a Pool() in the concurrent processes being executed. Unfortunately, this won't work because "cannot create a Pool in pooled function."

Unfortunately, this means that will have to re-architect the approach. Luckly for me, just concurrently executing level 2 categories is good enough at this point in time.

However, for those who have use cases that require further parallelization, one can use Managers. It provides a workaround for the inability to directly share a process pool. For the details, refer to the article "Share a Multiprocessing Pool With Workers" by Jason Brownlee.

The danger of using a Manager is that you might end up creating your own framework for concurrent execution. Perhaps, it woud be best to just use one of the existing frameworks? Below is a partial list of these types of frameworks to help you get started in their exploration.

Also, you might get lucky and meet performance goals by using Numba ("just" add the @jit decorator). For additional details, refer to the article "Explicit Parallel Loops" at Numba.ReadTheDocs.Io

Don't Use Python

Ultimately, if you want blazing performance, you'd have to convert from Python to some other language. One alternative is Cython. Another alternative is Julia (Parallel Map and Loops). There are many others. If this type of approach is used, please consider asking if the overhead added by another language in your infrastructure worth it?

Miscellaneous Technical Details

Since all the category processing has to complete before computing summary counts, have to issue a pool close() followed by a pool join(). For the details of why this is necessary, refer to the article "7 Multiprocessing Pool Common Errors in Python" by Jason Brownlee. Unfortunately, can't use the pool context manager to accomplish this goal. For the details of why this is, refer to the article "Multiprocessing Pool Context Manager" by Jason Brownlee.

The other thing is that to gurantee that print statements will actually show up, the associated buffer must be flushed. For the details refer to the following articles

To Dos

Unfortunately, there is an infinite number of things to do and a limited amount of time. Consequently, I have left out a few things which I hope to get to at a later point.

The first are tests to ensure that the parallel solution yields the same results as the naive nested loop solution. Currently, I manually compare the files nested_for_loops_output.txt and parallelize_level_2_categories_output.txt to see that the same exact category pair processing is happening. This can not easily be automated because the order of the category pair processing will be different in the two files.

The second is benchmarking/profiling. In my particular case, I will fix the number of lines of text to categorize and then vary the number of physical cores and the number of processes created by Pool(). For guidance on setting the number of processes created by Pool() refer to the following articles

Summary

The selected use case is slightly complex in order to better reflect a real world problem. Also, it enables a discussion of tradeoffs that always needs to be made.

One thing that has not been mentioned so far is that it is a must to write down the definition of done before doing anything. In my particular use case the definition of done is to map x number of lines of text to categories is one hour or less. The mapping is done by process_categories_lvl_1() and process_categories_non_lvl_1() in the common module.

The other thing that needs to be written down are the constraints. In my particular use case, we don't want to modify the code in the common_processing module because they are highly complex. Consequently, the easiest way to reduce processing time is to execute all the level 2 category processing in parallel.

Lastly, it should be noted that there is no general solution to parallelizing nested for-loops. Each use case is different. In the use case addressed in this blog we took advantage of the fact that we could process all level 2 categories in parallel. However, for some general guidelines on parallelizing nested for-loops, refer to the article "Parallel Nested For-Loops in Python" by Jason Brownlee. The other thing to note is that it is not always possibe to parallelize nested for loops.

References

The purpose of the reference section is to make it easy for the reader to find things. Also, the key passages within the link is provided. This way don't have to search an entire article to find the applicable material.

The reference section is lengthy and so it has been placed on a separate web page. To go to it, click here.

Tuesday, April 13, 2021

Python: Nested Dictionaries vs Dictionary Plus a Dataclass

The use case involves reading user attributes like name, status, and gender given the associated user UUID. This is a write-once and read many times scenario. Also, it is assumed that all the data will fit into memory.

Nested Dictionaries

The nested dictionary creation will look something like the following:

nested_dict_approach = {  
    '19acc7df-9c8b-11eb-9022-cc2f71aeb20c' : {
        'name': 'fred',
        'status': 'active',
        'gender': 'M' 
    },
    '19acc7df-9c8b-11eb-9022-cc2f71aeb20d' : {
        'name': 'barney',
        'status': 'inactive',
        'gender': 'M' 
    },
    '19acc7df-9c8b-11eb-9022-cc2f71aeb20e' : {
        'name': 'wilma',
        'status': 'unknown',
        'gender': 'F' 
    },
}

The constant repetition of the nested key names is annoying and results in verbose and error-prone code.

To access a particular user via a UUID is straightforward.

nested_dict_approach['19acc7df-9c8b-11eb-9022-cc2f71aeb20c']

{'name': 'fred', 'status': 'active', 'gender': 'M'}

The downside of the above output is that it is just a list of attributes with no unifying principle.

To access a particular attribute of a user via a UUID is also straightforward.

nested_dict_approach['19acc7df-9c8b-11eb-9022-cc2f71aeb20c']['name']

'fred'

Dictionary Plus a Dataclass

The dictionary plus a dataclass creation will look something like the following.

from dataclasses import dataclass

@dataclass
class User:
    name: str
    status: str
    gender: str

user_1 = User('fred', 'active', 'M')
user_2 = User('barney', 'inactive', 'M')
user_3 = User('wilma', 'unknown', 'F')

dict_plus_data_class_approach = {  
    '19acc7df-9c8b-11eb-9022-cc2f71aeb20c' : user_1,
    '19acc7df-9c8b-11eb-9022-cc2f71aeb20d' : user_2,
    '19acc7df-9c8b-11eb-9022-cc2f71aeb20e' : user_3,
}

The constant repetition of the nested key names is eliminated. The other benefit is that we can specify the data types of each of the attributes.

Accessing a particular user via a UUID is the same.

dict_plus_data_class_approach['19acc7df-9c8b-11eb-9022-cc2f71aeb20c']

User(name='fred', status='active', gender='M')

However, notice that the output now is not just a list of attributes. The attributes are organized into a User.

To access a particular attribute of a user via a UUID just use a dot (".") as opposed to square brackets ("[]").

dict_plus_data_class_approach['19acc7df-9c8b-11eb-9022-cc2f71aeb20c'].name

'fred'
Summary

In summary, if you are using nested dictionaries, stop and consider using a dictionary combined with a dataclass.

Friday, August 28, 2020

Python Iterable: Sequences, Non-Sequence Iterables and So On

Iterable is a key concept in Python. Think of it as allowing you to do a for loop over "stuff." This includes being able to do for loops over non-sequences like dictionaries and sets.

Trying to determine which of Python's various data types support the iterable concept can be tedious. Below is a table that helps with that. Also, the links in the table go to the corresponding Python documentation.

Please note that the table is not meant to be an exhaustive enumeration of where an iterable is used. The goal is to help people get started on their journey. For example, the table contains collections.abc.Sequence but it does not contain the other applicable abstract base classes for containers.

Technical Note: I couldn't get Blogger.Com to properly render the table. Consequently, I have included a screenshot of the table below. To go to the table with the links, click here.

Wednesday, February 12, 2020

How do you handle the situation when your data does not fit into memory?

If you don't have much time to devote to the topic, just watch the video Small Big Data: using NumPy and Pandas when your data doesn't fit in memory by Itamar Tuner-Trauring at PyData New York 2019.

This won't be a conventional blog post. It won't have paragraph after paragraph. Instead, it will be a list around which people can have conversations. The links within the list and the references at the end will provide the details.

The majority of the list items below come from Itamar Tuner-Trauring video mentioned earlier.

1. Small Big Data: too big to fit in memory but too small for a complex big data cluster

a. Load only what you need

i. Do you really need to load all the columns?

b. Use the appropriate data type

i. If possible, use NumPy 16 bit unsigned integers as opposed to its 64-bit version.

ii. If possible, use Pandas category rather than an object data type.

c. Divide data into chunks - Process 1 chunk at a time

i. pandas.read_csv( ... chunksize = None ...)

ii. Zarr: chunked, compressed, N-dimensional arrays

d. Use Generator Expression: lazily generated iterable objects

i. Python’s Generator Expressions: Fitting Large Datasets intoMemory by Luciano Strika

A. Commonly used example


with open('big.csv') as f:

for line in f:
process(line)

e. Indexing: only need a specific subset of the data

i. populate SQLite from Pandas

A. load from SQLite into DataFrame

ii. vroom R package

f. If your data is sparse, take advantage of it.

i. Pandas - Sparse Data Structures

ii. sparse.pydata.org

g. Approaches Involving Approximations

i. Requires application of domain knowledge

ii. Just sample your data.

A. pd.read_csv(..., skiprows=lambda x: x % 2 != 0)

iii. Probabilistic Data Structures

A. Introduction to Probabilistic Data Structures by Yin Niu

h. Streaming

i. Streamz

i. Use a single machine distributed framework

i. Single Machine: dask.distributed

j. Just load your data into a database like PostgreSQL.

k. Just use the disk.

i. It will be slow but perhaps acceptable until can implement a better solution.

l. Money Solution

i. Buy or rent computers with large amounts of RAM

2. Big Data: anything that is not "small big data"

a. Dask, Spark, ...

References

1. NumPy Reference

a. Array Objects

i. Data Type Objects (dtype)

2. Pandas

a. Getting Started

i. Essential Basic Functionality

A. dtypes

3. Python Documentation

a. Language Reference

i. Expressions

A. Atoms

I. Generator Expressions

4. Small Big Data: using NumPy and Pandas when your data doesn't fit in memory by Itamar Tuner-Trauring at PyData New York 2019

5. What are all the dtypes that pandas recognize? at StackOverFlow.Com

6. When to use Category rather than Object? at StackOverFlow.Com

Thursday, September 5, 2019

Quick Explanation of the Python Generator

The goal of this blog post is to provide a quick explanation of what a generator is and a short code snippet demonstrating it.

Generators are functions that can be used as iterators. Another way of saying the same thing is that a generator is a function which may return as many times as needed, instead of just once.

A good way to demonstrate a generator is via the Fibonacci Sequence.

def fib():

a,b = 0, 1

while True:

yield a

a, b = b, a+b

for f in fib():

if f > 100:

break

print(f)

The pro of generators is that they require less memory because you only generate the values as you need them. The con is that you are increasing the workload on your CPU and making your code more complex.

If you are interested in learning the details, please refer to the items:

1. CodeBasicsHub.Com - YouTube - Python - Video 24: Generators

2. Fibonacci Sequence at Math is Fun

3. Python Documentation

a. Glossary

i. Asynchronous Generator

ii. Asynchronous Generator Iterator

iii. Asynchronous Iterable

iv. Asynchronous Iterator

v. Generator

vi. Generator Expression

vii. Generator Iterator

viii. Iterable