Efficient Pandas: Using Chunksize for Large Datasets

Author(s): Lawrence Alaso Krukrubo 

Exploring giant information units effectively utilizing Pandas

Data Science professionals typically encounter very giant information units with a whole bunch of dimensions and hundreds of thousands of observations. There are a number of methods to deal with giant information units. We all know concerning the distributed file programs like Hadoop and Spark for dealing with massive information by parallelizing throughout a number of employee nodes in a cluster. But for this text, we will use the pandas chunksize attribute or get_chunk() operate.

Imagine for a second that you just’re engaged on a brand new film set and also you’d wish to know:-

1. What’s the commonest film ranking from 0.5 to five.0

2. What’s the common film ranking for many motion pictures produced.


To reply these questions, first, we have to discover a information set that comprises film scores for tens of hundreds of films. Thanks to Grouplens for offering the Movielens information set, which comprises over 20 million film scores by over 138,000 customers, overlaying over 27,000 totally different motion pictures.

This is a big information set used for constructing Recommender Systems, And it’s exactly what we’d like. So let’s extract it utilizing wget. I’m working in Colab, however any pocket book or IDE is ok.

!wget -O moviedataset.zip https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/moviedataset.zip
print('unziping ...')
!unzip -o -j moviedataset.zip

Unzipping the folder shows Four CSV information:


motion pictures.csv



Our curiosity is on the scores.csv information set, which comprises over 20 million film scores for over 27,000 motion pictures.

# First let's import just a few libraries
import pandas as pd
import matplotlib.pyplot as plt

Let’s take a peek on the scores.csv file

ratings_df = pd.read_csv('scores.csv')
  (22884377, 4)

As anticipated, The ratings_df information body has over twenty-two million rows. This is lots of information for our laptop’s reminiscence to deal with. To make computations on this information set, it’s environment friendly to course of the information set in chunks, one after one other. In form of a lazy trend, utilizing an iterator object.

Please be aware, we don’t must learn in the whole file. We may merely view the primary 5 rows utilizing the pinnacle() operate like this:


It’ s vital to speak about iterable objects and iterators at this level…

An iterable is an object that has an related iter() methodology. Once this iter() methodology is utilized to an iterable, an iterator object is created. Under the hood, that is what a for loop is doing, it takes an iterable like a listing, string or tuple, and applies an iter() methodology and creates an iterator and iterates by means of it. An iterable additionally has the __get_item__() methodology that makes it attainable to extract parts from it utilizing the sq. brackets.

See an instance beneath, changing an iterable to an iterator object.

# x beneath is an inventory. Which is an iterable object.
x = [1, 2, 3, 'hello', 5, 7]
# passing x to the iter() methodology converts it to an iterator.
y = iter(x)
# Checking sort(y)
<class 'list_iterator'>

The object returned by calling the pd.read_csv() operate on a file is an iterable object. Meaning it has the __get_item__() methodology and the related iter() methodology. However, passing an information body to an iter() methodology creates a map object.

df = pd.read_csv('motion pictures.csv').head()
# Let's cross the information body df, to the iter() methodology
df1 = iter(df)
<class 'map'>

An iterator is outlined as an object that has an related subsequent() methodology that produces consecutive values.

To create an iterator from an iterable, all we have to do is use the operate iter() and cross it the iterable. Then as soon as we have now the iterator outlined, we cross it to the subsequent() methodology and this returns the primary worth. calling subsequent() once more returns the subsequent worth and so forth… Until there aren’t any extra values to return after which it throws us a CeaseIterationError.

x = [1, 2, 3]
x = iter(x)  # Converting to an iterator object
# Let’s name the subsequent operate on x utilizing a for loop
for i in vary(4): print(subsequent(x))
# Error is displayed if subsequent is named in any case gadgets have been printed out from an iterator object

Note that the phrases operate and methodology have been used interchangeably right here. Generally, they imply the identical factor. Just {that a} methodology is often utilized on an object like the pinnacle() methodology on an information body, whereas a operate often takes in an argument just like the print() operate.

If you’d like to search out out about python comprehensions and mills see this link to my pocket book on Github. It’s not crucial for this text.

Ok. let’s get again to the ratings_df information body. We wish to reply two questions:

1. What’s the commonest film ranking from 0.5 to five.0

2. What’s the common film ranking for many motion pictures.

Let’s examine the reminiscence consumption of the ratings_df information body

ratings_memory = ratings_df.memory_usage().sum()
# Let's print out the reminiscence consumption
print('Total Current reminiscence is-', ratings_memory,'Bytes.')
# Finally, let's examine the reminiscence utilization of every dimension.
Total Current reminiscence is- 732300192 Bytes.
Index              128
userId       183075016
movieId      183075016
ranking       183075016
timestamp    183075016
dtype: int64

We can see that the whole reminiscence consumption of this information set is over 732.Three million bytes… Wow.

Since we’re within the scores, let’s get the totally different ranking keys on the size, from 0.5 to five.0

# Let's get an inventory of the ranking scale or keys
rate_keys = listing(ratings_df['rating'].distinctive())
# let's type the scores keys from highest to lowest.
rate_keys = sorted(rate_keys, reverse=True) 
  [5.0, 4.5, 4.0, 3.5, 3.0, 2.5, 2.0, 1.5, 1.0, 0.5]

We now know our ranking scale. Next, is to discover a method to get the variety of scores for every key on the size. Yet as a result of reminiscence dimension, we must always learn the information set in chunks and carry out vectorized operations on every chunk. Avoiding loops besides crucial.

Our first objective is to rely the variety of film scores per ranking key. Out of the 22 million-plus scores, what number of scores does every key maintain? Answering this query robotically solutions our first query:-

Question One:

1. What’s the commonest film ranking from 0.5 to 5.0

let’s create a dictionary whose keys are the distinctive ranking keys utilizing a easy for loop. Then we assign every key to worth zero.

ratings_dict = 
for i in rate_keys: ratings_dict[i] = 0
0.5: 0,  1.0: 0,  1.5: 0,  2.0: 0,  2.5: 0,  3.0: 0,  3.5: 0,  4.0: 0,  4.5: 0,  5.0: 0

Next, we use the python enumerate() operate, cross the pd.read_csv() operate as its first argument, then inside the read_csv() operate, we specify chunksize = 1000000, to learn chunks of 1 million rows of knowledge at a time.

We begin the enumerate() operate index at 1, passing begin=1 as its second argument. So that we will compute the common variety of bytes processed for every chunk utilizing the index. Then we use a easy for loop on the ranking keys and extract the variety of scores per key, for every chunk and sum these up for every key within the ratings_dict

The last ratings_dict will comprise every ranking key as keys and whole scores per key as values.

Using chunksize attribute we will see that :
Total variety of chunks: 23
Average bytes per chunk: 31.eight million bytes

This means we processed about 32 million bytes of knowledge per chunk as in opposition to the 732 million bytes if we had labored on the complete information body without delay. This is computing and memory-efficient, albeit by means of lazy iterations of the information body.

There are 23 chunks as a result of we took 1 million rows from the information set at a time and there are 22.eight million rows. So meaning the 23rd chunk had the ultimate 0.eight million rows of knowledge.

We also can see our ratings_dict beneath full with every ranking key and the whole variety of scores per key

5.0: 3358218, 4.5: 1813922, 4.0: 6265623, 3.5: 2592375, 3.0: 4783899, 2.5: 1044176, 2.0: 1603254, 1.5: 337605, 1.0: 769654, 0.5: 315651

Note that By specifying chunksize in read_csv, the return worth might be an iterable object of sort TextFileReader .Specifying iterator=True may also return the TextFileReader object:

# Example of passing chunksize to read_csv
reader = pd.read_csv(’some_data.csv’, chunksize=100)
# Above code reads first 100 rows, in case you run it in a loop, it reads the subsequent 100 and so forth
# Example of iterator=True. Note iterator=False by default.
reader = pd.read_csv('some_data.csv', iterator=True)
This will get the primary 100 rows, working by means of a loop will get the subsequent 100 rows and so forth.
# Both chunksize=100 and reader.get_chunk(100) return similar TextFileReader object.

This reveals that the chunksize acts identical to the subsequent() operate of an iterator, within the sense that an iterator makes use of the subsequent() operate to get its’ subsequent component, whereas the get_chunksize() operate grabs the subsequent specified variety of rows of knowledge from the information body, which has similarities to an iterator.

Before shifting on, let’s verify we acquired the entire scores from the train we did above. Total scores ought to be equal to the variety of rows within the ratings_df.

sum(listing(ratings_dict.values())) == len(ratings_df)

Let’s lastly reply query one by deciding on the important thing/worth pair from ratings_dict that has the max worth.

# We use the operator module to simply get the max and min values
import operator
max(ratings_dict.gadgets(), key=operator.itemgetter(1))
  (4.0, 6265623)

We can see that the ranking key with the very best ranking worth is 4.0 with a price of 6,265,623 film scores.

Thus, the commonest film ranking from 0.5 to five.Zero is 4.0

Let’s visualize the plot of ranking keys and values from max to min.

Let’s create an information body (ratings_dict_df) from the ratings_dict by merely casting every worth to an inventory and passing the ratings_dict to the pandas DataBody() operate. Then we type the information body by Count descending.

Question Two:

2. What’s the common film ranking for many motion pictures.

To reply this, we have to calculate the Weighted-Average of the distribution.

This merely means we multiply every ranking key by the variety of instances it was rated and we add all of them collectively and divide by the whole variety of scores.

# First we discover the sum of the product of fee keys and corresponding values.
product = sum((ratings_dict_df.Rating_Keys * ratings_dict_df.Count))
# Let's divide product by whole scores.
weighted_average = product / len(ratings_df)
# Then we show the weighted-average beneath.

So to reply query two, we will say

The common film ranking from 0.5 to five.Zero is 3.5.

It’s fairly encouraging that on a scale of 5.0, most motion pictures have a ranking of 4.0 and a mean ranking of 3.5… Hmm, Is anybody considering of film manufacturing?

If you’re like most individuals I do know, the subsequent logical query is:-

Hey Lawrence, what’s the possibility that my film would at the very least be rated common?

To discover out what proportion of films are rated at the very least common, we’d compute the Relative-frequency proportion distribution of the scores.

This merely means what proportion of film scores does every ranking key maintain?

Let’s add a proportion column to the ratings_dict_df utilizing apply and lambda.

ratings_dict_df['Percent'] = ratings_dict_df['Count'].apply(lambda x: (x / (len(ratings_df)) * 100))
Percentage of film scores per key.

Therefore to search out the proportion of films which are rated at the very least common (3.5), we merely sum the chances of film keys 3.5 to 5.0.

sum(ratings_dict_df[ratings_dict_df.Rating_Keys >= 3.5]['Percent'])


From these workouts, we will infer that on a scale of 5.0, most motion pictures are rated 4.0 and the common ranking for motion pictures is 3.5 and at last, over 61.3% of all motion pictures produced have a ranking of at the very least 3.5.


We’ve seen how we will deal with giant information units utilizing pandas chunksize attribute, albeit in a lazy trend chunk after chunk.

The deserves are arguably environment friendly reminiscence utilization and computational effectivity. While demerits embrace computing time and attainable use of for loops. It’s vital to state that making use of vectorised operations to every chunk can significantly pace up computing time.

Thanks on your time.

P.S See a hyperlink to the pocket book for this text in Github.

Bio: Lawrence Krukrubo is a Data Specialist at Tech Layer Africa, obsessed with honest and explainable AI and Data Science. Lawrence is licensed by IBM as an Advanced-Data Science Professional. He likes to contribute to open-source tasks and has written a number of insightful articles on Data Science and AI. Lawrence holds a BSc in Banking and Finance and pursuing his Masters in Artificial Intelligence and Data Analytics at Teesside, Middlesbrough U.Ok.

Efficient Pandas: Using Chunksize for Large Datasets was initially printed in Towards AI on Medium, the place persons are persevering with the dialog by highlighting and responding to this story.

Published by way of Towards AI


Please enter your comment!
Please enter your name here