Unzip Large file on AWS S3 using Lamda Functions - amazon-web-services

Question

I have a large file around 6GB and using AWS lambda trigger to unzip the file when it's uploaded to an S3 bucket using Python and Boto3 but I am getting Memory Error while unzipping the file in buffer using ByteIO.
# zip file is in output dir
if '-output' in file.key:
# get base path other the zip file name
save_base_path = file.key.split('//')[0]
# starting unzip process
zip_obj = s3_resource.Object(bucket_name=source_bucket, key=file.key)
buffer = BytesIO(zip_obj.get()["Body"].read())
z = zipfile.ZipFile(buffer)
print(f'Unziping....')
for filename in z.namelist():
file_info = z.getinfo(filename)
try:
response = s3_resource.meta.client.upload_fileobj(
z.open(filename),
Bucket=target_bucket,
Key=f'{save_base_path}/{filename}'
)
except Exception as e:
print(e)
print('unziping process completed')
# deleting zip file after unzip
s3_resource.Object(source_bucket, file.key).delete()
my_bucket.delete_object()
print("iteration completed")
else:
print('Zip file invalid position')
s3_resource.Object(source_bucket, file.key).delete
print(f'{file.key} deleted...')
Issue 1
when I am reading bytes its give me Memory Error
I have set Memory to 10240(10GB) in the General Configuration of AWS lambda function
Issue 2
I want to delete Object from s3 it runs code properly and doesn't give any error but also it is not deleting the file
is there any solution through which I can solve my Unzip issue

Answer 1

It is possible to wrap both the reading of the file in a small wrapper so that the entire zip file does not need to be downloaded from S3. From there it's straight forward enough to upload completed files back to S3 without keeping the entire contents in RAM:
# Download a zip file from S3 and upload it's unzipped contents back
# to S3
def s3_zip_to_s3(source_bucket, source_key, dest_bucket, dest_prefix):
s3 = boto3.client('s3')
# Use the S3Wrapper class to avoid having to transfer the entire
# file into RAM
with zipfile.ZipFile(S3Wrapper(s3, source_bucket, source_key)) as zip:
for name in zip.namelist():
print(f"Uploading {name}...")
# Use upload_fileobj to only stream to S3
s3.upload_fileobj(zip.open(name, 'r'), dest_bucket, dest_prefix + name)
# Create a file like object with a bare-bones implementation
# for reading only. Other than caching the file size, data is read
# from S3 for each call
class S3Wrapper:
def __init__(self, s3, bucket, key):
self.s3 = s3
self.bucket = bucket
self.key = key
self.pos = 0
self.length = s3.head_object(Bucket=bucket, Key=key)['ContentLength']
def seekable(self):
return True
def seek(self, offset, whence=0):
if whence == 0:
self.pos = offset
elif whence == 1:
self.pos += offset
else:
self.pos = self.length + offset
def tell(self):
return self.pos
def read(self, count=None):
if count is None:
resp = self.s3.get_object(Bucket=self.bucket, Key=self.key, Range=f'bytes={self.pos}-')
else:
resp = self.s3.get_object(Bucket=self.bucket, Key=self.key, Range=f'bytes={self.pos}-{self.pos+count-1}')
data = resp['Body'].read()
self.pos += len(data)
return data
While this works, this technique uses much less than the size of the zip file in my testing, it's not fast by any means.
I'd probably recommend some solution like a worker on EC2 or ECS to do the work for you.