By using AWS re:Post, you agree to the Terms of Use

Welcome to AWS re:Post

re:Post gives you access to a vibrant community that helps you become even more successful on AWS

Learn AWS faster by following popular topics

see all
1/18

Recent questions

see all
1/18

Boto3 - S3 - Downloads Occasionally failing

Hi Team, I have written a python program that copies videos from S3 bucket to an EC2 machine, videos that are more than 100 MB in size. While the program runs fine most of the time, but sometimes the download of the file fails with the error, wherein the name of the file is changed to video_name+<hash_characters>. After doing wee bit of research I found out the boto3 library of python, downloads the file temp with a hash name and then replaces it with the name of the file once the download is completed. Below is the log when the download fails : : ERROR/ForkPoolWorker-3] Task api.service.push_to_async_report_generation[25fdce51-a067-4893-bf4c-22013368e59f] raised unexpected: FileNotFoundError(2, 'No such file or directory') Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 451, in trace_task R = retval = fun(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 734, in __protected_call__ return self.run(*args, **kwargs) File "/home/ubuntu/src/api/service.py", line 100, in push_to_async_report_generation image_path = bucket.download_file(Kwargs["file_name"]) File "/home/ubuntu/src/utils/s3_services.py", line 81, in download_file self.bucket.download_file(file_name, os.path.join('./static',file_name)) File "/usr/local/lib/python3.8/dist-packages/boto3/s3/inject.py", line 244, in bucket_download_file return self.meta.client.download_file( File "/usr/local/lib/python3.8/dist-packages/boto3/s3/inject.py", line 170, in download_file return transfer.download_file( File "/usr/local/lib/python3.8/dist-packages/boto3/s3/transfer.py", line 307, in download_file future.result() File "/usr/local/lib/python3.8/dist-packages/s3transfer/futures.py", line 106, in result return self._coordinator.result() File "/usr/local/lib/python3.8/dist-packages/s3transfer/futures.py", line 265, in result raise self._exception File "/usr/local/lib/python3.8/dist-packages/s3transfer/tasks.py", line 126, in __call__ return self._execute_main(kwargs) File "/usr/local/lib/python3.8/dist-packages/s3transfer/tasks.py", line 150, in _execute_main return_value = self._main(**kwargs) File "/usr/local/lib/python3.8/dist-packages/s3transfer/download.py", line 571, in _main fileobj.seek(offset) File "/usr/local/lib/python3.8/dist-packages/s3transfer/utils.py", line 367, in seek self._open_if_needed() File "/usr/local/lib/python3.8/dist-packages/s3transfer/utils.py", line 350, in _open_if_needed self._fileobj = self._open_function(self._filename, self._mode) File "/usr/local/lib/python3.8/dist-packages/s3transfer/utils.py", line 261, in open return open(filename, mode) FileNotFoundError: [Errno 2] No such file or directory: './static/example.mp4.F5e1dAFE It seems like the S3transfer package is trying to open the file before it gets downloaded. Any help related to the topic is appreciated! Thanks!
0
answers
0
votes
3
views
asked 11 minutes ago