node.js – AWS SDK for Javascript v3 S3Client GetObjectCommand gives Error: unexpected end of file
In my AWS Lambda function, I have this code that extracts contents of a zip file in S3 bucket into local file system. It is using nodejs stream to avoid downloading the entire zip file first, to save memory:
import unzipper from 'unzipper';
import { Stream, Readable } from 'stream';
import { s3Client } from 'utils/AWS/s3Client';
import { GetObjectCommand } from '@aws-sdk/client-s3';
const util = require('util');
const pipeline = util.promisify(Stream.pipeline);
const streamUnzipFromS3ToLocal = async (
objectBucket: string,
objectKey: string,
extractDir: string,
) => {
logger.info(`Start to stream unzip ${objectBucket}:${objectKey} to local.`);
var start = new Date().getTime();
try {
const data = await S3.getFile(objectBucket, objectKey); // data.Body is a readable stream;
data.Body.on('error', (error) => {
logger.error(`Error while reading the stream: ${error}`);
console.log(JSON.stringify(error));
throw error;
});
await pipeline(data.Body, unzipper.Extract({ path: extractDir }));
logger.info(`${objectKey} is decompressed to ${extractDir}`);
} catch (error) {
logger.error(`Failed to stream unzip file ${objectKey}: ${error}`);
console.log(JSON.stringify(error));
throw error
}
var end = new Date().getTime();
var time = (end - start) / 1000;
logger.info(`Finished stream-unzipping ${objectKey} to local. execution time: ${time} sec;`);
return time;
};
and S3.getFile
is as follows:
export const getFile = async (bucket_name, object_key) => {
let res;
const params = {
Bucket: bucket_name,
Key: object_key,
};
try {
const x = new GetObjectCommand(params);
const data = await s3Client.send(new GetObjectCommand(params));
res = data;
} catch (err) {
logger.error(`Failed when s3Controller getFile: ${JSON.stringify(err)}`);
throw err;
}
return res;
};
I am getting this error when using the above code to stream unzip a very large file in S3 bucket:
Failed to stream unzip file myapp/a/s/d/2GBfile.zip: Error: unexpected end of file
2023-04-28T07:17:48.745Z 8b7c27dc-3548-42ba-ab40-a7052a1acbe7 INFO Error: unexpected end of file
at __node_internal_genericNodeError (node:internal/errors:867:15)
at Zlib.zlibOnError [as onerror] (node:zlib:189:17) {
errno: -5,
code: 'Z_BUF_ERROR'
}
and
Error while reading the stream: Error: aborted
2023-04-28T07:41:09.511Z bf844ca7-0876-4541-ad0a-b91d518361a1 ERROR Uncaught Exception
{
"errorType": "Error",
"errorMessage": "aborted",
"code": "ECONNRESET",
"stack": [
"Error: aborted",
" at connResetException (node:internal/errors:717:14)",
" at TLSSocket.socketCloseListener (node:_http_client:462:19)",
" at TLSSocket.emit (node:events:525:35)",
" at node:net:322:12",
" at TCP.done (node:_tls_wrap:588:7)"
]
}
What is going on?
How can I fix this?
Read more here: Source link