node.js – AWS SDK for Javascript v3 S3Client GetObjectCommand gives Error: unexpected end of file

In my AWS Lambda function, I have this code that extracts contents of a zip file in S3 bucket into local file system. It is using nodejs stream to avoid downloading the entire zip file first, to save memory:

import unzipper from 'unzipper';
import { Stream, Readable } from 'stream';
import { s3Client } from 'utils/AWS/s3Client';
import { GetObjectCommand } from '@aws-sdk/client-s3';
const util = require('util');
const pipeline = util.promisify(Stream.pipeline);


const streamUnzipFromS3ToLocal = async (
  objectBucket: string,
  objectKey: string,
  extractDir: string,
) => {
  logger.info(`Start to stream unzip ${objectBucket}:${objectKey} to local.`);
  var start = new Date().getTime();
  try {
    const data = await S3.getFile(objectBucket, objectKey); // data.Body is a readable stream;
    data.Body.on('error', (error) => {
      logger.error(`Error while reading the stream: ${error}`);
      console.log(JSON.stringify(error));
      throw error;
    });
    await pipeline(data.Body, unzipper.Extract({ path: extractDir }));
    logger.info(`${objectKey} is decompressed to ${extractDir}`);
  } catch (error) {
    logger.error(`Failed to stream unzip file ${objectKey}: ${error}`);
    console.log(JSON.stringify(error));
    throw error
  }
  var end = new Date().getTime();
  var time = (end - start) / 1000;
  logger.info(`Finished stream-unzipping ${objectKey} to local. execution time: ${time} sec;`);
  return time;
};

and S3.getFile is as follows:

export const getFile = async (bucket_name, object_key) => {
  let res;
  const params = {
    Bucket: bucket_name,
    Key: object_key,
  };

  try {
    const x = new GetObjectCommand(params);
    const data = await s3Client.send(new GetObjectCommand(params));
    res = data;
  } catch (err) {
    logger.error(`Failed when s3Controller getFile: ${JSON.stringify(err)}`);
    throw err;
  }

  return res;
};

I am getting this error when using the above code to stream unzip a very large file in S3 bucket:

Failed to stream unzip file myapp/a/s/d/2GBfile.zip: Error: unexpected end of file

2023-04-28T07:17:48.745Z    8b7c27dc-3548-42ba-ab40-a7052a1acbe7    INFO    Error: unexpected end of file
    at __node_internal_genericNodeError (node:internal/errors:867:15)
    at Zlib.zlibOnError [as onerror] (node:zlib:189:17) {
  errno: -5,
  code: 'Z_BUF_ERROR'
}

and

Error while reading the stream: Error: aborted

2023-04-28T07:41:09.511Z    bf844ca7-0876-4541-ad0a-b91d518361a1    ERROR   Uncaught Exception  
{
    "errorType": "Error",
    "errorMessage": "aborted",
    "code": "ECONNRESET",
    "stack": [
        "Error: aborted",
        "    at connResetException (node:internal/errors:717:14)",
        "    at TLSSocket.socketCloseListener (node:_http_client:462:19)",
        "    at TLSSocket.emit (node:events:525:35)",
        "    at node:net:322:12",
        "    at TCP.done (node:_tls_wrap:588:7)"
    ]
}

What is going on?

How can I fix this?

Read more here: Source link