node.js – ECONNRESET – Does not gracefully throw error, but crashes web app

admin

We have a NodeJS app running as a Azure Web App Service on a Linux based App Service Plan. (configured to be running as always on).

Setup:

  • NodeJS 16
  • App Service Plan (Linux)
  • Redis (Azure managed hosted service)
  • Application Insights (Azure managed hosted service)

Packages:

  • express 4.17.2
  • dotenv 14.2.0
  • redis 4.0.2
  • applicationinsights 2.2.0

The web service does basic data calculations returning a result as a REST API service. Redis is used to store previously calculated results.

Application Insights has been enabled on the App Service level in the portal.
For additional fault monitoring, we added the NPM Package applicationinsights version 2.2.0 in code.

Application insights is configured at start up of the app using:

const appInsights = require("applicationinsights");
appInsights.setup(process.env.APPLICATIONINSIGHTS_CONNECTION_STRING)
appInsights.start()

The app service runs for some time but then crashes unexpectedly with the following in the KUDU logs:

2022-01-20T00:41:19.028838008Z events.js:377
2022-01-20T00:41:19.029056811Z       throw er; // Unhandled 'error' event
2022-01-20T00:41:19.029073211Z       ^
2022-01-20T00:41:19.029079111Z 
2022-01-20T00:41:19.029084211Z SocketClosedUnexpectedlyError: Socket closed unexpectedly
2022-01-20T00:41:19.029089512Z     at TLSSocket.<anonymous> (/home/site/wwwroot/node_modules/@node-redis/client/dist/lib/client/socket.js:184:118)
2022-01-20T00:41:19.029095412Z     at Object.onceWrapper (events.js:520:26)
2022-01-20T00:41:19.029100512Z     at TLSSocket.emit (events.js:412:35)
2022-01-20T00:41:19.029105412Z     at net.js:675:12
2022-01-20T00:41:19.029110212Z     at TCP.done (_tls_wrap.js:563:7)
2022-01-20T00:41:19.029115112Z Emitted 'error' event on Commander instance at:
2022-01-20T00:41:19.029128012Z     at RedisSocket.<anonymous> (/home/site/wwwroot/node_modules/@node-redis/client/dist/lib/client/index.js:338:14)
2022-01-20T00:41:19.029149012Z     at RedisSocket.emit (events.js:400:28)
2022-01-20T00:41:19.029154512Z     at RedisSocket._RedisSocket_onSocketError (/home/site/wwwroot/node_modules/@node-redis/client/dist/lib/client/socket.js:207:10)
2022-01-20T00:41:19.029159212Z     at TLSSocket.<anonymous> (/home/site/wwwroot/node_modules/@node-redis/client/dist/lib/client/socket.js:184:107)
2022-01-20T00:41:19.029164013Z     at Object.onceWrapper (events.js:520:26)
2022-01-20T00:41:19.029168413Z     [... lines matching original stack trace ...]
2022-01-20T00:41:19.029172813Z     at TCP.done (_tls_wrap.js:563:7)

I then removed the use of Redis to test the scenario without a external connection, but after some time running, the application still crashes without triggering try/catch code.

I was able to trace the following debug information:

arg0:OperationalError {cause: Error: read ECONNRESET
    at TCP.onStreamRead…nternal/stream_base_commons:220:20)
    at TC…, isOperational: true, errno: -4077, code: 'ECONNRESET', syscall: 'read', …}
cause:Error: read ECONNRESETn    at TCP.onStreamRead (node:internal/stream_base_commons:220:20)n    at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -4077, code: 'ECONNRESET', syscall: 'read', stack: 'Error: read ECONNRESETn    at TCP.onStreamRea…Trampoline (node:internal/async_hooks:130:17)', message: 'read ECONNRESET'}
code:'ECONNRESET'
errno:-4077
isOperational:true
syscall:'read'
message:'read ECONNRESET'
name:'Error'
stack:'Error: read ECONNRESETn    at TCP.onStreamRead (node:internal/stream_base_commons:220:20)n    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)'

My local debug console points me to the file: /node_modulesdiagnostic-channel-publishersdistsrcconsole.pub.js:43:39 which as I understand is used to log console log events to Application Insights*.

I then removed Application Insights and the Web App has been running stable without any crashes. I re-enabled the use of Redis and no issues traced thus far. This points me to the issue being Application Insights not being able to gracefully handle a break in TCP Socket connection to the Application Insights Service.

Any way to confirm this or prevent the app to crash?

Read more here: Source link