rabbitmq – How do you turn on feature flags on a non running node to get it to start?

admin

Looks like RMQ updated last week. According to their docs

Moreover, feature flags have to be enabled before the upgrade. The upgrade will fail if you miss this step.

But it does not have any info that I can find or open tickets on Github regarding what to do if you don’t enable the feature flags prior to updating.

How do you turn on the flags on a non-running node to get it to start?

Notes:

I am on an AWS Ubuntu 22.04.5 LTS. I use ‘supervisor’ to run rabbit and other related services. Occasionally, it suggests running security updates via sudo apt update && sudo apt upgrade -y. I have never had any issues with this. My Erlang is at 26.2.5.4

Logs:

tail -n 355 /var/log/rabbitmq/startup_err

BOOT FAILED
===========
Error during startup: {error,failed_to_initialize_feature_flags_registry}

Runtime terminating during boot (terminating)

Crash dump is being written to: erl_crash.dump...done
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
[os_mon] memory supervisor port (memsup): Erlang has closed

tail -n 355 /var/log/rabbitmq/startup_log 

=INFO REPORT==== 18-Oct-2024::13:57:09.478721 ===
    alarm_handler: {set,{system_memory_high_watermark,[]}}
D/T [error]<0.254.0> Feature flags: `stream_filtering`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.
D/T [error]<0.254.0> Failed to initialize feature flags registry: {disabled_required_feature_flag,
D/T [error]<0.254.0>                                               stream_filtering}
D/T [error]<0.254.0> 
D/T [error]<0.254.0> BOOT FAILED
D/T [error]<0.254.0> ===========
D/T [error]<0.254.0> Error during startup: {error,failed_to_initialize_feature_flags_registry}
D/T [error]<0.254.0> 
D/T [error]<0.253.0>   crasher:
D/T [error]<0.253.0>     initial call: application_master:init/4
D/T [error]<0.253.0>     pid: <0.253.0>
D/T [error]<0.253.0>     registered_name: []
D/T [error]<0.253.0>     exception exit: {failed_to_initialize_feature_flags_registry,
D/T [error]<0.253.0>                         {rabbit,start,[normal,[]]}}
D/T [error]<0.253.0>       in function  application_master:init/4 (application_master.erl, line 142)
D/T [error]<0.253.0>     ancestors: [<0.252.0>]
D/T [error]<0.253.0>     message_queue_len: 1
D/T [error]<0.253.0>     messages: [{'EXIT',<0.254.0>,normal}]
D/T [error]<0.253.0>     links: [<0.252.0>,<0.44.0>]
D/T [error]<0.253.0>     dictionary: []
D/T [error]<0.253.0>     trap_exit: true
D/T [error]<0.253.0>     status: running
D/T [error]<0.253.0>     heap_size: 376
D/T [error]<0.253.0>     stack_size: 28
D/T [error]<0.253.0>     reductions: 170
D/T [error]<0.253.0>   neighbours:
D/T [error]<0.253.0>

sudo service rabbitmq-server start

Job for rabbitmq-server.service failed because the control process exited with error code.
See "systemctl status rabbitmq-server.service" and "journalctl -xeu rabbitmq-server.service" for details.

systemctl status rabbitmq-server.service

● rabbitmq-server.service - RabbitMQ broker
     Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
     Active: activating (start) since Fri 2024-10-18 23:42:16 UTC; 2s ago
   Main PID: 164057 (beam.smp)
     Status: "Standing by"
      Tasks: 25 (limit: 2338)
     Memory: 64.6M
        CPU: 1.495s
     CGroup: /system.slice/rabbitmq-server.service
             ├─164057 /usr/lib/erlang/erts-14.2.5.4/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs >
             ├─164067 erl_child_setup 32768
             ├─164084 sh -s disksup
             ├─164086 /usr/lib/erlang/lib/os_mon-2.9.1/priv/bin/memsup
             ├─164087 /usr/lib/erlang/lib/os_mon-2.9.1/priv/bin/cpu_sup
             ├─164088 /usr/lib/erlang/erts-14.2.5.4/bin/inet_gethost 4
             ├─164089 /usr/lib/erlang/erts-14.2.5.4/bin/inet_gethost 4
             └─164099 /usr/lib/erlang/erts-14.2.5.4/bin/epmd -daemon

D/T ip-187 rabbitmq-server[164057]: 2024-10-18 23:42:18.585056+00:00 [error] <0.255.0>
D/T ip-187 rabbitmq-server[164057]: Runtime terminating during boot (terminating)
D/T ip-187 rabbitmq-server[164057]: 
D/T ip-187 rabbitmq-server[164057]: Crash dump is being written to: erl_crash.dump...done
D/T ip-187 rabbitmq-server[164087]: [os_mon] cpu supervisor port (cpu_sup): Erlang has closed
D/T ip-187 rabbitmq-server[164086]: [os_mon] memory supervisor port (memsup): Erlang has closed
D/T ip-187 systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=1/FAILURE
D/T ip-187 systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
D/T ip-187 systemd[1]: Failed to start RabbitMQ broker.
D/T ip-187 systemd[1]: rabbitmq-server.service: Consumed 1.545s CPU time.

Read more here: Source link