Cross-compiling OpenSSL and FFMPEG for CM3+?

I have created myself a cross-compiler (look at this thread).

OS is 64bit, compiler is targeting CM3+.

To avoid clicking I repeat here my compiler’s capabilities:

  -mabi=                                lp64
  -march=                               armv8-a
  -mbig-endian                          [disabled]
  -mbionic                              [disabled]
  -mbranch-protection=
  -mcmodel=                             small
  -mcpu=                                cortex-a53+crc+simd
  -mfix-cortex-a53-835769               [enabled]
  -mfix-cortex-a53-843419               [enabled]
  -mgeneral-regs-only                   [disabled]
  -mglibc                               [enabled]
  -mharden-sls=
  -mlittle-endian                       [enabled]
  -mlow-precision-div                   [disabled]
  -mlow-precision-recip-sqrt            [disabled]
  -mlow-precision-sqrt                  [disabled]
  -mmusl                                [disabled]
  -momit-leaf-frame-pointer             [enabled]
  -moutline-atomics                     [enabled]
  -moverride=
  -mpc-relative-literal-loads           [enabled]
  -msign-return-address=                none
  -mstack-protector-guard-offset=
  -mstack-protector-guard-reg=
  -mstack-protector-guard=              global
  -mstrict-align                        [disabled]
  -msve-vector-bits=            scalable
  -mtls-dialect=                        desc
  -mtls-size=                           24
  -mtrack-speculation                   [disabled]
  -mtune=                               cortex-a53
  -muclibc                              [disabled]
  -mverbose-cost-dump                   [disabled]

  Known AArch64 ABIs (for use with the -mabi= option):
    ilp32 lp64

  Supported AArch64 return address signing scope (for use with -msign-return-address= option):
    all non-leaf none

  The code model option names for -mcmodel:
    large small tiny

  Valid arguments to -mstack-protector-guard=:
    global sysreg

  The possible SVE vector lengths:
    1024 128 2048 256 512 scalable

  The possible TLS dialects:
    desc trad

I wish to cross-compile both FFMPEG and OpenSSL to run as-fast-as-possible.

Can I add/change something to/in the compiler config?
Besides -O3 what else would be feasible to add to compile/link config?

I’m thinking on -flto maybe -ffast-math (not suitable for OpenSSL?), -funroll-loops, and -fno-plt (in case of static libs).

What would you add/remove from these?