Can red5 be used instead of FFmpeg in Android?

No you can’t compare those tools.
Red5 is simply for streaming media (MP4, MP3, FLV) and delivering media with the RTMP protocol.
If you are looking for a RTMP client for Android then maybe Red5 has something to offer.
But FFMPEG is focused on transcoding/converting streams from one format to another.
The only thing those tools have in common is that in recent versions of FFMPEG they have added support to stream over RTMP. But thats it.
Xuggler was (or is still) a project that integrates FFMPEG with Red5. But I think they removed Red5 since FFMPEG has RTMP streaming capabilities.