If you want to encrypt the external hard drive without formatting, you can choose the encrypt it via FileVault, encrypt the drive in Finder, or use a third-party tool.
FileVault only works to encrypt the internal and external bootable drive, therefore, if your external hard drive is not a bootable disk, without the macOS installer, you can not turn on FileVault on it from System Settings. However, iBoysoft DiskGeerker enables you to encrypt APFS and Mac OS Extended internal/external with FileVault.
The "Encrypt" option in Finder is only available when the non-startup disk is formatted with APFS or Mac OS Extended, therefore, if your drive is exFAT, FAT32, or NTFS, you can't encrypt it there. But, you can try iBoysoft BitLocker for Mac to encrypt exFAT/FAT32/NTFS drive on Mac.
Therefore, the specific method to encrypt a drive on your Mac depends on the file system, whether it is a bootable drive, and whether you want to erase the disk.