To encrypt a drive on a Mac, you can apply the macOS built-in tools or other third-party programs. The generated password is used to decrypt the drive so that you can access your files. And it is hard to decrypt the drive if you forget the password.
Which type of encryption did you use on the external hard drive? You should try to recall the password you used to encrypt the drive while formatting it in Disk Utility or via the Finder's Encrypt feature. If you turn on FileVault encryption for the drive, you can recover FileVault recovery key in multiple ways. If you apply another third-party app to encrypt the drive, go to the app to see whether you can recover the password with the hints you set.
Have you backed up the drive before encryption? If so, recover your files from the backup without a password. In case you can't find the encryption password, the drive data is inaccessible, the only way to use it is reformatting.
