In macOS Recovery Mode, you can't access the FileVault drive data directly, but you can disable FileVault with Terminal and clone it to another external drive so that you can access the drive files on another healthy Mac.
It seems that your iMac is unbootable, so you can't boot it up normally and turn off FileVault in System Settings. Follow this post to disable FileVault on Mac with Terminal in macOS Recovery Mode. Then, you can use Disk Utility to clone your drive to an external disk, here's how:
- Select Disk Utility from the macOS Utilities screen.
- Choose the destination volume from the left side list.
- Select Restore from the Edit menu of Disk Utility.
- Select the decrypted volume from the drop-down menu.
- Then click on the Restore button.
In case, you can't disable FileVault in recovery, the only option is to erase your startup disk and reinstall macOS, however erasing your Mac will delete all data on it, therefore you should first get your data off by running iBoysoft Data Recovery for Mac in macOS Recovery mode, the FileVault recovery key is needed to unlock the drive.