It seems that the APFS-locked volume you're trying to recover is possibly damaged so it can't be unlocked and recovered by macOS institutional recovery keys. To fix a corrupted or damaged APFS volume, you can use Disk Utility First Aid to help you. If the FileVault recovery is still not available, you can try data recovery software to recover your FileVault encrypted disk.
Recovering the FileVault locked drive with a data recovery app is much easier compared with using the macOS institutional recovery key. iBoysoft Data Recovery software for Mac is capable of FileVault data recovery, APFS data recovery, and corrupted/damaged file recovery. And this powerful FileVault recovery software is compatible with T2, M2, M1, M1 Pro, and M1 Max Mac.
To perform FileVault recovery with the iBoysoft Data Recovery for Mac program, do the following:
Step 1: Free download, install, and launch iBoysoft Data Recovery for Mac on your Mac.
Step 2: Select Storage Device or Disk Image under Data Recovery.
Step 3: Select the FileVault encrypted APFS volume and click Search for Lost Data to start scanning. iBoysoft Data Recovery software will ask you to enter a password to unlock it.
Here the password is the one you created when you encrypted the disk with FileVault. So just try to recall the correct password or recovery key to unlock the drive. The recovery key is a string of letters and numbers and is hard to remember, you must have kept it somewhere safe.
Note: If you put in an incorrect password, unlocking the drive will fail and the data can't be retrieved. You can try the Mac login password to unlock the FileVault-encrypted disk because the login password is by default the password for FileVault if you have never changed it.
Step 4: Once the scanning process is finished, you can filter these recoverable files by different categories and then preview the scanning results. Select all the files you want to recover and click "Recover" to retrieve them from the FileVault encrypted disk. Rember to not store these recovered files in the same encrypted volume to avoid data loss.