The issue was addressed with improved checks. This issue is fixed in Safari 17.5, macOS Sonoma 14.5, visionOS 1.2. A website's permission dialog may persist after navigation away from the site.
https://support.apple.com/kb/HT214108
https://support.apple.com/kb/HT214106
https://support.apple.com/kb/HT214103
https://support.apple.com/en-us/HT214108
https://support.apple.com/en-us/HT214106
https://support.apple.com/en-us/HT214103
https://support.apple.com/en-us/120906
https://support.apple.com/en-us/120903