Detailed info on the RSSI and BGNDRSSI register can be found on the AX5043 Programming Manual.
As shown in Figure 12 (page 18) the RSSI is evaluated by the receiver chain right after the AGC settles. The RSSI expressed in dBm is obtained from the AGCCOUNTER.
A first order RSSI Approximation is −AGCCOUNTER · 0.75dB − 38dBm. This approximation degrades at low S/N and low bitrates. A better RSSI approximation considers also the gain set in the digital domain: −AGCCOUNTER · 0.75dB + 20 log10(TRKAMPL)dB + constdBm .
The BGNDRSSI is calculated the same way, just that is done when there is no signal.