target/i386: fix fbstp handling of out-of-range values

The fbstp implementation fails to check for out-of-range and invalid
values, instead just taking the result of conversion to int64_t and
storing its sign and low 18 decimal digits. Fix this by checking for
an out-of-range result (invalid conversions always result in INT64_MAX
or INT64_MIN from the softfloat code, which are large enough to be
considered as out-of-range by this code) and storing the packed BCD
indefinite encoding in that case.

Backports commit 374ff4d0a3c2cce2bc6e4ba8a77eaba55c165252 from qemu
This commit is contained in:
Joseph Myers 2020-06-15 13:09:22 -04:00 committed by Lioncash
parent 477a0af161
commit ceaa77e576

View File

@ -698,6 +698,16 @@ void helper_fbst_ST0(CPUX86State *env, target_ulong ptr)
val = floatx80_to_int64(ST0, &env->fp_status);
mem_ref = ptr;
if (val >= 1000000000000000000LL || val <= -1000000000000000000LL) {
float_raise(float_flag_invalid, &env->fp_status);
while (mem_ref < ptr + 7) {
cpu_stb_data_ra(env, mem_ref++, 0, GETPC());
}
cpu_stb_data_ra(env, mem_ref++, 0xc0, GETPC());
cpu_stb_data_ra(env, mem_ref++, 0xff, GETPC());
cpu_stb_data_ra(env, mem_ref++, 0xff, GETPC());
return;
}
mem_end = mem_ref + 9;
if (SIGND(temp)) {
cpu_stb_data_ra(env, mem_end, 0x80, GETPC());