66 * 16777216 + 226 * 65536 + 18 * 256 + 71
I had relied on this formula in the past to suggest solutions to a customer in the past, which means I trusted it to do it's job!
I used the standard 'calc' utility from friendly Vista to do the calculation, and what answer does it return!!! :
18577352254558791
which at first sight I caught that was not in the 32 bit integer range!!! If that was the case, if this formula was wrong, the customer's app was screwed!! and implicitly, so was I!!!
I started checking, and cross-checking the calculation. First I turned to the Postgres database I had implemented this formula in:
select 66 * 16777216 + 226 * 65536 + 18 * 256 + 71;
The answer was 1122112071, which matched the customer's old (slow) function's result.
Then upon a few minutes of research, I figured that calc's scientific mode was producing correct result and the standard mode was not.
The difference, apparently, is because of the way both modes handle the operators. The Standard mode interprest the equation as:
((((((66 * 16777216 ) + 226 ) * 65536 ) + 18 ) * 256 ) + 71 )
and the scientific mode interpreted it as:
((66 * 16777216) + (226 * 65536) + (18 * 256) + 71 )
I tried the same expression with MinGW's 'expr' utility too and got the correct answer.
Must I say I was relieved to find that calc was crazy in "Standard" mode!!!
