truncation of values when casting a decimal(38,10) to float in pyspark - casting

I have a column of type decimal(38,10) with value as - 2.1400001000
I am trying to cast it to float using cast(FloatType()). this is resulting in truncation of the value to - 2.14
why is this happening and what can I do to load the value as it is. The final column type has to be of float type.
Regards and Thanks,
Lakshmi Venkata Sai Bollineni

Related

Warning about arithmetic overflow when multiplying numbers

I'm writing a program to calculate the result of numbers:
int main()
{
float a, b;
cin >> a >> b;
float result = b + a * a * 0.4;
cout << result;
}
but I have a warning at a * a and it said Warning C26451 Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2). Sorry if this a newbie question, can anyone help me with this? Thank you!
In the C language as described in the first edition of K&R, all floating-point operations were performed by converting operands to a common type (specifically double), performing operations with that type, and then if necessary converting the result to whatever type was needed. On many platforms, that was the most convenient and space-efficient way of handling floating-point math. While the Standard still allows implementations to behave that way, it also allows implementations to perform floating-point operations on smaller types to be performed using those types directly.
As written, the subexpression a * a * 0.5; would be performed by multiplying a * a together using float type, then multiply by a value 0.5 which is of type double. This latter multiplication would require converting the float result of a * a to double. If e.g. a had been equal to 2E19f, then performing the multiply using type float would yield a value too large to be represented using that type. Had the code instead performed the multiplication using type double, then the result 4E38 would be representable in that type, and the result of multiplying that by 0.5 (i.e. 2E38) would be within the range that is representable by float.
Note that in this particular situation, the use of float for the intermediate computations would only affect the result if a was within narrow ranges of very large or very small. If instead of multiplying by 0.5 one had multiplied by other values, however, the choice of whether to use float or double for the first multiplication could affect the accuracy of rounding. Generally, using double for both multiplies would yield slightly more accurate results, but at the expense of increased execution time. Using float for both may yield better execution speed, but at the result of reduced precision. If the floating-point constant had been something that isn't precisely representable in float, converting to double and multiplying by a double constant may yield slightly more accurate results than using float for everything, but in most cases where one would want that precision, one would also want the increased position that would be achieved by using double for the first multiply as well.
Let's look at the error message.
Using operator '*' on a 4 byte value
It is describing this code:
a * a
Your float is 4 bytes. The result of the multiplication is 4 bytes. And the result of a multiplication may overflow.
and then casting the result to a 8 byte value.
It is describing this code:
(result) * 0.4;
Your result is 4 bytes. 0.4 is a double, which is 8 bytes. C++ will promote your float result to a double before performing this multiplication.
So...
The compiler is observing that you are doing float math that could overflow and then immediately converting the result to a double, making the potential overflow unnecessary.
Change the code to this to remove the float to double to float conversions.
float result = b + a * a * 0.4f;
I read the question as "how to change the code to remove the warning?".
If you take the advice in the warning's text literally:
float result = b + (double)a * a * 0.4;
But this is nonsense — if an overflow happens, your result will probably not fit into float result.
It looks like in your case overflow is not possible, and you feel perfectly fine doing all calculations with float. If so, just write 0.4f instead of 0.4 — then the constant will have float type (instead of double), and the result will also be float.
If you want to "fix" the problem with overflow
double result = b + (double)a * a * 0.4;
But then you must also change the following code, which uses the result. And you don't remove the possibility of overflow, you just make it much less likely.

Multiplying float with double overflow in C++

I am a bit confused of the point of having this warning:
Arithmetic overflow: Using operator '' on a 4byte value and then casting the result to a 8byte value. Cast the value to the wider type before calling '' operator to avoid overflow.
#include <iostream>
using std::cin;
using std::cout;
using std::ios_base;
int main() {
cout.setf(ios_base::fixed, ios_base::floatfield);
double mints = 10.0 / 3.0;
const float c_MILLION = 1e6;
cout << "\n10 million mints: " << 10 * c_MILLION * mints;
cin.get();
}
According to my understanding when we multiply a float value with a double value we are basicaly multiplying a 4byte value to an 8byte value and it we will hence, lose some precision according to the links that I have read:
Cannot implicitly convert type 'double' to 'float'
Multiply a float with a double
http://www.cplusplus.com/articles/DE18T05o/#:~:text=Overflow%20is%20a%20phenomenon%20where,be%20larger%20than%20the%20range
However, when I do output this, I get a double value
https://i.stack.imgur.com/EOQzm.png
If that is the case, why does it bother to warn me to cast c_MILLION to double value if it is automatically changing it to a double result? It cant convert an 8byte value to a 4byte value anyways. So, why does it bother to warn the programmers when it is already saving us from this trouble? Or can it convert an 8byte value to a 4byte value as well. If so, how does it determine what type to print? This is a question that I cannot find the answer to from the links I read.
If it automatically converting the result to 8byte value, what is the point of displaying this warning?
Here is the warning:
https://i.stack.imgur.com/L2szy.png
Severity Code Description Project File Line Suppression State
Warning C26451 Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2)
The problem was that I was multiplying an int value with a double value. But still the warning should not exist when it automatically converts the multiplication of an int to double to a double value.
The warning is because of this multiplication: 10 * c_MILLION. There can be some values of c_MILLION where some precision is lost that would not have been lost if c_MILLION was first converted to a double. Since the result of this multiplication is converted to double, a mistaken programmer might assume that no precision was lost beyond what might be expected if the operands were double in the first place. Hence the warning.

Why is that type double instead of float?

Problem: In a homework problem (it is to be done on paper with a pen so no coding) I must determine the type and value of an addition performed in C++.
1 + 0.5
What I've answered is:
Type float (because I thought that integer + float = float)
Value 1.5 (As far as I know when two different datatypes are added,
the result of the addition is going to be converted to the datatype that does not loose any information.)
Solution says:
Type: double
Value: 1.5
My Question: Why is 0.5 a double and not a float? How can I distingish between a float and a double? I mean, 0.5 looks to me like a float and a double.
First of all, yes. integer + float = float. You are correct about that part.
The issue is not with that, but rather your assumption that 0.5 is a float. It is not. In C++, float literals are followed by an f meaning that 0.5f is a float. However, 0.5 is actually a double. That means that your equation is now:
integer + double = double
As you can see, this result is a double. That is why the correct answer to your question is that the resulting type is a double.
By the way, to clear the record, technically what's going on here isn't integer + double = double. What is happening is that the 1 is falling subject to implicit conversion. Essentially, the 1 is converted to a double, since the other side of the operation is a double as well. This way, the computer is adding the same types and not different ones. That means that the actual addition taking place here is more like:
double + double = double
In C++, floating point literals without a type suffix are double by default. If you want it to be float, you need to specify the f suffix, like 0.5f.

Confusion about float data type declaration in C++

a complete newbie here. For my school homework, I was given to write a program that displays -
s= 1 + 1/2 + 1/3 + 1/4 ..... + 1/n
Here's what I did -
#include<iostream.h>
#include<conio.h>
void main()
{
clrscr();
int a;
float s=0, n;
cin>>a;
for(n=1;n<=a;n++)
{
s+=1/n;
}
cout<<s;
getch();
}
It perfectly displays what it should. However, in the past I have only written programs which uses int data type. To my understanding, int data type does not contain any decimal place whereas float does. So I don't know much about float yet. Later that night, I was watching some video on YouTube in which he was writing the exact same program but in a little different way. The video was in some foreign language so I couldn't understand it. What he did was declared 'n' as an integer.
int a, n;
float s=0;
instead of
int a
float s=0, n;
But this was not displaying the desired result. So he went ahead and showed two ways to correct it. He made changes in the for loop body -
s+=1.0f/n;
and
s+=1/(float)n;
To my understanding, he declared 'n' a float data type later in the program(Am I right?). So, my question is, both display the same result but is there any difference between the two? As we are declaring 'n' a float, why he has written 1.0f instead of n.f or f.n. I tried it but it gives error. And in the second method, why we can't write 1(float)/n instead of 1/(float)n? As in the first method we have added float suffix with 1. Also, is there a difference between 1.f and 1.0f?
I tried to google my question but couldn't find any answer. Also, another confusion that came to my mind after a few hours is - Why are we even declaring 'n' a float? As per the program, the sum should come out as a real number. So, shouldn't we declare only 's' a float. The more I think the more I confuse my brain. Please help!
Thank You.
The reason is that integer division behaves different than floating point division.
4 / 3 gives you the integer 1. 10 / 3 gives you the integer 3.
However, 4.0f / 3 gives you the float 1.3333..., 10.0f / 3 gives you the float 3.3333...
So if you have:
float f = 4 / 3;
4 / 3 will give you the integer 1, which will then be stored into the float f as 1.0f.
You instead have to make sure either the divisor or the dividend is a float:
float f = 4.0f / 3;
float f = 4 / 3.0f;
If you have two integer variables, then you have to convert one of them to a float first:
int a = ..., b = ...;
float f = (float)a / b;
float f = a / (float)b;
The first is equivalent to something like:
float tmp = a;
float f = tmp / b;
Since n will only ever have an integer value, it makes sense to define it as as int. However doing so means that this won't work as you might expect:
s+=1/n;
In the division operation both operands are integer types, so it performs integer division which means it takes the integer part of the result and throws away any fractional component. So 1/2 would evaluate to 0 because dividing 1 by 2 results in 0.5, and throwing away the fraction results in 0.
This in contrast to floating point division which keeps the fractional component. C will perform floating point division if either operand is a floating point type.
In the case of the above expression, we can force floating point division by performing a typecast on either operand:
s += (float)1/n
Or:
s += 1/(float)n
You can also specify the constant 1 as a floating point constant by giving a decimal component:
s += 1.0/n
Or appending the f suffix:
s += 1.0f/n
The f suffix (as well as the U, L, and LL suffixes) can only be applied to numerical constants, not variables.
What he is doing is something called casting. I'm sure your school will mention it in new lectures. Basically n is set as an integer for the entire program. But since integer and double are similar (both are numbers), the c/c++ language allows you to use them as either as long as you tell the compiler what you want to use it as. You do this by adding parenthesis and the data type ie
(float) n
he declared 'n' a float data type later in the program(Am I right?)
No, he defined (thereby also declared) n an int and later he explicitly converted (casted) it into a float. Both are very different.
both display the same result but is there any difference between the two?
Nope. They're the same in this context. When an arithmetic operator has int and float operands, the former is implicitly converted into the latter and thereby the result will also be a float. He's just shown you two ways to do it. When both the operands are integers, you'd get an integer value as a result which may be incorrect, when proper mathematical division would give you a non-integer quotient. To avoid this, usually one of the operands are made into a floating-point number so that the actual result is closer to the expected result.
why he has written 1.0f instead of n.f or f.n. I tried it but it gives error. [...] Also, is there a difference between 1.f and 1.0f?
This is because the language syntax is defined thus. When you're declaring a floating-point literal, the suffix is to use .f. So 5 would be an int while 5.0f or 5.f is a float; there's no difference when you omit any trailing 0s. However, n.f is syntax error since n is a identifier (variable) name and not a constant number literal.
And in the second method, why we can't write 1(float)/n instead of 1/(float)n?
(float)n is a valid, C-style casting of the int variable n, while 1(float) is just syntax error.
s+=1.0f/n;
and
s+=1/(float)n;
... So, my question is, both display the same result but is there any difference between the two?
Yes.
In both C and C++, when a calculation involves expressions of different types, one or more of those expressions will be "promoted" to the type with greater precision or range. So if you have an expression with signed and unsigned operands, the signed operand will be "promoted" to unsigned. If you have an expression with float and double operands, the float operand will be promoted to double.
Remember that division with two integer operands gives an integer result - 1/2 yields 0, not 0.5. To get a floating point result, at least one of the operands must have a floating point type.
In the case of 1.0f/n, the expression 1.0f has type float1, so the n will be "promoted" from type int to type float.
In the case of 1/(float) n, the expression n is being explicitly cast to type float, so the expression 1 is promoted from type int to float.
Nitpicks:
Unless your compiler documentation explicitly lists void main() as a legal signature for the main function, use int main() instead. From the online C++ standard:
3.6.1 Main function
...
2 An implementation shall not predefine the main function. This function shall not be overloaded. It shall have a declared return type of type int, but otherwise its type is implementation-defined...
Secondly, please format your code - it makes it easier for others to read and debug. Whitespace and indentation are your friends - use them.
1. The constant expression 1.0 with no suffix has type double. The f suffix tells the compiler to treat it as float. 1.0/n would result in a value of type double.

Unwanted rounding C++

I have the following equations:
//get thermistor resistor value
temp=(THERMISTOR_R0)/((temp2/temp)-1);
//get temperature value in Kelvins and convert to Celsiuis
temp=(THERMISTOR_BETA)/log(temp/(THERMISTOR_R0*exp((-THERMISTOR_BETA)/298)));
temp-=273;
desiredVoltage =((15700-(25*temp))/10);
THERMISTOR_R0 and THERMISTOR _BETA are constant.
temp, temp2 and desiredVoltage are unsigned int and are defined before calculations.
The problem is, for example, when the term ((temp2/temp)-1) falls below 1, it rounds down to 0. I want to get rid of this rounding as it is causing huge problems with my calculations.
How do I do this?
It's not rounding, it's integer division. If both operands of the / operator are of integer types the behavior of C++ is to perform an integer division, which keeps only the integer part of the result (this is often needed in some algorithms because it's faster).
To get a "regular" division make sure that at least one of the operands involved is of a floating point type (float, double or long double); you can do this either declaring the variables involved as FP types
double temp2, temp;
either sticking a cast in front of one of the operands.
temp=(THERMISTOR_R0)/((double(temp2)/temp)-1);
(notice that here you'll incur in truncation if temp is still of integral type).
Most probably, here you'll simply want to declare temp and temp2 as double (or float if you are working in a really resource-tight environment).
Also, when dividing by a numeric literal, keep in mind that if you don't write the decimal point it will be an int literal, if you write it it will be a double. E.g., 298 is an int, 298. is a double, so 1/2 is 0, but 1/2. is 0.5.
Make sure that you use floating point types if you want floating point division behaviour.
The compiler rounds floating point values down.
Generally this problem is solved by adding 0.5 to the value before the conversion to an integer. You may want to explicitly use floating point values (or cast), then add 0.5 to the result before you cast back to integers.