After setting up ColdFusion 2021 I found that when ParseDateTime is being used to format a datetime value from SQL Server it won't format the date as an ODBC literal like ColdFusion 2016 does, e.g. {ts '2021-05-15 13:20:51'}. Instead it just outputs the value unchanged from the database.
Here is my environment:
Coldfusion 2016 Server:
Version: 2016,0,17,325979
Tomcat Version: 8.5.61.0
Edition: Developer
Operating System: Windows 7
Coldfusion 2021 Server:
Version: 2021,0,01,325996
Tomcat Version: 9.0.41.0
Edition: Developer
Operating System: Windows 10
The database is SQL Server 2008 R2 and ColdFusion datasource is using the MS SQL Server driver.
An example query:
<cfquery name="qryDates">
select id, expiry_date
from purchases
</cfquery>
<cfoutput>#ParseDateTime(qryDates.expiry_date)#</cfoutput>
Coldfusion 2016 output:
{ts '2021-05-15 13:20:51'}
Coldfusion 2021 output:
2021-05-15 13:20:51.0
Is there a way to get ParseDateTime to behave the way it does on my CF 2016 and every other CF server I have used in the past? I do not want to have to change the formatting to accomplish this (e.g. #DateTimeFormat(ParseDateTime(qryDates.expiry_date))#).
Thank you for any advice.
I don't have any way to test my assumptions right now, but I assume you're seeing the side effects of the way ParseDateTime() handles a Java object, and the differences of that function between CF2016 and CF2021/18.
Since expriry_date is an actual datetime object in your database, cfquery from the database will give you a sql object instead of just a string that looks like a date. Since it has both date and time components, I believe that JDBC will send a java.sql.Timestamp object back to CF.
The purpose of ParseDateTime() is for converting a string to a date object, and even though it can "format" a date object, that's not what it's intended for. It's initial assumption is that it's being given a string, and because CF is dynamically typed, it will usually try to implicitly convert a value into the correct type. And since the database value is date-ish, CF will try to make it appropriately look like a string. At least that's the way it was intended to work. And it looks like that is what is happening in <CF2016 and Lucee, and the value is getting a "{ts...".
It appears that CF2021/18 is receiving a date object and not really doing much with it. It seems to be just passing the Java object (without the "{ts...") back out, and when ColdFusion tries to convert it to a string, it's not a ColdFusion timestamp variable so there is no "{ts...".
DEMO CODE:
<cfscript>
/*** CREATE QUERY OBJECT ***/
theDateQuery = QueryNew("dt","timestamp",[{"dt":createODBCDateTime('2021-04-09 12:01:02.345')}]) ;
writeDump(theDateQuery);
/*** SET QUERY RESULT TO A VARIABLE ***/
dobj = theDateQuery.dt[1] ;
writeOutput("dobj: " & dobj & " ("& getMetadata(dobj).getName() & ")<br>");
/*** CONVERT QUERY VARIABLE TO A STRING ***/
dstr = theDateQuery.dt[1].toString() ;
writeOutput("dstr: " & dstr & " ("& getMetadata(dstr).getName() & ")<br>");
/*** PARSEDATETIME OF OBJECT ***/
x = parseDateTime(dobj) ;
writeOutput("x: " & x & " ("& getMetadata(x).getName() & ")<br>");
/*** PARSEDATETIME OF STRING ***/
y = parseDateTime(dstr) ;
writeOutput("y: " & y & " ("& getMetadata(y).getName() & ")<br>");
</cfscript>
CF 2016 and lower
https://trycf.com/gist/c9c35dccb04f91cd6c06e4082ed306ca/acf2016?theme=monokai
Your database is giving the ColdFusion query back a Java object, which is going into the ParseDateTime function, and CF is doing its magic then giving you back a ColdFusion date object, which looks like a CF timestamp that is easily coerced into what you want it to display.
dobj: 2021-04-09 12:01:02.345 (java.sql.Timestamp)
x: {ts '2021-04-09 12:01:02'} (coldfusion.runtime.OleDateTime)
CF2021 (and CF2018)
https://trycf.com/gist/4c9bbc036a63a135962f0912b8591e00/acf2021?theme=monokai (trycf link for CF2021 seems to be defaulting to Lucee5, so you'll have to reselect CF2021)
It looks like the behavior changed slightly in CF2018. I don't know the inner workings of the ParseDateTime functions, but I'm assuming that in earlier versions, it would convert the date object to a date string before it converted it into a ColdFusion object on return. As of 2018, it appears that if it receives a Java data object, it recognizes that it's already a date object and just returns it (bypassing a conversion to a ColdFusion date string and then object). This is seems to be more in line with the originally stated behavior. And since the Java timestamp value doesn't have "{ts...", but CF does, it changes the way a string us ultimately created from the function's return value. This is possibly a compatibility bug (since the behavior is subtly different), but I'm not completely sure.
Regardless, it seems to be caused by the simple fact that ParseDateTime is being used here for something other than its intended purpose. Just like using TimeFormat(now(),"yyyy-MMM-dd hh:nn:ss"). ACF will return a string with the Year and Day masked, even though that's not something that TimeFormat() is intended for (NOTE: Lucee seems to handle it correctly). If a function is used in an unexpected manner, it shouldn't be expected to return consistent results in different versions.
And that's where I get kinda iffy on whether it should be a bug or not. In the past, most ColdFusion functions didn't much care if a "date" was a string or an object. It seems that CF2018+ does. Or at least treats them differently.
Ultimately, your code should probably be fixed to use functions the way they were meant to be used. However, I understand that might be a bit much. You might be able to fix your current issue by explicitly converting the query object into a string before you parse it.
ParseDateTime(qryDates.expiry_date.toString())
One other thing to note is that in CF2016, if your datetime value had a millisecond component, that would be stripped off in your final ParseDateTime value. However, in CF2021/18, since the Java value doesn't appear to be changed, the milliseconds component will still be in your return value. toString() should also fix that, and you'll get exactly what you had before.
Related
<cfscript>
SetTimeZone("Asia/Karachi"); //same as our windows server timezone
</cfscript>
<cfoutput>
<cfset mydate = 'June 01, 2008'>
<cfset JobStartDate=CreateODBCDateTime(mydate)>
#JobStartDate#
</cfoutput>
Error: Date value passed to date function createDateTime is unspecified or invalid. Specify a valid date in createDateTime function.
I am using ColdFusion 2021 (update 4) on windows server. Under JVM details Java Default Locale is en_US.
Error can reproduced on: cffiddle.org
Would work just fine with other dates for e.g. July 01, 2008 (okay), May 15, 2009 (okay) etc. But shows error with June 01, 2008 (error) and April 07, 2002 (error). Not sure if there might be other dates.
Additional note: Can this issue be associated with the daylight saving in Pakistan?
Daylight Saving Revival
In 2008 Pakistan used daylight saving time for the first time since 2002 to address its energy crisis. The clocks moved one hour ahead (to UTC+6) at midnight between May 31 and June 1 in 2008. The need for daylight saving time during the peak summer season in Pakistan came in light of the country’s struggle for an approximate 4000-megawatt power shortfall. (reference)[https://www.timeanddate.com/news/time/pakistan-extends-dst-2008.html]
Any help would be greatly appreciated. Thanks
I'm afraid the (originally ~) accepted answer to this isn't correct. The problem is entirely down to the daylight savings issue the original poster mentioned.
The original code is this:
<cfset mydate = 'June 01, 2008'>
<cfset JobStartDate=CreateODBCDateTime(mydate)>
As mentioned, CreateODBCDateTime expects a date/time object, not a string, so the first thing CF needs to do is to convert 'June 01, 2008' to date/time, so the equivalent of this:
<cfset mydate = createDateTime(2008,6,1,0,0,0)>
I've added the hour, minute and seconds part there because they are necessary to create a date/time object. You've given it no time part, so CF has to assume zeros there.
And guess what? on June 1 2008 under the daylight savings rules in Pakistan, there is no such thing as 00:00:00. At the stroke of midnight, time shunted forward to 01:00:00. Hence the error:
Date value passed to date function createDateTime is unspecified or invalid
It's telling you exactly what the problem is. One will always get this when trying to use a time that doesn't exist due to daylight savings vagaries. It's exactly the same situation as if one tried to specify the date part as "Feb 32" or something more obviously invalid.
One will get the same error on 2009-04-15 for the same reason: that's when daylight saving started that year.
This demonstrates why servers should always be set to UTC. There are very seldom "unexpected" gaps in the time (the odd corrective leap-second notwithstanding), so these problems simply don't arise. If you use UTC and then adjust the timezone for display for humans when necessary, CF will always get it right.
Another point. Saying that code worked fine in older versions of CF is incorrect (this came up in comments to the earlier answer). SetTimeZone was only added to CFML on ColdFusion for CF2021, and the code in the question errors on earlier versions. So whatever you were or were not experiencing / testing with on older versions of CF was not this issue.
Can this issue be associated with the daylight saving in Pakistan?
No, this issue isn't associated with the daylight saving in Pakistan.
Update: As Adam Camaron correctly mentioned...
The problem is entirely down to the daylight savings issue the original poster mentioned.
My initial findings turned out to be incorrect. Please read Adam Camerons very interesting answer to this issue for further deep explanation. His answer should be the accepted one.
I'll keep my answer post active for further reference, because it might be helpfull giving a better overview about similar issues dealing with dateTime objects in CFML.
===========================
Orignal answer:
According to the cfml documentation about CreateODBCDateTime() you need to pass a dateTime-Object as an argument to CreateODBCDateTime(). Your variable mydate = 'June 01, 2008' is just a string that represents a date, but it's not a dateTimeobject.
July 01, 2008 (okay), May 15, 2009 (okay) etc. But shows error with June 01, 2008 (error) and April 07, 2002 (error)
The cfml engine will try somehow to deal with the submitted string and cast it to the correct data type, but this may or may not work: It's simply isn't offically supported, so I'd rather not do it that way.
To create a dateTimeObject you need to parse that string to a dateObject first, e.g. using lsParseDateTime(), and like #sos correctly commented, if you are using different locales, better to always pass the correct locale that the string content represents as an attribute:
<cfoutput>
<cfset mydate = 'June 01, 2008'>
<cfset JobStartDate=CreateODBCDateTime( lsParseDateTime( mydate, "en_US" ))>
#JobStartDate#
</cfoutput>
If your dateTime data is suitable, an alternative would be creating a dateTimeObject from scratch by using createDateTime() function first, e.g.:
<cfoutput>
<cfset mydate = createDateTime(2008,5,1,0,0,0)>
<cfset JobStartDate=CreateODBCDateTime( mydate )>
#JobStartDate#
</cfoutput>
Regarding timeshifts across the world, it depends how you are getting and saving your time data and to whom in the world you are delivering it to. In worldwide environments I'd usually save dates to UTCs and output it accordingly by using TimeZone functions.
Side note... because this is commonly missunderstood so I'm posting it here just for posterity: "locales" adapts the string to typical readable (traduced) strings, as they are commonly read and identified by the respective cultures, but they don't change timeZones.
To understand that a little more, I can warmly recommend watching this video about timeZones from Lucee. It's not from ColdFusion but it explains a lot about time internationalization and timezones in CFML and some of it pitfalls.
I am writing a Test Case for a REST Controller. Code below:
private SimpleDateFormat dateFormat = new SimpleDateFormat("dd/MM/yy");
#Test
public void getByExternalTransactionId() throws Exception {
EquityFeeds equityFeeds = new EquityFeeds(423,"SAPEXTXN1", "GS", "ICICI", "BUY", dateFormat.parse("22/11/13"), 101.9f, "BLO", "Y",0);
when(equityFeedsService.findByExternalTransactionId("SAPEXTXN1")).thenReturn(equityFeeds);
mockMvc.perform(MockMvcRequestBuilders.get("/equityFeeds/getByExternalTransactionId/{externalTransactionId}", "SAPEXTXN1"))
.andExpect(status().isOk())
.andExpect(content().contentType(MediaType.APPLICATION_JSON_VALUE))
.andExpect(jsonPath("$.*", Matchers.hasSize(10)))
.andExpect(jsonPath("$.id", Matchers.is(423)))
.andExpect(jsonPath("$.externalTransactionId", Matchers.is("SAPEXTXN1")))
.andExpect(jsonPath("$.clientId", Matchers.is("GS")))
.andExpect(jsonPath("$.securityId", Matchers.is("ICICI")))
.andExpect(jsonPath("$.transactionType", Matchers.is("BUY")))
// .andExpect(jsonPath("$.transactionDate", Matchers.is(dateFormat.parse("22/11/13"))))
.andExpect(jsonPath("$.transactionDate", Matchers.is(Date.parse("22/11/13"))))
.andExpect(jsonPath("$.marketValue", Matchers.is(101.9f)))
.andExpect(jsonPath("$.sourceSystem", Matchers.is("BLO")))
.andExpect(jsonPath("$.priorityFlag", Matchers.is("Y")))
.andExpect(jsonPath("$.processingFee", Matchers.is(0)));
verify(equityFeedsService, times(1)).findByExternalTransactionId("1");
verifyNoInteractions(equityFeedsService);
}
Issue:
It's breaking in transactionDate which is java.util.Date in POJO. I have tried below in the test case:
.andExpect(jsonPath("$.transactionDate", Matchers.is(dateFormat.parse("22/11/13"))))
This gives me the output
java.lang.AssertionError: JSON path "$.transactionDate"
Expected: is <Fri Nov 22 00:00:00 IST 2013>
but: was <1385058600000L>
Expected :is <Fri Nov 22 00:00:00 IST 2013>
Actual :<1385058600000L>
Then I tried:
.andExpect(jsonPath("$.transactionDate", Matchers.is(Date.parse("22/11/13"))))
This gave me the output:
java.lang.AssertionError: JSON path "$.transactionDate"
Expected: is <1412965800000L>
but: was <1385058600000L>
Expected :is <1412965800000L>
Actual :<1385058600000L>
This looks to be very close. I understand that the difference in value is because the time the date was created in POJO is milliseconds is different from the time the date was created in the .andExpect(jsonPath("$.transactionDate", Matchers.is(Date.parse("22/11/13")))) lines and hence the values are different, as also the values are long since the values are in milliseconds.
My TestCase is failing only because of this field. I am really out of my mind as to how should I solve this. I have to use java.util.Date. Any others solution for this?
Edit: If I understand correctly (not familiar with Hamcrest), you need to feed an old-fashioned java.util.Date object into the EquityFeeds constructor and test that the long value that you get back out — 1385058600000L or 1 385 058 600 000 — agrees with the Date you specified.
java.time
To construct the java.util.Date that you need:
Instant transactionTime = LocalDate.of(2013, Month.NOVEMBER, 22)
.atStartOfDay(ZoneId.systemDefault())
.toInstant();
Date oldfashionedDate = Date.from(transactionTime);
System.out.println(oldfashionedDate);
Output so far when running in Asia/Kolkata time zone:
Fri Nov 22 00:00:00 IST 2013
Now just pass oldfashionedDate as the 6th argument to the EquityFeeds constructor. The point in the above lines of code is not only that we’re using the java.time, the modern Java date and time API, and it is very clear to read from the code what it does; it also makes testing easy. To produce the expected long value:
long expectedTransactionDateMillis = transactionTime.toEpochMilli();
System.out.println(expectedTransactionDateMillis);
1385058600000
Not being familiar with Hamcrest and not having tried I would expect that you just need to fill this value into your assertion:
.andExpect(jsonPath("$.transactionDate", Matchers.is(expectedTransactionDateMillis)))
The java.util.Date class is poorly designed and long outdated. I recommend you don’t use it, and I even more strongly recommend you don’t use the notoriously troublesome SimpleDateFormat class. Even though Date is used in your production code for historical reasons, I see no reason why you should repeat that mistake in your tests. Use java.time, it is so much nicer to work with. And conversions exist for when you need an object of one of the outdated types like Date.
What went wrong in your test?
Your first attempt:
.andExpect(jsonPath("$.transactionDate", Matchers.is(dateFormat.parse("22/11/13"))))
The result of dateFormat.parse("22/11/13") is a java.util.Date. Even though this is equal to the Date in your POJO, the value in your JSON is a long representing a count of milliseconds since the epoch (a practice you may want to change if you can). Since a Date and a long can never be equal, your test fails.
You then tried:
.andExpect(jsonPath("$.transactionDate", Matchers.is(Date.parse("22/11/13"))))
Now you are getting a long value alright. We can see that the expected and actual values have the same format in the printed output. There are two things wrong:
You are using the Date.parse method that has been deprecated since 1997. It has been so because it works unreliably across time zones. Even if you insisted on using Date, you should still stay far away from those deprecated methods and constructors.
Your date string gets parsed into 1 412 965 800 000, equal to 2014-10-11T00:00+05:30, so it’s nearly a year off. That method is not only deprecated, it is also confusing and parses your date string as the 11th day of the 22nd month of 2013. There is no 22nd month of the year, but the parse method just extrapolates, keeps counting months into 2014 and ends up with October that year. One more reason not to use that method.
Link
Oracle tutorial: Date Time explaining how to use java.time.
In my application, server side date validations were done through IsDate which is very inconsistent in behavior. I used isValid("USdate",DateVar), that works fine with incoming dates, but when DateVar is a date time stamp it fails. Values coming in DateVar could be anything, a date, a time, a date & time or even some invalid data. If I use Date mask with isValid, that behaves like isDate and of no use. How I can accomplish this.
All "dates" that will be arriving via a request - be they via a URL parameter, a form submission, a cookie, etc - will be strings, not dates.
What you need to do is to work out what string formats you will allow, and validate them accordingly.
EG: you might decide that yyyy-mm-dd is OK, but you won't accept m/d/yy. You might pass them as three separate components for y, m and d. But you really oughtn't try to accept any old format, as you will need to have a validator for each format, and there's a law of diminishing returns there: people won't expect to use any format they like; they'll be expecting you to guide them. You also need to be mindful that if you ask me to type in today's date, I'd give you 4/5/2015. But to you that might represent April 5.
Given various month-length and leap-year rules, really the easiest and most reliable way to see if and input string represents a date in an acceptable format do this:
Validate the format mask, eg: if you're accepting yyyy-mm-dd, then the input needs to be \d{4}-\d{2}-\d{2}. Then at least you know the string has been formed properly.
Then extract the components from the string, and attempt to create a date object with them. If it doesn't error: it's OK.
Last: check any date boundaries within / outwith which the date needs to fall.
We have a web server running Windows 2003 Standard and the time zone is set to GMT-06:00 Central Time, and the box is checked to adjust for daylight saving changes.
A web service on this server queries a datetime field and when the datetime is fetched it is correct. When the dataset is returned to the client 1 hour is subtracted from the datetime if the date is Mar 9-31, Apr 1-5, Oct 26-31, or Nov 1-5. These are the dates that the DST time change can happen on.
It does not matter when the data was saved to the database. If I save a date of 4/1/2013 today, it will be returned to the client minus the 1 hour.
We have verified that SQL is storing the date correctly since it is being returned to the web service correctly.
If I convert the date to a date string at the web server before returning it to the client, the correct date string is returned.
All dates outside of the possible DST dates are good.
All dates during DST that are not on a possible change date are good.
As an example, a date saved as 4/1/2013 12:00:00 AM will be returned as 3/31/2013 11:00:00 PM.
A date saved as 4/6/2013 12:00:00 AM will be returned as 4/6/2013 12:00:00 AM.
I added a web method to the web server to return a date value of Now() and it returns the correct date.
The only thing I have found that is similar was something about a XML rule that says all datetime values are to be transmitted using an offset. I'm not 100% sure but I don't think this is it because only some dates are changed.
Any thoughts/suggestions of what else to look at to get this resolved?
Edit: I have found some dates on the dates listed above that are correct.
I'm going to take a shot at expanding this question, since i've had a similar issue in serialization of data objects.
Take a System.DateTime object that you want to write into a DataColumn. The DateTime object will by default return the DateTime.Kind property as [Unspecified]. When the DataColumn is then set to this DateTime object the DataColumn has a DateTimeMode property that resets to [UnspecifiedLocal].
There is not offest during serialization of DataColumn when the DateTimeMode is [Unspecified] or [UTC]. But when the DateTimeMode is set to [UnspecifiedLocal] or [Local], the offeset is applied. This is where the time can go up or down depending on the timezone or daylight savings configuration.
Sad news is that I can share your problem, but don't have a decent solution. Hope this helps your search.
I can only think of some ugly solutions, but i have not tested any. If i find an elegant solution i will try post again.
I am quering a (.net) web service from Excel (takes cover from the backlash), which returns a array of dates. I have noticed that any date that is on the day boundry gets shifted forward 1 hour, I have checked the web service on the server and the original date is correct.
I then used Fiddler to view the call on the client machine, and the dates which are at midnight have a Z on the end (to mark it as UTC) where as all other date times don't.
UPDATE The above statement is wrong, it is the first and last date that is marked with a Z, it doesn't matter what the time is. The rest is correct that the client only adjusts the first and last date
I take it SOAP is then changing this from UTC to local (BST) and therefore adding an hour.
My question is how can I stop the Z from being added onto these dates (or change it to local)?
I have checked the culture in IIS 7.5 and it is set to en-GB, the server is set to UK.
It's a little fiddly, and this isn't really an answer but should give you some things to try.
One way might be to remove the offset. I don't have the code to had but in psudeo-like code it might be
TimeZone tz = GlobalizationNamespace.GetTimeZone("Greenwich Mean Time");
SomeDateTimeNoOffset = SomeDateTime.AddHours(tz.GetUtcOffset(SomeDateTime));
This should might stop the service from putting a Z at the end as it has no TimeZone offset.
Another other option would be to expose the property in the Service Contract as a String
public string DateTimeNoTzInfo
{
get { DateTime.ToString('yyyy-MM-ddThh:mm:ss.ttttt') }
}