I'm creating a c++ project that should works on several timezone. The application receives an event, with a reference timezone, and this event is shown graphically to the user at the correct hour, in his local timezone. For example, an user working in Berlin receives an event written in Tokyo. The event from Tokyo is first converted in UTC time, then reconverted from UTC to the computer local time in Berlin, and finally shown to the user on his graphical interface.
To convert from UTC to local computer time I have several functions of the Windows API at my disposal to do the job. But in order to convert a time from another timezone to UTC, I need to get the timezone information from the Windows registry.
Now some timezone have also a Daylight Saving Time to consider. I'm able to create a recurrence rule from the Windows info without problems. However I noticed that the day the DST should occur is sometimes incorrect on several timezone. For example, the "E. South America Standard Time". With the recurrence provided by Windows, the DST start day begins 1 week sooner.
If I understood right, the recurrence rule returned by Windows for this specific timezone says "every year, on the 2nd month, on the 2nd week of the month". However this rule matches rarely with the correct date published on the internet for the time changing, whereas the dates are all correct if the rule would be "every year, on the 2nd month, on the 3nd week of the month". Furthermore, as you can see on the provided screenshot, the Windows registry data shows 2 weeks for the DST start time (highlighted in blue), but 3 weeks for the DST end time (surrounded in red), which is calculated correctly by my code. The description of the data content may be found here: http://msdn.microsoft.com/en-us/library/windows/desktop/ms725481(v=vs.85).aspx
I have several questions
Have I understood the recurrence rule correctly? (Here is what MSDN says about it: https://msdn.microsoft.com/en-us/library/windows/desktop/ms725481(v=vs.85).aspx)
Is there known issues about several timezone, especially the "E. South America Standard Time" one?
Is there a reason why a DST start date, that obviously occurs regularly every year, on the 3rd week of the 10th month, have a value set on the 2nd week?
Are timezone written in Windows registry reliable, if not, which function of the Windows API should I use to convert a timezone with DST from a date written in a different timezone than the one set on the local machine?
NOTE I have strongly verified if the data I read from the registry were correct before posting this message. I'm pretty sure that is not an error of this type.
NOTE I'm working with Windows 7, but the issue remains the same on Windows 10
As a Microsoft employee with significant involvement with Windows time zone data, please allow me to assure you that Microsoft works very hard to ensure that it releases updates to keep Windows time zone data as accurate as it possibly can be.
There are several challenges, including the timing of time zone changes as given by governments. For example, we recently posted a notice about upcoming Windows time zone changes for Fiji, Cyprus, Sudan, Tonga, Namibia, and Turks & Caicos. In some cases, we can meet the effective-dates established by these different governments. In some cases we cannot because they don't offer enough lead time.
Consider the recent case of Sudan, who told the the IANA tz mailing list on October 17th, 2017 of a change effective November 1st 2017. IANA has processed this change, and so has Microsoft. But due to the short notice, and the time it takes to process such a change in the Windows operating system, it will be a little while before there is a new "Sudan Standard Time" time zone created in the Windows time zone data. Thus, we issue interim guidance to use a different time zone for the time being, then switch back once we have data that properly reflects the full history of time in the affected region.
With regard to the Windows time zone you mentioned, "E. South America Standard Time", the time zone data is correct. If you expand the registry entry, you'll see that there is a sub-key called "Dynamic DST" which contains the year-by-year changes to the data. For some time zones, this isn't necessary at all, and for other time zones you'll find a very small amount of data because the same rule repeats year after year. But in the case of Brazil, you'll notice Dynamic DST entries for 2009 through 2040.
(click the image to see the full resolution, and note the areas I manually marked in red that change year-over-year)
The Dynamic DST entries support the Win32 DYNAMIC_TIME_ZONE_INFORMATION structure, and corresponding APIs with "dynamic" in their name, such as GetDynamicTimeZoneInformation. (They also support the System.TimeZoneInfo class in the .NET Framework.) Most of these APIs have been in places since Windows Vista / Server 2008, with some others coming in Windows 7 / Server 2012.
Note that the TZI entry that is in the parent key is copied in from the current year's dynamic rule by internal Windows processes. This supports the APIs that work with Win32 TIME_ZONE_INFORMATION structures, which have been in place since Windows 2000.
To answer the specific questions you asked:
Have I understood the recurrence rule correctly?
The specific rule you cited is for 2017 only, and it says that DST ends on the 2nd month (February), on the 3rd Saturday, at 23:59:59.999 local time, and starts again on the 10th month (October), on the 2nd Saturday, at 23:59:59.999 local time. Keep in mind that Brazil is in the southern hemisphere, so DST starts late in the year and ends early in the next year.
Also, you may wonder why it's 23:59:59.999 on Saturday instead of 00:00:00.000 on Sunday. This is an artifact of history. There have in the past been certain programs and processes that incorrectly used <= instead of < to evaluate the transition, and thus would accidently move the clock one millisecond into the next day, back out of it, then back into it again. The events generated by the day changing erroneously could then lead to further problems. Microsoft has done their best to fix those bugs, but still opts to be 1ms off instead of risk the problem popping up in some new place one day. (This only applies for transitions that occur exactly at midnight.)
Is there known issues about several timezone, especially the "E. South America Standard Time" one?
There are no known issues for that time zone, however there is a known issue that Windows time zones (even with the dynamic DST data) can only support a maximum of two time zone transitions in a single year. So, in places like Morocco that transition four times per year there are some internal workarounds in place that keep the current time zone data in sync such that "now" is an accurate representation of local time, but cannot represent all points of the year correctly at any given time.
We also don't currently have a time zone that maps cleanly for Troll Station, Antarctica. The reason is that from 2005 to 2015, this research station (population under 50) transitioned through three different offsets (UTC+0, UTC+1, and UTC+2) every year.
If your application is critically dependent on either of the above scenarios, then I recommend using an API with IANA data sources instead of the Win32 APIs.
Is there a reason why a DST start date, that obviously occurs regularly every year, on the 3rd week of the 10th month, have a value set on the 2nd week?
Yes, as mentioned above, it's due to the 1ms intentional error. DST in 2017, in the parts of Brazil that have DST, starts on Sunday October 15th at 00:00:00.000. That's the third Sunday of the month. 1ms prior is 23:59:59.999 on Saturday October 14th, which is the second Saturday of the month. This can be different every year, which is why there is Dynamic DST data.
Are timezone written in Windows registry reliable, if not, which function of the Windows API should I use to convert a timezone with DST from a date written in a different timezone than the one set on the local machine?
If you are using the Win32 APIs, they use the registry data themselves, so there is no need to work with the registry data directly. You should prefer the "Dynamic" versions of the APIs, as they properly account for the year-over-year changes in the Dynamic DST data. Sometimes these are labeled as "Ex". For example, the function you asked about is best handled by the TzSpecificLocalTimeToSystemTimeEx function.
...
That all said, if you are able to avoid using Windows time zone data in your application, I recommend doing so. Prefer IANA data sources, or those derived from them. There are many routes to working with IANA time zone data. Newer Windows APIs like Windows.Globalization.Calendar and Windows.Globalization.DatetimeFormatting.DateTimeFormatter in WinRT/UWP do indeed use IANA time zones, and that is clearly the path forward. In the standard C++ space, I highly recommend using Howard Hinnant's date/tz libraries, or those provided by the ICU project. There are several other viable choices as well.
Great to see the level of detail in Matt’s post.
On the question of "Is there known issues about several timezone, especially the "E. South America Standard Time" one?"
Nothing much more to add save that in addition to Matt’s incredibly detailed and thorough contributions to the Windows time zone data, there’s considerable vetting with different agencies, NGOs and government bodies that’s been outlined elsewhere. As an example, there are additional challenges in places like Morocco which not only transitions four times a year, but the information is subject to the government setting the official observance of DST in the region, and sometimes with little lead time, leading to additional publishing and deployment challenges. (Not to mention when a government revises a decision multiple times.)
So, for some applications with critical dependencies, an API to IANA source may be preferred, or calling the time zone data in Windows.Globalization.
All this is timely, if you’ll pardon the pun, just before we Fall Back in the northern hemisphere as well.
I want to be able to pull data depending on the time zone I'm currently located in.
For example, I have this query at the moment:
$response = $fb->get('/pageid/insights/page_impressions?period=day')
And I get this response:
But how would I go about showing the data so that it's in Eastern Time (ET) format. Otherwise known as EDT or UTC -4? I'm assuming it's an additional "parameter" added to the query, but what would it be?
Unfortunately, you can't. All data is aggregated by days according to a fixed offset of UTC-7 (even when the Pacific time zone is on UTC-8).
You could adjust the time zone of the resulting timestamp, but that would be misleading, as the value totals would now not truly be matching the days total for the time zone specified.
Really, an API like this (or any operation grouping timestamps by date) should consider a time zone - and that time zone should be specified by full IANA time zone identifier, such as America/New_York. Consider that UTC-4 is not sufficient, because US Eastern Time alternates between EST (UTC-5) and EDT (UTC-4).
You could request Facebook add this feature, but AFAIK they do not currently offer it.
See also this related answer.
This strange behavior has recently came to my attention, while I was testing my Rails app on local environment in which I use around_filter to set the timezone to registered user (the default timezone is UTC).
What I did was that I registered a new user in my app. My current time was 10pm GMT-5 (March 3), and this user's created_at time was saved to database to 4am UTC (March 4). Now, I know that this time is saved in database with the timezone settings, but here comes the problem:
I use a graph for visual representation of daily registered users, and when I called the following function to tell me number of users registered in the last few days:
from ||= Date.today - 1.month
to ||= Date.today
where(created_at: from..to).group('DATE(created_at)').count
It would say that this user was registered in March 4, while it was in fact registered on March 3 from my perspective.
My question is:
How should I call where function and group by a created_at column, so that the dates with be affected correctly (according to my timezone) ?
Or is there something else that I should be doing differently?
I'm not a rubyist, so I'll let someone else give the specific code, but I can answer from a general algorithmic perspective.
If you're storing UTC in the database, then you need to query by UTC as well.
In determining the range of the query (the from and to), you'll need to know the start and stop times for "today" in your local time zone, and convert those each to UTC.
For example, I'm in the US Pacific time zone, and today is March 7th, 2015.
from: 2015-03-07T00:00:00-08:00 = 2015-03-07T08:00:00Z
to: 2015-03-08T00:00:00-08:00 = 2015-03-08T08:00:00Z
If you want to subtract a month like you showed in the example, do it before you convert to UTC. And watch out for daylight saving time. There's no guarantee the offsets will be the same.
Also, you'll want to use a half-open interval range that excludes the upper bound. I believe in Ruby that this is done with three dots (...) instead of two (at least according to this).
Grouping is usually a bit more difficult. I assume this is a query against a database, right? Well, if the db you're querying has time zone support, then you could use it convert the date to your time zone before grouping. Something like this (pseudocode):
groupby(DATE(CONVERT_TZ(created_at,'UTC','America/Los_Angeles')))
Since you didn't state what DB you're using, I can't be more specific. CONVERT_TZ is available on MySQL, and I believe Oracle and Postgres both have time zone support as well.
Date.today will default to your system's set timezone (which by the way should always be UTC, here's why) so if you want to use UTC, simply do Time.zone.now.to_date if rails is set to UTC
Otherwise you should do
Time.use_zone('UTC') do
Time.zone.now.to_date
end
After this you should display the created_at dates by doing object.created_at.in_time_zone('EST')
to show it in your current timezone
Say, if it's 13:00 at new york (EST) , then it's 06:00 at new zealand (NZST).
If new zealand goes into summer time, then when it's 13:00 at new york (still EST), it's going to be 07:00 at new zealand (now NZDT).
I read the boost time library, but it seems to me one has to determine the daylight saving rules oneself to find out the time in foreign country from the 'localtime' perspective..
e.g.
nyc_string = "EST-05:00:00EDT+01:00:00,M4.1.0/02:00:00,M10.5.0/02:00:00";
// above basically defines the daylight saving rule
time_zone_ptr nyc_2(new posix_time_zone(nyc_string));
std::cout << "The second zone is in daylight savings from:\n "
<< nyc_2->dst_local_start_time(2004) << " through "
<< nyc_2->dst_local_end_time(2004) << std::endl;
source: http://www.boost.org/doc/libs/1_39_0/doc/html/date_time/examples.html
Maybe there's something I'm not yet aware of? Does boost use any database that keeps track of daylight saving rules? I wonder if there's a nice way to adjust local time to different time zone in c++, taking the daylight saving rules into account..If I could have an example, that'd be so great!
Boost.DateTime has a timezone database named date_time_zonespec.csv located in libs/date_time/data. The Flight Time Example in the documentation shows how to access and use it. This database doesn't contain the history of timezone changes. There also doesn't seem to be a place hosting updates to this database (other than the Boost library itself).
If you need accurate, up-to-date, time zone data, then check out the ICU Time Zone Classes of the popular ICU internationalization library from IBM. As mentioned in the Updating the Time Zone Data section:
The time zone data in ICU is generated from the industry-standard TZ
database using the tzcode
(http://source.icu-project.org/repos/icu/icu/trunk/source/tools/tzcode/
) tool.
The ICU data files with recent time zone data can be downloaded from
the update URL,
http://source.icu-project.org/repos/icu/data/trunk/tzdata/icunew .
The ICU timezone database is derived from the tz database now maintained by ICANN.
To download a file via http in a C++ program, you can use libcurl or the cURLpp C++ wrapper. It might be easier to setup a scheduler on your OS to regularly download the latest database.
As already mentionned in the comments, use UTC consistently for storage and business logic. Only convert to/from local times for display/input purposes.
Customers from around the world can send certain 'requests' to my server application. All these customers are located in many different time zones.
For every request, I need to map the request to an internal C++ class instance. Every class instance has some information about its 'location', which is also indicated by a time zone.
Every customer can send requests relating to instances belonging to different time zones. To prevent my customers from converting everything themselves to the time zone of the 'target' instance, I have to convert everything myself from one time zone to another. However, I only find in C++ (unmanaged, native) functions to convert times between local time and GTM, but not from/to a time zone that is not your current time zone.
I could ask my customers to send every date time in UTC or GTM, but that does not solve my problem as I still have to convert this to the time zone of the 'instance', which can be any time zone in the world.
I also don't seem to find a Windows function that does this. What I do find is a managed .Net class that does this, but I want to keep my application strictly unmanaged.
Are there any Windows (XP, Vista, 7, 2003, 2008) functions that I can use (and which I overlooked in the documentation), or are there any other free algorithms that can convert between one time zone and the other?
Notice that it is not the GMT-difference that is posing the problem, but the actual DST-transition moment that seems to depend on the time zone. E.g:
Western Europe goes from non-DST to DST the last Sunday before April 1st.
USA goes from non-DST to DST the 2nd Sunday after March 1st.
China has no DST.
Australia goes from non-DST to DST the 1st Sunday after October 1st.
All this DST-transition information is available somewhere in the Windows registry. Problem is: which Windows function can I use to exploit this information.
I don't know of a way to extract information about other TimeZones via the API: I've seen it done by querying the registry though (we do this in a WindowsCE-based product).
The TimeZones are defined as registry keys under
HKLM\Software\Microsoft\Windows NT\Current Version\Time Zones
Each key contains several values, and the one which tells you about offsets & Daylight Savings is the TZI key. This is a binary blob, and it represents this structure:
typedef struct
{
LONG m_nBias;
LONG m_nStandardBias;
LONG m_nDaylightBias;
SYSTEMTIME m_stcStandardDate;
SYSTEMTIME m_stcDaylightDate;
} TZI;
Look up MSDN's TIME_ZONE_INFORMATION page (http://msdn.microsoft.com/en-us/library/ms725481(v=VS.85).aspx) for how to interpret the Bias fields, and especially the StandardDate and DaylightDate fields -- they are gently abused to support constructs like "the last Saturday in April".
HTH