I fetch necessary data from Google Trends using the following URL:
http://www.google.com/trends/fetchComponent?q=doctor&cid=TIMESERIES_GRAPH_0&export=3&date=4/2013+3m&hl=en-US
But the output of Google Trends does not contain data for yesterday and today (even if extrapolated by the past hours). For example, the abovementioned URL on 2013-04-28 returned JavaScript code with the following fragment:
...,
{"c":[{"v":new Date(2013,3,26),"f":"Friday, April 26, 2013"},{"v":65.0,"f":"65"}]},
{"c":[{"v":new Date(2013,3,27),"f":"Saturday, April 27, 2013"},{"v":null}]},
{"c":[{"v":new Date(2013,3,28),"f":"Sunday, April 28, 2013"},{"v":null}]},
...
Notice null values for April 27, 28.
But, as we know, "Hot Trends" are available with hourly granularity. That assumes Google possesses enough data to give us the "complete answer" even for the trends request related to specific query not only for the "hottest".
Does anybody know a hack how to fetch from Google Trends up-to-date daily data for specific query? Or, may be, workarounds to derive similar trends data from other sources?
Related
While training my predictor I came across this error and I got stuck how to fix it.
I have two data-series, a "Target time-series data" with 9234 rows and a single "item_id" and a second one that is "Related time-series data" with the same number of rows as I only have a single id.
I'm setting de data with a window of 180 days, what is exactly the difference between the second and the first number that has appeared on the error, 9414 - 9234 = 180.
We were unable to train your predictor.
Please ensure there are no missing values for any items in the related time series, All items need data until 2020-03-15 00:00:00.0. For example, following items have missing data: item: brl only has 9234/9414 required datapoints starting 1994-06-07 00:00:00.0, please refer to documentation for additional details.
Once my data don't have missing data and it's on a daily basis why is it returning this error?
My data starts on 1994-06-07 and ends on 2019-09-17. Why should I have 9414 data points rather than 9234?
Should I take out 180 days in my "Target time-series data"?
The future values of the related time-series data must be known.
Example of a good related-time series: You know past and future days in which marketing has or will send email newsletters promoting the product you're forecasting. You can use this data as a related-time series.
Example of a bad related-time series: You notice that Google searches for your brand correlated with the sale of your product. As a result you want to use it as a related-time series. Since you don't know how many searches will occur in the future, so you can't use this as a related time series.
In you case, You have TARGET_TIME_SERIES data for 9414 days and you want to predict demand for the next 180 days. That means your RELATED_TIME_SERIES data should be 9594 days.
Edit: I have not tested this with amazon's forecasting product. I'm basing my answer on working with Facebook Prophet (which is one of the models amazon forcast uses). Please let me know if my solution worked.
I have a text as given:
(3) Reflects the adoption of SFAS No. 128, EARNINGS PER SHARE.
<PAGE>
ITEM 7. MANAGEMENT'S DISCUSSION AND ANALYSIS OF FINANCIAL CONDITION AND RESULTS
OF OPERATION
YEAR ENDED DECEMBER 28, 1997 COMPARED TO THE YEAR ENDED DECEMBER 29, 1996
In November 1996, the Company initiated a major restructuring and growth
plan designed to substantially reduce its cost structure and grow the business
in order to restore higher levels of profitability for the Company. By July
1997, the Company completed the major phases of the restructuring plan. The
$225.0 million of annualized cost savings anticipated from the restructuring
results primarily from the consolidation of administrative functions within
Here, I want to extract "MANAGEMENT'S DISCUSSION AND ANALYSIS" which occurs after <PAGE>. There are many other "MANAGEMENT'S DISCUSSION AND ANALYSIS" in the document ( I have not copied the document as it is 1000+ pages long).
I used the following Regex expression:
pattern = ('?<=<PAGE>')('.*')('?=Management\'s Discussion')
but it is giving this error
TypeError: 'str' object is not callable
What's wrong, where and how to rectify it?
In Power BI (Desktop) we use a Power BI Query (M) to get data from Application Insights Analytics. We published the Power BI Report to Power BI online configured with a daily refresh. It worked fine until it stopped working on 25-1-2017 (UTC).
The error we get is:
DataSource.Error: Web.Contents failed to get contents from '.....' (502): Bad Gateway
This is the complete error:
DataSource.Error: Web.Contents failed to get contents from 'https://management.azure.com/subscriptions/<subscriptionId>/resourcegroups/fps.fsa/providers/microsoft.insights/components/4PS%20Field%20Service%20iOS%20-%20iOS/api/query?api-version=2014-12-01-preview&csl=customEvents%0A%7C%20where%20timestamp%20%3E%20ago%2830d%29%0A%7C%20order%20by%20timestamp%20desc%0A%7C%20extend%20dimensionUserId%20%3D%20tostring%28customDimensions.%5B%27userId%27%5D%29%0A%7C%20extend%20dimensionHost%20%3D%20tostring%28customDimensions.%5B%27url%27%5D%29%0A%7C%20extend%20measurementQuantity%20%3D%20iff%28%20isnotempty%28customMeasurements.%5B%27value%27%5D%29%2C%20todouble%28customMeasurements.%5B%27value%27%5D%29%2C%200.0%29%0A%7C%20extend%20measurementKey%20%3D%20tostring%28customDimensions.%5B%27key%27%5D%29%0A%7C%20extend%20platform%20%3D%20%27iOS%27%0A&x-ms-app=AAPBI' (502): Bad Gateway
Details:
DataSourceKind=Web
DataSourcePath=https://management.azure.com/subscriptions/<subscriptionId>/resourcegroups/fps.fsa/providers/microsoft.insights/components/4PS%20Field%20Service%20iOS%20-%20iOS/api/query
Url=https://management.azure.com/subscriptions/<subscriptionId>/resourcegroups/fps.fsa/providers/microsoft.insights/components/4PS%20Field%20Service%20iOS%20-%20iOS/api/query?api-version=2014-12-01-preview&csl=customEvents%0A%7C%20where%20timestamp%20%3E%20ago%2830d%29%0A%7C%20order%20by%20timestamp%20desc%0A%7C%20extend%20dimensionUserId%20%3D%20tostring%28customDimensions.%5B%27userId%27%5D%29%0A%7C%20extend%20dimensionHost%20%3D%20tostring%28customDimensions.%5B%27url%27%5D%29%0A%7C%20extend%20measurementQuantity%20%3D%20iff%28%20isnotempty%28customMeasurements.%5B%27value%27%5D%29%2C%20todouble%28customMeasurements.%5B%27value%27%5D%29%2C%200.0%29%0A%7C%20extend%20measurementKey%20%3D%20tostring%28customDimensions.%5B%27key%27%5D%29%0A%7C%20extend%20platform%20%3D%20%27iOS%27%0A&x-ms-app=AAPBI
Does anyone know how to solve this?
The 502 Bad Gateway Message is usually due to the AA Query returning too much data. The gateway is limited to 8MB of data, period.
Example
You created a dashboard that worked in December 2016 and gave you all requests from the start of the month. Now it's January 2017 and it's failing. You use the PowerBI Dashboard to calculate some metrics from the raw results using a query like the one below.
requests | where timestamp > datetime(2016-12-01)
The fix
Determine how many days you really care about. If your intention was to get all requests and the timing from the first of the month you can cut out a lot of extra data by limiting the time range to this month AND by only projecting the columns you need
requests | where timestamp > startofmonth(now()) | project name, duration
Another fix
Assuming you are calculating things like averages and percentiles you could also just have Analytics do this for you and PowerBI just display the results.
requests | where timestamp > startofmonth(now()) | summarize count(), avg(duration), min(duration), max(duration), stdev(duration), percentiles(duration, 50, 75, 90, 95, 99) by name
More Examples
You might want a meaningful graph, so you could split up the aggregations by a time period. This would give you a lot of data in return, but severely less than you'd get if you queried the raw data points.
By Day
requests | where timestamp > startofmonth(now()) | summarize count(), avg(duration), min(duration), max(duration), stdev(duration), percentiles(duration, 50, 75, 90, 95, 99) by bin(timestamp, 1d), name
By Hour
requests | where timestamp > startofmonth(now()) | summarize count(), avg(duration), min(duration), max(duration), stdev(duration), percentiles(duration, 50, 75, 90, 95, 99) by bin(timestamp, 1h), name
I've only given a few examples and you'd have to make sure they fit in with the intention of your dashboard.
The recommendation as James suggested is to limit the result size, yet if you still need to return a larger data set you can work directly with AI API instead of ARM.
1) You will need to create an API Key, see https://dev.applicationinsights.io/documentation/Authorization/API-key-and-App-ID
2) Next you will need to update the Power BI M script that you exported from Analytics by replacing the ARM URL with AI API:
Replace ARM call:
.....
Source =
Json.Document(Web.Contents("https://management.azure.com/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/<RESOURCE GROUP>/providers/microsoft.insights/components/<APP NAME>/api/query?api-version=2014-12-01-preview",
[Query=[#"csl"="requests",#"x-ms-app"="AAPBI"],Timeout=#duration(0,0,4,0)])),.....
With AI API call:
.... Source =
Json.Document(Web.Contents("https://api.applicationinsights.io/beta/apps/<APPLICATION_ID>/query?api-version=2014-12-01-preview",
[Query=[#"csl"="requests",#"x-ms-app"="AAPBI"],Timeout=#duration(0,0,4,0)])),.....
3) Finally, update credentials to basic, and use your API Key
enter image description here
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
I know that Yahoo has a great API for accessing detailed financial metrics about a company documented at http://www.gummy-stuff.org/Yahoo-data.htm. Yahoo also provides historical pricing data, documented at http://code.google.com/p/yahoo-finance-managed/wiki/csvHistQuotesDownload.
However, I'm trying to find a place where I can programmatically access detailed historical data, like what was a company's earnings 10 years ago, and not just the price of the stock. Does anyone know of such a site? I'm willing to pay, and I think http://www.mergent.com/servius, but they seem very, very expensive. A single standardized financial report from a company costs 50 units, which is $2.50 under their pay-as-you-go plan.
Google seems to have pretty good historical financial data that appears to go back 5 years. I may try scraping them, but I'd like to go back much, much further. Any ideas?
Quandl provides a huge amount of different databases with all sorts of data, not only EOD but e.g. earnings per share and a lot of other stuff like US employment data.
API is easy to use and well documented. It also provides an Excel plug-in, a Matlab plug-in, a Python package, an R package, and a number of languages has a support through community maintained libraries.
Not all the data are free though. For more advanced data bases there is a subscription fee. I think the price is different depending on the database and the number of potential users.
Check out this page: ADVFN Financial Data Scraper. You can download a spreadsheet with built-in macro that scrapes up 22 years of financial earnings data for any publically traded company that ADVFN posts historic data for. Just keep in mind that it's not an quick process, for the 3000 odd companies pre-listed in the spreadsheet, the macro will need to run for a couple of days (obviously you can download less if you like though). But, you'll end up with over 8 million data values and you'll have them saved locally in a spreadsheet for quick and easy analysis.
ADVFN posts up to 307 rows of data per company per year and this spreadsheet can capture them all, yielding a very comprehensive data base of historical financial data.
Wolfram Alpha has the data you desire
Examples:
http://www.wolframalpha.com/input/?i=msft+earnings
http://www.wolframalpha.com/input/?i=aapl+earnings+2001
http://www.wolframalpha.com/input/?i=ko+price+1983
I have not used it, but I see they provide a free API with an option to upgrade if you exceed their monthly limits.
Intrinio provides income statements, balance sheets, and statement of cashflows going back 10 years, in addition to stock prices and valuation ratios, via API. You can programmatically query the API to pull the data into your app.
Some examples:
https://api.intrinio.com/financials/standardized?identifier=YUM&statement=income_statement&fiscal_period=Q2&fiscal_year=2015
This grabs YUM's income statement from Q2, 2015.
https://api.intrinio.com/companies?latest_filing_date=2017-03-06
That shows all companies with a new filing date on or after 2017-03-06, which is useful for determining which fundamentals need to be updated.
https://api.intrinio.com/data_point?ticker=AAPL,MSFT&item=pricetoearnings
That pulls the current price to earnings ratio for Apple and Microsoft. You could swap out last_price to get the current stock price.
https://api.intrinio.com/data_point?ticker=$FEDFUNDS&item=level
This call returns the current federal funds interest rate from the federal reserve.
https://api.intrinio.com/prices?ticker=AAPL
That returns the price history for AAPL.
Intrinio gives away 500 daily API calls to any developer.
Depends what you want. Lets say, if you looking for FX historical data you can take a look on Dukascopy historical data feed(http://www.dukascopy.com/swiss/english/data_feed/historical/)
It is possible to write some scripts to download data into your app.
You can get what you want from financialmodelingprep they have quarterly income statement, balance sheet and cash flow. I include a sample code so you can see how I took the data in jquery.
They also offer historical quote according to their documentation.
Fiddle: https://jsfiddle.net/7g238qrp/
$(document).ready(function() {
var url = "https://financialmodelingprep.com/api/financials/income-statement/AAPL?period=quarter";
$.ajax({
url: url,
type: "GET",
crossDomain: true,
success: function (response) {
let resp = response;
resp = resp.substring(5);
resp = resp.substring(0, resp.length - 5);
// if you want to convert to JSON
//resp = JSON.parse(resp)
//console.log(resp);
$('#JonContent').text(resp);
},
error: function (xhr, status) {
alert("error");
}
});
});
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.8.3/jquery.min.js"></script>
<pre>
<div id="JonContent"></div>
</pre>
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
Until now, I've been using the INDU ticker to follow the DOW with the Yahoo! API. For whatever reason you were unable to directly follow ^dji ^djia or any other reasonable combination. Up until yesterday, INDU was working fine. However now I receive no data when requesting indu.
What other ticker can I use with the Yahoo! finance API that will return the DJIA?
This index is not available under any other name.
However, this problem was just a temporary glitch, now resolved by Yahoo. Unfortunately, their financial data availability is very flaky lately. E.g. data available on the web page, but CSV downloads give "N/A" for all fields, etc. There were similar incidents in recent months, with stock prices for random stocks given wrong values, and more.
So, if you're building a new service around these Yahoo services, be aware that:
These services are not reliable.
You're breaking Yahoo ToS, so there's nothing you can do if they are broken / not working, you cannot even complain to Yahoo in good faith.
According to Yahoo (post by Yahoo Developer Network Community Manager Robyn Tippins on Yahoo developer forums):
The reason for the lack of documentation is that we don't have a Finance API. It appears some have reverse engineered an API that they use to pull Finance data, but they are breaking our Terms of Service (no redistribution of Finance data) in doing this so I would encourage you to avoid using these webservices.
The formula for the DJIA isn't very complicated. If you are still able to pull quotes from individual stocks, you could use your code to pull the prices of the existing 30 components of the DJIA, add them up and divide by the current divisor. Of course, this has several disadvantages.
You need to make 30 requests instead of one.
You will have to adjust the divisor if there is a stock-split.
You will have to change the the queries when the components
change.
The components of the DJIA are
AA AXP BA BAC CAT CSCO CVX DD DIS GE HD
HPQ IBM INTC JNJ JPM KFT KO MCD MMM MRK
MSFT PFE PG T TRV UTX VZ WMT XOM
The current divisor is 0.132129493.
The divisor changes whenever there is a stock split in on of the components. The components of the DOW changed 48 times from 1896-2009.
It seems like Yahoo Finance does not support the web service to query ^DJI or INDU.
Check out this discussion:
http://developer.yahoo.com/forum/General-Discussion-at-YDN/Dow-Jones-Industrial-Average-Quote-Error/1317052217631-f9173931-04fd-4519-b1b3-efb65d7ff8fa/1317065435082
Assuming that your application does not need to be real time market data (to the second), you can use the RAW data that is provided to build the interactive graph on yahoo. This data is comma separated and updates about once every minute. The downside: it will include all the data from the trading day. The time given is in Unix time so a conversion would be needed. I tried this out for the ticker symbols you listed and the only one I was able to get data with was ^dji. Hopefully this is what you are looking for!
You can mess with the link and see what happens to the data. For example you can change the amount of days.
http://chartapi.finance.yahoo.com/instrument/1.0/%5Edji/chartdata;type=quote;range=1d/csv/
I think Yahoo Finance All Currencies quote API Documentation will help you.
I found a Yahoo forum answer that says we cannot download CSV data for ^DJI.
Check also YQL console. This console will fetch values in JSON format.
The DIA ticker (SPDR Dow Jones Industrial Average) closely imitates the Dow.