I work with supply chain networks and I'd like to represent/visualize the flow of products through the network. I'm using Power BI.
Each pair of origin/destination can have more than one link. This link represents a particular transportation mode or a particular group of products.
Besides categories, there are fields that represents numerical values like the flow through the link, or a flow constrain through the link.
These values may be used as weights at the charts.
So far, all Power BI visual that I tested only represents one link per origin/destination pair.
Force-Directed Graph shown more than one, but it superposes the links.
Network Navigator has a similar approach but only shows one link per origin/destination.
Is there a Power BI visual capable of represent a flow through a network with "great circle" kind of links between nodes and do not superposes them?
The following table is an example of the data I'd like to visualize:
Link_ID
Link_NN
Origin_ID
Dest_ID
Link_TP
Product_TP
Min_Vol
Max_Vol
ABC
Pipe_A
LOC_1
LOC_2
Pipe
Diesel
0
1000
ABD
Pipe_B
LOC_1
LOC_2
Pipe
Gasoline
0
2000
ABE
Truck_A
LOC_1
LOC_2
Truck
Diesel
0
100
ABF
Ship_A
LOC_1
LOC_3
Ship
Fuel Oil
0
10000
Link_TP and Product_TP would be used in slicers. The main point is to visualize all the links that exists between an origin/destination set.
I work for a commercial graph drawing library provider and I can tell you that we have been getting similar requests a few times, already. Thus I believe there is no such good visual available out-of-the box.
However Power BI can be extended easily and we have had customers doing exactly that: Building a custom Power BI visual with our third party diagramming control to add the missing features to their dashboards.
To give you some ideas, here are visualizations that sound similar to what you are looking for.
Interactive Process Mining Diagram Online Demo
Sankey Flow Diagram Online Demo
Network Flow visualization Online Demo
Also there is a GitHub repo that shows how to create a custom Power BI visual with the above technology:
https://github.com/yWorks/graphs-in-power-bi
As mentioned: DISCLAIMER: I work for the company that creates that library, however I do not represent my company on SO. My statements and thoughts on SO are my own.
PowerBI, no.
And, in general, I do not see any library to provide exactly this. The closest is probably deck.gl's ArcLayer. I think ArcLayer is flashy and all but it's not as clear as I want.
There is an excellent academic paper on the subject that describes exactly what I want.
"Force-directed layout of origin-destination flow maps" 2017 by Jenny et. al.
Related
Firstly apologies if this has been answered elsewhere.
I am using QuantLib (via Excel) to build a "standard" bond pricing sheet: prices, yields, spline AND matched-maturity ASW.
I can price the bonds, and have successfully built a forecast (Euribor) and discount (EONIA) curve. I can use qlMakeVanillaSwap() to define a spot-start swap by tenor (eg "1y","2Y" etc) and it works fine. However I am struggling to define a "broken date" swap, ie one which starts T+2 and ends on a given date (and so usually has a short stub on the first payment), to match the bond maturity. All the examples I can find have integer year tenors.
I would be grateful if someone could point me to the right method (can be in python, C++ or Excel). Or do I have to go down the route of creating explicit fixed and floating rate schedules for the swaps?
The answer seems to be: Yes, I do have to create explicit fixed and floating rate schedules, using qlSchedule(), but it turns out to be not too onerous. NB. I am pricing a vanilla EUR ABB vs 6m Euribor swap.
As for pricing, it seems the qlMakeVanillaSwap() is doing a few helpful things in one call, but only IF your swap has a whole-period tenor (eg "1y"). I found the answer for what I wanted to do in the example sheet that came with the QuantLibXL download package.
The other thing that qlMakeVanillaSwap() is doing (in addition to creating the schedules) is setting the Pricing Engine (which is used to discount the cashflows). In the longer version you have to (a) set it yourself using qlInstrumentSetPricingEngine() and (b) pass the result of that call to the Trigger parameter of qlVanillaSwapFairRate(), to establish the calculation order.
I am using the Weka GUI for classifying sensor data.
I have measures of 10 people, the data is sorted. So the first 10% correspond to participant 1, the second 10% to participant 2 etc.
I would like to use 10 fold cross validation to build a model on 9 participants and test it on the remaining participant. In my case I believe I could accomplish this by simply not randomizing the data splits.
How would I best go about doing this?
I don't know how to do this in the Explorer.
In the KnowledgeFlow GUI, there is a CrossValidationFoldMaker used to create cross-validation folds. This has an option to Preserve instances order, which says it preserves the order of instances rather than randomly shuffling.
There's a video describing the KnowledgeFlow interface here:
https://www.youtube.com/watch?v=sHSgoVX9z-8&t=7s
I have unstructured twitter data which is retrieved by the apache flume and stored it into the HDFS. So now I want to convert this unstructured data into structured one using the mapreduce.
Task wanted to do using the mapreduce:
1. conversion Unstructured to structure one.
2. I just want the text part which contain tweet part.
3. I want to identify the tweets for particular topic and grouped according to their sub part.
e.g. I have tweets of samsung handset so i want to make a group according to their handsets like groups of Samsung Note 4, Samsung galaxy etc.
It is my college project so my guide suggested me to use k means algorithm, I search a lot on k means but failed to understand how to identifies the Centroid for this basically i failed to understand how to apply K means to this situation in MapReduce.
Please gude me if I am doing wrong as I am new to this concept
K-means is clustering algorithm. It cluster or group similar data and calculate the common centroid. You can create time-series for the above questions you have mention. Group the tweets according to the topic.
K-mean implementation in MapReduce.
https://github.com/himank/K-Means
Using K-means in Twitter datasets.
You can check the following links
https://github.com/JulianHill/R-Tutorials/blob/master/r_twitter_cluster.r
http://www.r-bloggers.com/cluster-your-twitter-data-with-r-and-k-means/
http://rstudio-pubs-static.s3.amazonaws.com/5983_af66eca6775f4528a72b8e243a6ecf2d.html
i want to list all available industries ( like: http://biz.yahoo.com/p/ ) and show all corresponding stocks.
Until now I'm using YAHOO.Finance.SymbolSuggest.ssCallback for the symbol suggestion and http://finance.yahoo.com/d/quotes.csv?s=... for getting the stock's data.
Does anyone have any idea how to get all industries and corresponding stocks?
Is there another hidden Yahoo API?
Lists of all available industries are called GICS Sectors for Standard and Poor's (S&P500 will use that) and ICB for Dow Jones and FTSE. Hence it used by Nasdaq, Nyse and others markets.
It seems like Yahoo uses a third industry classification by Morning Star, but since I'm not quite sure I will give both ways of retrieving data.
Morning Star
I don't know if Yahoo really sticks to this classification, but some names were really close so let's see it:
You need to go to their Index Data and in each sector, click on it and then at the bottom View complete index holdings.
It's not as precise as in Yahoo industry list, but it's all you can do with Morning Star. Not very convincing, I know...
GICS Sectors
GICS Sectors are now a trademark of Standard and Poor's and then data have to be sought for in S&P's website.
Short answer: take a look at this page, you will need to be registered (it's free and easy) and you can download spreadsheets (xls) with stocks and corresponding sectors. Nevertheless, things aren't always easy, and you will have to do a bit of a search to retrieve all stocks with their corresponding industries. For example, the file INDICATED_RATE_CHANGE.xls will give you some companies and their sectors in each month of 2012. Using that and SP500_DividendAristocrats_2012.xls you should be able to retrieve at least a large part of S&P 500 companies.
ICB
ICB is used by NYSE, NASDAQ etc... Then it's a lot simpler than S&P and MorningStar. Here is your answer. BOOM! Direct link!
Link is dead :(
Finally
I strongly advise you to use the simpler and most-used industry classification index: the ICB. It will always be available and publicly displayed since millions of investors relay everyday on it, without having to use S&P financial services or MorningStar brokerage services...
EDIT
You can look at nasdaq.com to retrieve all companies and their corresponding sector: here for Nasdaq and here for Nyse
Get all industry-IDs from here:
http://biz.yahoo.com/ic/ind_index.html
(look at the links)
Then use YQL ( https://developer.yahoo.com/yql/console/ )
with a query like this:
select * from yahoo.finance.industry where id=912
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
Until now, I've been using the INDU ticker to follow the DOW with the Yahoo! API. For whatever reason you were unable to directly follow ^dji ^djia or any other reasonable combination. Up until yesterday, INDU was working fine. However now I receive no data when requesting indu.
What other ticker can I use with the Yahoo! finance API that will return the DJIA?
This index is not available under any other name.
However, this problem was just a temporary glitch, now resolved by Yahoo. Unfortunately, their financial data availability is very flaky lately. E.g. data available on the web page, but CSV downloads give "N/A" for all fields, etc. There were similar incidents in recent months, with stock prices for random stocks given wrong values, and more.
So, if you're building a new service around these Yahoo services, be aware that:
These services are not reliable.
You're breaking Yahoo ToS, so there's nothing you can do if they are broken / not working, you cannot even complain to Yahoo in good faith.
According to Yahoo (post by Yahoo Developer Network Community Manager Robyn Tippins on Yahoo developer forums):
The reason for the lack of documentation is that we don't have a Finance API. It appears some have reverse engineered an API that they use to pull Finance data, but they are breaking our Terms of Service (no redistribution of Finance data) in doing this so I would encourage you to avoid using these webservices.
The formula for the DJIA isn't very complicated. If you are still able to pull quotes from individual stocks, you could use your code to pull the prices of the existing 30 components of the DJIA, add them up and divide by the current divisor. Of course, this has several disadvantages.
You need to make 30 requests instead of one.
You will have to adjust the divisor if there is a stock-split.
You will have to change the the queries when the components
change.
The components of the DJIA are
AA AXP BA BAC CAT CSCO CVX DD DIS GE HD
HPQ IBM INTC JNJ JPM KFT KO MCD MMM MRK
MSFT PFE PG T TRV UTX VZ WMT XOM
The current divisor is 0.132129493.
The divisor changes whenever there is a stock split in on of the components. The components of the DOW changed 48 times from 1896-2009.
It seems like Yahoo Finance does not support the web service to query ^DJI or INDU.
Check out this discussion:
http://developer.yahoo.com/forum/General-Discussion-at-YDN/Dow-Jones-Industrial-Average-Quote-Error/1317052217631-f9173931-04fd-4519-b1b3-efb65d7ff8fa/1317065435082
Assuming that your application does not need to be real time market data (to the second), you can use the RAW data that is provided to build the interactive graph on yahoo. This data is comma separated and updates about once every minute. The downside: it will include all the data from the trading day. The time given is in Unix time so a conversion would be needed. I tried this out for the ticker symbols you listed and the only one I was able to get data with was ^dji. Hopefully this is what you are looking for!
You can mess with the link and see what happens to the data. For example you can change the amount of days.
http://chartapi.finance.yahoo.com/instrument/1.0/%5Edji/chartdata;type=quote;range=1d/csv/
I think Yahoo Finance All Currencies quote API Documentation will help you.
I found a Yahoo forum answer that says we cannot download CSV data for ^DJI.
Check also YQL console. This console will fetch values in JSON format.
The DIA ticker (SPDR Dow Jones Industrial Average) closely imitates the Dow.