If sas is installed on windows os then How to troubleshoot or catch its performance of sas DI jobs on unix? using any tools or commands or By using nmon?
Thank you...
Performance of each job in SAS Data Integration studio can be measured using a few techniques
ARM logging - Enabling/utilizing the Audit and Performance Measurement capabilities with SAS (commonly referred to as ARM logging).
You can add a parameter FULLSTIMER=I in the autoexec.sas for the session or for each session which will help in giving performance stats, there are many a code snippets or tools out there which can parse and give you fancy performance stats on the jobs.
I have been exploring a few more things now a days which is provided with SAS 9.4 called Environment Manager. This is a daemon which can be configured and give a lot of performance stats and other things. It is web based and a very handy tool for Admins.
Hope this information helps!
Related
I can see number in thousand as Application Insights in Visual Studio 2017. I read about the Application Insights but does not exactly understood what is use of it.
As shown in below screenshot, its showing 25K in Application Insights and continuously increasing. Messaging_TestApp is SQL Database name used for NService bus.
Can anyone explain what is Application Insights and why number increasing ?
First, let me answer the basic "What is App Insights?":
Application insights is essentially an activity monitoring tool from Microsoft for any type of application.
From Microsoft's documentation on What is Application Insights?
Application Insights is an extensible Application Performance Management (APM) service for web developers on multiple platforms. Use it to monitor your live web application. It will automatically detect performance anomalies. It includes powerful analytics tools to help you diagnose issues and to understand what users actually do with your app. It's designed to help you continuously improve performance and usability. It works for apps on a wide variety of platforms including .NET, Node.js and J2EE, hosted on-premises or in the cloud. It integrates with your DevOps process, and has connection points to a variety of development tools. It can monitor and analyze telemetry from mobile apps by integrating with Visual Studio App Center.
Now, the more specific parts of your question:
What you are seeing in Visual Studio?
Application insights is normally run on a server sitting somewhere or purchased as a service from Azure, but it was recently also built into Visual Studio for monitoring apps running out of the IDE. The VS IDE is essentially running a sort of minified version of Application Insights that has then hooked into the SQL Database project you are currently debugging.
What is the Banner number?
From my own testing this number appears to be the total number of telemetry logs Application Insights has recorded since I started the application. Perhaps most of them are being filtered out by your 30m time range.
We are running into quota limits for a small data set which is less than 1Gb in Bigquery. Google cloud gives us no indication of what queries are running on the backend which isn't allowing us to tune the setup. We have a Bigquery dataset and a dashboard built in data studio which is querying on the data set.
I've used relational databases like Oracle in the past and they have excellent tooling to diagnose issues. But with Bigquery, I feel like I am staring into the dark.
I'd appreciate any help/pointers you can give.
The concurrent queries limit makes reference to the number of statements that are executed simultaneously in BigQuery. The quota limit for on-demand, interactive queries is 100 concurrent queries (updated).
Based on this, it is seems that your Data Studio is hitting this quota when running your reports in which case is suggested to re-design your dashboard build in order to avoid exceeding those limits.
Additionally, you can use the bq ls -j -a PROJECTNAME command to list the jobs that have been run in your project in order to identify the queries you need to work with, as well mentioned by Elliott Brossard
This publication:
Pdf
mentions that SAS can only run (MapReduce?) some modeling PROCs on Hadoop. Does anyone know these PROCs or even better is there an exhaustive list? Thanks.
Christian
I haven't seen a list of procedures that will run against Hadoop, it also changes quickly due to additions to SAS Software along with changes in Hadoop.
There is a SAS Support for Hadoop page at SAS 9.4 Support for Hadoop | RESOURCES / THIRD-PARTY SOFTWARE REFERENCE
There are a lot of articles off this page but the link for the Hadoop Support Matrix is helpful see what SAS Products support Hadoop including the distribution, version, and any known issues.
There are several papers available but I think this one about Data Modeling Considerations in Hadoop and Hive by Clark Bradley, Ralph Hollinshead, Scott Kraus, Jason Lefler, Roshan Taheri October 2013 might be a good fit for you.
SAS has a variety of products that interact with many database type systems including Hadoop, and the options are growing along with Hadoop. There are some newer products such as SAS Data Loader for Hadoop (and others) that support running jobs using the Spark Engine instead of MapReduce. Also just to clarify most Hadoop clusters run yarn/mapreduce2, not just Mapreduce 1. SAS can support Mapreduce, but most of the time clusters are running yarn/mapreduce2 which SAS also supports. In addition some SAS programs running against Hadoop may not require a Mapreduce job at all sometimes, depending on what you are trying to do.
There are only a few common engines in Hadoop to use honestly. You have the older Mapreduce (1) which is much older. Then came yarn/mapreduce2 after that which is likely the most common execution engine to date. The Spark Engine has been available for a few years but it still pretty new, it is supposed to be faster yet not as flexible as Mapreduce2 from what I've heard. I think Hortonworks has an engine called Apache Tez which will work with SAS from my experience. Apache Tez still uses yarn just instead of Mapreduce2 engine it uses Apache Tez. It seems for the most part your SAS client won't even know the difference, and so far I haven't ran into any issues with yarn running Tez. There may be a few smaller projects out there, but these are the only ones I run across.
We currently have a Cloud SQL instance with about 600 databases (10Gb total) and we have had several problems with crashes of the instance, so we are thinking about moving to a 2nd generation instance. However, I have found no tool in the console that does this.
Is there some way to do this other than exporting everything as SQL and then executing all queries in the new instance?
And as a side note, is there some limit to the amount of databases per instance? I have found no information on how many databases are recommended to avoid performance and reliability issues.
Thank you
Export and import is the way to do it currently.
Google Cloud SQL uses practically unmodified MySQL binaries, so you can find the limits in the MySQL doc. This one is for 5.6: https://dev.mysql.com/doc/refman/5.6/en/database-count-limit.html
The underlying OS, however, is a custom variant of Linux, and the limits is not documented at this point, but you are probably doing something wrong if you exceed the limits of the OS.
I have a web-service application that exposes lot of web-methods ( internally calling internal APIs or external web-services). I would like to have a tool to monitor this whole application running under IIS. To be clear I just don't want to inspect the requests that are failing, I also would like to see things like how long different calls are taking . I guess it has to be something like SQL Profiler tool
where you can look at all types of queries ( those that run successfully and those that fail too) and then apply all sorts of filter in focusing on the areas that interests you more.
Is there anything built in IIS that could be used for this purpose?
What you are looking for is a runtime profiler for the CLR. The answer is ETW tracing. The tool that can help is PerfMonitor.
IIS 7 has failed request tracing which can help in diagnosing the cause for the
failure.
Also IIS logs have lot of information on request, response,duration and few more
which can be helpful in getting metrics. I usually use Log Parser for getting
metrics from IIS logs
And if you are in IIS7+ then you could use built-in ETW tracing for webservices.
Here is the ETW provider for webservice
e04fe2e0-c6cf-4273-b59d-5c97c9c374a4: Microsoft-Windows-WebServices Microsoft-Windows-WebServices
For any application running under ASP.NET, you can use perfmon to look at different performance numbers.
If you're using ASP.NET 4, there are even better tools available to you
Also if your web services are WCF based, it gets even better Not only do you have very rich tracing, there's also a gui that displays the logs very nicely.
You can try get official IIS Extensions with the Microsoft Web Platform Installer called Web Performance Monitor
Microsoft sells System Center Operation Manager, which has been used by many big firms. But that's too expensive if you cannot afford it.
Web Performance Monitor of course worth a try, and Perfmon is already an option.