We have a django application server monitored by new relic. We have have used Mysql and MongoDb for data storage in our app. In rpm.newrelic we are having the transaction details of Mysql. We also want to get the transaction details of mongoDb too.
We are using pymongo module for interacting with mongo. I read here that they have included the support of pymongo in their latest python agent. But I am not able to find the documentation for the same. Can anyone point me to some docs ?
At one point we had the same question, and so we built this: https://github.com/Livefyre/pymongo-newrelic
This has some rough edges, but you'll see queries (roughly mapped to SQL terms), and time spent in granular detail.
And while the newer New Relic agents support pymongo directly: https://docs.newrelic.com/docs/python/instrumented-python-packages#nosql-database-clients
They do include this caveat (as of this writing):
Note that MongoDB and Redis calls are currently only recorded as transaction breakdown metrics. That is, no rollup metrics are produced and so they will still be shown on the overview dashboard as Python time and not as a separate segment or even as database calls. Also, no specific details of MongoDB queries are captured at this time and so no information will be displayed on the databases page in the UI corresponding to these queries.
Related
I am new to MERN stack and managed to build an app. I want to deploy it in AWS. But the problem I have to use document DB instead of Mongo DB. Do I need to rewrite my code to do this. Can I use the same mongoose methods? Please help. I am very new to this.
DocumentDB is API compatible with MongoDB for the most part, that's its whole claim to fame, so you most likely won't have to change anything.
There are however some limitations and differences between the systems, which are documented here (Unfortunately the article is too long to briefly summarize it here, so I'm just going to include the list of subtopics - check out the docs for more details).
Admin Databases and Collections
cursormaxTimeMS
explain()
Field Name Restrictions
Index Builds
Lookup with empty key in path
MongoDB APIs, Operations, and Data Types
mongodump and mongorestore Utilities
Result Ordering
Retryable Writes
Sparse Index
Storage Compression
Using $elemMatch Within an $all Expression
$distinct and $elemMatch Indexing
$lookup
I have a django app with redis which is currently used as the broker for Celery, and nothing beyond that.
I would like to utilize it further for lookup caching.
Let's say I had a widely used table in my database that I keep hitting for lookups. For the same of example, let's say it's a mapping of U.S. zip codes to city/state names, or any lookup that may actually change over time that's important to my application.
My questions are:
Once the server starts (in my case, Gunicorn), how do I one-time load the data from the database table to Redis. I mean- where and how do I make this one time call? Is there a place in the django framework for such "onload" calls? or do I simply trigger it lazy-style, upon the first request which will be served from the database, but trigger a Redis load of the entire table?
What about updates? If the database table is updated somehow, (e.g. row deleted, row updated, row added) how do I catch that in order to update the Redis representation of it?
Is there a best-practice or library already geared toward exactly that?
how do I one-time load
For the one time load you can find answer here (from those answers only urls.py worked for me). But I prefer another scenario. I would create manage command and I would add this script to the command you start your Gunicorn. For example if you're using systemd you could add this to service service config. You can also combine those, like add command and call it from urls.py
What about updates
It really depends on your database. For example if you use postgresql, you can create trigger for update/insert/delete and external table as redis. Also django has signal mechanism so you can implement that in django as well. You can also write your custom wrapper. In this wrapper you implement you operations + syncing with redis. And you would call wrapper instead of. But I prefer the first scenario.
Is there a best-practice or library already geared toward exactly
that?
Sorry I can't help you with this one.
I just installed Sitecore Experience Platform and configured it according to the Sitecore scaling recommendations for processing servers.
But I want to know the following things:
1.How can I use the sitecore processing server?
2.How can I check whether processing server is working fine?
3.How collections DB data is processed and send to reporting server?
The processing server is a piece of the whole analytics (xDB) part of the Sitecore solution. More info can be found here.
Snippet:
"The processing and aggregation component extracts information from
captured, raw analytics data and transforms it into a form suitable
for use in reporting applications. It also performs specific tasks on
the collection database that involve mass updates.
You implement processing and aggregation on a Sitecore application
server connected to both the collection and reporting databases. A
processing server can run independently on a dedicated server, or on
the same server together with other Sitecore components. By
implementing multiple processing or aggregation servers, it is
possible to achieve higher performance on high-traffic solutions."
In short: the processing server will aggregate the data in Mongo and processes it (to the reporting database). This can be put on a separate server in order to spare resources on your other servers. I'm not quite sure what it all does behind the scenes and how to check exactly and only that part of the process, but you could check the the reporting tools in the Sitecore backend, like Experience Analytics. If those are working, you probably are fine. Also, check the logs on the processing server - that will give you an indication what he is doing and if any errors occur.
I am trying to understand how Django and Appengine work together?
First, question: Is this a good team?
Experience, what is possible and what not, would be great.
I also read some modules like auth, admin wont work.
But the article is rather old, so maybe there is an update.
And in that tutorial one has to import bforms.
What is that?
Django Module? Appengine? Python? Bigtable?
How is Bigtable different from regular SQL, MySQL?
Thanks
Regular SQL and MySQL are designed for one computer only and fail in cloud computing where you need 1,000 computers for one database. Thus the next generation databases, like bigtable, were created to distribute the data over many database servers. They are called NoSQL databases for "Not Only SQL." See http://nosql-database.org/ for a list of NoSQL databases. The google app engine apparently allows you to use the bigtable structure so you data is distributed over a dozen database servers in the cloud. So does Amazon's simple db.
I'm working on a django website that needs to track popularity of items within a given date/time range. I'll need the ability to have a most viewed today, this week, all time, etc...
There is a "django-popularity" app on github that looks promising but only works with mysql (I'm using postgresql).
My initial thoughts are to create a generic ViewCounter model that logs views for all the objects that are tracked and then run a cron that crunches those numbers into the relevant time-based statistics for each item.
Looking forward to hearing your ideas.
Did you try django-popularity with postgres? The github page just says that the developer has not tested it with anything other than MySQL.
The app has only been tested with MySQL but it should fully work for Postgres with few adjustments. If you do manage to get it to work: please inforrm me. (I'm the developer.) I would love to be able to tell people this product is useable for Postgres as well.
Moreover; all the funcitonality relying on raw SQL checks whether there is actually a MySQL database in use. If not, it should throw an assertion error.
Also, the generic viewcounter is already in my package (it's called ViewTracker, but hell). The cron job seems too much of a hassle to me if we could do either SQL or Django caching as well.