I am developing a distributed processing API in Java, Erlang and Ruby. What basic commands can I include from which I can build mapreduce, pipelining, and all the most used parallell algorithms on top of it.
You may get some hints from this blog post, which I found quite useful.
http://ox.no/posts/minimalistic-mapreduce-in-net-4-0-with-the-new-task-parallel-library-tpl
Related
I am new to network programming and got started with boost, REST etc. I wanted to know if I could use REST API's with boost-asio such as using Google Maps' Distance Matrix in my program. But I couldn't find a proper documentation for boost.
I don't expect you to give me complete working code rather I need idea or some sort of guidance as to what to do, where to find things etc.Also this program will be in C++ purely (I don't know if it can even be done in C++, given this answer https://stackoverflow.com/a/28736632/4846740). Thanks
Note: This post was not very helpful Integrating Google maps with C++ Program
You'd want to
use a REST library (e.g. cpprestsdk or some other frameworks like autobahn-cpp?), or
= at least write the REST requests on top of a HTTP library, such as Boost Beast
The library examples show you everything you need to send requests and receive responses. If you want, you can use additional libraries like https://github.com/0xdead4ead/BeastHttp to make it even more high-level/instant.
I would 100% recommend that you do not do this in C++. While I'm not a huge fan of Python, it's undoubtedly the hammer that is made for this nail. Check out BeautifulSoup, Mechanize, and Scrapy (+XPath), for really convenient ways of obtaining/parsing HTML, filling out web forms, and gaining responses. Typically, unless you're doing realtime target tracking, you do not need the latency gained from running everything in C/C++. You can get away with quarter-second, or even half-second updates.
I'm not sure what you're trying to do, but I would say save yourself the headache, and just work with Python.
I am new to GAE (cloud based development in general actually) and I am looking for some advice.
I am looking to use GAE only as a service (REST + JSON) with my client-side in Sproutcore. I am looking to find a light-weight service-based framework that will provide me with strong security (authentication, protection against XSS etc).
Java or Python is fine, the priority being speed and security.
Does anybody have any ideas on this? Any links to relevant information. I am going through an information overload phase. Any help in this regard would be much appreciated!
Thank you.
How about tipfy, a open source lightweight python-based framework made for GAE?
Its core can be enhanced through usage of extensions.
Combining some of those extensions
would allow you to perform auth
against different systems (Google,
Facebook, Twitter...) or against your
own baked one. An example of this can
be found here, source code is
available here.
A really simple sample of JSON
rendering is available here.
Of course, other python frameworks run on top of AppEngine. You'll find in the AppEngine-Python google group, a thread discussing pros/cons of most of them.
Considering speed, those links may shed some light on the subject
Choosing Java vs Python on Google App Engine
Google AppEngine Language Performance Comparison Followup
proceed with caution if you want to use a framework. since GAE is still evolving, frameworks tend to break. i have tried both tipfy and django on large projects.
http://blog.bygsoft.com/2011/04/27/why-we-should-avoid-tipfy-a-gae-framework/
in many places tipfy tends to just write meaningless (read no value add) python wrappers around what the core gae frameworks provide just causing code bloat.
last i checked, the map reduce package did not work well with tipfy. sorry, i may sound very biased but my experience with tipfy has been average.
So looks like I can't comment on your answer but the link was very helpful! Looks very much like something I am looking for:
Server-side sessions
Custom user authentication
JSON based service with no View layer
Fast and secure
Any other python framework that you might possibly know of along the lines of Tipfy? Just so I have something to compare with?
Does anybody know of any Java libraries along the same lines of Tipfy?
Thanks again for the link!
Edit 1:
Hmm...I have no idea how I ended up with 2 accounts. I'll try to iron it out..
Your links were very helpful..Thanks!
I am inching towards Tipfy at this point, but I think I am going to wait a little longer on this thread to see if I get some more opinions..
Thanks again...
does anyone know of a good profiling tool or library for Clojure?
I would prefer something that could be used from the REPL, along the lines of (with-profiling ...) in Allegro Common Lisp back in the day.
Is there anything along those lines?
Or do you have any experience with (non-commercial) Java profilers that work well with Clojure?
I found VisualVM (see here and here) to be very convenient. Its use with Clojure has been described about a year ago in this blog post; as far as I can see, it's not outdated in any way.
Note that the GUI from which one starts the VisualVM profiler has a prominent text area where one can enter classes / packages to be excluded from profiling -- I find the results rather more useful when clojure.* is on that list.
There's a newish Clojure library which offers profiling: https://github.com/ptaoussanis/timbre
Just found profile in Clojure contrib.
It doesn't work for large amounts of code (it blew up with OutOfMemoryError on a Project Euler solution which VisualVM handled just fine) and it requires you to insert profiling calls in the functions you want to profile.
Still, it's a better alternative to VisualVM in the cases where you just want to profile a couple of functions.
Quick heads-up that I've deprecated Timbre's profiling for a new dedicated Clojure + ClojureScript profiling lib at https://github.com/ptaoussanis/tufte.
That's basically a refinement of the stuff from Timbre, plus dedicated docs.
The README includes a comparison with JVM tools like VisualVM, YourKit, etc.
I was told I have to use winsock, but I dont know where to start. For example, I am trying to access, lets say http://www.newegg.com/, I am trying to get the text title of just the three front page products. Any help is greatly appreciated. :D
I'd also recommend libcurl for this sort of thing.
You can use the cURL command line tool to generate sample code as well, which is helpful for experimentation.
W3.org themselves provide sample C / C++ librarys for Http requests.
Find them here
Specifically, look for HTTPReq.c
Use boost library and poco. They both provide solutions for network programming. Boost also provide spirit library which you can use for parsing data from websites. Poco libraru also provides NetSSL, crypto solutions.
P.S. boost::spirit is not a library for parsing data from websites, it provides solution for parsing strings ...
you need to open a socket.
then you need to do an http get
somewhat like :-
http://www.esqsoft.com/examples/troubleshooting-http-using-telnet.htm
You could use the QNetworkAccessmanager class from Qt framework.
I'm assuming you need to use c++ for a reason, such as integration with existing software, otherwise, as per some of the other suggestions, choosing a language with a more convenient framework (eg: scripting language) would be better suited for the task.
If you would like to avoid getting your hands dirty with WINSOCK, or have the need to run on a platform other than windows, you could look at the using the boost asio library.
The following page contains links to simple sync and async http clients:
http://www.boost.org/doc/libs/1_37_0/doc/html/boost_asio/examples.html
You can find documentation on the library itself at:
http://www.boost.org/doc/libs/1_37_0/doc/html/boost_asio.html
Use c++ if you must, but it might be a lot less painful to use python.
Look at the Python httplib module for how to set the host you want to pull from etc. Python's available for free for most platforms and is enough like C++ that you can probably learn python a heck of a lot faster than you can learn to write a program controlled browser in c++. Well, maybe that's not true for everyone on this site, but I'll bet it's true for "most" of us. I used to get stock quotes updated in near real time from CNN Money years ago and IIRC it was around 100 lines of python code.
Hotei
Ideally, and you will think I am crazy, I can code some basic logic into a bash or korn script and open that functionality up to clients hitting them. There is a lot of plumbing involved in web services and I was wondering what tools and techniques more experienced developers have been using to prototype systems where a backend webservice may not be yet available. Do such tools exist for bash and ksh? What languages are the easiest to develop mockups in. BTW I am staring at an ecplipse IDE feeling a bit disgusted by what I am looking at. I just now got JBOSS installed and running... at its heart all these services are are socket connections to the client through a port ... or am I mistaken?
BTW: I am currently reading through this. And my disgust is increasing.
I share your disgust with eclipse. A language like Python is perfect for rapid prototyping. If you combine it with one of the many web frameworks (Pylons or Django would be my recommendation for Python), the amount of work you can accomplish quickly is astounding.
Seriously consider some other scripting language. I've been using Python and Ruby to build quick prototypes, and been very happy with them.
If you're free to make this choice, consider using a REST architecture instead of a WSDL and SOAP solution. RPC has its place, but if you can live with the restrictions of REST, life will be much easier. Even if you can't, it's a lot quicker for prototyping to use a dynamic language.