Related
Is is possible to set batch size on function level within a webjob?
I have multiple functions in a webjob, some of them depend on other external APIs which does not allow a high degree of parallelization.
I have seen only the Singleton attribute which is not exactly what I am looking for.
just figured out that this is possible with a custom QueueProcessorFactory I already use.
An example from MS is here:
https://github.com/Azure/azure-webjobs-sdk-samples/blob/master/BasicSamples/MiscOperations/CustomQueueProcessorFactory.cs
Having attributes for this would be nice ;-)
Alex
Yeah, custom QueueProcessor instances are designed were designed to be the "escape hatch" allowing you full control in advanced scenarios. We want to keep the mainline paths simple and easy to use, while allowing you to drop down and deeply customize when needed. Adding a bunch of override options on QueueTriggerAttribute itself would be possible, but could also complicate the programming model.
If you would like to suggest a change, I suggest you log issues in the public repo: https://github.com/Azure/azure-webjobs-sdk/issues
Thanks :)
I have a C++ process running in the background that will be generating 'events' infrequently that a Python process running on the same box will need to pick up.
The code on the C side needs to be as lightweight as possible.
The Python side is read-only.
The implementation must be cross-platform.
The data being sent is very simple.
What are my options?
Thanks
zeromq -- and nothing else. encode the messages as strings.
However, If you want to get serialiazation from a library use protobuf it will generate classes for Python and C++. You use the SerializeToString() and ParseFromString() functions on either end, and then pipe the strings via ZeroMq.
Problem solved, as I doubt any other solution is faster, and neither will any other solution be as easy to wire-up and simple to understand.
If want to use specific system primitives for rpc such as named pipes on Windows and Unix Domain Sockets on unix then you should look at Boost::ASIO. However, unless you have (a) a networking background, and (b) a very good understanding of C++, this will be very time consuming
Use zeromq, it's about as simple as you can get.
Google's protobuf is a great library for RPC between programs. It generates bindings for Python and C++.
If you need a distributed messaging system, you could also use something like RabbitMQ, zeromq, or ActiveMQ. See this question for a discussion on the message queue libraries.
Another option is to just call your C code from your Python code using the ctypes module rather than running the two programs separately.
How complex is your data? If it is simple I would serialize it as a string. If it was moderately complex I would use JSON. TCP is a good cross-platform IPC transport. Since you say that this IPC is rare the performance isn't very important, and TCP+JSON will be fine.
You can use Google GRPC for this
I will say you create a DLL that will manage the communication between the two. The python will load DLL and call method like getData() and the DLL will in turn communicate with process and get the data.
That should not be hard.
Also you can use XML file or SQLite database or any database to query data. The daemon will update DB and Python will keep querying. There might be a filed for indicating if the data in DB is already updated by daemon and then Python will query.
Of course it depends on performance and accuracy factors!
Is there a relatively easy way to display the output of a C++ program on a webpage? And I don't mean manually, in other words, you see it on a webpage as it runs not as in I make a code tag and write it in myself.
EDIT: Just so everybody can get this clear I am going to post this up here. I am NOT trying to make a webpage in C++. Please excuse me if this sounds spiteful or anything but I am getting a lot of answers relating to that.
Step one, get yourself a server-side language. Be that PHP, ASP, Python, Ruby, whatever. Get it set up so you can serve it.
Step two, find your language's exec equivalent. Practically all of them have them. It'll let you run a command as if it were from the command line, usually with arguments and capture the output. Here's PHP's:
http://php.net/manual/en/function.exec.php
Of course, if you're passing user-input as arguments, sanitise!
I've just seen that you accepted Scott's answer. I usually wouldn't chase up a SO thread so persistently but I fear you're about to make a mistake that you'll come to regret down the line. Giving direct access to your program and its own built-in server is a terrible idea for two reasons:
You waste a day implementing this built-in server and then getting it to persist and testing it
More importantly, you've just opened up another attack vector into your server. When it comes to security, keep it simple.
You're far better having your C++ app running behind another (mature) server side language as all the work is done for you and it can filter the input to keep things safe.
You could write a CGI app in C++, or you could use an existing web server language to execute the command and send the output to the client.
You want to use Witty.
Wt (pronounced 'witty') is a C++
library for developing interactive web
applications.
The API is widget-centric and similar
to desktop GUI APIs. To the developer,
it offers complete abstraction of any
web-specific implementation details,
including event handling, graphics
support, graceful degradation (or
progressive enhancement), and pretty
URLs.
Unlike many page-based frameworks, Wt
was designed for creating stateful
applications that are at the same time
highly interactive (leveraging
techinques such as AJAX to their
fullest) and accessible (supporting
plain HTML browsers), using automatic
graceful degradation or progressive
enhancement.
The library comes with an application
server that acts as a stand-alone web
server or integrates through FastCGI
with other web servers.
I am not sure this is what you are looking for but you may want CGI You may want to look at this SO question, C++ may not be the best language for what you want to do.
based off the questions you posted Writing a web app like what you want is no simple task. What I would recommend is use some other library (this is one i found with a quick google) to get a web console on your server and give the user it is running under execute deny permissions on every folder except the folder you have your app installed.
This is still is a risky method if you don't set up the security correctly but it is the easiest solution without digging around too much on existing libraries to just have the application interactive.
EDIT --
The "Best" solution is learn AJAX and have your program post its own pages with it but like I said, it will not be easy.
It sounds like you want something like a telnet session embedded in a webpage. A quick google turns up many Java telnet apps, though I'm not qualified to evaluate which would be most ideal to embed in html.
You would set up the login script on the host machine to run your c++ app and the user would interact with it through the shell window. Note though that this will only work for pure command line apps. If you want to use a GUI app in this way, then you should look into remote desktop software or VNC.
It may be worth looking into Adobe's "Alchemy" project on Adobe Labs
This may help you with what you're trying to achieve.
:)
Are you looking for something like what codepad.org does? I believe they explain how they did it here.
There is a library called C++ Server Pages - Poco. I used it for one of my college project, its pretty good. There is also good documentation to get started with, u can find it here http://pocoproject.org/docs/
I was told I have to use winsock, but I dont know where to start. For example, I am trying to access, lets say http://www.newegg.com/, I am trying to get the text title of just the three front page products. Any help is greatly appreciated. :D
I'd also recommend libcurl for this sort of thing.
You can use the cURL command line tool to generate sample code as well, which is helpful for experimentation.
W3.org themselves provide sample C / C++ librarys for Http requests.
Find them here
Specifically, look for HTTPReq.c
Use boost library and poco. They both provide solutions for network programming. Boost also provide spirit library which you can use for parsing data from websites. Poco libraru also provides NetSSL, crypto solutions.
P.S. boost::spirit is not a library for parsing data from websites, it provides solution for parsing strings ...
you need to open a socket.
then you need to do an http get
somewhat like :-
http://www.esqsoft.com/examples/troubleshooting-http-using-telnet.htm
You could use the QNetworkAccessmanager class from Qt framework.
I'm assuming you need to use c++ for a reason, such as integration with existing software, otherwise, as per some of the other suggestions, choosing a language with a more convenient framework (eg: scripting language) would be better suited for the task.
If you would like to avoid getting your hands dirty with WINSOCK, or have the need to run on a platform other than windows, you could look at the using the boost asio library.
The following page contains links to simple sync and async http clients:
http://www.boost.org/doc/libs/1_37_0/doc/html/boost_asio/examples.html
You can find documentation on the library itself at:
http://www.boost.org/doc/libs/1_37_0/doc/html/boost_asio.html
Use c++ if you must, but it might be a lot less painful to use python.
Look at the Python httplib module for how to set the host you want to pull from etc. Python's available for free for most platforms and is enough like C++ that you can probably learn python a heck of a lot faster than you can learn to write a program controlled browser in c++. Well, maybe that's not true for everyone on this site, but I'll bet it's true for "most" of us. I used to get stock quotes updated in near real time from CNN Money years ago and IIRC it was around 100 lines of python code.
Hotei
What simple tools would you recommend to read a text file of addresses, send each record separately to a web service for geocoding, and save the batch of results as a text file?
Looking for no-frills component(s) with usage examples, for minimal code-from-scratch. Language irrelevant as long as dev environment is easy to install.
Requirements
- usable by unsophisticated programmer
- low or no cost
- runs under Windows.
Second thought:
How easy would this be to do inside a browser using JavaScript and a library or two?
I'd go for Java and use a flat file parsing library like jFFP or Flatworm
These libraries are pretty easy to understand and to use (I've worked with both of them in the past) and they both provide code samples.
Spring Integration would be another good option but the learning curve might be too big if you are not familiar with Spring and it might be overkill for your simple workflow.
Actually, in your case, I think I'd choose Flatworm for the parsing. You'll find code samples on its website or in How to read and parse flat files in Java. And you could even use it to write your output file like in Writing flat files in Java with Flatworm).
For the SOAP part, I'd use the JAX-WS Reference Implementation (which is included in the JDK 6 so you won't have to add any library if you are using Java 6) and Netbeans IDE. Netbeans IDE has very good support for developing JAX-WS Web Services Client (or here for later versions of Netbeans) and should really ease the process. Once the various classes generated, calling the web service is a matter of 3 lines of code as shown in the examples of the provided links:
// Call Web Service Operation
com.cdyne.ws.Check service = new com.cdyne.ws.Check();
com.cdyne.ws.CheckSoap port = service.getCheckSoap();
// TODO initialize WS operation arguments here
java.lang.String bodyText = "";
java.lang.String licenseKey = "";
// TODO process result here
com.cdyne.ws.DocumentSummary result = port.checkTextBody(bodyText, licenseKey);
Given the generic nature of the requirements, the relatively simple workflow, but its potential to bring a few twists and turns in the design (for example, the need of using https rather than http for webservices, the need of producing some odd token for authentication, or some fancy marshaling or conversion etc.) it might be best to use a modern script language. A very basic plan could be to use a plain shell script (a bat file), base on curl and a few other command line utilities, but this approach may not be flexible enough to deal with some requirements; instead languages such as Perl, PHP, Python, Ruby would be much preferable.
This would provide a low entry barrier, the ability to test elements of the application interactively before putting them into a formal script, and to leverage extensive libraries to deal with the various requirements that may arise, such as the storage of configuration parameters, parsing detail, output format, webservices, maths associated with geo positions etc. etc.
My inclination would be to use Python, but as said most other modern dynamic languages would do.
I would use XMLunit with Eclipse IDE, + JUnit, and JDK1.6 . A finished program that does this might only be 100 lines of code. It's doable by someone who is a novice programmer...
When the program is done you can compile as an .exe file for future use.
I would choose "Strawberry Perl" as my second choice for programming language. Python is slightly harder to use I think.