Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I am looking to build a webapp to improve user experience in booking railway tickets in India. The API is impossible to get due to hefty charge to procure it. I have seen many apps that provide details of the trains etc through their apps.
My Question is how are they scraping data from the website.In general how can I legally get data shown to user (I don't want payment and stuff that are impossible without API) on any website. How do people scrape such data? Any tools/methods?
Bear with me if question is naive. I'm pretty new to this stuff.
They can get the train schedule information using any one of several programming languages though it is most likely done with ordinary PHP and any good webserver host. For example all indian train schedules can be found on the indianrail.gov website.
Sending a specially built URL to ..
http://www.indianrail.gov.in/cgi_bin/inet_trnnum_cgi.cgi?lccp_trnname=1123
using the POST method of sending form data should give you all the details for train number 1123 After that it becomes just a simple task of tidying up the results for storage in a database.
Update: well armoured site its checking both the user agent and referer of inbound requests.
Ammendum: the indianrail.gov site is changing to http://www.trainenquiry.com/ -> will have to take another look
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I am a freelance software engineer, I have worked in few areas of computer science, I have made some e-commerce websites in the past. Now I have an opportunity to build a big enterprise level system. I can not disclose specifics about the application due to NDA I signed so pardon me if my question seems broad, let me know in the comments if you require clarification. I appreciate your help.
About Application:
In this application, I would require building a system like uber, there will be people at my client's end for
resolving customer issues, so a CRM is also needed.
customers will be using this app, so I have
to design a separate system that can manage tickets and access
database.
My question is where to start designing such application. I guess I would require DynamoDB and AWS, I have divided modules into parts such as Client App, Database, Dashboard etc. I want to know if there is some case study that can help me decide how to design such large application.
I found this link useful, it gave me an idea of work, but still, I believe it's a long way from money shot.
[EDIT]
To narrow down the scope of the question, What Backend server should be chosen for an application that will serve one hundred thousand users per hour. I will use Mongo DB as the database, and Python as backend scripting language.
IBM has a nice article on Enterprise Architecture,
https://www.ibm.com/developerworks/rational/library/enterprise-architecture-maximum-value/
Before building the software, design how it should work and choose your software components according to that.
Previously you might need costly infrastructure to think something, with recent technologies, you can do them at a lower cost. You need to apply the right architecture and engineering when designing your application.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I have a webapp running with Django. I would like to make an Enterprise Edition of this website. In exchange for a yearly fee, I would like to allow companies to host the webapp on their own server and benefit from unique features
The problem is that I have no idea how to proceed. How can I execute this, without sending my whole code over to their computer and launching the django app there?
Is there something like generating an .exe that would run the django app without sending my code? How do companies usually proceed to make their tools available on another company's intranet?
well I haven't heard something like that, but what you can do is using a license in your code, there are many types of licenses, from open source to trade secret license, read more about it
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
How can I make our software's data available as an app in Power BI so our customers can choose to integrate their data in Power BI?
I know there is a Rest API and I have seem many samples of how to use this for internal software. However I don't want each customer to have to create an api application. I want our app to appear like Salesforce & Zendesk etc. do so the customer can simply pick us from the list.
Hope this makes sense.
I have helped a client of mine work through the process and it involves working with Microsoft to get the public content pack published. The process is outlined here. I would start by following the steps indicated there like filling out the nomination form and reaching out to support before you begin. There are a number of constraints around authentication and APIs and refresh you need to be aware of before you begin development and it is best to discuss this with support.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I am looking for a way to pass log events from AWS application to my company site.
The thing is that the AWS application is 100% firewalled from everything except only one IP address because it's encryption related service.
I just don't know what service I should use to do this. There's so many services so I do really have no idea what is it.
I think I'd just use simple message service, does this makes sense? The thing is there's plenty of events (let's say 1M per day), so I don't want big extra costs for this.
Sorry for the generic question, but I think it's quite concrete - "What is the most optimal way to pass event message from AWS when volume is approx 1M per day each 256 bytes on average?".
I'd like to connect to AWS service instead to any of the EC2 hosts...
On both sides I have tomcats with AWS-SDK.
I just want to avoid rewriting. Maybe I should do it with S3? The files are immutable, but I could upload files every 1h. I don't need real-time events. I just need to have logfiles on site for analysis of user experience and that customers can access it, but having log in 1M chunks would either require further assembling etc, I am really confused, sorry.
Kinesis is good for streaming event data. S3 is good if you already have files that you want stored.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 11 months ago.
Improve this question
I wish to write a web service server (using http protocol) in OCaml.
For a simple example, I want to write a http server. User can access it and provide parameter like http://myservice.com?i=1&j=2. Then my server get the request and parameters and calculate i+j and return the result.
Of course, my service will be more complicated for the calculation part. Instead of this simple example of calculation, what I really need to do is to
access the database (MongoDB) to get some data
access another 3rd party web service to get more data
calculate all data to get a result and return to the user.
So, I also need to consider parallelism / multi-threading, although I want to start with simple case first.
My questions are:
Which library should I use to first set up such a http server? I have looked into Ocamlnet 3 and think it might be a good candidate, but it lacks good tutorial and I still don't know how to use nethttpd or netplex etc.
How should I design the architecture of my web application? I know OCaml is not good at parallelism, then how can I make each service instance not blocking?