I'm running an embedded Jetty (9.1.0.v20131115) setup, and have several Handlers setup to process requests on several different contexts.
One of these Handlers performs login functionality when the user submits a form. It's setup as follows:
ContextHandler loginContext = new ContextHandler("/login");
loginContext.setHandler(new LoginHandler());
// Other handlers go here...
contexts.setHandlers(new Handler[]{rootContext, logoutContext, loginContext, resourceHandler});
server.setHandler(contexts);
That should be pretty standard and is nothing special. What perplexes me is that when I run the LoginHandler through a debugger, the HttpServletRequest object has no parameters, even though the form clearly has two form input elements!
here's a copy of the request, which I caught via netcat:
POST /login HTTP/1.1
Host: localhost:52520
Connection: keep-alive
Content-Length: 31
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://localhost:52520
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://localhost:52520/dashboard/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
username=myuser&password=mypass
On top of that, if I change the form action to GET instead of POST the parameters show up just fine!
Is there anything special that must be done to get a Handler to accept POST parameters?
Seems to work just fine.
package jetty;
import static org.hamcrest.CoreMatchers.*;
import static org.junit.Assert.*;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
import java.io.PrintWriter;
import java.io.StringReader;
import java.io.StringWriter;
import java.net.InetSocketAddress;
import java.net.Socket;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.eclipse.jetty.server.Request;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
import org.eclipse.jetty.server.handler.ContextHandler;
import org.eclipse.jetty.server.handler.DefaultHandler;
import org.eclipse.jetty.server.handler.HandlerCollection;
import org.eclipse.jetty.server.handler.HandlerWrapper;
import org.eclipse.jetty.util.IO;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
public class JettyPostTest
{
public static class LoginHandler extends HandlerWrapper
{
#Override
public void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response) throws IOException, ServletException
{
PrintWriter out = response.getWriter();
response.setContentType("text/plain");
out.printf("username = %s\n",request.getParameter("username"));
out.printf("password = %s\n",request.getParameter("password"));
baseRequest.setHandled(true);
}
}
private static Server server;
private static int port;
#BeforeClass
public static void startServer() throws Exception
{
server = new Server();
ServerConnector connector = new ServerConnector(server);
connector.setPort(0);
server.addConnector(connector);
// collection for handlers
HandlerCollection handlers = new HandlerCollection();
server.setHandler(handlers);
// login context
ContextHandler loginContext = new ContextHandler("/login");
loginContext.setHandler(new LoginHandler());
handlers.addHandler(loginContext);
// default handler
handlers.addHandler(new DefaultHandler());
// start server
server.start();
// grab port
port = connector.getLocalPort();
}
#AfterClass
public static void stopServer() throws Exception
{
server.stop();
}
#Test
public void testPostParameters() throws IOException
{
StringBuilder req = new StringBuilder();
req.append("POST /login/ HTTP/1.1\r\n");
req.append("Host: localhost:").append(port).append("\r\n");
req.append("Connection: close\r\n");
req.append("Content-Length: 31\r\n");
req.append("Cache-Control: max-age=0\r\n");
req.append("Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n");
req.append("Origin: http://localhost:").append(port).append("\r\n");
req.append("User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36\r\n");
req.append("Content-Type: application/x-www-form-urlencoded\r\n");
req.append("Referer: http://localhost:").append(port).append("/dashboard/\r\n");
req.append("Accept-Encoding: gzip,deflate,sdch\r\n");
req.append("Accept-Language: en-US,en;q=0.8\r\n");
req.append("\r\n");
req.append("username=myuser&password=mypass\r\n");
try (Socket socket = new Socket())
{
socket.connect(new InetSocketAddress("localhost",port));
// Write request
try (OutputStream out = socket.getOutputStream();
OutputStreamWriter writer = new OutputStreamWriter(out);
InputStream in = socket.getInputStream();
InputStreamReader reader = new InputStreamReader(in))
{
StringReader reqStream = new StringReader(req.toString());
IO.copy(reqStream,writer);
writer.flush();
out.flush();
StringWriter respStream = new StringWriter();
IO.copy(reader,respStream);
System.out.println(respStream.toString());
String expected = "username = myuser\npassword = mypass\n";
assertThat("Response",respStream.toString(),containsString(expected));
}
}
}
}
Results in the output:
2013-12-18 13:23:08.856:INFO:oejs.Server:main: jetty-9.1.0.v20131115
2013-12-18 13:23:08.888:INFO:oejsh.ContextHandler:main: Started o.e.j.s.h.ContextHandler#49ada86{/login,null,AVAILABLE}
2013-12-18 13:23:08.897:INFO:oejs.ServerConnector:main: Started ServerConnector#3f14b553{HTTP/1.1}{0.0.0.0:34456}
HTTP/1.1 200 OK
Content-Type: text/plain; charset=ISO-8859-1
Connection: close
Server: Jetty(9.1.0.v20131115)
username = myuser
password = mypass
2013-12-18 13:23:08.994:INFO:oejs.ServerConnector:main: Stopped ServerConnector#3f14b553{HTTP/1.1}{0.0.0.0:0}
2013-12-18 13:23:08.995:INFO:oejsh.ContextHandler:main: Stopped o.e.j.s.h.ContextHandler#49ada86{/login,null,UNAVAILABLE}
Only change I made from your request is to change Connection: keep-alive to Connection: close to let jetty close the connection. This change is minor and only made to allow the test to execute swiftly, using the original value does not change the results of the test.
Something piece of information is missing from your question.
Related
I need to install the RediSearch module on top of a GCP memorystore redis instance.
I followed the steps:
docker run -p 6379:6379 redislabs/redisearch:latest
I pushed this docker image to a Kubernetes cluster and exposed the external IP. I used that external IP and the 6379 port as configuration for my application but I'm not able to connect to RediSearch.
code:
import java.io.IOException;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.Description;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import io.redisearch.client.Client;
import io.redisearch.*;
public class RediSearch {
static Client client = new Client("testdoc1", "clusteripaddress", 8097);
private static final Logger LOG = LoggerFactory.getLogger(RediSearch.class);
public interface Options extends PipelineOptions {
#Description("gcp project id.")
#Default.String("XXXX")
String getProjectId();
void setProjectId(String projectId);
}
public static PipelineResult run(Options options) throws IOException {
Pipeline pipeline = Pipeline.create(options);
pipeline.apply(Create.of("test"))
.apply(ParDo.of(new DoFn<String, String>() {
private static final long serialVersionUID = 1L;
#ProcessElement
public void processElement(ProcessContext c) throws Exception {
String pubsubmsg = c.element();
Schema sc = new Schema()
.addTextField("title", 5.0)
.addTextField("body", 1.0)
.addNumericField("price");
client.createIndex(sc, Client.IndexOptions.Default());
Map<String, Object> fields = new HashMap<String, Object>();
fields.put("title", "hello world");
fields.put("body", "lorem ipsum");
fields.put("price", 800);
fields.put("price", 1337);
fields.put("price", 2000);
client.addDocument("searchdoc3", fields);
SearchResult[] res = client.searchBatch(new Query("hello world").limit(0, 5).setWithScores());
for (Document d : res[0].docs) {
LOG.info("redisearchlog{}",d.getId().startsWith("search"));
LOG.info("redisearchlog1{}",d.getProperties());
LOG.info("redisearchlog2{}",d.toString());
}
}
}));
return pipeline.run();
}
public static void main(String[] args) throws IOException {
Options options = PipelineOptionsFactory.fromArgs(args).as(Options.class);
run(options);
}
}
Error :
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.jedis.util.Pool.getResource(Pool.java:59)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:234)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:15)
at io.redisearch.client.Client._conn(Client.java:137)
at io.redisearch.client.Client.getAllConfig(Client.java:275)
at com.testing.redisearch.RediSearch$1.processElement(RediSearch.java:59)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to host xxxxxxxxxxx:6379
at redis.clients.jedis.Connection.connect(Connection.java:204)
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:100)
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1894)
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:117)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349)
at redis.clients.jedis.util.Pool.getResource(Pool.java:50)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:234)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:15)
at io.redisearch.client.Client._conn(Client.java:137)
at io.redisearch.client.Client.getAllConfig(Client.java:275)
at com.testing.redisearch.RediSearch$1.processElement(RediSearch.java:59)
at com.testing.redisearch.RediSearch$1$DoFnInvoker.invokeProcessElement(Unknown Source)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)
at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:183)
at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:335)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:411)
at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:380)
at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:305)
at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at redis.clients.jedis.Connection.connect(Connection.java:181)
... 31 more
Any solution is appreciated.
There are multiple causes as per the error JedisConnectionException: Could not get a resource from the pool. According to the answers in this question the problem is that the connection to RediSearch couldn't be established, be it because Redis is not running, the connection times out or it cannot be allocated.
Regardless, I have noticed that even though you deploy Redis on port 6379, in your code you are trying to access it on port 8097. Please change your Client declaration to the following and retry the connection.
static Client client = new Client("testdoc1", "<cluster_ip_address>", 6379);
If you are looking to have the RediSearch module in your memorystore instance, it appears that may not be supported yet. You can see in the google cloud docs here that at the time of writing this, even for version 5.0, redis modules are not supported.
Dependencies:
RestLet 2.3
Jetty 9.4
MyServer.Java
import org.apache.log4j.Logger;
import org.apache.log4j.xml.DOMConfigurator;
import org.restlet.Component;
import org.restlet.Server;
import org.restlet.data.Protocol;
public class MyServer {
public static void main(String[] a) {
Component component = new Component();
Server server = component.getServers().add(Protocol.HTTP, 5000);
server.getContext().getParameters().add("useForwardedForHeader", "true");
component.getDefaultHost().attach(new MyApplication());
try {
System.out.println("Going to start MyServer");
component.start();
System.out.println("Server started on port " + port);
} catch (Exception e) {
log.error(e, e);
}
}
}
MyApplication.java
import com.myApp.resources.*;
import org.restlet.Application;
import org.restlet.Restlet;
import org.restlet.routing.Router;
public class MyApplication extends Application {
#Override
public Restlet createInboundRoot() {
Router router = new Router(getContext());
router.attach("/App", MyResource.class);
return router;
}
}
MyResource.Java
import org.restlet.resource.ServerResource;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.util.Date;
#Path("/getDetails")
public class MyResource extends ServerResource{
#GET
#Path("/{number}")
public Object getData(#PathParam("number") String number) {
try {
System.out.println("number: " + number);
} catch (Exception e) {
e.printStackTrace();
}
Response.ResponseBuilder builder = Response.ok("Received", MediaType.TEXT_PLAIN);
return builder.build();
}
Now I'm starting my application it starting with the following console log...
Going to start MyServer
[main] INFO org.eclipse.jetty.util.log - Logging initialized #291ms to org.eclipse.jetty.util.log.Slf4jLog
[main] WARN org.eclipse.jetty.server.AbstractConnector - Ignoring deprecated socket close linger time
Starting the Jetty [HTTP/1.1] server on port 5000
[main] INFO org.eclipse.jetty.server.Server - jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_181-b13
[main] INFO org.eclipse.jetty.server.AbstractConnector - Started ServerConnector#3327bd23{HTTP/1.1,[http/1.1]}{0.0.0.0:5001}
[main] INFO org.eclipse.jetty.server.Server - Started #446ms
Starting com.myApp.server.MyApplication application
Server started on port 5000
Now when I'm trying to get the result by using localhost:5000/App/getDetails/5426. I'm unable to get the response.
Even I'm getting following message on the console...
[qtp586617651-20] WARN org.eclipse.jetty.server.HttpChannel - /App/getDetails/GJ0125 java.io.IOException: Restlet exception
Any help would be appreciated. Also if any other way to implement then let me know.
I am trying to build a scraper, but I keep getting the 503 blocking error. I can still access the website manually, so my IP address hasn't been blocked. I keep switching user agents and still can't get my code to run all the way through. Sometimes I get up to 15, sometimes I don't get any, but it always fails eventually. I have no doubt that I'm doing something wrong in my code. I did shave it down to fit, though, so please keep that in mind. How do I fix this without using third parties?
import requests
import urllib2
from urllib2 import urlopen
import random
from contextlib import closing
from bs4 import BeautifulSoup
import ssl
import parser
import time
from time import sleep
def Parser(urls):
randomint = random.randint(0, 2)
randomtime = random.randint(5, 30)
url = "https://www.website.com"
user_agents = [
"Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)",
"Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)",
"Opera/9.80 (Windows NT 6.1; U; cs) Presto/2.2.15 Version/10.00"
]
index = 0
opener = urllib2.build_opener()
req = opener.addheaders = [('User-agent', user_agents[randomint])]
def ReadUPC():
UPCList = [
'upc',
'upc2',
'upc3',
'upc4',
'etc.'
]
extracted_data = []
for i in UPCList:
urls = "https://www.website.com" + i
randomtime = random.randint(5, 30)
Soup = BeautifulSoup(urlopen(urls), "lxml")
price = Soup.find("span", { "class": "a-size-base a-color-price s-price a-text-bold"})
sleep(randomtime)
randomt = random.randint(5, 15)
print "ref url:", urls
sleep(randomt)
print "Our price:",price
sleep(randomtime)
if __name__ == "__main__":
ReadUPC()
index = index + 1
sleep(10)
554 class HTTPDefaultErrorHandler(BaseHandler):
555 def http_error_default(self, req, fp, code, msg, hdrs):
556 raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
557
558 class HTTPRedirectHandler(BaseHandler):
HTTPError: HTTP Error 503: Service Unavailable
What website you are scraping? most websites uses cookies to recognize the user as well. Please enable cookies in your code.
Also open that link in browser and along with Firebug and see Headers being sent to server by your browser while making request. and then try to fake all those headers.
PS:
In my view, sending random user-agent strings from SAME IP wont make any difference, unless you are rotating IPs.
Behave like a normal human being using a browser. That website seems to be designed to analyze your behaviour and sees that you're a scraper, and wants to block you; in the easiest case, a minimal JavaScript that changes link URLs on the fly would be enough to disable "dumb" scrapers.
There's elegant ways to solve this dilemma, for example by instrumenting a browser, but that won't happen without external tools.
I have a Java EE server/client architecture which communicate with each other by using SSL connection. When the connection is made, the client can interrogate the server web services. My question is how can I access to client certificate information in the server web service ? My server controller below :
import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
#Path("mycontroller")
#Consumes(MediaType.APPLICATION_XML)
#Produces(MediaType.APPLICATION_XML)
public class Controller {
#GET
#Path("dosomething")
public Response doSomething() {
// How can I have access to certificate information here ?
return Response.ok().build();
}
}
I found a way to do what I wanted.
First, the server has to be configurated to require client certificate authentication. In my case I use a JBoss server and had to add this in the standalone.xml file :
...
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
...
<connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" enable-lookups="false" secure="true">
<ssl name="localhost" key-alias="localhost" password="server" certificate-file="${jboss.server.config.dir}/server.jks" certificate-key-file="${jboss.server.config.dir}/server.jks" ca-certificate-file="${jboss.server.config.dir}/truststore.jks" protocol="TLSv1" verify-client="true" />
</connector>
...
</subsystem>
...
And then in my controller I had to inject HttpServletRequest and finally I could obtain an instance of X509Certificate which contains the certificate information :
import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.servlet.http.HttpServletRequest;
import java.security.cert.X509Certificate;
#Path("mycontroller")
#Consumes(MediaType.APPLICATION_XML)
#Produces(MediaType.APPLICATION_XML)
public class Controller {
#Context
private HttpServletRequest request;
#GET
#Path("dosomething")
public Response doSomething() {
X509Certificate[] certChain = (X509Certificate[]) request.getAttribute("javax.servlet.request.X509Certificate");
X509Certificate certificate = certChain[0];
return Response.ok().build();
}
}
If you are looking for the standard certificate information that would be found in the HTTP Headers and the HTTP Servlet Reqeust object, such as Client certificate information from Apache HTTP reverse proxy. You can inject these
For Example:
#Context private HttpServletRequest servletRequest;
#Context private HttpServletContext servletContext;
( see Get HttpServletRequest in Jax Rs / Appfuse application? or in the Java EE tutorial )
If you wish to access the keystore file and load the private key of the certificate, then file access should be done through a JNDI file resource or a JCA adaptor.
But i would advise caution, the application server should handle all the SSL/TLS connection security your WAR component just declares that it wants the connection to be "confidential" in the web.xml file. Mixing the message level security and authentication with the applicaition or transport protocol security can break separation of concerns. i.e. keeping authentication attached to the message in a bus or hub scenario.
I am using mechanize in python to log into a HTTPS page. The login is successful but the output is just a SAML response. I am unable to get the actual page source which i get when opening with my browser.
import mechanize
import getpass
import cookielib
br=mechanize.Browser()
br.set_handle_robots(False)
b=[]
cj = cookielib.CookieJar()
br.set_cookiejar(cj)
pw=getpass.getpass("Enter Your Password Here: ")
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)
br.addheaders = [('User-agent','Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11'),
('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'),
('Accept-Encoding', 'gzip,deflate,sdch'),
('Accept-Language', 'en-US,en;q=0.8'),
('Accept-Charset', 'ISO-8859-1,utf-8;q=0.7,*;q=0.3')]
br.open("https:***single sign on login url***")
br.select_form(name='login-form')
br.form['userid']='id'
br.form['password']=pw
response=br.submit()
print response.read()
a=br.open("https:****url****")
for i in range(1000):
b.append(a.readline())
print b
I get SAML output which is encrypted but i dont know how to reply with that SAML post to get to the actual page.