I am implementing small Java ME app. This app gets some data from 3rd patty resource and needs to be authenticated before. I do first call for get cookies (it was easy), and the second call with this cookies for get data. I googled a little how to do it, and found next solution - Deal with cookie with J2ME
I have changed this code to next for my purpose:
public void getData(String url,String cookie) {
HttpConnection hpc = null;
InputStream is = null;
try {
hpc = (HttpConnection) Connector.open(url);
hpc.setRequestProperty("cookie", cookie);
hpc.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
hpc.setRequestProperty("Accept-Encoding", "gzip, deflate");
hpc.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
is = hpc.openInputStream();
int length = (int) hpc.getLength();
byte[] response = new byte[length];
is.read(response);
String strResponse = new String(response);
} catch (Exception e) {
System.out.println(e.getMessage() + " " + e.toString());
} finally {
try {
if (is != null)
is.close();
if (hpc != null)
hpc.close();
} catch (Exception e) {}
}
}
I get something like to the next
??ÑÁNÃ0à;O±(²§M}A-?#
.?PYS¨Ôe¥Í#\üìde??XÊo}Vâ]hk?6ëµóA|µvÞz'Íà?wAúêmw4í0?ÐÆ?ÚMW=?òêz CÛUa:6Ö7¼T?<oF?nh6[_0?l4?äê&)?çó³?ÅÕúf¨ä(.? ªDÙ??§?ÊP+??(:?Á,Si¾ïA¥ã-jJÅÄ8ÊbBçL)gs.S.þG5ÌÀÆéX}CÁíÑ-þ?BDK`²?\¶?ó3I÷ô±e]°6¬c?q?Ó?¼?Y.¯??Y?%?ÏP1è?ìw;?È Ò??e
|ôh0?
How can I decode this?
Stupid me. I didn't take to consideration next code: hpc.setRequestProperty("Accept-Encoding", "gzip, deflate"); I get coded in ZIP response and everything that I need it decode it.
Related
Please see the below code sample
JavaRDD<String> mapRDD = filteredRecords
.map(new Function<String, String>() {
#Override
public String call(String url) throws Exception {
BufferedReader in = null;
URL formatURL = new URL((url.replaceAll("\"", ""))
.trim());
try {
HttpURLConnection con = (HttpURLConnection) formatURL
.openConnection();
in = new BufferedReader(new InputStreamReader(con
.getInputStream()));
return in.readLine();
} finally {
if (in != null) {
in.close();
}
}
}
});
here url is http GET request. example
http://ip:port/cyb/test?event=movie&id=604568837&name=SID×tamp_secs=1460494800×tamp_millis=1461729600000&back_up_id=676700166
This piece of code is very slow . IP and port are random and load is distributed so ip can have 20 different value with port so I dont see bottleneck .
When I comment
in = new BufferedReader(new InputStreamReader(con
.getInputStream()));
return in.readLine();
The code is too fast.
NOTE: Input data to process is 10GB. Using spark to read from S3.
is there anything wrong I am doing with BufferedReader or InputStreamReader any alternative .
I cant use foreach in spark as I have to get the response back from server and need to save JAVARdd as textFile on HDFS.
if we use mappartition code something as below
JavaRDD<String> mapRDD = filteredRecords.mapPartitions(new FlatMapFunction<Iterator<String>, String>() {
#Override
public Iterable<String> call(Iterator<String> tuple) throws Exception {
final List<String> rddList = new ArrayList<String>();
Iterable<String> iterable = new Iterable<String>() {
#Override
public Iterator<String> iterator() {
return rddList.iterator();
}
};
while(tuple.hasNext()) {
URL formatURL = new URL((tuple.next().replaceAll("\"", ""))
.trim());
HttpURLConnection con = (HttpURLConnection) formatURL
.openConnection();
try(BufferedReader br = new BufferedReader(new InputStreamReader(con
.getInputStream()))) {
rddList.add(br.readLine());
} catch (IOException ex) {
return rddList;
}
}
return iterable;
}
});
here also for each record we are doing same .. isnt it ?
Currently you are using
map function
which creates a url request for each row in the partition.
You can use
mapPartition
Which will make the code run faster as it creates connection to the server only once , that is only one connection per partition.
A big cost here is setting up TCP/HTTPS connections. This is exacerbated by the fact that Even if you only read the first (short) line of a large file, in an attempt to re-use HTTP/1.1 connections better, modern HTTP clients try to read() to the end of the file, so avoiding aborting the connection. This is a good strategy for small files, but not for those in MB.
There is a solution there: set the content-length on the read, so that only a smaller block is read in, reducing the cost of the close(); the connection recycling then reduces HTTPS setup costs. This is what the latest Hadoop/Spark S3A client does if you set fadvise=random on the connection: requests blocks rather than the entire multi-GB file. Be aware though: that design is actually really bad if you are going byte-by-byte through a file...
I am new to AWS, Can any one please help me how to create users on virtual machine using java code. I have created windows ec2 instance and able to connect using sshClient, but unable to create users. I dont have any idea about how it can be done.
[I am new to stack overflow please forgive me if wrong formatting]
Answering my own question... It may help someone..
You can use execute command to create user on remote windows...
Example:
public boolean executeCommands(String commands, SSHRequestParam param)
throws Exception,IOException {
boolean response = false;
try {
Connection conn = getConnection(param);
Session sess = conn.openSession();
sess.execCommand(commands);
sess.waitForCondition(ChannelCondition.EXIT_SIGNAL, 0);
InputStream stdout = new StreamGobbler(sess.getStdout());
BufferedReader br = new BufferedReader(
new InputStreamReader(stdout));
while (true) {
String line = br.readLine();
if (line == null)
break;
response=true;
System.out.println(line);
}
InputStream stderr = new StreamGobbler(sess.getStderr());
BufferedReader br1 = new BufferedReader(new InputStreamReader(
stderr));
while (true) {
String line = br1.readLine();
if (line == null)
break;
response=false;
System.out.println(line);
}
/* Show exit status, if available (otherwise "null") */
System.out.println("ExitCode: " + sess.getExitStatus());
/* Close this session */
System.out.println("closing session");
sess.close();
closeConnection(conn);
} catch (IOException e) {
System.out.println("Exception..while executing command.: " + commands);
e.printStackTrace();
}
System.out.println("Command executed : "+response);
return response;
}
public void createUser(String userName, String password,SSHRequestParam param)
throws Exception,IOException {
String command = "net user " +userName+" "+password+" /add" ;
if(executeCommands(command,param))
System.out.println("User created successfully");
else
System.out.println("User didn't create....");
}
In above example param variable stores the parameters to connect to the server i.e. hostname, username, password....
I have been trying to implement a video upload facebook feature for my mobile app for a while now but never really succeeded with rest. I learnt yesterday that the graph alternative was available.
After getting a few errors to do with access key mainly i have gotten to the point where output stream succesfully writes the movie file and the input stream just receives an empty json array once i have written the 3gp file.
Anyone any idea why I would get an empty json array and no video gets published when i get all my code to run, i get 200 response code and the server sends me a non error response?
Any help greatly appreciated.
Here is the class that gets the blank json array (send method). I have appended the token to the url and in the table to be sure. I am sorry if the code is untidy buts it just been a day of trial and error.
import java.io.ByteArrayOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Enumeration;
import java.util.Hashtable;
import javax.microedition.io.Connector;
import javax.microedition.io.HttpConnection;
import net.rim.device.api.io.http.HttpProtocolConstants;
public class HttpMultipartRequest2
{
static final String BOUNDARY = "----------V2ymHFg03ehbqgZCaKO6jy";
byte[] postBytes = null;
String url = null;
Hashtable paramsTable;
public HttpMultipartRequest2(String url, Hashtable params,
String fileField, String fileName, String fileType, byte[] fileBytes) throws Exception
{
this.url = url;
String boundary = getBoundaryString();
paramsTable = params;
String boundaryMessage = getBoundaryMessage(boundary, params, fileField, fileName, fileType);
String endBoundary = "\r\n--" + boundary + "--\r\n";
ByteArrayOutputStream bos = new ByteArrayOutputStream();
bos.write(boundaryMessage.getBytes());
bos.write(fileBytes);
bos.write(endBoundary.getBytes());
this.postBytes = bos.toByteArray();
bos.close();
}
String getBoundaryString() {
return BOUNDARY;
}
String getBoundaryMessage(String boundary, Hashtable params, String fileField, String fileName, String fileType)
{
StringBuffer res = new StringBuffer("--").append(boundary).append("\r\n");
Enumeration keys = params.keys();
while(keys.hasMoreElements())
{
String key = (String)keys.nextElement();
String value = (String)params.get(key);
res.append("Content-Disposition: form-data; name=\"").append(key).append("\"\r\n")
.append("\r\n").append(value).append("\r\n")
.append("--").append(boundary).append("\r\n");
}
res.append("Content-Disposition: form-data; name=\"").append(fileField)
.append("\"; filename=\"").append(fileName).append("\"\r\n")
.append("Content-Type: ").append(fileType).append("\r\n\r\n");
Log.info(("res "+res.toString()));
return res.toString();
}
public String send() throws Exception
{
StringBuffer sb = new StringBuffer();
HttpConnection hc = null;
InputStream is = null;
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] res = null;
try
{
Log.info("before hc open"+ url);
hc = (HttpConnection) Connector.open(url);
hc.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + getBoundaryString());
hc.setRequestProperty("access_token", (String)paramsTable.get("access_token"));
hc.setRequestProperty(HttpProtocolConstants.HEADER_CONTENT_LENGTH, String.valueOf(postBytes.length));
hc.setRequestProperty( "x-rim-transcode-content", "none" );
ByteArrayOutputStream out = new ByteArrayOutputStream();
OutputStream dos = hc.openOutputStream();
is = hc.openInputStream();
Log.info("before dos write responsecode");// + hc.getResponseCode());
out.write(postBytes, 0, postBytes.length);
//Log.info("flushing"+ hc.getResponseCode());
Log.info("after doswrite responsecode");
dos.write(out.toByteArray());
dos.flush();
Log.info("after flush");
if(dos!=null)
dos.close();
int ch;
Log.info("before openinput ");
Log.info("after openinput ");
while ((ch = is.read()) != -1)
{
bos.write(ch);
sb.append((char)ch);
Log.info("char"+(char)ch);
}
res = bos.toByteArray();
Log.info("Response recieved from Server is : " + sb.toString() );
}
catch(Exception e)
{
Log.info(hc.getResponseCode() + "sexce"+e);
}
catch(OutOfMemoryError error)
{
Log.info("outofmemory " + error);
System.gc();
}
finally
{
try
{
if(bos != null)
bos.close();
if(is != null)
is.close();
if(hc != null)
hc.close();
}
catch(Exception e2)
{
Log.info("finally exception"+ e2);
}
}
return sb.toString();
}
}
Are you trying to upload to a user's feed or to a page? There is an Open Bug regarding posting to pages.
Also, could you post some code?
Assuming that you've read the documentation:
Facebook Graph API->Video
And that you are using graph-video.facebook.com, not graph.facebook.com.
Environment: C#, .Net 3.5, Sql Server 2005
I have a method that works in a stand-alone C# console application project. It creates an XMLElement from data in the database and uses a private method to send it to a web service on our local network. When run from VS in this test project, it runs in < 5 seconds.
I copied the class into a CLR project, built it, and installed it in SQL Server (WITH PERMISSION_SET = EXTERNAL_ACCESS). The only difference is the SqlContext.Pipe.Send() calls that I added for debugging.
I am testing it by using an EXECUTE command one stored procedure (in the CLR) from an SSMS query window. It never returns. When I stop execution of the call after a minute, the last thing displayed is "Calling GetResponse() using http://servername:53694/odata.svc/Customers/". Any ideas as to why the GetResponse() call doesn't return when executing within SQL Server?
private static string SendPost(XElement entry, SqlString url, SqlString entityName)
{
// Send the HTTP request
string serviceURL = url.ToString() + entityName.ToString() + "/";
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(serviceURL);
request.Method = "POST";
request.Accept = "application/atom+xml,application/xml";
request.ContentType = "application/atom+xml";
request.Timeout = 20000;
request.Proxy = null;
using (var writer = XmlWriter.Create(request.GetRequestStream()))
{
entry.WriteTo(writer);
}
try
{
SqlContext.Pipe.Send("Calling GetResponse() using " + request.RequestUri);
WebResponse response = request.GetResponse();
SqlContext.Pipe.Send("Back from GetResponse()");
/*
string feedData = string.Empty;
Stream stream = response.GetResponseStream();
using (StreamReader streamReader = new StreamReader(stream))
{
feedData = streamReader.ReadToEnd();
}
*/
HttpStatusCode StatusCode = ((HttpWebResponse)response).StatusCode;
response.Close();
if (StatusCode == HttpStatusCode.Created /* 201 */ )
{
return "Created # Location= " + response.Headers["Location"];
}
return "Creation failed; StatusCode=" + StatusCode.ToString();
}
catch (WebException ex)
{
return ex.Message.ToString();
}
finally
{
if (request != null)
request.Abort();
}
}
The problem turned out to be the creation of the request content from the XML. The original:
using (var writer = XmlWriter.Create(request.GetRequestStream()))
{
entry.WriteTo(writer);
}
The working replacement:
using (Stream requestStream = request.GetRequestStream())
{
using (var writer = XmlWriter.Create(requestStream))
{
entry.WriteTo(writer);
}
}
You need to dispose the WebResponse. Otherwise, after a few calls it goes to timeout.
You are asking for trouble doing this in the CLR. And you say you are calling this from a trigger? This belongs in the application tier.
Stuff like this is why when the CLR functionality came out, DBAs were very concerned about how it would be misused.
We've created a custom webservice in Umbraco to add (async) files and upload them. After upload the service is called with node and file-information to add a new node to the content tree.
At first our main problem was that the service was running outside of the Umbraco context, giving strange errors with get_currentuser.
Now, we inherit the umbraco BaseWebService from the umbraco.webservices dll and we've set all acces information in the settings file; we authenticatie before doing anything else using (correct and ugly-hardcoded) administrator.
When we now execute the webservice (from the browser or anything else) we get:
at umbraco.DataLayer.SqlHelper`1.ExecuteReader(String commandText, IParameter[] parameters)
at umbraco.cms.businesslogic.CMSNode.setupNode()
at umbraco.cms.businesslogic.web.Document.setupNode()
at umbraco.cms.businesslogic.CMSNode..ctor(Int32 Id)
at umbraco.cms.businesslogic.Content..ctor(Int32 id)
at umbraco.cms.businesslogic.web.Document..ctor(Int32 id)
at FileUpload.AddDocument(String ProjectID, String NodeID, String FileName)*
Where AddDocument is our method. The node (filename w/o extension) does not exist in the tree (not anywhere, it's a new filename/node). We've cleared the recycle bin, so it's not in there either.
Are we missing something vital, does anyone has a solution?
Below is the source for the webservice;
using umbraco.cms.businesslogic.web;
using umbraco.BusinessLogic;
using umbraco.presentation.nodeFactory;
using umbraco.cms.businesslogic.member;
using umbraco.cms;
/// <summary>
/// Summary description for FileUpload
/// </summary>
[WebService(Namespace = "http://umbraco.org/webservices/")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
public class FileUpload : umbraco.webservices.BaseWebService //System.Web.Services.WebService
{
private string GetMimeType(string fileName)
{
string mimeType = "application/unknown";
string ext = System.IO.Path.GetExtension(fileName).ToLower();
Microsoft.Win32.RegistryKey regKey = Microsoft.Win32.Registry.ClassesRoot.OpenSubKey(ext);
if (regKey != null && regKey.GetValue("Content Type") != null)
mimeType = regKey.GetValue("Content Type").ToString();
return mimeType;
}
[WebMethod]
public string HelloWorld() {
return "Hello World";
}
[WebMethod]
public void AddDocument(string ProjectID, string NodeID, string FileName)
{
Authenticate("***", "***");
string MimeType = GetMimeType(FileName); //"application/unknown";
// Create node
int nodeId = 1197;
string fileName = System.IO.Path.GetFileNameWithoutExtension(#"*****\Upload\" + FileName);
string secGroups = "";
//EDIT DUE TO COMMENT: Behavior remains the same though
Document node = umbraco.cms.businesslogic.web.Document.MakeNew(fileName.Replace(".", ""), new DocumentType(1049), umbraco.BusinessLogic.User.GetUser(0), nodeId);
secGroups = "Intern";
StreamWriter sw = null;
try
{
//EXCEPTION IS THROWN SOMEWHERE HERE
Document doc = NodeLevel.CreateNode(fileName, "Bestand", nodeId);
doc.getProperty("bestandsNaam").Value = fileName;
byte[] buffer = System.IO.File.ReadAllBytes(#"****\Upload\" + FileName);
int projectId = 0;
int tempid = nodeId;
//EXCEPTION IS THROWN TO THIS POINT (SEE BELOW)
try
{
Access.ProtectPage(false, doc.Id, 1103, 1103);
Access.AddMembershipRoleToDocument(doc.Id, secGroups);
}
catch (Exception ex)
{
// write to file
}
try
{
doc.Publish(umbraco.BusinessLogic.User.GetUser(0));
umbraco.library.UpdateDocumentCache(doc.Id);
umbraco.content.Instance.RefreshContentFromDatabaseAsync();
}
catch (Exception ex)
{
// write to file
}
System.IO.File.Delete(FileName);
}
catch (Exception ex)
{
// THIS EXCEPTION IS CAUGHT!!
}
}
public override umbraco.webservices.BaseWebService.Services Service
{
get { return umbraco.webservices.BaseWebService.Services.DocumentService; }
}
}
If anyone has a solution, pointer, hint or whatever; help is appreciated!!
TIA,
riffnl
We've rewritten the whole procedure (dumped all code and restart) and we've got it working now.
I think we've been messing around with the old code so much in trying to get it to work we were missing some key issues, because it functions.
Thanks for thinking along anyway!