No headers in HAR response - python-2.7

I parse website 'http://ok.ru'. To get data from the post request I need to send a specific token that is generated by Javascript on the website and this token is contained in headers.
So I thought maybe one solution would be to open the website, let it generate token, grab headers and that's it.
One tool that can implement Java scripts is Selenium, however, to get headers I need to use brosermob-proxy (or equivalent). That is where I'm stuck.
There's no headers in response and I can't figure it out. Maybe someone who worked with browsermob can see what's wrong? I would also be glad to hear another solutions to my task. The code itself is below:
from browsermobproxy import Server
from selenium import webdriver
from ast import literal_eval
import json, os
os.chdir('C:/browsermob-proxy-2.1.0-beta-2/bin')
server = Server()
server.start()
proxy = server.create_proxy()
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har('test')
driver.get('http://ok.ru')
driver.find_element_by_xpath('//input[#name="st.email"]').send_keys('****#****.com')
driver.find_element_by_xpath('//input[#name="st.password"]').send_keys('****')
driver.find_element_by_xpath(u'//input[contains(#value,"Log in")]').click()
result = literal_eval(json.dumps(proxy.har, ensure_ascii=False))
driver.close()
for entry in result['log']['entries']:
if len(entry['response']['headers']) > 0:
print entry['response']['headers']

The answer turned to be easy: just to add options to new_har:
proxy.new_har('test', options={'captureHeaders': True})
However, there is no token in headers, which is a new puzzle to me...

Related

TypeError: can only concatenate str (not "DeferredAttribute") to str

I am wondering anyone can help me in one of the thing I cannot get my around it and really bothering as I spent last two on it but couldnt make it.
Basically, I am building an App (Django Python) to restore the information regarding all the network device e.g hostname, IP Address, S/N, Rack info etc. but I also to enable few options like Add, Edit, Delete and Connect next to device entery. I was able to create all the options except Connect option where I am completely stuck, I am trying to query database to get the IP address information and then using Popen module to open a putty window with the ssh to that IP Address device related, I tried everything I could but nothing worked, thereofrore, asking your help if you have any idea about this ? or any other alternative method for a user when he click on connect the putty or similar app will open and he just put the login credentials and get into the device.
I am sharing my code here, let me know if I am doing something wrong.
on the show all device page, i have this code, show.html
<td>Connect</td>
<!--<td>Connect</td>-->
I tried both ways, with id and ip address entry in the database
on view.py
def connect(request, ip_address):
hostlist_ip = HostList.ip_address
print(hostlist_ip)
Popen("putty.exe" + hostlist_ip)
return redirect('/show')
and in the url.py
path('connect/<str:ip_address>', views.connect),
or
path('connect/<str:ip_address>', views.connect),
Since I am also printing the the output on the terminal I notice that it is not returning the actually IP address but return this;
<django.db.models.query_utils.DeferredAttribute object at 0x04B77C50>
and on the web I receiving this error
TypeError at /connect/10.10.32.10
can only concatenate str (not "DeferredAttribute") to str
Request Method: GET
Request URL: http://localhost:8000/connect/10.10.32.10
Django Version: 2.2.3
Exception Type: TypeError
Exception Value:
can only concatenate str (not "DeferredAttribute") to str
let me know if you can help.
Just a F.Y.I I already tested the Popen via python but since we not getting the actual IP address from the database I am seeing this and I am a complete newbie with html/css and Djano, however I have some basic knowledge of python, so please ignore my any stupid comments in the post.
Many thanks
ahh I cannot believe I spend two day to troubleshoot this and just changed the name from ip_address to ip_add and it is working now :) i think as I mentioned above in the comment, it probably confusing with the built in module
here is simple solution:
views.py
def connect(request, ip_add):
import os
from subprocess import Popen
Popen("powershell putty.exe " + ip_add)
return redirect('/show')
url.py
path('connect/<str:ip_add>', views.connect),
I may have to find out a way if user is using the mac or linux, how I am going to change this powershell to something else. but anyhow it is working for windows
thanks all for the responses.

How to extract browser cookie(s) using Katalon Studio

I am trying to get all the cookies of the browser which is opened while we execute automation test using Katalon.
Please suggest how we can extract a specific cookie/list of cookies using Katalon.
I have checked in traditional Selenium/Java; its pretty easy -
driver.manage().getCookieNamed("Cookie Name").getValue();
But I am not getting any clue how to do this in Katalon.
Resolved it....
import com.kms.katalon.core.webui.driver.DriverFactory
import org.openqa.selenium.WebDriver
WebDriver driver = DriverFactory.getWebDriver() println driver.manage().getCookieNamed("Cookie Name").getValue()

How to refresh training in chatterbot bot in python?

I made a simple chatbot using chatterbot library and python. The way I trained it is, I made it read a few text files containing chat examples, and it learns how to reply to messages based on those training examples. The problem I am facing is - Even if I erase the contents of the training text files, and run the application, the chatbot still behaves in the same manner as before, i.e. it's memory doesn't get refreshed. I tried starting a new file and copy pasted the same code and changed the name of the program, but it still doesn't help. How do I solve this problem? Here is the code for reference:
from chatterbot.trainers import ListTrainer
from chatterbot import ChatBot
from chatterbot.trainers import ChatterBotCorpusTrainer
import os
bot = ChatBot('trialBot')
bot.set_trainer(ListTrainer)
#directory containing training text files
mainDir = 'C:\\Users\\xyz\\Desktop\\trainfiles\\'
for _file in os .listdir(mainDir):
chats = open(mainDir + _file, 'r').readlines()
bot.train(chats)
while True:
request = raw_input('You: ')
response = bot.get_response(request)
print('Bot: ' + str(response))
It sounds like you might want to use an in-memory database so that the content is only persisted as long as the chat bot is running.
bot = ChatBot(
'trialBot',
database_uri=None
)
Setting database_uri to None will cause the chat bot to use a Sqlite database that is stored in-memory so store the knowledge that it is trained with. As a result, you will have a fresh database to work with each time you run your program.

Problems with MSF4J and #MatrixParam

Folks, I have found what seems to be a problem with / (bug in ?) MSF4J as including an #MatrixParam annotated variable in a URI causes the affected (micro)service to either 'hang' indefinitely, or if accessed via a browser, to give a "404 Not Found" message for the path/endpoint, even when correct.
Here is a code fragment that illustrates the problem - it compiles ok (eclipse/maven) and deploys without errors using microservicesrunner() in the usual way.
package org.test.service;
import javax.ws.rs.GET;
import javax.ws.rs.MatrixParam;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Response;
#Path("books")
public class MPTest { // MatrixParam Test
#GET
#Produces(MediaType.TEXT_PLAIN)
#Path("/query")
// method to respond to 'GET' requests
public Response getListOfBooks(#MatrixParam("Author") String author) {
// do something in here to get book data from DB and sort by titles
List<String> titles = .......;
return Response.status(200) .entity("List of Books by " +author+ "ordered by title " + titles).build();
}
}
With this code fragment, accessing the URL "(host:8080)/books/query;Author=MickeyMouse" should cause a list of books by that author to be retrieved from the DB (I have omitted the actual code that does so for clarity, as it is not relevant to this post).
However, it does not get there, so that code isnt executed. As far as I can tell with a debugger, no #MatricParam value is retrieved - it remains null until the process times out. Things like curl and wget just hang until they time out, and from a browser, the best I can get is a 404 not found error for the URI, even though it is valid.
However, if I replace the #MatrixParam with a #PathParam it works perfectly, and can I get the URL string retrieved in its entirity. The URI that I get is as expected - no odd hex characters, no typos, and so forth. The URI entered is what you get back. So, no problem there.
Behaviour is also consistent across platforms (couple of flavours of Linux, and three versions of Windoze), so it is not anything to do with the OS itself. Similarly, I get the same behavior with multiple clients and tools, so it isnt a problem there either.
So, it appears to be a problem within the MSF4J framework / domain, and I could use some support / help / suggestions here as I've reached the point of tearing my hair out..... Any ideas, folks?
The only reference I can find to a similar problem was closed as 'off topic' without a reply (see Rest API Matrix param annotation) so I think that this needs re-opening as it seems to be a genuine problem....
Regards, and thanks in advance for any help,
Rick
#MatrixParam is not supported with MSF4J at the moment. You can create a GitHub issue. So we can implement that support in future releases.

Getting an 500 internal error when uploading file using cgi and python

Hi I have a project that involves uploading documents. I am using html for the front end and python for the backend. I've managed to link my html and python file but I'm having a problem with the server. At first I though it was a random thing but I'm pretty sure it's because of what I added to the python code. I have:
import cgi
import sys
import os
htmlform = cgi.FieldStorage()
file_data = htmlform['myfile']
if not fileitem.file:
return
(name,ext) = os.path.splitext( fileitem.filename)
#if ext == “.jpg” or ext == “.png” or ext == “.gif”:
#ioFlag = “wb”
#else:
#ioFlag = “w”
I was able to log into my page go to the html form submit the form and got to a basic success html page I had below the above input. Now Im pretty new to python and didnt realise that the if statements should be indented. And I get a 500 internal error when I uncommented the if statement. I did it once and then went through commenting out my code being completely confused as to why I was getting error but after a while it just started working again. My guess is the incorrect if statement somehow got it stuck. I expect after about an hour it'll be working again but ideally I'd like to know if I could stop the process on the server if possible. I was following this guide http://www.alwaysgetbetter.com/blog/2009/01/02/python-file-upload/
Fixed it! The problem seems to be the indentation. If you're ever unsure about this stuff look at the error logs. I'm using an apache server and I dont have access to the error logs so I used
sudo cat /etc/log/apache2/error.log
It gave me the answer and this should hopefully help you even if your question is unrelated.
EDIT: An for completeness sake file_data should be fileitem