Scrapy FormRequest authentication with user, password plus form_key - python-2.7

Question

So I need to parse a set of pages from an authenticated user's point of view and get some values from there...
So I need scrapy to authenticate itself before starting to parse.
In order to do that I understand that I would need to use InitSpider and init_request.
My problem is that in order to authenticate, using a FormRequest, I must send to the form a certain value that is some kind of a session id (form_key) that is generated automatically.
So first, scrapy must access the self.login_page, get the value of form_key (like in the xpath bellow), then post the LOGIN FORM to self.login_post
The 2 URL's are described bellow in the script section
The HTML containing my form key looks like this:
<input name="form_key" type="hidden" value="2pos7YUekQz6Y9rD">
So in order to obtain my form_key value, I did this (updated scrapy script):
class MyScript(InitSpider):
name = "logged-in"
allowed_domains = ["example.com"]
login_post = 'https://www.example.com/customer/account/loginPost/'
def init_request(self):
""" Called before crawler starts """
return Request(url=self.login_post, callback=self.login, dont_filter = True)
def login(self, response):
""" Generate login request """
login_page = 'https://www.example.com/customer/account/login/'
myformkey= self.pax_key(login_page)
return FormRequest.from_response(response,
formxpath="//div[#class='account-login']/form[#id='login-form']",
formdata={'login[username]':'user',
'login[password]':'pass',
'form_key':myformkey
},
callback=self.check_login_response)
def check_login_response(self,response):
""" Check the response returned by login request to see if we are logged in """
logging.log(logging.INFO,'... checking login response ...')
if "Invalid login or password." in response.body:
logging.log(logging.INFO,'... BAD LOGIN ...')
else:
logging.log(logging.INFO, 'GOOD LOGIN... initialize:'+response.url)
self.initialized()
def pax_key(self, url):
data = urllib.urlopen(url).read()
hxs = Selector(text=data) #HtmlXPath
lista = hxs.xpath('//input[#type="hidden"][#name="form_key"]/#value').extract()
logging.log(logging.INFO,lista)
return lista
def parse(self, response):
...
So the problem now is...:
INFO: do login...
2015-12-04 21:57:18 [root] INFO: [u'yFVS9fR8Jteuo6Co']
2015-12-04 21:57:19 [root] INFO: http://www.example.com/enable-cookies
2015-12-04 21:57:19 [scrapy] ERROR: Spider error processing <GET http://www.example.com/enable-cookies> (referer: None)
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 588, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "d:\Calin\WorkSpace\example-spider\products\spiders\logged-in.py", line 49, in login
callback=self.check_login_response)
File "c:\python27\lib\site-packages\scrapy\http\request\form.py", line 37, in from_response
form = _get_form(response, formname, formnumber, formxpath)
File "c:\python27\lib\site-packages\scrapy\http\request\form.py", line 81, in _get_form
raise ValueError('No <form> element found with %s' % formxpath)
ValueError: No <form> element found with //div[#class='account-login']/form[#id='login-form']
So first of all... I have no idea why I get redirected towards the enable-cookies page...
Spider error processing <GET http://www.example.com/enable-cookies>
I do not tell it to go there... at all.
I assume that is why there is no form there to xpath it... so I get the error in the end.
ValueError: No <form> element found with
//div[#class='account-login']/form[#id='login-form']