Scraping No Link - django

I'm learning xpath & web scraping using django-dynamic-scraper aka DSS (django+scrapy) and try to retrieve data from a website with following code:
<tr valign="top">
<td align="center" valign="top">
<p><img src="someimage.jpg"></p>
</td>
<td>
</td>
<td>
<div align="left">
<span class="style1">
<strong>Title1</strong>
</span>
<span class="style2">Title2:</span>ContentA<br />
<span class="style2">Title3:</span>ContentB<br />
<span class="style2">Title4:</span>ContentC<br />
</div>
</td>
</tr>
My questions:
What's the xpath for an URL object of DSS if there's no link at that code?
What's the xpath to retrieve image file if there's no class for first <td>?
How to retrieve data for each data from ContentA, ContentB, & ContentC if the span's class is same?

What's the xpath for an URL object of DSS if there's no link at that code?
Can't get the question, could you please explain?
What's the xpath to retrieve image file if there's no class for first ?
//tr[1]/td[1]//img/#src
How to retrieve data for each data from ContentA, ContentB, & ContentC
if the span's class is same?
//text()[preceding-sibling::span[#class="style2"]]

Related

XPath to find specific rows and related values based on a passed parameter

I'm trying to grab the value from the lights node, based on a house number set in a parameter. The problem is, based on certain conditions, houses may be in different row positions.
If the parameter being sent to me for the house number is House237, then how to I get the number of lights located within the row-2-Lights node?
Also, how do I do the same if the next run, the house number is House867? Below is my HTML:
<?xml version='1.0' encoding='utf-8'?>
<table id="neighborhood">
<tr onmouseover="leave('1')">
<td id="row-1-house">
<div class="houseCol">
<a href="#" onClick="goHome('867');return false">
House867
</a>
</div>
</td>
<td id="row-1-Lights">
<div class="decimal">14</div>
</td>
</tr>
<tr onmouseover="leave('2')">
<td id="row-2-house">
<div class="houseCol">
<a href="#" onClick="goHome('237');return false">
House237
</a>
</div>
</td>
<td id="row-2-Lights">
<div class="decimal">12</div>
</td>
</tr>
</table>
You can try the following XPath-1.0 expression. The parameter is the 'HouseXXX' string, the child of the a element.
/table[#id='neighborhood']/tr[td/div[#class='houseCol']/a[normalize-space(text())='House237']]/td[contains(#id,'Lights')]/div[#class='decimal']/text()
The output of this is
12
In this example the parameter is set to 'House237'. How you incorporate the parameter into the XPath expression depends on your usecase scenario.
For example, in XSLT you would replace 'House237' with a variable like $HouseNumber to set the parameter.

Python Selenium Click on object inside specific row

I have site and the HTML looks like this:
<tr role="row" class="odd">
<td class="sorting_1">555</td>
<td>
FruitType1 : Fruit1
</td>
<td>Fruit1</td>
<td>FruitType1</td>
<td>Somwhere</td>
<td></td>
<td>0</td>
<td>
<button class="copy_button btn_gray_inverse" id="555">Copy</button>
</td>
<td>
<button class="fruit check_btn" id="555" href="" value="0">
<i class="bt_check"></i>
</button>
</td>
<td>
<a class="fruit remove_btn" id="555" href="#">
<i class="bt_remove">
::before
</i>
</a>
</td>
</tr>
Im trying to click on button (<button class="fruit check_btn" id="555" href="" value="0">) inside this specific row. Rows can be different only in text under tr (Fruit1, Fruit2), with this code and its not working:
FruitList = self.driver.find_elements_by_xpath\
('//tr[#role="row" and contains (., "Fruit1")]')
for Fruit in FruitList:
Enabled = Fruit.find_element_by_xpath('//i[#class="bt_check"]')
Enabled.click()
It allways clicks on button from first awailable row on the page, not the one that is containing text "Fruit1".
Please help
To find the check button for Fruit1, I suggest you to do it in two steps.
First find the row for Fruit1:
fruit_row = driver.find_element_by_xpath("//td[text()='Fruit1']/..")
Note that I use /.. at the end of the Xpath in order to select the tr element that contains the td with text 'Fruit1' instead of the td himself.
Second step, find the button and click on it:
fruit_row.find_element_by_class_name("check_btn").click()
To click on <td> with text as Fruit1, you can use the following line of code :
driver.find_element_by_xpath("//td/a[contains(#href,'/fruit_list/555')]//following::td[text()='Fruit1']")
You can be more generic with :
driver.find_element_by_xpath("//td/a[contains(#href,'/fruit_list/555')]//following::td[1]")

How to extract parent and nested items in single regular expression?

I want to use regular expression to match the following html table:
<tbody class=\"DocTableBody \">
<tr data-fastRow=\"1\" class=\"DataRow TDRE\">
<td id=\"g-f-1\" class=\"TDC FieldDisabled Field TCLeft CellText g-f\" >
<div class=\"DTC\">
<label id=\"c_g-f-1\" class=\"DCC\" >01-Apr-2015</label>
</div>
</td>
<td id=\"g-g-1\" class=\"TDC FieldDisabled Field TCLeft CellTextHtml g-g\" >
<div class=\"DTC\">
<label id=\"c_g-g-1\" class=\"DCC\" >ACTIVE</label>
</div>
</td>
</tr>
<tr data-fastRow=\"2\" class=\"DataRow TDRO\">
<td id=\"g-f-2\" class=\"TDC FieldDisabled Field TCLeft CellText g-f\" >
<div class=\"DTC\">
<label id=\"c_g-f-2\" class=\"DCC\" >01-Apr-2015</label>
</div>
</td>
<td id=\"g-g-2\" class=\"TDC FieldDisabled Field TCLeft CellTextHtml g-g\" >
<div class=\"DTC\">
<label id=\"c_g-g-2\" class=\"DCC\" >ACTIVE</label>
</div>
</td>
</tr>
</tbody>
I expected to extract the following value:
"1"
01-Apr-2015
ACTIVE
"2"
01-Apr-2015
ACTIVE
I tried the following to extract the value in data-fastRow:
(?sUi)<tr data-fastRow=\\"(\d+)\\".+>.*<\/tr>
But I couldn't extract the nested items in <label.+>(.*)</label> in single regular expression.
Is that possible to extract parent and nested items in single regular expression?
It's a really bad idea to parse HTML with regular expressions.
Each languahe has its own libraries to parse HTML.
In Python for example you have BeautifulSoup.
It's by far much better to use such libraries.
Usually, such libraries has jQuery-Selector-like interface (or something like that), which allows you to find your data with extremely easy queries.

CSS/jQuery extractor Usage to extract content in td tag which doesn't have tag id

<tr class="paxDetailRow" pax_seq="3" pax_sts="CUR">
<td class="none"> </td>
<td class="borderFirst">3</td>
<td>Master</td>
<td class="tooltipParent" tooltipWidth="400">
F
</td>
<td class="tooltipParent" tooltipWidth="400">
Rowland
</td>
<td> </td>
<td>8</td>
<td class="price borderLast">
1723.84
<div class="tooltipContent"></div>
</td>
</tr>
From this I want to extract the value 8 which is there inside td tag
I tried some regex/css extractor but its not working out for me. As I'm new to usage of css/jQuery extractor I'm finding it difficult. Can anyone help me with this?
You can extract it through:
tr[class=paxDetailRow] > td:eq(6)
It will be much more efficient than using XPath.
I doubt you can get text inside a HTML tag using CSS selectors so I would suggest going for XPath Extractor instead.
The relevant XPath query will look like:
//tr[#class='paxDetailRow']/td[7]/text()
Demo:
References:
XPath 1.0 Language Reference
XPath Tutorial
Using the XPath Extractor in JMeter

How to handle dynamically changing id's with similar starting name using Webdriver

I am automating the test for web application. I have a scenario for creating an admin, for which i have to enter the name, email address and phone number text boxes. But ids of this text boxes are dynamic.
userName, id='oe-field-input-41'
Email, id='oe-field-input-42'
phone number, id='oe-field-input-43'
First Query:
The numbers in the ids are dynamic, it keep changes
I tired to use the xpath for handling the dynamic value.
xpath = //*[starts-with(#id,'oe-field-input-')]
In this it enter the text into first text box successfully
Second Query:
I am not able use the same xpath for next two text boxes, as it enters the email and phone number into name field only
Please help me to resolve this dynamic value handling.
Edited: added the html code,
<table class="oe_form_group " cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr class="oe_form_group_row">
<td class="oe_form_group_cell oe_form_group_cell_label" width="1%" colspan="1">
<td class="oe_form_group_cell" width="99%" colspan="1">
<span class="oe_form_field oe_form_field_many2one oe_form_field_with_button">
<a class="oe_m2o_cm_button oe_e" tabindex="-1" href="#" draggable="false" style="display: inline;">/</a>
<div>
</span>
</td>
</tr>
<tr class="oe_form_group_row">
<td class="oe_form_group_cell oe_form_group_cell_label" width="1%" colspan="1">
<td class="oe_form_group_cell" width="99%" colspan="1">
<span class="oe_form_field oe_form_field_email">
<div>
<input id="oe-field-input-35" type="text" maxlength="240">
</div>
</span>
</td>
</tr>
<tr class="oe_form_group_row">
<td class="oe_form_group_cell oe_form_group_cell_label" width="1%" colspan="1">
<td class="oe_form_group_cell" width="99%" colspan="1">
<span class="oe_form_field oe_form_field_char">
<input id="oe-field-input-36" type="text" maxlength="32">
</span>
</td>
</tr>
<tr class="oe_form_group_row">
<td class="oe_form_group_cell oe_form_group_cell_label" width="1%" colspan="1">
<td class="oe_form_group_cell" width="99%" colspan="1">
<span class="oe_form_field oe_form_field_char">
<input id="oe-field-input-37" type="text" maxlength="32">
</span>
</td>
</tr>
<tr class="oe_form_group_row">
</tbody>
you can try alternate way for locating unique element by label or so. For example:
css=.oe_form_group_row:contains(case_sensitive_text) input
xpath=//tr[#class = 'oe_form_group_row'][contains(.,'case_sensitive_text')]//input
If you are using ISFW you should create custom component for such form fields.
You do have some classes which are good for identification, e.g. oe_form_field_email, oe_form_field_char. It's a little complicated to use them because they're not on the input fields themselves, and the second one is not unique; but it's quite possible:
.//span[contains(#class, 'oe_form_field_email')]//input
That is an xpath which identifies the Email field as being the input which is a descendant of a span with the oe_form_field_email class. You could also use the same logic in a css selector like this, more efficiently:
span.oe_form_field_email input
For the two other fields, there is no unique class which can tell them apart so you're going to have to rely on the order (I'm assuming username comes before phone number), and that means you have to use xpaths:
(//tr//span[contains(#class, 'oe_form_field_char')])[1]//input
(//tr//span[contains(#class, 'oe_form_field_char')])[2]//input
Those xpaths pick out the first and second fields respectively, which are inputs which are descendants of a span of class oe_form_field_char.
P.S. I used Firepath in firefox to verify the xpath and css locators.
The problem here is, that your XPath does the correct selection, but Selenium will always pick the first one if multiple results are returned for your query.
You can select each of the input fields directly by using:
//input[1]
//input[2]
//input[3]
If there are other input fields, you can tighten your selection by selecting only input nodes with oe-field-input in their id attribute like this:
//input[starts-with(#id,'oe-field-input-')][1]
//input[starts-with(#id,'oe-field-input-')][2]
//input[starts-with(#id,'oe-field-input-')][3]
Use the following xpath works like a charm. Although I don't recommend this kind of an xpath. Since we don't have text against the text box no other choice.
//div/input[contains(#id, 'oe-field-input')] - First text box
//tr[#class = 'oe_form_group_row'][2]//input - Second text box
//tr[#class = 'oe_form_group_row'][3]//input - Third text box
You can use below XPATH.
//tr[#class = 'oe_form_group_row'][2]//input for First Text box
//tr[#class = 'oe_form_group_row'][3]//input for Second Text box
//tr[#class = 'oe_form_group_row'][4]//input for Third text box.
I have tested avove xpath.
But the better way if you have development access then ask developers to make is standaralized and recommand tags like "name" , "value", or attach text e.g. Email:, Password. So you can use these in your xpath.