How to parse thorugh a list of links using a loop using beautiful soup? - list

Question

I am newbie in coding so please go easy. I have this work thing that I want to automate a bit.
I have to collect data from this government website.
But I think it is designed this way to prevent it from bots and DDOS attacks. What I have to do as part of my work is to click individually on these links, record the case
Filling date
Last listed date
Party name
Case status
If case status ='disposed' then I read and check the pdfs for reason of disposal. (which can't automated).
Now, I have to go through many pages of this and its hard to even copy paste this much info. So for the first 4 details, I tried to create a script. One of them retrieves the hyperlinks from table page, and the second script to go through the list of links and get the listed above details. Its the second script where I am facing the problems.
List of changes in the URL:
case_list = [
"case-details?bench=YW1yYXZhdGk=&filing_no=MjgxMjEyOTAwMjA4MjAyMA==",
"case-details?bench=YW1yYXZhdGk=&filing_no=MjgxMjEyOTAwMjA5MjAyMA==",
"case-details?bench=YW1yYXZhdGk=&filing_no=MjgxMjEyOTAwMjEwMjAyMA==",
"case-details?bench=YW1yYXZhdGk=&filing_no=MjgxMjEyOTAwMjExMjAyMQ==",
"case-details?bench=YW1yYXZhdGk=&filing_no=MjgxMjEyOTAwMjEyMjAyMQ==",
"case-details?bench=YW1yYXZhdGk=&filing_no=MjgxMjEyOTAwMjEzMjAyMA==",
"case-details?bench=YW1yYXZhdGk=&filing_no=MjgxMjEyOTAwMjE0MjAyMA==",
"case-details?bench=YW1yYXZhdGk=&filing_no=MjgxMjEyOTAwMjE2MjAyMQ==",
"case-details?bench=YW1yYXZhdGk=&filing_no=MjgxMjEyOTAwMjE3MjAyMA==",
"case-details?bench=YW1yYXZhdGk=&filing_no=MjgxMjEyOTAwMjIwMjAyMA=="]
URL:
response= requests.get(url)
soup= BeautifulSoup(response, 'html.parser').text
tr= soup.find_all('td')
status = tr[19].text
#gets Filing date
filing_date = tr[3].text
#gets title
case_title = tr[5].text
#gets case disposed date
disposal_date = tr[15].text
Function for grabbing details from the URL:
def get_case_components(url):
response = requests.get(url).text
soup = BeautifulSoup(response, 'html.parser')
tr = soup.find_all('td')
status = tr[19].text
#gets Filing date
filing_date = tr[3].text
#gets title
case_title = tr[5].text
#gets disposed date
disposal_date = tr[15].text
return filing_date, case_title, status, disposal_date
Function for appending the DataFrame:
def get_case(df):
# loop for going through the case_list
for links in url_list :
url = links
#putting the URL in the function
get_case_components(url)
df = df.append( {"filing_date": filing_date,
"case_title" : case_title, "status": status,
"disposal_date": disposal_date}, ignore_index=True)
time.sleep(1)
return df
Calling the get_info() function using Dataframe.
df= pd.DataFrame(columns = ["filing_date", "case_title", "status", "disposal_date"])
df = get_case(df)
df.head()
For some reason I just keep getting the same thing over and over again as output, like only one case fills the entire Dataframe.
0 14-12-2020 Rajani Jagarlamudi VS Sharadakrupa Cold Storag... Pending 16-03-2022 \t
1 14-12-2020 Rajani Jagarlamudi VS Sharadakrupa Cold Storag... Pending 16-03-2022 \t
2 14-12-2020 Rajani Jagarlamudi VS Sharadakrupa Cold Storag... Pending 16-03-2022 \t
3 14-12-2020 Rajani Jagarlamudi VS Sharadakrupa Cold Storag... Pending 16-03-2022 \t
4 14-12-2020 Rajani Jagarlamudi VS Sharadakrupa Cold Storag... Pending 16-03-2022 \t

Answer 1

I made a script that gets the cases and stores them in an array.
The problem in your script was that you were manually selecting the cases when you were doing:
tr = soup.find_all('td')
status = tr[19].text
#gets Filing date
filing_date = tr[3].text
#gets title
case_title = tr[5].text
#gets disposed date
disposal_date = tr[15].text
My code dynamically scrapes them, so you don't have to worry about it.
import requests
from bs4 import BeautifulSoup
from dataclasses import dataclass
HOST = "https://nclt.gov.in/"
LINK = "https://nclt.gov.in/order-judgement-date-wise-search?bench=Y2hlbm5haQ%3D%3D&start_date=MDEvMDEvMjAyMQ%3D%3D&end_date=MDEvMDEvMjAyMg%3D%3D"
#dataclass
class Case:
number: str
filing_num: str
case_no: str
pvr: str
listing_date: str
status: str
def get_cases(url):
res = requests.get(url)
if res.status_code == 200:
print("getting cases")
return res.text
def extract_case(html):
soup = BeautifulSoup(html, "html.parser")
cases = [Case(*[td.text for td in tr.select("td")]) for tr in soup.select("table tbody tr")]
next, *rest = [link["href"] for link in soup.select(".page-link") if link.text.strip() == "Next"]
if len(next):
next = HOST + next
return cases, next
def main():
total_cases = []
next = ""
amount = int(input("How many pages do you want to scrape?"))
oamount = amount
while amount:
if len(next) and oamount != amount:
html = get_cases(next)
else:
html = get_cases(LINK)
cases, next = extract_case(html)
total_cases.extend(cases)
amount -= 1
print(cases, next)
print("Scraped", len(total_cases), "cases")
if __name__ == "__main__":
main()