BeautifulSoup and List indexing - list

Question

As I am still quite new to web scraping I am currently practicing some basics such as this one. I have scraped the categories from 'th' tag and the players from the 'tr' tag and appended it to a couple empty lists. The categories come out fine from get_text(), but when I try printing the players it has a number rank before the first letter of the name, and the player's team abbreviation letters after the last name.
3 things I am trying to do:
1)output only the first and last name of each player by doing some slicing from the list but I cannot figure out any easier way to do it. There is probably a quicker way inside the tags where I can call the class or using soup.findAll again in the html, or something else I am unware of, but I currently do not know how or what I am missing.
2)take the number ranks before the name and append it to an empty list.
3)take the 3 last abbreviated letters and append it to an empty list
Any suggestions would be much appreciated!
from bs4 import BeautifulSoup as bs4
import requests
import pandas as pd
from time import sleep
players = []
categories = []
url ='https://www.espn.com/nba/stats/player/_/table/offensive/sort/avgPoints/dir/desc'
source = requests.get(url)
soup = bs4(source.text, 'lxml')
for i in soup.findAll('th'):
c = i.get_text()
categories.append(c)
for i in soup.findAll('tr'):
player = i.get_text()
players.append(player)
players = players[1:51]
print(categories)
print(players)

Answer 1

Apis are always the best way to go in my opinion.
However, this can also be done with pandas .read_html() (it uses beautifulsoup under the hood to parse the table).
import pandas as pd
url = 'https://www.espn.com/nba/stats/player/_/table/offensive/sort/avgPoints/dir/desc'
dfs = pd.read_html(url)
dfs[0][['Name','Team']] = dfs[0]['Name'].str.extract('^(.*?)([A-Z]+)$', expand=True)
df = dfs[0].join(dfs[1])
Output:
print (df[['RK','Name','Team','POS']])
RK Name Team POS
0 1 James Harden HOU SG
1 2 Stephen Curry GS PG
2 3 Bradley Beal WSH SG
3 4 Trae Young ATL PG
4 5 Kevin Durant BKN SF
5 6 CJ McCollum POR SG
6 7 Kyrie Irving BKN PG
7 8 Jaylen Brown BOS SG
8 9 Giannis Antetokounmpo MIL PF
9 10 Jayson Tatum BOS PF
10 11 Damian Lillard POR PG
11 12 Luka Doncic DAL PG
12 13 Collin Sexton CLE PG
13 14 Paul George LAC SG
14 15 Brandon Ingram NO SF
15 16 Nikola Jokic DEN C
16 17 LeBron James LAL SF
17 18 Zach LaVine CHI SG
18 19 Christian Wood HOU PF
19 20 Kawhi Leonard LAC SF
20 21 Joel Embiid PHI C
21 22 Jerami Grant DET PF
22 23 Anthony Davis LAL PF
23 24 Jamal Murray DEN PG
24 25 Julius Randle NY PF
25 26 Malcolm Brogdon IND PG
26 27 Fred VanVleet TOR SG
27 28 Nikola Vucevic ORL C
28 28 Donovan Mitchell UTAH SG
29 30 Terry Rozier CHA PG
30 31 Devin Booker PHX SG
31 32 Khris Middleton MIL SF
32 33 Terrence Ross ORL SG
33 33 Victor Oladipo IND SG
34 35 Russell Westbrook WSH PG
35 36 Domantas Sabonis IND PF
36 36 De'Aaron Fox SAC PG
37 38 Zion Williamson NO SF
38 39 Tobias Harris PHI SF
39 40 Bam Adebayo MIA C
40 41 DeMar DeRozan SA SG
41 41 D'Angelo Russell MIN SG
42 43 Gordon Hayward CHA SF
43 44 Kyle Lowry TOR PG
44 44 Shai Gilgeous-Alexander OKC SG
45 46 Mike Conley UTAH PG
46 47 Malik Beasley MIN SG
47 48 RJ Barrett NY SG
48 49 Thomas Bryant WSH C
49 50 Pascal Siakam TOR PF

Answer 2

Is this what you want?
import requests
from bs4 import BeautifulSoup
from tabulate import tabulate
url = "https://www.espn.com/nba/stats/player/_/table/offensive/sort/avgPoints/dir/desc"
soup = BeautifulSoup(requests.get(url).text, "html.parser")
table_data = [
[r // 2, i.find("a").getText(), i.find("span").getText()] for r, i in
enumerate(soup.find_all("td", class_="Table__TD"), start=1)
if i.find("a") and i.find("span")
]
print(tabulate(table_data, headers=["Rank", "Name", "Team"], tablefmt="pretty"))
Output:
| Rank | Name | Team |
+------+-------------------------+------+
| 1 | James Harden | HOU |
| 2 | Stephen Curry | GS |
| 3 | Bradley Beal | WSH |
| 4 | Trae Young | ATL |
| 5 | Kevin Durant | BKN |
| 6 | CJ McCollum | POR |
| 7 | Kyrie Irving | BKN |
| 8 | Jaylen Brown | BOS |
| 9 | Giannis Antetokounmpo | MIL |
| 10 | Jayson Tatum | BOS |
| 11 | Damian Lillard | POR |
| 12 | Luka Doncic | DAL |
| 13 | Collin Sexton | CLE |
| 14 | Paul George | LAC |
| 15 | Brandon Ingram | NO |
| 16 | Nikola Jokic | DEN |
| 17 | LeBron James | LAL |
| 18 | Zach LaVine | CHI |
| 19 | Christian Wood | HOU |
| 20 | Kawhi Leonard | LAC |
| 21 | Joel Embiid | PHI |
| 22 | Jerami Grant | DET |
| 23 | Anthony Davis | LAL |
| 24 | Jamal Murray | DEN |
| 25 | Julius Randle | NY |
| 26 | Malcolm Brogdon | IND |
| 27 | Fred VanVleet | TOR |
| 28 | Nikola Vucevic | ORL |
| 29 | Donovan Mitchell | UTAH |
| 30 | Terry Rozier | CHA |
| 31 | Devin Booker | PHX |
| 32 | Khris Middleton | MIL |
| 33 | Terrence Ross | ORL |
| 34 | Victor Oladipo | IND |
| 35 | Russell Westbrook | WSH |
| 36 | Domantas Sabonis | IND |
| 37 | De'Aaron Fox | SAC |
| 38 | Zion Williamson | NO |
| 39 | Tobias Harris | PHI |
| 40 | Bam Adebayo | MIA |
| 41 | DeMar DeRozan | SA |
| 42 | D'Angelo Russell | MIN |
| 43 | Gordon Hayward | CHA |
| 44 | Kyle Lowry | TOR |
| 45 | Shai Gilgeous-Alexander | OKC |
| 46 | Mike Conley | UTAH |
| 47 | Malik Beasley | MIN |
| 48 | RJ Barrett | NY |
| 49 | Thomas Bryant | WSH |
| 50 | Pascal Siakam | TOR |
+------+-------------------------+------+

Answer 3

Always ask you - Is there an easier way?
Yes it is and you should go it :)
If you wanna scrape, first take a look if you really have to scrape content from the website or if there is an api that provide the information well structured.
Example requesting api
import requests
import pandas as pd
url = "https://site.web.api.espn.com/apis/common/v3/sports/basketball/nba/statistics/byathlete?region=us&lang=en&contentorigin=espn&isqualified=true&page=1&limit=50&sort=offensive.avgPoints%3Adesc&season=2021&seasontype=2"
headers = {"user-agent": "Mozilla/5.0"}
response = requests.get(url, headers=headers)
response.raise_for_status()
ranking=[]
for i,player in enumerate(response.json()['athletes'], start=1):
rank = i
name = player['athlete']['displayName']
team = player['athlete']['teamShortName']
category = player['athlete']['position']['abbreviation']
ranking.append({'rank':rank, 'name':name, 'team':team, 'category':category})
df = pd.DataFrame(ranking)
df
Output data frame
rank name team category
1 James Harden HOU SG
2 Stephen Curry GS PG
3 Bradley Beal WSH SG
4 Trae Young ATL PG
5 Kevin Durant BKN SF
6 CJ McCollum POR SG
7 Kyrie Irving BKN PG
8 Jaylen Brown BOS SG
9 Giannis Antetokounmpo MIL PF
10 Jayson Tatum BOS PF
But to answer your question
You can also do it with BeautifulSoup, but it is much more error-prone in my opinion:
from bs4 import BeautifulSoup
import requests
import pandas as pd
data = []
url ='https://www.espn.com/nba/stats/player/_/table/offensive/sort/avgPoints/dir/desc'
source = requests.get(url)
soup = BeautifulSoup(source.text, 'lxml')
for i in soup.select('tr')[1:]:
if i.select_one('td'):
rank = i.select_one('td').get_text()
if i.select_one('div > a'):
player = i.select_one('div > a').get_text()
if i.select_one('div > span'):
team =i.select_one('div > span').get_text()
data.append({'rank':rank, 'player':player, 'team':team})
pd.DataFrame(data)
If you do not wanna use css selectors, you can also do
for i in soup.find_all('tr')[1:]:
if i.find('td'):
rank = i.find('td').get_text()
if i.find('a'):
player = i.find('a').get_text()
if i.find('span'):
team =i.find('span').get_text()