Beautifulsoup extraction using for loop into table in Python 2 - python-2.7

Question

Platform: Python 2.7.13 on Win 7 with spyder IDE
Please I'm totally new to both beautifulsoup and python. I am stuck at the last two lines.
Q. I want to import the details on the url below and put them in a table. That is the information with dd tags:
The first part of the code works well to get the link and get all the school details. However, i'm having trouble running the for command to get the remaining elements.
full code is below
# coding: utf-8
import urllib2
url = "http://tools.canlearn.ca/cslgs-scpse/cln-cln/rep-fit/p/af.p.clres.do?institution_id=default&searchType=ALL&searchString=&progLang=A&instType=B&prov_1=prov_1&progNameOnly=N&start=0&finish=999&section=1"
#try:
page = urllib2.urlopen(url)
#except (httplib.HTTPException, httplib.IncompleteRead, urllib2.URLError):
# missing.put(tmpurl)
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
rooturl = "http://tools.canlearn.ca/cslgs-scpse/cln-cln/rep-fit/p/"
from bs4 import BeautifulSoup
soup = BeautifulSoup(page)
info = soup.find_all("div", class_="wb-frm")
names = [x.ol.find_all("li") for x in info][0]
def f(string):
return str(string[0] + ', ' + string[-1])
names2 = [names[i:i+3] for i in range(0, len(names), 3)]
diploma = [ [x[0].findAll("a")[0].find(text=True).strip() ,x[1].string ,f(x[2].find(text=True).strip().split()) ] for x in names2]
links = [x.ol.find_all("a") for x in info][0]
links2 = [y.get('href') for y in links]
links3 = [rooturl + z for z in links2]
for i in xrange(len(links3)) :
url_link = urllib2.urlopen(links3[i])
link_html = BeautifulSoup(url_link)
#Changed the code here based on good answer given by heyiamt ..
#it was
# link_html2 = link_html.find_all("div", class_="wb-frm")
# website = link_html2[0].a.get('href')
# dd[y]=link2[y].get('dd')
# diploma[i].append(dd) diploma[i].append(link_html2[0].a.get('href'))
# diploma[i].append(website)
#Get the whole box for the general info
# general_info_html = link_html.find_all("div", class_="panel-body")
# general_info_html2 = [y.findAll('dd') for y in general_info_html[2:]]
# general_info = {}
# for x in general_info_html2 :
# general_info.update({x[0].find(text='dt') : x[1].find(text='dd')})
# general_info.update({x[0].get('dd')})
# diploma[i].append(general_info)
for d in link_html.find_all('dd'):
if d.a is not None:
diploma[i].append(d.a.string)
continue
if d.string is not None:
diploma[i].append(d.string)
continue
diploma[i].append(d.contents[0])
import pandas as pd
col1 = [x[1] for x in diploma]
col2 = [x[0] for x in diploma]
col3 = [x[2] for x in diploma]
col4 = [x[3] for x in diploma]
col5 = [x[4] for x in diploma]
col55 = {'Program Level' : [x.get('Program Level:') for x in col5], 'Credential Type' : [x.get('Credential Type:') for x in col5],
'Joint Program Level' : [x.get('Joint Program Level:') for x in col5],
'Joint Credential Type' : [x.get('Joint Credential Type:') for x in col5],
'Address' : [x.get('Address:') for x in col5],
'Telephone' : [x.get('Telephone:') for x in col5],
'Email' : [x.get('Email:') for x in col5],
'Fax' : [x.get('Fax:') for x in col5],
'Toll Free' : [x.get('Toll Free:') for x in col5]
}
df = pd.DataFrame(col1, columns = ['University'])
df2 = pd.DataFrame(col55)
df['Type'] = col2
df['City'] = col3
df['Website'] = col4
df['Address'] = df2['Address']
df['Credential Type'] = df2['Credential Type']
df['Email'] = df2['Email']
df['Fax'] = df2['Fax']
df['Joint Credential Type'] = df2['Joint Credential Type']
df['Joint Program Level'] = df2['Joint Program Level']
df['Program Level'] = df2['Program Level']
df['Telephone'] = df2['Telephone']
df['Toll Free'] = df2['Toll Free']
df.to_csv('data1.csv', encoding='utf-8')
Expected result: (i.e with "dd" tags)
http://www.rosewoodcollege.ca/program-information/
Apprenticeship Program Certificate
Not entered
Not entered
Calgary, Alberta T3J 5H3
[phone]
mail#rosewoodcollege.ca

Answer 1

For this site, you can just use BeautifulSoup to find the tags within the divs without actually scrolling through the divs themselves. These particular dd tags have a bit of fishiness to them, though. Here's a shot at managing the different possibilities.
# Using link_html from your code above.
dd_strs = []
for d in link_html.find_all('dd'):
if d.a is not None:
dd_strs.append(d.a.string)
continue
if d.string is not None:
dd_strs.append(d.string)
continue
dd_strs.append(d.contents[0])
for dd_str in dd_strs:
print dd_str
Output is
http://www.rosewoodcollege.ca/program-informatio n/
Apprenticeship Program
Certificate
Not entered
Not entered
Rosewood College
[phone]
mail#rosewoodcollege.ca
2015-12-30
If you can rely on the dt tags to always be mated, in order, to the dd tags, you can just repeat the above but for dt instead of dd and merge the lists accordingly.