I have been up and down these pages looking for how to do this and there are many similar posts but I can't seem to get it to work, so I find myself having to ask specifically how to do this.
I am trying to gather metrics about my software project in git hub. For many of these metrics you can use the API. However, one of the most interesting items are the unique visitors and view count on the github graphs/traffic and unfortunately this info is not located in the Github API. So, to get this I am trying to log into my github account navigate to the site then get the numbers. Located below is my code. I can't seem to get logged into github to do anything however (my url request continues to show a login page rather then the traffic page). I think it probably has something to do with the variables that need to be posted but I'm not sure whats wrong with them.
from requests import session
from bs4 import BeautifulSoup as bs
USER = 'MYID'
PASSWORD = 'MYPASSWORD'
URL1 = 'https://github.com/login'
URL2 = 'https://github.com/MYPROJ/graphs/traffic'
with session() as s:
req = s.get(URL1).text
html = bs(req)
token = html.find("input", {"name": "authenticity_token"}).attrs['value']
com_val = html.find("input", {"name": "commit"}).attrs['value']
login_data = {'login_field': USER,
'password': PASSWORD,
'authenticity_token' : token,
'commit' : com_val}
r1 = s.post(URL1, data = login_data)
r2 = s.get(URL2)
print(r2.url)
print bs(r2.text).find('span', {'class':'num js-uniques uniques'})
Any help is appreciated.
Thanks,
-Jeff
Figured it out.
I was using the wrong address to post my login and username, as well as some other wrong bits.
This is the updated code that worked for me:
from requests import session
from bs4 import BeautifulSoup as bs
USER = 'MyUserName'
PASSWORD = 'Mypassword'
URL1 = 'https://github.com/session'
URL2 = 'https://github.com/MyProj/graphs/traffic-data'
with session() as s:
req = s.get(URL1).text
html = bs(req)
token = html.find("input", {"name": "authenticity_token"}).attrs['value']
com_val = html.find("input", {"name": "commit"}).attrs['value']
login_data = {'login': USER,
'password': PASSWORD,
'commit' : com_val,
'authenticity_token' : token}
r1 = s.post(URL1, data = login_data)
r2 = s.get(URL2)
Cut1 = r2.text.split(',"summary":{"total":',2)
ViewsTot = Cut1[1].split(',"unique":',1)
ViewsUnq = ViewsTot[1].split('}}',1)
Related
I attempted to request a REST request to see the document below. But do not work. https://superset.apache.org/docs/rest-api
request: curl -XGET -L http://[IP:PORT]/api/v1/chart
response: {"msg":"Bad Authorization header. Expected value 'Bearer <JWT>'"}
The Superset installation has been on PIP and was also Helm Chart. But all are the same. helm: https://github.com/apache/superset
How should I order a REST API?
Check the security section of the documentation you have linked. It has this API /security/login, you can follow the JSON parameter format and get the JWT bearer token. Use that token to send in the Header of your other API calls to superset.
open http://localhost:8080/swagger/v1, assuming http://localhost:8080 is your Superset host address
then find this section
the response would be like this
{
"access_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmcmVzaCI6dHJ1ZSwiaWF0IjoxNjU0MzQ2OTM5LCJqdGkiOiJlZGY2NTUxMC0xMzI1LTQ0NDEtYmFmMi02MDc1MzhjZDcwNGYiLCJ0eXBlIjoiYWNjZXNzIiwic3ViIjoxLCJuYmYiOjE2NTQzNDY5MzksImV4cCI6MTY1NDM0NzgzOX0.TfjUea3ycH77xhCWOpO4LFbYHrT28Y8dnWsc1xS_IOY",
"refresh_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmcmVzaCI6ZmFsc2UsImlhdCI6MTY1NDM0NjkzOSwianRpIjoiNzBiM2EyZDYtNDFlNy00ZDNlLWE0NDQtMTRiNTkyNTk4NjUwIiwidHlwZSI6InJlZnJlc2giLCJzdWIiOjEsIm5iZiI6MTY1NDM0NjkzOSwiZXhwIjoxNjU2OTM4OTM5fQ.OgcctNnO4zTDfTgtHnaEshk7u-D6wOxfxjCsjqjKYyE"
}
Thank #andrewsali commented on this github issue, I finally figure out how to access the superset REST API by python code.
import requests
from bs4 import BeautifulSoup
import json
def get_supetset_session():
"""
# http://192.168.100.120:8088/swagger/v1
url = f'http://{superset_host}/api/v1/chart/'
r = s.get(url)
# print(r.json())
"""
superset_host = '192.168.100.120:8088' # replace with your own host
username = 'YOUR_NAME'
password = 'YOUR_PASSWORD'
# set up session for auth
s = requests.Session()
login_form = s.post(f"http://{superset_host}/login")
# get Cross-Site Request Forgery protection token
soup = BeautifulSoup(login_form.text, 'html.parser')
csrf_token = soup.find('input',{'id':'csrf_token'})['value']
data = {
'username': username,
'password': password,
'csrf_token':csrf_token
}
# login the given session
s.post(f'http://{superset_host}/login/', data=data)
print(dict(s.cookies))
return s
DEMO
# s = get_supetset_session()
base_url = 'http://192.168.100.120:8088'
def get_dashboards_list(s, base_url=base_url):
"""## GET List of Dashboards"""
url = base_url + '/api/v1/dashboard/'
r = s.get(url)
resp_dashboard = r.json()
for result in resp_dashboard['result']:
print(result['dashboard_title'], result['id'])
s = get_supetset_session()
# {'session': '.eJwlj8FqAzEMRP_F5z1Islay8jOLJcu0NDSwm5xK_r0uPQ7DG978lGOeeX2U2_N85VaOz1FuxVK6JIHu1QFhGuEOk5NG8qiYGkJ7rR3_Ym-uJMOzJqySeHhIG8SkNQK6GVhTdLf0ZMmG6sZGQtiQ1Gz0qYiUTVoHhohZthLXOY_n4yu_l0-VKTObLaE13i2Hz2A2rzBmhU7WkkN1cfdH9HsuZoFbeV15_l_C8v4F4nBC9A.Ypn16Q.yz4E-vz0gp3EmJwv-6tYIcOGavU'}
get_dashboards_list(s)
Thanks #Ferris for this visual solution!
To add to this, you can also create the appropriate API call with Python just like following:
import requests
api_url = "your_url/api/v1/security/login"
payload = {"password":"your password",
"provider":"db",
"refresh":True,
"username":"your username"
}
response = requests.post(api_url, json=payload)
# the acc_token is a json, which holds access_token and refresh_token
access_token = response.json()['access_token']
# no get a guest token
api_url_for_guesttoken = "your_url/api/v1/security/guest_token"
payload = {}
# now this is the crucial part: add the specific auth-header
response = request.post(api_url_for_guesttoken , json=payload, headers={'Authorization':f"Bearer {access_token}"})
I have a Django site in which I want to use spotipy to look for statistics of the song like popularity and views. I have this code right now:
import spotipy
import spotipy.util as util #luxury
import json
import webbrowser
username = 'dgrqnco2rx8hdu58kv9if9eho'
scope = 'user-read-private user-read-playback-state user-modify-playback-state'
token = util.prompt_for_user_token(username, scope, client_id='08bb526962574a46b359bffc56048147',
client_secret='bf6d4184c8ae40aca207714e02153bad', redirect_uri='http://google.com/')
sp_obj = spotipy.Spotify(auth=token)
ss = 'name of song'
if ss.__contains__('('):
q = ss[0:ss.index('(')]
elif ss.__contains__('['):
q = ss[0:ss.index('[')]
elif ss.__contains__('['):
q = ss[0:ss.index('{')]
else:
q = ss
query = sp_obj.search(q, 1, 0, 'track')
#<<<<<<<<<<SONG>>>>>>>>>>
#FIND THE SONG URI
song_uri = query['tracks']['items'][0]['uri']
track = sp_obj.track(song_uri)
track_data = sp_obj.audio_features(song_uri)
song_popularity = track['popularity']
song_danceability = track_data[0]['danceability']
song_energy = track_data[0]['energy']
song_loudness = track_data[0]['loudness']
song_tempo = track_data[0]['tempo']
However spotipy redirects me to a page for authorization and I need to paste the url in the console. The regular user however does not have access to this console. So how can I do the authorization in an alternative way or even bypass it?
I was thinking about getting a spotify account in which every user will be getting logged in so that the user won't have to do the authorization and won't have to have a spotify account. Is this possible? If not what else can I try?
You can't use util.prompt_for_user_token because it's just a helper for local usage only.
You need to arrange your code as API endpoints so that multiple users can sign in. Here is a full working example that would allow multiple users to sign in https://github.com/plamere/spotipy/blob/master/examples/app.py.
It uses Flask but you can easily adapt it to Django.
I have a terminal that served in webbrowser with wetty. I want to authenticate the user from gitlab to let user with interaction with the terminal(It is inside docker container. When user authenticated i ll allow him to see the containers terminal).
I am trying to do OAuth 2.0 but couldn't manage to achieve.
That is what i tried.
I created an application on gitlab.
Get the code and secret and make a http call with python script.
Script directed me to login and authentication page.
I tried to get code but failed(Their is no mistake on code i think)
Now the problem starts in here. I need to get the auth code from redirected url to gain access token but couldn't figure out. I used flask library for get the code.
from flask import Flask, abort, request
from uuid import uuid4
import requests
import requests.auth
import urllib2
import urllib
CLIENT_ID = "clientid"
CLIENT_SECRET = "clientsecret"
REDIRECT_URI = "https://UnrelevantFromGitlabLink.com/console"
def user_agent():
raise NotImplementedError()
def base_headers():
return {"User-Agent": user_agent()}
app = Flask(__name__)
#app.route('/')
def homepage():
text = 'Authenticate with gitlab'
return text % make_authorization_url()
def make_authorization_url():
# Generate a random string for the state parameter
# Save it for use later to prevent xsrf attacks
state = str(uuid4())
save_created_state(state)
params = {"client_id": CLIENT_ID,
"response_type": "code",
"state": state,
"redirect_uri": REDIRECT_URI,
"scope": "api"}
url = "https://GitlapDomain/oauth/authorize?" + urllib.urlencode(params)
print get_redirected_url(url)
print(url)
return url
# Left as an exercise to the reader.
# You may want to store valid states in a database or memcache.
def save_created_state(state):
pass
def is_valid_state(state):
return True
#app.route('/console')
def reddit_callback():
print("-----------------")
error = request.args.get('error', '')
if error:
return "Error: " + error
state = request.args.get('state', '')
if not is_valid_state(state):
# Uh-oh, this request wasn't started by us!
abort(403)
code = request.args.get('code')
print(code.json())
access_token = get_token(code)
# Note: In most cases, you'll want to store the access token, in, say,
# a session for use in other parts of your web app.
return "Your gitlab username is: %s" % get_username(access_token)
def get_token(code):
client_auth = requests.auth.HTTPBasicAuth(CLIENT_ID, CLIENT_SECRET)
post_data = {"grant_type": "authorization_code",
"code": code,
"redirect_uri": REDIRECT_URI}
headers = base_headers()
response = requests.post("https://MyGitlabDomain/oauth/token",
auth=client_auth,
headers=headers,
data=post_data)
token_json = response.json()
return token_json["access_token"]
if __name__ == '__main__':
app.run(host="0.0.0.0",debug=True, port=65010)
I think my problem is on my redirect url. Because it is just an irrelevant link from GitLab and there is no API the I can make call.
If I can fire
#app.route('/console')
that line on Python my problem will probably will be solved.
I need to make correction on my Python script or different angle to solve my problem. Please help.
I was totally miss understand the concept of auth2. Main aim is to have access_token. When i corrected callback url as localhost it worked like charm.
I am trying to download the entire html code from
http://www.ivolatility.com/options/AMZN/NASDAQ/
The output does not include the data in the tables.
This is the code I am using
url = 'http://www.ivolatility.com/options/AMZN/NASDAQ/'
r = requests.get(url, allow_redirects=True)
open('C:.../Downloads/amzn.html', 'wb').write(r.content)
I think it might be related to registration issues.
Anything I can do?
Thanks
Your request returns a login form, which means you'll have to login in order to access the data.
The login process is relatively easy - all we have to do is submit the form data to the login page (and use a session object to store the cookies).
Then we can use that authenticated session to retrieve the table contents.
The code,
import requests
url = 'http://www.ivolatility.com/options/AMZN/NASDAQ/'
login_url = 'https://www.ivolatility.com/login.j'
usr = 'my username'
pwd = 'my password'
data = {
'username':usr, 'password':pwd,
'ref_url':login_url, 'service_name':'Home Page',
'step':1, 'login__is__sent':1
}
s = requests.session()
s.post(login_url, data)
r = s.get(url)
with open('my file', 'wb') as f:
f.write(r.content)
Suppose user 'ABCD' has tweeted on topic, how to find the comments for that tweet ?
I am using tweepy to get twitter data.
It will also be good, If it possible using any other python packages.
Code for the following is:
Import tweepy
consumer_key = ""
consumer_secret = ""
access_key = ""
access_secret = ""
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api = tweepy.API(auth)` # This is authentication process.
new_tweets = api.user_timeline(screen_name ='ABCD',count=20) # Code to download 20 timelines for user 'ABCD'
Till here i can get 20 timelines from user ABCD.
Q: How to get tweets for each timeline.
Thanks.