Code snippet
from dbinit import session
from geoalchemy2 import Geometry, func
result = session.query(func.ST_AsText('POINT(100 100)'))
How to retrieve the data from this result object?
I have figured out the solution.
re = result.all()
Related
I have the following code below
#app.route('/all', methods=['GET'])
def get_all_feedback():
all_feedback = db.session.query(Feedback).all()
result = feedbacks_schema.dump(all_feedback)
return jsonify(result)
I am using postman to make a request to get all feedback from my feedback table I have (fyi: I am using postgreSQL). I am getting all my feedback but only the ids from my table. What I want is all the columns for each row.
What is the correct way of doing this?
Try
all_feedback = Feedback.query.all()
I need some help regarding the error in code. My Code consists of retrieving the zomato reviews and storing it in HDFS and again reading it performing Recommender Analtyics on it. I am getting a problem regarding my function is not recognizing in pyspark code. I am not entirely pasting the code as it might be confusing so i am writing a small similar use case for your easy understanding.
I am trying to read a file from local and converting it to dataframe from rdd and performing some operations and again converting it to rdd and performing map operation to have delimiter by '|' and then save it to HDFS.
When i try to call self.filter_data(y) in lambda func of check function its not recognizing and giving me error as
Exception: It appears that you are attempting to reference
SparkContext from a broadcast variable, action, or transformation.
SparkContext can only be used on the driver, not in code that it run
on workers. For more information, see SPARK-5063.
****CAN ANY ONE HELP ME WHY MY FILTER_DATA FUNCTION IS NOT RECOGNISING? SHOULD I NEED TO ADD ANY THING OR ANY THING WRONG IN THE WAY I AM CALLING. PLEASE HELP ME. THANKS IN ADVANCE****
INPUT VALUE
starting
0|0|ffae4f|0|https://b.zmtcdn.com/data/user_profile_pictures/565/aed32fa2eb18bb4a5a3ba426870fd565.jpg?fit=around%7C100%3A100&crop=100%3A100%3B%2A%2C%2A|https://www.zomato.com/akellaram87?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1|2.5|FFBA00|Well...|unknown|16946626|2017-08-01T00-25-43.455182Z|30059877|Have been here for a quick bite for lunch, ambience and everything looked good, food was okay but presentation was not very appealing. We or...|2017-04-15 16:38:38|Big Foodie|6|Venkata Ram Akella|akellaram87|Bad Food|0.969352505662|0|0|0|0|0|0|1|1|0|0|1|0|0|0.782388212399
ending
starting
1|0|ffae4f|0|https://b.zmtcdn.com/data/user_profile_pictures/4d1/d70d7a57e1bfdf296ff4db3d8daf94d1.jpg?fit=around%7C100%3A100&crop=100%3A100%3B%2A%2C%2A|https://www.zomato.com/users/sm4-2011696?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1|1|CB202D|Avoid!|unknown|16946626|2017-08-01T00-25-43.455182Z|29123338|Giving a 1.0 rating because one cannot proceed with writing a review, without rating it. This restaurant deserves a 0 star rating. The qual...|2017-01-04 10:54:53|Big Foodie|4|Sm4|unknown|Bad Service|0.964402034541|0|1|0|0|0|0|0|1|0|0|0|1|0|0.814540622345
ending
My code:
if __name__== '__main__':
import os,logging,sys,time,pandas,json;from subprocess
import PIPE,Popen,call;from datetime import datetime, time, timedelta
from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName('test')
sc = SparkContext(conf = conf,pyFiles=['/bdaas/exe/nlu_project/spark_classifier.py','/bdaas/exe/spark_zomato/other_files/spark_zipcode.py','/bdaas/exe/spark_zomato/other_files/spark_zomato.py','/bdaas/exe/spark_zomato/conf_files/spark_conf.py','/bdaas/exe/spark_zomato/conf_files/date_comparision.py'])
from pyspark.sql import Row, SQLContext,HiveContext
from pyspark.sql.functions import lit
sqlContext = HiveContext(sc)
import sys,logging,pandas as pd
import spark_conf
n = new()
n.check()
class new:
def __init__(self):
print 'entered into init'
def check(self):
data = sc.textFile('file:///bdaas/src/spark_dependencies/classifier_data/final_Output.txt').map(lambda x: x.split('|')).map(lambda z: Row(restaurant_id=z[0], rating = z[1], review_id = z[2],review_text = z[3],rating_color = z[4],rating_time_friendly=z[5],rating_text=z[6],time_stamp=z[7],likes=z[8],comment_count =z[9],user_name = z[10],user_zomatohandle=z[11],user_foodie_level = z[12],user_level_num=z[13],foodie_color=z[14],profile_url=z[15],profile_image=z[16],retrieved_time=z[17]))
data_r = sqlContext.createDataFrame(data)
data_r.show()
d = data_r.rdd.collect()
print d
data_r.rdd.map(lambda x: list(x)).map(lambda y: self.filter_data(y)).collect()
print data_r
def filter_data(self,y):
s = str()
for i in y:
print i.encode('utf-8')
if i != '':
s = s + i.encode('utf-8') + '|'
print s[0:-1]
return s[0:-1]
I am trying to run a VAR model with the following script.
import statsmodels
import statsmodels.tsa.api as sm
from statsmodels.tsa.api import VAR
tsBitcoin_frame = tsBitcoin.to_frame()
tsSP500_frame = tsSP500.to_frame()
forVar = [tsBitcoin_frame, tsSP500_frame]
dataForVar = pd.concat(forVar, axis =1)
model = VAR(dataForVar)
results = model.fit(2)
results.summary()
However python is giving me the following error "name 'VAR' is not defined"
I am using statsmodels version 0.8.0. I even tried using command sm.VAR instead VAR but then python wouldn't print the statistics of VAR model. Does anyone know why this is happening, how can I solve it or how can implement the VAR model in python? Thanks!
Sorry I figured out my mistake. I was not putting print before results.summary and should have left the line from statsmodels.tsa.api import VAR. Thanks though!
My gateway say I need to URL encode all my data, in Django how is this possible?
requests.post(default_gateway.keyword_forwarding_url, data=raw_data,
stream=True, verify=True)
I have tried
import urllib
requests.post(default_gateway.keyword_forwarding_url, data=urllib.urlencode(raw_data),
stream=True, verify=True)
Your data needs to be passed to urlencode() as a dict or in a sequence of paired tuples:
import urllib
formatted_raw_data = {raw_data[0]: raw_data[1], raw_data[2]: raw_data[3]} // or however it needs to be formatted
requests.post(default_gateway.keyword_forwarding_url, data=urllib.urlencode(formatted_raw_data),
stream=True, verify=True)
When I use the Urllib module, I can call/print/search the html of a website the first time, but when I try again it is gone. How can I keep the html throughout the program.
For example, when I try:
html = urllib.request.urlopen('http://www.bing.com/search?q=Mike&go=&qs=n&form=QBLH&filt=all&pq=mike&sc=8-2&sp=-1&sk=')
search = re.findall(r'Mike',str(html.read()))
search
I get:
['Mike','Mike','Mike','Mike']
But then when I try to do this a second time like so:
results = re.findall(r'Mike',str(html.read()))
I get:
[]
when calling 'result'.
Why is this and how can I stop it from happening/fix it?
Without being very well versed in python, I'm guessing html.read() reads the http stream, so when you call it the second time there is nothing to read.
Try:
html = urllib.request.urlopen('http://www.bing.com/search?q=Mike&go=&qs=n&form=QBLH&filt=all&pq=mike&sc=8-2&sp=-1&sk=')
data = str(html.read())
search = re.findall(r'Mike',data)
search
And then use
results = re.findall(r'Mike',data)
In addition to the correct guess of #rvalik that you can only read a stream once, data = str(html.read()) is incorrect. urlopen returns a bytes object and str returns the display representation of that object. An example:
>>> data = b'Mike'
>>> str(data)
"b'Mike'"
What you should do is either decode the bytes object using the encoding of the HTML page (UTF-8 in this case):
from urllib.request import urlopen
import re
with urlopen('http://www.bing.com/search?q=Mike&go=&qs=n&form=QBLH&filt=all&pq=mike&sc=8-2&sp=-1&sk=') as html:
data = html.read().decode('utf8')
print(re.findall(r'Mike',data))
or search with a bytes object:
from urllib.request import urlopen
import re
with urlopen('http://www.bing.com/search?q=Mike&go=&qs=n&form=QBLH&filt=all&pq=mike&sc=8-2&sp=-1&sk=') as html:
data = html.read()
print(re.findall(rb'Mike',data))