I have below python method which insert data in to a table. the first column is json_data and the 2nd column is file name. I am getting both the values to this function while calling this method from main.
def insert(sf_handler,data,file_name):
query = """INSERT INTO my_table (DATA,FILE_NAME)
(select (PARSE_JSON('%s'),'%s'))""" % {json.dumps(data),file_name)}
pd.read_sql(query,sf_handler)
But while executing this i am getting below error. Can someone help on this.
TypeError: not enough arguments for format string
I got the answer. Just type cast the file name and remove the flower bracket with parenthesis it will work.
query = """INSERT INTO my_table (DATA,FILE_NAME)
(select (PARSE_JSON('%s'),'%s'))""" % (json.dumps(data),str(file_name))
pd.read_sql(query,sf_handler)
Related
controller.py
def consolidated_universities_data_by_country(countries,universities):
cursor = connection.cursor()
query = None
if countries == str(1):
query = f"""
#sql_query#
"""
result_data=cursor.execute(query)
result=dict_fetchall_rows(result_data)
consolidated_df_USA=pd.DataFrame(result).fillna('NULL').replace( {True : 1, False : 0}).groupby('CourseId')['ApplicationDeadline'].apply(', '.join).reset_index()
return consolidated_df_USA
With the mentioned code i am able to get desired output i.e., i wanted to merge n rows deadline in one row for given courseid, but i am not able to get rest of the columns.
consolidated_df_USA=pd.DataFrame(result).fillna('NULL').replace( {True : 1, False : 0}).groupby('CourseId')['ApplicationDeadline','CourseName'].agg(', '.join).reset_index()
return consolidated_df_USA
With this i am able to get some columns but some of the columns are getting depricated. Also getting below warning.
FutureWarning: Dropping invalid columns in SeriesGroupBy.agg is deprecated. In a future version, a TypeError will be raised. Before calling .agg, select only columns which should be valid for the aggregating function.
How to get all the columns which is given by sql query?
I have one table in snowflake, I am performing bulk load using.
one of the columns in table is date, but in the source table which is on sql server is having null values in date column.
The flow of data is as :
sql_server-->S3 buckets -->snowflake_table
I am able to perform the sqoop job in EMR , but not able to load the data into snowflake table, as it is not accepting null values in the date column.
The error is :
Date '' is not recognized File 'schema_name/table_name/file1', line 2, character 18 Row 2,
column "table_name"["column_name":5] If you would like to continue loading when an error is
encountered, use other values such as 'SKIP_FILE' or 'CONTINUE' for the ON_ERROR option.
can anyone help, where I am missing
Using below command you can able to see the values from stage file:
select t.$1, t.$2 from #mystage1 (file_format => myformat) t;
Based on the data you can change your copy command as below:
COPY INTO my_table(col1, col2, col3) from (select $1, $2, try_to_date($3) from #mystage1)
file_format=(type = csv FIELD_DELIMITER = '\u00EA' SKIP_HEADER = 1 NULL_IF = ('') ERROR_ON_COLUMN_COUNT_MISMATCH = false EMPTY_FIELD_AS_NULL = TRUE)
on_error='continue'
The error shows that the dates are not arriving as nulls. Rather, they're arriving as blank strings. You can address this a few different ways.
The cleanest way is to use the TRY_TO_DATE function on your COPY INTO statement for that column. This function will return database null when trying to convert a blank string into a date:
https://docs.snowflake.com/en/sql-reference/functions/try_to_date.html#try-to-date
I want to print a column values from a table using this query -
cursor = self.conn.execute ("select column1 from table_name where column2 =='%d'"%(number))
Value = cursor.fetchall()
Print value
The mentioned query format is not secure, you can try binding in this way:
self.conn.execute('SELECT column1 FROM table_name WHERE column2 = ?', (number,))
According the docs (Sqlite3 Docs):
# Never do this -- insecure!
symbol = 'RHAT'
c.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol)
# Do this instead
t = ('RHAT',)
c.execute('SELECT * FROM stocks WHERE symbol=?', t)
You have declared your database item as ATOMIC NUMBER so either change that to ATOMIC_NUMBER or change the code to put inverted commas around the two words "ATOMIC NUMBER".
AND for the 5th time now read the StackOverflow Tour guide so that you can behave in a way that doesn't annoy the people that are trying to answer your questions.
https://stackoverflow.com/tour
I have a C++ program that inserts values into a database table. I can't directly hardcode the values in because the data is constantly being updated, but I'm really confused about the syntax.
When I try to do this:
l.exec("INSERT INTO course VALUES(cid, term, 'subj',crse, sec, 'units', 'instructors');");
l.exec("INSERT INTO meeting VALUES(cid, term, 'type', 'days', 'time', 'build', room);");
l.exec("INSERT INTO enrolledin VALUES(cid, term, sid, 'major', 'classlevel', 'level', 'status', seat, numunits, 'grade');");
l.exec("INSERT INTO student VALUES(sid, 'surname', 'prefname', 'email');");
I get this error:
terminate called after throwing an instance of 'pqxx::undefined_column'
what(): ERROR: column "cid" does not exist
LINE 1: INSERT INTO course VALUES(cid, term, 'subj',crse, se...
^
HINT: There is a column named "cid" in table "course", but it cannot be referenced from this part of the query.
--
I was told that it's because I was inserting the literal string name instead of the values inside the string, and I'm confused as to how to insert the values inside the string via C++ while still using variable names.
Syntax of the used SQL INSERT query is incorrect. It should be:
INSERT INTO course (cid, subj) VALUES(1, 'subj');
You should specify table name together with columns to insert into and values after that. I reduced number of columns for simplicity. For a complete syntax of INSERT query check the PostgreSQL documentation.
To insert values from your variables you can do the following:
int cidValue = 1;
std::string subjValue = "subj";
l.exec("INSERT INTO course (cid, subj) VALUES(" + std::to_string(cidValue) + ", '" + l.esc(subjValue) + "')");
esc() function helps to prevent SQL injection attack.
I'm having some strange feeling abour sqlite3 parameters that I would like to expose to you.
This is my query and the fail message :
#query
'SELECT id FROM ? WHERE key = ? AND (userid = '0' OR userid = ?) ORDER BY userid DESC LIMIT 1;'
#error message, fails when calling sqlite3_prepare()
error: 'near "?": syntax error'
In my code it looks like:
// Query is a helper class, at creation it does an sqlite3_preprare()
Query q("SELECT id FROM ? WHERE key = ? AND (userid = 0 OR userid = ?) ORDER BY userid DESC LIMIT 1;");
// bind arguments
q.bindString(1, _db_name.c_str() ); // class member, the table name
q.bindString(2, key.c_str()); // function argument (std::string)
q.bindInt (3, currentID); // function argument (int)
q.execute();
I have the feeling that I can't use sqlite parameters for the table name, but I can't find the confirmation in the Sqlite3 C API.
Do you know what's wrong with my query?
Do I have to pre-process my SQL statement to include the table name before preparing the query?
Ooookay, should have looked more thoroughly on SO.
Answers:
- SQLite Parameters - Not allowing tablename as parameter
- Variable table name in sqlite
They are meant for Python, but I guess the same applies for C++.
tl;dr:
You can't pass the table name as a parameter.
If anyone have a link in the SQLite documentation where I have the confirmation of this, I'll gladly accept the answer.
I know this is super old already but since your query is just a string you can always append the table name like this in C++:
std::string queryString = "SELECT id FROM " + std::string(_db_name);
or in objective-C:
[#"SELECT id FROM " stringByAppendingString:_db_name];