I am exporting data to excel(.xls) in django using xlwt module but datetime is exporting in this 43239.6389467593 format. I dont know what is this format and how to change it to datetime in excel sheet. I followed the following
https://simpleisbetterthancomplex.com/tutorial/2016/07/29/how-to-export-to-excel.html
This is a numerical representation of the date value. If you don't want to change the export code on django you can fix the issue within Excel.
If you have an entry like this in cell A1 for example then set Cell B1 =A1 and change the format of the cell B1 to the Dateformat you want. It should then appear as a normal date.
You can do this
import datatime
if isinstance(row[col_num], datetime.datetime):
date_time = row[col_num].strftime('%Y-%m-%d %H:%M:%S')
ws.write(row_num, col_num, date_time, font_style)
else:
ws.write(row_num, col_num, row[col_num], font_style
Related
I am trying to load a CSV from GCS which contains timestamps in one of the columns.
When I upload via BQ interface, I get the following error:
Could not parse '2018-05-03 10:25:18.257000000' as DATETIME for field creation_date (position 6) starting at location 678732930 with message 'Invalid datetime string "2018-05-03 10:25:18.257000000"'
Is the issue here the trailing 0's? How would I fix the issue using Python?
Thanks in advance
Yes you are correct. The issue is the trailing 0s. DATETIME field only allows 6 digits at the subsecond value.
Name | Range
DATETIME | 0001-01-01 00:00:00 to 9999-12-31 23:59:59.999999
To remove the trailing 0s, you can use Pandas to convert it to a proper DATETIME format so it can be used in BigQuery. For testing purposes, I used a CSV file that contains a dummy value at column 0 and DATETIME with trailing 0s at column 1.
Test,2018-05-03 10:25:18.257000000
Test1,2018-05-03 10:22:18.123000000
Test2,2018-05-03 10:23:18.234000000
Using this block of code, Pandas will convert column 1 to the proper DATETIME format:
import pandas as pd
df = pd.read_csv("data.csv",header=None) #define your CSV file here
first_column = df.iloc[:, 1] # Change to the location of your DATETIME column
df.iloc[:, 1] = pd.to_datetime(first_column,format='%Y-%m-%d %H:%M:%S.%f') # convert to correct datetime format
df.to_csv("data.csv", header=False, index=False) # write the new values to data.csv
print(df) #print output for testing
This will result to:
Test,2018-05-03 10:25:18.257
Test1,2018-05-03 10:22:18.123
Test2,2018-05-03 10:23:18.234
You can now use the updated CSV file to write to BQ via BQ interface. See result of BQ testing:
What is the correct way to import specific sheet of excel by using Django-Import-Export Module??
or if possible. all the sheets in a workbook one by one...
I referred their documentation and it was not much helpful
https://django-import-export.readthedocs.io/en/latest/
the same way I would like to export data into one workbook from multiple sheets...
how do achieve it??
Here is the complete answer for my question
databook = Databook()
imported_data= databook.load(file.read( ), format= 'xlsx')
for dataset in imported_data.sheets():
print(dataset.title) # returns the names of the sheets
print(dataset) # returns the data in each sheet
This is how you can export your multiple datasets in one excel file
book = tablib.Databook((data1, data2, data3))
with open('students.xls', 'wb') as f:
f.write(book.export('xls'))
documentation
You can also try using use pyexcel_xls package on django it's fairly easy to use.
from pyexcel_xls import get_data as xls_get
def import_excel(request):
excel_file = request.FILES['file']
#uploading the excel file
if (str(excel_file).split('.')[-1] == "xls"):
data = xls_get(excel_file, column_limit=15)
elif (str(excel_file).split('.')[-1] == "xlsx"):
data = xlsx_get(excel_file, column_limit=15)
if (len(data['sheet1']) > 2): #reading of records begins from second row
name = data['sheet1'] #excel sheet name you wish to get
for l in range(2, len(name)): #loop through the number of rows in the sheet
data = name[l][0] #extract the first column's data
I have a continuous stream of data coming in so I want to define the DataFrame before hand so that I don't have tell pandas to format data or set index
So I want to create a DataFrame like
df = pd.DataFrame(columns=["timestamp","stockname","price","volume"])
but I want to tell Pandas that index of data should be timestamp and that the format would be
"%Y-%m-%d %H:%M:%S:%f"
and one this it set, then I would read through file and pass data to the DataFrame initialized
I get data in variables like these populated every time in loop like
for line in filehandle:
timestamp, stockname, price, volume = fetch(line)
here I want to update the "df"
this update would go on while I would keep using the copy of
df
let us say into a
tempdf
to do re-sampling or any other task at any given point in time because original dataframe
df
is getting updated continuously
import numpy as np
import pandas as pd
import datetime as dt
import time
# create df with timestamp as index
df = pd.DataFrame(columns=["timestamp","stockname","price","volume"], dtype = float)
pd.to_datetime(df['timestamp'], format = "%Y-%m-%d %H:%M:%S:%f")
df.set_index('timestamp', inplace = True)
for i in range(10): # for the purposes of functioning demo code
i += 1 # counter
time.sleep(0.01) # give jupyter notebook a moment
timestamp = dt.datetime.now() # to be used as index
df.loc[timestamp] = ['AAPL', np.random.randint(1000), np.random.randint(10)] # replace with your database read
tempdf = df.copy()
If you are reading a file or database continuously, you can replace the for: loop with what you described in your question. #MattR's questions should also be addressed; if you need to continuously log or update data, I am not sure if pandas is the best solution.
I need to enter data's in a particular column in excel using python.
I am new to python and i am eager to involve with this scripting.
I expect the script that ask for a dialog to enter the data and the given data should import in the excel in the particular column.Can anyone help me?
You shall use openpyxl to manipulate excel.
Example:
from openpyxl import Workbook
wb = Workbook()
ws = wb.active
ws.title = "Sheet Name"
ws['B2'] = "Test Data 1"
# or
d = ws.cell(row=1, column=2, value="Test Data 2")
wb.save('D:\\Prabhakar\\Sample\\balances.xlsx')
Ref: Openpyxl
I have a date string that fails to import because it is in a different format to that expected my the machines locale (i.e. US dates to a UK machine).
How do I tell DAX to convert this string into a date, but using a specified format or locale, different to the machines default.
For example, I would like to import
3/27/2008 11:07:31 AM
as
27/3/2008 11:07:31 AM
You have two options.
First option, use the basic Formatting tab functionality in Power BI.
Select the column and use the below settings in the Formatting tab:
Second option (recommended), use PowerQuery to import the text column in datetime data type.
The following expression will split the text by "/" character, then will convert dd/mm/yyyy string to the datetime data type.
Table.AddColumn(#"Changed Type", "DateTime",
each Text.Split([#"#(001A)Date Import"],"/"){1} & "/"
& Text.Split([#"#(001A)Date Import"],"/"){0} & "/" &
Text.Split([#"# (001A)Date Import"],"/"){2})
In this case I've added an additional column in order to import the column in the required datetime type, you can apply the changes to the same column though.
Date import column is the actual text column, DateTime is the column I've added to import Date Importas Datetime type.
If you get stuck check the official documentation about PowerQuery.
Let me know if this helps.
I think the most practical solution is in the Query Editor, but complex formula are not required.
I would Right-click the column and choose Change Type / Using Locale. Then I would specify Data Type = Date and Locale = English (United States).