I performed data cleaning using pandas and created a csv file. When I loaded the file in power query, I saw there were null values in few columns. I am sure that all null values were removed when performing cleaning task. I even re verified it.
UPDATE:- Instead of csv file, I tried creating an excel file and when I loaded it there were no null values.
Can anybody point out what's the issue with csv file method?
Typically associated with the format of the CSV. Such as single quotes inside text fields also using single quotes to start and stop the text field.
Related
I need to export data of a BigQuery table into csv on Google Cloud Storage.
I used the following:
EXPORT DATA
OPTIONS(
uri=concat(path_file_output,'_*.csv'),
format='CSV',
overwrite=true,
header=true,
field_delimiter=';'
)
AS
SELECT * FROM my_bigquery_table
In my_bigquery_table there are string columns with the character '€' that are badly changed during the export
for example: a field with '1234.56 €' is changed with '1234.56 â'.
Exist a way to avoid this?
on the google documentation :https://cloud.google.com/bigquery/docs/reference/standard-sql/other-statements
there aren't any other options for the export
Microsoft will be always Microsoft... Be reading the comments, the problem comes from Excel, and the default encoding format.
So, let me explain. Your system doesn't use a UTF-8 encoding format. In France, my system uses ISO8859 encoding type, and when you open a file with Excel, it doesn't understand. Same thing if you have a coma separated value (the meaning of CSV) that you import in Excel, it doesn't work in France (we have the habit to use semi-colon separated value).
Anyway. There isn't straight forward solution to open the file with Excel. But you can do it.
Open Excel, and open a blank notebook
Go to Data, Get Data, from text
Select your file and click on "get data"
Then you can configure your import. Select UTF-8 as File Origin
And then continue with other parameters. You can see a sample of your file and the result that you will get.
Note: I have nothing against microsoft, but when it comes to development, Microsoft is a trap nest...
I'm trying to find the best way to upload, parse and work with text file in Oracle APEX (current version 20.1). Bussiness case: I must upload text file, first line will be saved to table A.
Rest lines contains some records (columns are pipe delimited) should be validated. After that correct recordes should be saved to table B or if there is some error it should be saved to table C (error log).
I tried to do something with the Data Loading wizard but it doesn't fit to my requirements.
Right now I added a "File browse..." item to page, and after page submit I can find this file in APEX_APPLICATION_TEMP_FILES in blob_content.
Is there any other option to work with that file than working with blob_content from APEX_APPLICATION_TEMP_FILES. I find it difficoult to work with type of data.
Text file look something like that:
2020-06-05 info: header line
2020-06-05|columnAValue|columnBValue|
2020-06-05|columnAValue||columnCValue
2020-06-05|columnAValue|columnBValue|columnCValue
have a look into the APEX_DATA_PARSER.PARSE table function. It parses the CSV file and returns the values as rows and columns. It's described in more detail within this blog posting:
https://blogs.oracle.com/apex/super-easy-csv-xlsx-json-or-xml-parsing-about-the-apex_data_parser-package
Simply pass "file.csv" (literally) as the p_file_name argument. APEX_DATA_PARSER does not care about the "real" file name....
The function uses the file extension only to differentiate between delimited, XLSX, XML or JSON files. So simply pass in a static file name like "file.csv". That should be enough.
I have a CSV file from which i need to extract two values like 'UutId & Test' and place it in the last column (filename).How can i pull/extract values from same CSV file and place it in the last column of the same CSV file called 'filename' which is blank(without any values right now).
If possible please tell what processor and configs to use?TIA
For reference i am attaching CSV file snapshot of value i want to add in the last column called 'filename' repeated in the whole CSV records.
Values to be extracted
The column in which the values should be placed
List queue after GetFile
The list queue after playing the UpdateRecord processor makes it blank file.
List queue after playing UpdateRecord
please help
Use UpdateRecord processor below configs
in CSVRecordSetWriter controller service add filename column with string type as last field in the avro schema.
UpdateRecord Configs:
Add new property in UpdateRecord processor as
/filename
concat(/UutId,/Test)
As we are using Record Path Value as Replacement Value Strategy
now update record processor will concat UutId,Test values to filename column value.
Refer to this link for more details regarding Update Record processor.
I'm new to this, so I am sure it is a silly question, but I have read through every question related on the site and can't find anything!
I am exporting from pgadmin. A few of the columns have line breaks within the cells, so the exported data is very choppy. Does anyone know how to fix this? Is there a way to make it so the line breaks within cells are not read?
I know I am doing the right settings for exporting, but basically what happens is that the header names are there, along with one row of content for each column and then Column A will have 20 more rows beneath it because of line breaks from the first cell in column E.
Any help would be much appreciated!
I assume that you're referring to the Query --> Execute to file command in the Query window. I don't think it's a bug that pgAdmin doesn't escape line breaks within strings in its csv output, but Excel can read it correctly anyway.
In the export options, please make sure that you use commas as column separators and double quotes as quote chars. Here are my settings:
Additionally, when you load your CSV into Excel, please don't use Data -> From Text. This one doesn't parse CSV with line breaks correctly. Just open the file directly in Excel (via Open within Excel, or by right clicking it in Windows Explorer and choosing Open With -> Microsoft Excel).
How can I read a CSV file, parse the values, and then output it to a particular database table?
That's the basic problem.
Here is a 'bigger picture' of what I'm trying to do:
I'm trying to either read from multiple CSV files (every minute) and or read from an ever-updating CSV file (with additional row entries every update) every minute.