I want to convert the string data type to numpy array of 2-D.
I'm importing a .txt file from a directory which contains:
[[18,1,2018,12,15],
[07,1,2018,12,15],
[03,1,2018,12,15]]
and the code is:
import numpy as np
f = open("/home/pi/timer_database.txt","r")
read = f.read()
x = np.array(list(read))
print(x.size)
print(type(x))
print(x.ndim)
The output is :
47
type <numpy.ndarray>
1
Please help me in this issue.
Use This code
import numpy as np
f = open("/home/pi/timer_database.txt","r")
read = f.read()
read = read.replace("[" , "")
read = read.replace("]" , "")
read = read.replace(",\n" , "\n")
f= open("New_Array.txt","w+")
f.write(read)
f.close()
Array = np.loadtxt("New_Array.txt" , delimiter=',')
print(Array)
You can use ast to evaluate your string, which is much easier than parsing the whole thing:
import ast
x=np.array(ast.literal_eval(read))
Or simply eval:
x=np.array(eval(read))
But this will raise an error because of the leading zeros you have, so first simply remove them:
import re
read=re.sub(r'\b0','',read)
Also if you are writing the file, it is much more advisable to use other approaches, first I would suggest to simply use pickle.
Related
I have two parameters like filename and time and I want to write them in a column in a csv file. These two parameters are in a for-loop so their value is changed in each iteration.
My current python code is the one below but the resulting csv is not what I want:
import csv
import os
with open("txt/scalable_decoding_time.csv", "wb") as csv_file:
writer = csv.writer(csv_file, delimiter=',')
filename = ["one","two", "three"]
time = ["1","2", "3"]
zipped_lists = zip(filename,time)
for row in zipped_lists:
print row
writer.writerow(row)
My csv file must be like below. The , must be the delimeter. So I must get two columns.
one, 1
two, 2
three, 3
My csv file now reads as the following picture. The data are stored in one column.
Do you know how to fix this?
Well, the issue here is, you are using writerows instead of writerow
import csv
import os
with open("scalable_decoding_time.csv", "wb") as csv_file:
writer = csv.writer(csv_file, delimiter=',')
level_counter = 0
max_levels = 3
filename = ["one","two", "three"]
time = ["1","2", "3"]
while level_counter < max_levels:
writer.writerow((filename[level_counter], time[level_counter]))
level_counter = level_counter +1
This gave me the result:
one,1
two,2
three,3
Output:
This is another solution
Put the following code into a python script that we will call sc-123.py
filename = ["one","two", "three"]
time = ["1","2", "3"]
for a,b in zip(filename,time):
print('{}{}{}'.format(a,',',b))
Once the script is ready, run it like that
python2 sc-123.py > scalable_decoding_time.csv
You will have the results formatted the way you want
one,1
two,2
three,3
import csv
reader = csv.reader(post.text, quotechar="'")
with open('source91.csv', 'wb') as f:
writer = csv.writer(f)
writer.writerows(list(reader))
output is showing vertically i need to print the data horizantally in CSV
Simple Answer : if you have only one array
with open('source91.csv', 'wb') as f:
writer = csv.writer(f, delimiter='\n')
writer.writerows(list(reader))
Complicated answer:
you may need numpy to make is happen.
transpose will simply converts row to column
import numpy as np
a = np.array(list(reader))
a = np.append(a, list(reader)) # if you have multiple lines
a = np.transpose(a)
np.savetxt('source91.csv', a)
This question already has answers here:
Python: Fast and efficient way of writing large text file
(3 answers)
Closed 6 years ago.
I am trying to write to append to a list using cPickle in python 2.7 but it does not append.
Code:
import cPickle
import numpy
a = numpy.array([[1, 2],[3, 4]]);
output = open("1.pkl",'wb');
cPickle.dump(a,output);
a = numpy.array([[4, 5],[6, 7]]);
output = open("1.pkl",'ab');
cPickle.dump(a,output);
print(cPickle.load(open("1.pkl",'rb')));
Output:
[[1 2]
[3 4]]
I was using this method to append the arrays in text files before
Code:
a = numpy.array([[1, 2],[3, 4]]);
text_file = open("1.txt", "w");
numpy.savetxt(text_file, a);
text_file.close();
a = numpy.array([[4, 5],[6, 7]]);
text_file = open("1.txt", "a");
numpy.savetxt(text_file, a);
text_file.close();
text_file = open("1.txt", "r");
print(text_file.read());
Output:
1.000000000000000000e+00 2.000000000000000000e+00
3.000000000000000000e+00 4.000000000000000000e+00
4.000000000000000000e+00 5.000000000000000000e+00
6.000000000000000000e+00 7.000000000000000000e+00
I Was using this to write the data of a python simulation I setup for Power Systems. The output data is huge around 7GB. And the writing process was slowing down the simulation a lot. I read that cPickle can make writing process faster.
How do I append to the cPickle output file without having to read the whole data?
Or is there a better alternative to cPickle to make writing faster?
I don't believe you can just append to a pickle, or in a way that makes sense anyway.
If you just get the current serialized version of an object and add another serialized object at the end of the file, it wouldn't just magically append the second object to the original list.
You would need to read in the original object, append to it in Python, and then dump it back.
import cPickle as pickle
import numpy as np
filename = '1.pkl'
a = np.array([[1, 2],[3, 4]])
b = np.array([[4, 5],[6, 7]])
# dump `a`
with open(filename,'wb') as output_file:
pickle.dump(a, output, -1)
# load `a` and append `b` to it
with open(filename, 'rb') as output_file:
old_data = pickle.load(output_file)
new_data = np.vstack([old_data,a])
# dump `new_data`
with open(filename, 'wb') as output_file:
pickle.dump(new_data, output_file, -1)
# test
with open(filename, 'rb') as output_file:
print(pickle.load(output_file))
After reading your question a second time, you state that you don't want to read in the whole data again. I suppose this doesn't answer your question then, does it?
I'm trying to get the temperature from a json source. It's nested and I can't figure out how to get a nested value from a json file or url
So here comes my code so far:
#! /usr/bin/python
import urllib2
import json
f = urllib2.urlopen('http://api.openweathermap.org/data/2.5/find?q=London&units=metric')
json_string = f.read()
parsed_json = json.loads(json_string)
temp = parsed_json['list']
print "Current temperature is: %s" % (temp)
f.close()
Right now I can get all values at once but not just a particular value (temp in my case)
I prefer to get the value clean without u'temp': if possible.
u'temp' is how Python represents unicode objects which is what JSON strings get parsed into in Python. Is this what you're looking for?
print temp[0]['main']['temp']
I don't know the structure of the API you're calling, so this may be making quite a few assumptions, but it will get you the raw temperature.
You are getting several values back. To list them all do:
import urllib2
import json
f = urllib2.urlopen('http://api.openweathermap.org/data/2.5/find?q=London&units=metric')
json_string = f.read()
parsed_json = json.loads(json_string)
for each in parsed_json['list']:
country = each['sys']['country']
temperature = each['main']['temp']
print "Current Temperature in {} is {}".format(country, temperature)
Output
Current Temperature in CA is 11.73
Current Temperature in GB is 11.8
Using the python 2.7 shell on osx lion. The .csv file has 12 columns by 892 rows.
import csv as csv
import numpy as np
# Open up csv file into a Python object
csv_file_object = csv.reader(open('/Users/scdavis6/Documents/Kaggle/train.csv', 'rb'))
header = csv_file_object.next()
data=[]
for row in csv_file_object:
data.append(row)
data = np.array(data)
# Convert to float for numerical calculations
number_passengers = np.size(data[0::,0].astype(np.float))
And this is the error I get:
Traceback (most recent call last):
File "pyshell#5>", line 1, in <module>
number_passengers = np.size(data[0::,0].astype(np.float))
TypeError: list indices must be integers, not tuple
What am I doing wrong.
Don't use csv to read the data into a NumPy array. Use numpy.genfromtxt; using dtype=None will cause genfromtxt to make an intelligent guess at the dtypes for you. By doing it this way you won't have to manually convert strings to floats.
data[0::, 0] just gives you the first column of data.
data[:, 0] would give you the same result.
The error message
TypeError: list indices must be integers, not tuple
suggests that for some reason your data variable might be holding a list rather than a ndarray. For example, the same Exception can produced like this:
In [73]: data = [1,2,3]
In [74]: data[1,2]
TypeError: list indices must be integers, not tuple
I don't know why that is happening, but if you post a sample of your CSV we should be able to help fix that.
Using np.genfromtxt, your current code could be simplified to:
import numpy as np
filename = '/Users/scdavis6/Documents/Kaggle/train.csv'
data = np.genfromtxt(filename, delimiter=',', skiprows=1, dtype=None)
number_passengers = np.size(data, axis=0)