NMEA data parsing using python 2.7 - python-2.7

i want readings of present GPS location, when i run the below code in raspberry pi, program prints 10-12 outputs and then it displays the error as below:
Traceback (most recent call last):
File "simplegpsparsing.py", line 24, in
get_present_gps()
File "simplegpsparsing.py", line 16, in get_present_gps
lat, _, lon= line.split(',')[2:5]
ValueError: need more than 0 values to unpack
i want present value of GPS( buffers should be updated with immediate GPS) so that present GPS values can be known.
my code goes as below :
import os
import serial
def get_present_gps:
ser= serial.Serial('/dev/ttyAMA0',9600)
ser.open()
while True :
f=open('/home/pi/Desktop/gps1','w')
data=ser.read(ser.inWaiting()) # read no.of bytes in waiting
f.write(data) #write data into file
f.flush() # flush(method) from buffer into os buffer
os.fsync(f.fileno()) #ensure to write from os buffer(internal buffers)into disk
f = open('/home/pi/Desktop/gps1','r') # fetch the required file
for line in f.read().split('\n') :
if line.startswith( '$GPGGA' ) :
lat, _, lon = line.strip().split(',')[2:5]
try :
lat = float( lat )
lon = float( lon )
print lat
print lon
except :
pass
# something wrong happens with your data, print some error messages
get_present_gps()
if the serial port is left open without closing, will it create any problem? will the above code meet my requirement i.e getting the instantaneous value?

Here is a thought: parse the data string into chunks that are newline terminated; leave "unfinished" strings in the buffer until the next time around the loop. Untested, this will look something like this:
keepThis = ''
while True :
data=keepThis + ser.read(ser.inWaiting()) # read no.of bytes in waiting
m = data.split("\n") # get the individual lines from input
if len(m[-1]==0): # true if only complete lines present (nothing after last newline)
processThis = m
keepThis = ''
else:
processThis = m[0:-1] # skip incomplete line
keepThis = m[-1] # fragment
# process the complete lines:
for line in processThis:
if line.startswith( '$GPGGA' ) :
try :
lat, _, lon = line.strip().split(',')[2:5] # move inside `try` just in case...
lat = float( lat )
lon = float( lon )
print lat
print lon
except :
pass

Related

pyserial data gaps

I have python code that acquires serial data from 2 devices and writes to a .txt file. Every 4-15 minutes there is approx 30-45 seconds of data missing in the .txt file and this is not acceptable for our use case. I've spent hours googling and searching SO about multiprocessing and serial port data acquisition and haven't come up with a solution.
Here is my code
gpsser = input(("Enter GPS comport as 'COM_': "))
ser = serial.Serial(port=gpsser,
baudrate=38400,
timeout=2,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
bytesize=serial.EIGHTBITS)
root = Tk()
root.title("DualEM DAQ")
path = filedialog.asksaveasfilename() + ".txt"
file = glob.glob(path)
filename = path
with open(filename, 'wb') as f:
w = csv.writer(f, dialect='excel')
w.writerow(['header'])
def sensor():
while True:
try:
NMEA1 = dser.readline().decode("ascii")
while dser.inWaiting() == 0:
pass
NMEA1_array = NMEA1.split(',')
NMEA2_array = NMEA2.split(',')
NMEA3_array = NMEA3.split(',')
NMEA4_array = NMEA4.split(',')
if NMEA1_array[0] == '$PDLGH':
value1 = NMEA1_array[2]
value2 = NMEA1_array[4]
if NMEA1_array[0] == '$PDLG1':
value3 = NMEA1_array[2]
value4 = NMEA1_array[4]
if NMEA1_array[0] == '$PDLG2':
value5 = NMEA1_array[2]
value6 = NMEA1_array[4]
return (float(value1), float(value2), float(value3),
float(value4), float(value5), float(value6),
except (IndexError, NameError, ValueError, UnicodeDecodeError):
pass
def gps():
while True:
try:
global Status, Latitude, Longitude, Speed, Truecourse, Date
global GPSQuality, Satellites, HDOP, Elevation, Time
while ser.inWaiting() == 0:
pass
msg = ser.readline()
pNMEA = pynmea2.parse(msg)
if isinstance(pNMEA, pynmea2.types.talker.RMC):
Latitude = pynmea2.dm_to_sd(pNMEA.lat)
Longitude = -(pynmea2.dm_to_sd(pNMEA.lon))
Date = pNMEA.datestamp
Time = datetime.datetime.now().time()
if () is not None:
return (Longitude, Latitude, Date, Time)
except (ValueError, UnboundLocalError, NameError):
pass
while True:
try:
with open(filename, "ab") as f:
data = [(gps() + sensor())]
writer = csv.writer(f, delimiter=",", dialect='excel')
writer.writerows(data)
f.flush()
print(data)
except (AttributeError, TypeError) as e:
pass
The program is writing to the file but I need help understanding why I'm losing 30-45 seconds of data every so often. Where is my bottle neck that is causing this to happen?
Here is an example of where the breaks are, note the breaks are approx 50 seconds in this case.
Breaks in writing data to csv
DB
Back when I used PySerial, I did this:
nbytes = ser.inWaiting()
if nbytes > 0:
indata = ser.read(nbytes)
#now parse bytes in indata to look for delimiter, \n in your case
#and if found process the input line(s) until delimiter not found
else:
#no input yet, do other processing or allow other things to run
#by using time.sleep()
Also note that new versions (3.0+) of PySerial have .in_waiting as a property not a method, so no (), it used to be .inWaiting().
You should not flush the serial port input. Data is arriving on its own timing into a buffer in the driver, not when your read happens, so you are throwing away data with your flush. You may need to add code to synchronize with the input stream.
I used threading with a queue and changed my mainloop to look like this.
while True:
try:
with open(filename, "ab") as f:
writer = csv.writer(f, delimiter=",", dialect='excel')
data = []
data.extend(gpsdata())
data.extend(dualemdata())
writer.writerows([data])
f.flush()
f.close()
dser.flushInput()
ser.flushInput()
print(data)
sleep(0.05)
except (AttributeError, TypeError) as e:
pass
I had to flush the serial port input data before the looping back to the read functions so it was reading new realtime data(this eliminated any lag of the incoming data stream). I've ran a 30 minute test and the time gaps appear to have go away. Thankyou to Cmaster for giving me some diagnostic ideas.

PyAudio recorder script IOError: [Errno Input overflowed] -9981

The code below is what I use to record audio until the "Enter" key is pressed it returns an exception,
import pyaudio
import wave
import curses
from time import gmtime, strftime
import sys, select, os
# Name of sub-directory where WAVE files are placed
current_experiment_path = os.path.dirname(os.path.realpath(__file__))
subdir_recording = '/recording/'
# print current_experiment_path + subdir_recording
# Variables for Pyaudio
chunk = 1024
format = pyaudio.paInt16
channels = 2
rate = 48000
# Set variable for the labelling of the recorded WAVE file.
timestamp = strftime("%Y-%m-%d-%H:%M:%S", gmtime())
#wave_output_filename = '%s.wav' % self.get('subject_nr')
wave_output_filename = '%s.wav' % timestamp
print current_experiment_path + subdir_recording + wave_output_filename
# pyaudio recording stuff
p = pyaudio.PyAudio()
stream = p.open(format = format,
channels = channels,
rate = rate,
input = True,
frames_per_buffer = chunk)
print "* recording"
# Create an empty list for audio recording
frames = []
# Record audio until Enter is pressed
i = 0
while True:
os.system('cls' if os.name == 'nt' else 'clear')
print "Recording Audio. Press Enter to stop recording and save file in " + wave_output_filename
print i
if sys.stdin in select.select([sys.stdin], [], [], 0)[0]:
line = raw_input()
# Record data audio data
data = stream.read(chunk)
# Add the data to a buffer (a list of chunks)
frames.append(data)
break
i += 1
print("* done recording")
# Close the audio recording stream
stream.stop_stream()
stream.close()
p.terminate()
# write data to WAVE file
wf = wave.open(current_experiment_path + subdir_recording + wave_output_filename, 'wb')
wf.setnchannels(channels)
wf.setsampwidth(p.get_sample_size(format))
wf.setframerate(rate)
wf.writeframes(''.join(frames))
wf.close()
The exception produced is this
Recording Audio. Press Enter to stop recording and save file in 2015-11-20-22:15:38.wav
925
Traceback (most recent call last):
File "audio-record-timestamp.py", line 51, in <module>
data = stream.read(chunk)
File "/Library/Python/2.7/site-packages/pyaudio.py", line 605, in read
return pa.read_stream(self._stream, num_frames)
IOError: [Errno Input overflowed] -9981
What is producing the exception? I tried changing the chunk size (512,256,8192) it doesn't work. Changed the while loop condition and it didn't work.
I had a similar problem; there are 3 ways to solve it (that I could find)
set rate=24000
add option "exception_on_overflow=False" to the "read()" call, that is, make it "stream.read(chunk, exception_on_overflow=False)"
use callbacks
Here is, for your convenience, an example o "using callbacks"
#!/usr/bin/python
import sys, os, math, time, pyaudio
try:
import numpy
except:
numpy = None
rate=48000
chan=2
sofar=0
p = pyaudio.PyAudio()
def callback(in_data, frame_count, time_info, status):
global sofar
sofar += len(in_data)
if numpy:
f = numpy.fromstring(in_data, dtype=numpy.int16)
sys.stderr.write('length %6d sofar %6d std %4.1f \r' % \
(len(in_data), sofar, numpy.std(f)))
else:
sys.stderr.write('length %6d sofar %6d \r' % \
(len(in_data), sofar))
data = None
return (data, pyaudio.paContinue)
stream = p.open(format=pyaudio.paInt16, channels=chan, rate=rate,
input=True, stream_callback=callback)
while True:
time.sleep(1)

Parsing txt file using python : List index out of range

Hello I have written a python program to parse data specific data from txt file
my code is:
f = open('C:/Users/aikaterini/Desktop/Ericsson_PARSER/BSC_alarms_vf_OSS.txt','r')
from datetime import datetime
import MySQLdb
def firstl():
with f as lines:
lines = lines.readlines()
print len(lines)
for i,line in enumerate(lines):
if line.startswith("*** Disconnected from"):
conline = line.split()
bsc = conline[-2]
print "\n"*5
print bsc
print "*"*70
break
for i,line in enumerate(lines):
if line.startswith("*** Connected to"):
conline = line.split()
bsc = conline[-2]
print "\n"*5
print bsc
print "*"*70
elif line[:3] == "A1/" or line[:3] == "A2/":
if lines[i+1].startswith("RADIO"):
fal = line.split()
first_alarm_line = [fal[0][:2],fal[-2],fal[-1]]
year = first_alarm_line[1][:2]
month = first_alarm_line[1][2:4]
day = first_alarm_line[1][4:]
hours = first_alarm_line[2][:2]
minutes = first_alarm_line[2][2:]
date = datetime.strptime( day + " " + month + " " + year + " " + \
hours+":"+minutes,"%d %m %y %H:%M")
print first_alarm_line
print date, "\n"
print lines[i+1]
print lines[i+4]
print lines[i+5]
desc_line = lines[i+4]
desc_values_line = lines[i+5]
desc = desc_line.split(None,2)
print desc
desc_values = desc_values_line.split(None,2)
rsite = ""
#for x in desc_values[1]:
# if not (x.isalpha() or x == "0"):
# rsite += x
rsite = desc_values[1].lstrip('RBS0')
print "\t"*2 + "rsite:" + rsite
if desc[-1] == "ALARM SLOGAN\n":
alarm_slogan = desc_values[-1]
print alarm_slogan
x = i
print x # to check the line
print len(line) #check length of lines
while not lines[x].startswith("EXTERNAL"):
x+=1
if lines[x].startswith("EXTERNAL"):
while not lines[x] == "\n":
print lines[x]
x+=1
print "\n"*5
elif lines[i+1].startswith("CELL LOGICAL"):
fal = line.split()
first_alarm_line = [fal[0][:2],fal[-2],fal[-1]]
#print i
print first_alarm_line
type = lines[i+1]
print type
cell_line = lines[i+3]
cell = cell_line.split()[0]
print cell
print "\n"*5
##########Database query###########
#db = MySQLdb.connect(host,user,password,database)
firstl()
when i run the program the results are correct
but it prints until line 50672 while there are 51027
and i get the last printed result with the following error:
['A2', '130919', '0309']
2013-09-19 03:09:00
RADIO X-CEIVER ADMINISTRATION
MO RSITE ALARM SLOGAN
RXOCF-18 RBS03668 OML FAULT
['MO', 'RSITE', 'ALARM SLOGAN\n']
rsite:3668
OML FAULT
50672
51027
Traceback (most recent call last):
File "C:\Python27\parser_v3.py", line 106, in <module>
firstl()
File "C:\Python27\parser_v3.py", line 72, in firstl
while not lines[x].startswith("EXTERNAL"):
IndexError: list index out of range
if i comment the while not line i get :
Traceback (most recent call last):
File "C:\Python27\parser_v3.py", line 106, in <module>
firstl()
File "C:\Python27\parser_v3.py", line 60, in firstl
rsite = desc_values[1].lstrip('RBS0')
IndexError: list index out of range
The txt content is like :
A1/EXT "FS G11B/25/13/3" 382 150308 1431
RADIO X-CEIVER ADMINISTRATION
BTS EXTERNAL FAULT
MO RSITE CLASS
RXOCF-16 RBS02190 1
EXTERNAL ALARM
ALARM SYSTEM ON/OFF G2190 DRAMA CNR
A1/EXT "FS G11B/25/13/3" 755 150312 1434
RADIO X-CEIVER ADMINISTRATION
BTS EXTERNAL FAULT
MO RSITE CLASS
RXOCF-113 RBS00674 1
EXTERNAL ALARM
IS.BOAR FAIL G0674 FALAKRO
I don't understand since i do a split with maxnumber 2 and i get 3 elements as u can see and i am picking the 2nd and if i comment that i get another error when i pick an element from a list and the thing is that returning the correct result.Please help me.
Sorry for the long post thank you in advance.
I'm haven't dug deep into your code, but have you tried validating that x does not exceed the number of elements in lines before trying to access that index? Also, for readability I'd suggest using lines[x] != rather than not lines[x] ==
while x < len(lines) and lines[x] != "\n":
I solved it although i don't know if it is correct way but it works.
I think the problem was that the x was exceeding the length of the list lines containing the file and there had to be a check after the split that the list had length larger or equal to number of elements so :
if len(desc_values) > 2 and len(desc) > 2:
rsite = desc_values[1].lstrip('RBS0')
print "\t"*2 + "rsite:" + rsite
if desc[-1] == "ALARM SLOGAN\n":
alarm_slogan = desc_values[-1]
print alarm_slogan
x = i
print x #to check the line
print len(lines) # check length of lines
while [x] < len(lines): #check so that x doesnt exceed the length of file list "line"
while not lines[x].startswith("EXTERNAL"):
x+=1
if lines[x].startswith("EXTERNAL"):
while lines[x] != "\n":
print lines[x]
x+=1
Thank you man you really helped me although i am trying to find a way to stop the iteration of x to gain some computation time i tried break but it throws you completely of the loop.
Thanks anyway

Problems with global variables ((python))

I have a problem with my global variables in my code. In SCRIPT1.py I use many variables from a little document config.py which only contains variables which I also need in other modules of my code. But when running my SCRIPT1.py I get an error (ERROR). I have no idea why it doesn't work with config.(name of variable)... I found this solution to have your variables in all of your modules on stack overflow with a lot of good votes. What am I doing wrong?
First my code contained config.costSurfaceA in stead of costSurfaceArray (for ex in 'def createPath') but when running it with this variable, it gave me a syntax error because of the dot in 'config.costSurfaceA'. I replaced it all by 'costSurfaceArray' and did this in the if statement 'config.costSurfaceA = costSurfaceArray' just to get it as a variable. But I have the feeling this is all to much work for nothing..
Thanks in avance for helping me! I know it is a lot of code but I think it's all important for understanding..
SCRIPT1.py
from osgeo import gdal, osr
from skimage.graph import route_through_array
import numpy as np
import Save_Array_To_Excel_01
import config
def ask_costsurfacepath_path():
config.costsurfacepath = input('please enter the system path where to find the cost-surface-IMG file (ex: /Users/PeterVanvoorden/Documents/GroepT/Thesis/Branched_Testfile.img): ')
def ask_outputpath_path():
config.outputpath = input('please enter the system path where to save the outputpath IMG file (ex: /Users/PeterVanvoorden/Documents/GroepT/Thesis/Branched_Testfile.img): ')
def raster2array(rasterfn):
print 'raster2array'
raster = gdal.Open(rasterfn)
band = raster.GetRasterBand(1)
array = band.ReadAsArray()
return array
def coord2pixelOffset(rasterfn,x,y):
print 'coord2pixelOffset'
raster = gdal.Open(rasterfn)
geotransform = raster.GetGeoTransform()
originX = geotransform[0] # East/West location of Upper Left corner
originY = geotransform[3] # North/South location of Upper Left corner
pixelWidth = geotransform[1] # X pixel size
pixelHeight = geotransform[5] # Y pixel size
xOffset = int((x - originX)/pixelWidth)
yOffset = int((y - originY)/pixelHeight)
return xOffset,yOffset
def createPath(CostSurfacefn,costSurfaceArray,startCoord,stopCoord):
print 'creatpath'
# coordinates to array index
startCoordX = startCoord[0]
startCoordY = startCoord[1]
startIndexX,startIndexY = coord2pixelOffset(CostSurfacefn,startCoordX,startCoordY)
stopCoordX = stopCoord[0]
stopCoordY = stopCoord[1]
stopIndexX,stopIndexY = coord2pixelOffset(CostSurfacefn,stopCoordX,stopCoordY)
# create path
indices, weight = route_through_array(costSurfaceArray, (startIndexY,startIndexX), (stopIndexY,stopIndexX),geometric=True,fully_connected=True)
indices = np.array(indices).T
path = np.zeros_like(costSurfaceArray)
path[indices[0], indices[1]] = 1
return path
def array2raster(newRasterfn,rasterfn,array):
print 'array2raster'
raster = gdal.Open(rasterfn)
geotransform = raster.GetGeoTransform()
originX = geotransform[0] # East/West location of Upper Left corner
originY = geotransform[3] # North/South location of Upper Left corner
pixelWidth = geotransform[1] # X pixel size
pixelHeight = geotransform[5] # Y pixel size
cols = array.shape[1]
rows = array.shape[0]
driver = gdal.GetDriverByName('GTiff')
outRaster = driver.Create(newRasterfn, cols, rows, gdal.GDT_Byte)
outRaster.SetGeoTransform((originX, pixelWidth, 0, originY, 0, pixelHeight))
outband = outRaster.GetRasterBand(1)
outband.WriteArray(array)
outRasterSRS = osr.SpatialReference()
outRasterSRS.ImportFromWkt(raster.GetProjectionRef())
outRaster.SetProjection(outRasterSRS.ExportToWkt())
outband.FlushCache()
def main(CostSurfacefn,outputPathfn,startCoord,stopCoord):
print 'main'
costSurfaceArray = raster2array(CostSurfacefn) # creates array from cost surface raster
config.costSurfaceA = costSurfaceArray
config.pathArray = createPath(CostSurfacefn,costSurfaceArray,startCoord,stopCoord) # creates path array
Save_Array_To_Excel_01.Save_Array(config.pathArray) # Save Array to csv file
array2raster(outputPathfn,CostSurfacefn,config.pathArray) # converts path array to raster
if __name__ == "__main__":
ask_costsurfacepath_path()
ask_outputpath_path()
CostSurfacefn = config.costsurfacepath
print config.costsurfacepath
startCoord = (config.startX,config.startY)
stopCoord = (config.stopX,config.stopY)
outputPathfn = config.outputpath
main(CostSurfacefn,outputPathfn,startCoord,stopCoord)
config.py
# Configuration file with all global variables
# number of properties
number = None
# different permutations of properties
permutations = list()
# properties array containing:
# * first column = ID first property [0]
# * second column = ID second property [1]
# * third column = distance between two properties [2]
# * forth column = estimated cost [3]
properties_array = None
# lowest price until now
lowest_price = 10**10000
# path with this lowest price
lowest_path = None
# current price (needs to be compared with lowest price)
current_price = 0
# current path (needs to be compared with lowest path)
current_path = [1]
# path to place where to save properties list
plist_path = None
# Array of the path
pathArray = None
# Array of the map
costSurfaceA = None
# current start X coordinate
startX = 0
# current start Y coordinate
startY = 0
# current stop X coordinate
stopX = 0
# current stop Y coordinate
stopY = 0
# path to costsurface IMG file
costsurfacepath = 0
# path to output path from Least cost path analysis
outputpath = 0
ERROR
please enter the system path where to put the file as a STRING (ex: /Users/PeterVanvoorden/Documents/GroepT/Thesis/Branched_Testfile.csv): '/User/PeterVanvoorden/Desktop/Shell.csv'
You entered: /User/PeterVanvoorden/Desktop/Shell.csv
please enter the system path where to find the cost-surface-IMG file (ex: /Users/PeterVanvoorden/Documents/GroepT/Thesis/Branched_Testfile.img): '/User/PeterVanvoorden/Desktop/clipsmall.img'
please enter the system path where to save the outputpath IMG file (ex: /Users/PeterVanvoorden/Documents/GroepT/Thesis/Branched_Testfile.img): '/User/PeterVanvoorden/Desktop/Shellimg.img'
/User/PeterVanvoorden/Desktop/clipsmall.img
main
raster2array
Traceback (most recent call last):
File "/Users/PeterVanvoorden/Documents/GroepT/Thesis/f_python_standalone/python_files/Working_Files/Least_cost_path_analysis_01_outputArray.py", line 97, in <module>
main(CostSurfacefn,outputPathfn,startCoord,stopCoord)
File "/Users/PeterVanvoorden/Documents/GroepT/Thesis/f_python_standalone/python_files/Working_Files/Least_cost_path_analysis_01_outputArray.py", line 76, in main
costSurfaceArray = raster2array(CostSurfacefn) # creates array from cost surface raster
File "/Users/PeterVanvoorden/Documents/GroepT/Thesis/f_python_standalone/python_files/Working_Files/Least_cost_path_analysis_01_outputArray.py", line 17, in raster2array
band = raster.GetRasterBand(1)
AttributeError: 'NoneType' object has no attribute 'GetRasterBand'
You can pass around like this.
def ask_costsurfacepath_path():
costsurfacepath = input('please enter ...')
return costsurfacepath
... in __name__ == '__main__'
CostSurfacefn = ask_costsurfacepath_path()
...

IndexError, but more likely I/O error

Unsure of why I am getting this error. I'm reading from a file called columns_unsorted.txt, then trying to write to columns_unsorted.txt. There error is on fan_on = string_j[1], saying list index out of range. Here's my code:
#!/usr/bin/python
import fileinput
import collections
# open document to record results into
j = open('./columns_unsorted.txt', 'r')
# note this is a file of rows of space-delimited date in the format <1384055277275353 0 0 0 1 0 0 0 0 22:47:57> on each row, the first term being unix times, the last human time, the middle binary indicating which machine event happened
# open document to read from
l = open('./columns_sorted.txt', 'w')
# CREATE ARRAY CALLED EVENTS
events = collections.deque()
i = 1
# FILL ARRAY WITH "FACTS" ROWS; SPLIT INTO FIELDS, CHANGE TYPES AS APPROPRIATE
for line in j: # columns_unsorted
line = line.rstrip('\n')
string_j = line.split(' ')
time = str(string_j[0])
fan_on = int(string_j[1])
fan_off = int(string_j[2])
heater_on = int(string_j[3])
heater_off = int(string_j[4])
space_on = int(string_j[5])
space_off = int(string_j[6])
pump_on = int(string_j[7])
pump_off = int(string_j[8])
event_time = str(string_j[9])
row = time, fan_on, fan_off, heater_on, heater_off, space_on, space_off, pump_on, pump_off, event_time
events.append(row)
You are missing the readlines function, no?
You have to do:
j = open('./columns_unsorted.txt', 'r')
l = j.readlines()
for line in l:
# what you want to do with each line
In the future, you should print some of your variables, just to be sure the code is working as you want it to, and to help you identifying problems.
(for example, if in your code you would print string_j you would see what kind of problem you have)
Problem was an inconsistent line in the data file. Forgive my haste in posting