I have a DataStream <pyflink.datastream.data_stream.DataStream> coming from a CoFlatMapFunction (simplified here):
%flink.pyflink
# join two streams and update the rule-set
class MyCoFlatMapFunction(CoFlatMapFunction):
def open(self, runtime_context: RuntimeContext):
state_desc = MapStateDescriptor('map', Types.STRING(), Types.BOOLEAN())
self.state = runtime_context.get_map_state(state_desc)
def bool_from_user_number(self, user_number: int):
'''Retunrs True if user_number is greater than 0, False otherwise.'''
if user_number > 0:
return True
else:
return False
def flat_map1(self, value):
'''This method is called for each element in the first of the connected streams'''
self.state.put(value[1], self.bool_from_user_number(value[2]))
def flat_map2(self, value):
'''This method is called for each element in the second of the connected streams (exchange_server_tickers_data_py)'''
current_dateTime = datetime.now()
dt = current_dateTime
x = value[1]
y = value[2]
yield Row(dt, x, y)
def generate__ds(st_env):
# interpret the updating Tables as DataStreams
type_info1 = Types.ROW([Types.SQL_TIMESTAMP(), Types.STRING(), Types.INT()])
ds1 = st_env.to_append_stream(table_1 , type_info=type_info1)
type_info2 = Types.ROW([Types.SQL_TIMESTAMP(), Types.STRING(), Types.STRING()])
ds2 = st_env.to_append_stream(table_2 , type_info=type_info2)
output_type_info = Types.ROW([ Types.PICKLED_BYTE_ARRAY() ,Types.STRING(),Types.STRING() ])
# Connect the two streams
connected_ds = ds1.connect(ds2)
# Apply the CoFlatMapFunction
ds = connected_ds.key_by(lambda a: a[0], lambda a: a[0]).flat_map(MyCoFlatMapFunction(), output_type_info)
return ds
ds = generate__ds(st_env)
The output, however, I am unable to view, either via registering it as a view / table, writing to a sink table or (the best case) using a Kinesis Streams sink to write data from the Flink stream into a Kinesis stream. Firehouse would also not fit my use case as the 30 second latency would be too long. Any help would be appreciated, thanks!
What I have tried:
Registering it as a view / table like so:
# interpret the DataStream as a Table
input_table = st_env.from_data_stream(ds).alias("dt", "x", "y")
z.show(input_table, stream_type="update")
Which gives an error of:
Query schema: [dt: RAW('[B', '...'), x: STRING, y: STRING]
Sink schema: [dt: RAW('[B', ?), x: STRING, y: STRING]
I have also tried writing to a sink table, like so:
%flink.pyflink
# create a sink table to emit results
st_env.execute_sql("""DROP TABLE IF EXISTS table_sink""")
st_env.execute_sql("""
CREATE TABLE table_sink (
dt RAW('[B', '...'),
x VARCHAR(32),
y STRING
) WITH (
'connector' = 'print'
)
""")
# convert the Table API table to a SQL view
table = st_env.from_data_stream(ds).alias("dt", "spread", "spread_orderbook")
st_env.execute_sql("""DROP TEMPORARY VIEW IF EXISTS table_api_table""")
st_env.create_temporary_view('table_api_table', table)
# emit the Table API table
st_env.execute_sql("INSERT INTO table_sink SELECT * FROM table_api_table").wait()
I get the error:
org.apache.flink.table.api.ValidationException: Unable to restore the RAW type of class '[B' with serializer snapshot '...'.
I have also tried to use a sink and add_sink to write the data to a sink, which would be an AWS kinesis data stream like in these Docs, like so:
%flink.pyflink
from pyflink.common.serialization import JsonRowSerializationSchema
from pyflink.datastream.connectors import KinesisStreamsSink
output_type_info = Types.ROW([Types.SQL_TIMESTAMP(), Types.STRING(), Types.STRING()])
serialization_schema = JsonRowSerializationSchema.Builder().with_type_info(output_type_info).build()
# Required
sink_properties = {
'aws.region': 'eu-west-2'
}
kds_sink = KinesisStreamsSink.builder()
.set_kinesis_client_properties(sink_properties)
.set_serialization_schema(SimpleStringSchema())
.set_partition_key_generator(PartitionKeyGenerator
.fixed())
.set_stream_name("test_stream")
.set_fail_on_error(False)
.set_max_batch_size(500)
.set_max_in_flight_requests(50)
.set_max_buffered_requests(10000)
.set_max_batch_size_in_bytes(5 * 1024 * 1024)
.set_max_time_in_buffer_ms(5000)
.set_max_record_size_in_bytes(1 * 1024 * 1024)
.build()
ds.sink_to(kds_sink)
Which i assume would work, but KinesisStreamsSink is not found in pyflink.datastream.connectors and I am unable to find any documentation on how to do this within AWS Kinesis Analytics Studio. Any help would be much much appreciated, thank you! How would I go about writing the data to a Kinesis Streams sink / converting it to a table?
Okay, i have figured it out. There were a couple issues with the particular Pyflink version available on AWS Kinesis Analytics Studio (1.13). The error messages themselves were not that useful, so for anyone who is having issues themselves I would really recommend viewing the errors in the Flink Web UI. Firstly, the MapStateDescriptor datatypes must be specified using Types.PICKLED_BYTE_ARRAY(). Secondly, not shown in the Qn, but each MapStateDescriptor must have a distinct name. I also found that using Row from pyflink.common threw errors for me. It worked better for me to switch to using use Tuples by specifying Types.TUPLE() as is done in this example. I also had to switch to specifying the output as a tuple.
Another thing I have not done is specify a watermark strategy for the DataStream, which could potentially be done by extracting the timestamp from the first field, and assign watermarks based on knowledge of the stream:
class MyTimestampAssigner(TimestampAssigner):
def extract_timestamp(self, value, record_timestamp: int) -> int:
return int(value[0])
watermark_strategy = WatermarkStrategy.for_bounded_out_of_orderness(Duration.of_seconds(5)).with_timestamp_assigner(MyTimestampAssigner())
ds = ds.assign_timestamps_and_watermarks(watermark_strategy)
# the first field has been used for timestamp extraction, and is no longer necessary
# replace first field with a logical event time attribute
table = st_env.from_data_stream(ds, col("dt").rowtime, col('f0'), col('f1'))
But i have instead created a sink table for writing to a Kinesis Data Stream again as an output. In total, the corrected code would look something like this:
from pyflink.table.expressions import col
from pyflink.datastream.state import MapStateDescriptor
from pyflink.datastream.functions import RuntimeContext, CoFlatMapFunction
from pyflink.common.typeinfo import Types
from pyflink.common import Duration as Time, WatermarkStrategy, Duration
from pyflink.common.typeinfo import Types
from pyflink.common.watermark_strategy import TimestampAssigner
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.datastream.functions import KeyedProcessFunction, RuntimeContext
from pyflink.datastream.state import ValueStateDescriptor
from datetime import datetime
# Register the tables in the env
table1 = st_env.from_path("sql_table_1")
table2 = st_env.from_path("sql_table_2")
# interpret the updating Tables as DataStreams
type_info1 = Types.TUPLE([Types.SQL_TIMESTAMP(), Types.STRING(), Types.INT()])
ds1 = st_env.to_append_stream(table2, type_info=type_info1)
type_info2 = Types.TUPLE([Types.SQL_TIMESTAMP(), Types.STRING(), Types.STRING()])
ds2 = st_env.to_append_stream(table1, type_info=type_info2)
# join two streams and update the rule-set state
class MyCoFlatMapFunction(CoFlatMapFunction):
def open(self, runtime_context: RuntimeContext):
'''This method is called when the function is opened in the runtime. It is the initialization purposes.'''
# Map state that we use to maintain the filtering and rules
state_desc = MapStateDescriptor('map', Types.PICKLED_BYTE_ARRAY(), Types.PICKLED_BYTE_ARRAY())
self.state = runtime_context.get_map_state(state_desc)
# maintain state 2
ob_state_desc = MapStateDescriptor('map_OB', Types.PICKLED_BYTE_ARRAY(), Types.PICKLED_BYTE_ARRAY())
self.ob_state = runtime_context.get_map_state(ob_state_desc)
# called on ds1
def flat_map1(self, value):
'''This method is called for each element in the first of the connected streams '''
list_res = value[1].split('|')
for i in list_res:
time = datetime.utcnow().replace(microsecond=0)
yield (time, f"{i}_one")
# called on ds2
def flat_map2(self, value):
'''This method is called for each element in the second of the connected streams'''
list_res = value[1].split('|')
for i in list_res:
time = datetime.utcnow().replace(microsecond=0)
yield (time, f"{i}_two")
connectedStreams = ds1.connect(ds2)
output_type_info = Types.TUPLE([Types.SQL_TIMESTAMP(), Types.STRING()])
ds = connectedStreams.key_by(lambda value: value[1], lambda value: value[1]).flat_map(MyCoFlatMapFunction(), output_type=output_type_info)
name = 'output_table'
ds_table_name = 'temporary_table_dump'
st_env.execute_sql(f"""DROP TABLE IF EXISTS {name}""")
def create_table(table_name, stream_name, region, stream_initpos):
return """ CREATE TABLE {0} (
f0 TIMESTAMP(3),
f1 STRING,
WATERMARK FOR f0 AS f0 - INTERVAL '5' SECOND
)
WITH (
'connector' = 'kinesis',
'stream' = '{1}',
'aws.region' = '{2}',
'scan.stream.initpos' = '{3}',
'sink.partitioner-field-delimiter' = ';',
'sink.producer.collection-max-count' = '100',
'format' = 'json',
'json.timestamp-format.standard' = 'ISO-8601'
) """.format(
table_name, stream_name, region, stream_initpos
)
# Creates a sink table writing to a Kinesis Data Stream
st_env.execute_sql(create_table(name, 'output-test', 'eu-west-2', 'LATEST'))
table = st_env.from_data_stream(ds)
st_env.execute_sql(f"""DROP TEMPORARY VIEW IF EXISTS {ds_table_name}""")
st_env.create_temporary_view(ds_table_name, table)
# emit the Table API table
st_env.execute_sql(f"INSERT INTO {name} SELECT * FROM {ds_table_name}").wait()
I'm trying to store temporal embedding in pytable. There are 12 tables and each table has more than 130,000 rows, where each table has two columns (word varchar, embedding float(numpy.arry(300,))). What I want is to calculate cosine similarity for a given against all the word in a given table and repeat this for all 12 tables. Presently I'm doing it sequentially by iterating on each table but it takes around 15 minutes to calculate for all 12 tables.
So my question is, is it possible to read all the table concurrently? I used multithreading but I error
Segmentation fault: 11
Below is my code snippet
def synchronized_open_file():
with lock:
return tb.open_file(FILENAME, mode="r", title="embedding DB")
def synchronized_close_file(self, *args, **kwargs):
with lock:
return self.close(*args, **kwargs)
outqueue = queue.Queue()
for table in list_table :
thread = threading.Thread(target=self.top_n_similar, args=(table,))
thread.start()
threads.append(thread)
try:
for _ in range(len(threads)):
result = outqueue.get()
if isinstance(result, Exception):
raise result
else:
top_n_neighbor_per_period[result[0]] = result[1]
finally:
for thread in threads:
thread.join()
def top_n_similar(table_name):
H5FILE = synchronized_open_file()
do work()
outqueue.put(result)
finally :
synchronized_close_file(H5FILE)
Yes, you can access multiple pytable objects simultaneously. A simple example is provided below that creates 3 tables using a (300,2) numpy record array created with random data. It demonstrates you can access all 3 tables as table objects -OR- as numpy arrays (or both).
I have not done multi-threading with pytables, so can't help with that. I suggest you get your code to work in serial before adding multi-threading. Also, review the pytables docs. I know h5py has specific procedures to use mpi4py for multi-threading. Pytables may have similar requirements.
Code Sample
import tables as tb
import numpy as np
h5f = tb.open_file('SO_55445040.h5',mode='w')
mydtype = np.dtype([('word',float),('varchar',float)])
arr = np.random.rand(300,2)
recarr = np.core.records.array(arr,dtype=mydtype)
h5f.create_table('/', 'table1', obj=recarr )
recarr = np.core.records.array(2.*arr,dtype=mydtype)
h5f.create_table('/', 'table2', obj=recarr )
recarr = np.core.records.array(3.*arr,dtype=mydtype)
h5f.create_table('/', 'table3', obj=recarr )
h5f.close()
h5f = tb.open_file('SO_55445040.h5',mode='r')
# Return Table ojbects:
tb1 = h5f.get_node('/table1')
tb2 = h5f.get_node('/table2')
tb3 = h5f.get_node('/table3')
# Return numpy arrays:
arr1 = h5f.get_node('/table1').read
arr2 = h5f.get_node('/table2').read
arr3 = h5f.get_node('/table3').read
h5f.close()
I've been using python-docx to produce large documents full of tables and figures conforming with a standard template. I have discovered how to make them cross-referenceable using https://github.com/python-openxml/python-docx/issues/359 . However this labels my figures/tables starting at 1 within each section and continuing until the next section where it restarts from 1.
I would like the figure numbers to be dependent on the section number (i.e. 1st figure in 2nd section = Figure 2.1 etc.). Does anyone know if this is possible?
Currently the numbering is produced by the function:
def Table(paragraph):
from docx.oxml import OxmlElement
from docx.oxml.ns import qn
run = run = paragraph.add_run()
r = run._r
fldChar = OxmlElement('w:fldChar')
fldChar.set(qn('w:fldCharType'), 'begin')
r.append(fldChar)
instrText = OxmlElement('w:instrText')
instrText.text = ' SEQ TableMain \* ARABIC \s 1 '
print instrText
r.append(instrText)
fldChar = OxmlElement('w:fldChar')
fldChar.set(qn('w:fldCharType'), 'end')
r.append(fldChar)
Called by the following code which also populates the table and table title and footer
table3 = document.add_table(rows=1, cols=1)
table3.cell(0,0).text="Table "
for paragraph in table4.cell(0,0).paragraphs:
paragraph.style = document.styles['Caption']
Table(paragraph)
paragraph.add_run(text="this is the full table name")
row_cells = table3.add_row().cells
call_func_that_makes_actual_table(row_cells[0],...)
row_cells = table3.add_row().cells
row_cells[0].text="Source: ..."
for paragraph in row_cells[0].paragraphs:
paragraph.style = document.styles['Source']
This produces a table like
this
Whereas I would like the table numbering like
this
Managed to work this out myself the solution is adding a further function:
def section(paragraph):
from docx.oxml import OxmlElement
from docx.oxml.ns import qn
run = run = paragraph.add_run()
r = run._r
fldChar = OxmlElement('w:fldChar')
fldChar.set(qn('w:fldCharType'), 'begin')
r.append(fldChar)
instrText = OxmlElement('w:instrText')
instrText.text = ' STYLEREF 1 \s '
r.append(instrText)
fldChar = OxmlElement('w:fldChar')
fldChar.set(qn('w:fldCharType'), 'end')
r.append(fldChar)
and changing the call to:
for paragraph in table.cell(1,0).paragraphs:
paragraph.style = document.styles['Caption']
section(paragraph)
paragraph.add_run(text=".")
Figure(paragraph)
paragraph.add_run(text=": this is the full table name")
This is the code I'm running in Python. The table has been created in the DB already. I'm doing a commit, so I don't know why it's working.
The code executes just fine, but no data is inserted into the table. I ran the same insert statement directly via sqlite command line and it worked just fine.
import os
import sqlite3
current_dir = os.path.dirname(__file__)
db_file = os.path.join(current_dir, '../data/trips.db')
trips_db = sqlite3.connect(db_file)
c = trips_db.cursor()
print 'inserting data into aggregate tables'
c.execute(
'''
insert into route_agg_data
select
pickup_loc_id || ">" || dropoff_loc_id as ride_route,
count(*) as rides_count
from trip_data
group by
pickup_loc_id || ">" || dropoff_loc_id
'''
)
trips_db.commit
trips_db.close
I changed the last 2 lines of my code to this:
trips_db.commit()
trips_db.close()
Thanks #thesilkworm
Could you write stored procedure inside sqlite:
Then Try In Python:
cur = connection.cursor()
cur.callproc('insert_into_route_agg_data', [request.data['value1'],
request.data['value2'], request.data['value3'] ] )
results = cur.fetchone()
cur.close()
import arcpy
fc = r'H:\H-ONUS UTILITY DATA GIS\As_Builts\2014\RandolphPoint_Phase2\789-AS-BUILT 8-7-13.dwg\Polyline'
out_gdb = r'H:\H-ONUS UTILITY DATA GIS\As_Builts\2014\RandolphPoint_Phase2\RandolphPoint.gdb.gdb'
field = 'Layer'
values = [row[0] for row in arcpy.da.SearchCursor(fc, (field))]
uniqueValues = set(Values)
for value in uniqueValues:
sql = """Layer" = '{0}'""".format(Value)
name = arcpy.ValidateTableName(value,out_gdb)
arcpy.FeatureClassToFeatureClass_conversion(fc, out_gdb, name, sql)
I am trying to convert CAD(dwg) to ArcGIS 10.2.2 Feature Classes using a file geodatase as the workspace. I was just taught this code at an ESRI conference and of course it worked beautifully for the insturtor.
My error I am getting is "NameError:name'Values' is not defined" however I did define it as values = [row[0] for row in arcpy.da.SearchCursor(fc, (field))] I have been working hours on this, it would help out my job considerably.
Python variables are case-sensitive.
You've declared values with a lower-case v, but you're referring to it on the next line with an upper-case V.
(Same with value/Value further down.
import arcpy
fc = r'H:\H-ONUS UTILITY DATA GIS\As_Builts\2014\RandolphPoint_Phase2\789ASBUILT.dwg\Polyline'
out_gdb = r'H:\H-ONUS UTILITY DATA GIS\As_Builts\2014\RandolphPoint_Phase2\RandolphPoint.gdb'
field = 'Layer'
value = [row[0] for row in arcpy.da.SearchCursor(fc, (field))]
uniquevalues = set(value)
for value in uniquevalues:
sql = """"Layer" = '{0}'""".format(value)
name = arcpy.ValidateTableName(value,out_gdb)
arcpy.FeatureClassToFeatureClass_conversion(fc, out_gdb, name, sql)
Here is the solution, I had an extra .gdb in the geodatabase path
my word value was values so had to take the s off
and also in my sql statement I was missing a " before the word Layer
If anyone is reading this just change the individual parameters and it works beautifully!
thanks Juffy for responding and trying to help me out
Cartogal