I have a a few list that I am wanting to send through a Kafka producer.
listA [1,2,3]
listB ["cat", "dog", "fish"]
the producer sends the messages as bytes so I'm unsure how to properly set up the message so the list sends, when quotes are needed to send the message. This is what I currently have.
producer = KafkaProducer(bootstrap_servers='localhost:9092')
for _ in range(1):
print(producer.send('test', b'"worker_id": listA , "worker_name" : listB'))
this method just give me a SyntaxError.
I've also tried this method below and I get a similar result
print(producer.send('test', b("worker_uuid": worker_uuid))
Have you considered JSON encoding? If you configure your KafkaProducer w/ a value_serializer like so:
KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))
then you should be able to send your lists directly like so:
producer.send('test', [1, 2, 3])
producer.send('test', ["cat", "dog", "fish"])
Consumers will need to be configured to decode via json as well. If you are using kafka-python, you might do something like:
KafkaConsumer(value_deserializer=lambda v: json.loads(v.decode('utf-8')))
this is solved by first converting the list into byte arrays.
rList = [1, 2, 3, 4, 5]
arr = bytes(rList)
print(arr)
the result is
b'\x01\x02\x03\x04\x05'
Related
I am still learning and I have a question. What excatly in flutter means? -
`'...[]'`
Just wondering.. When im using 'for in' to fetch data from my database I have to use this '...[]' but why? What does it doing?
I couldn't find answer in the google, probably im pretty bad in googling stuff but ofcourse I got you guys!
Thanks for reply!
This is the spread operator, is used when you want to separate the items, imagine a situation that you want to join two lists:
List<int> a = [1, 2, 3];
List<int> b = [4, 5, 6];
If you just put them inside a list, you will get a List of lists.
List<int> combinedLists = [a, b];
// [[1, 2, 3], [4, 5, 6]]
But when you use spread operator, you will get the items out of list, like this:
List<int> combinedLists = [...a, ...b];
// [1, 2, 3, 4, 5, 6]
I have the following Stream Dataframe
+------------------------------------+
|______sentence______________________|
| Representative is a scientist |
| Norman did a good job in the exam |
| you want to go on shopping? |
--------------------------------------
I have list as follows
val myList
as the final output i need myList contain above three sentences in the stream dataframe
output
myList = [Representative is a scientist, Norman did a good job in the exam, you want to go on shopping? ]
I tried the following which gives stream error
val myList = sentenceDataframe.select("sentence").rdd.map(r => r(0)).collect.toList
Error thrown with above method
org.apache.spark.sql.AnalysisException: Queries with streaming sources
must be executed with writeStream.start()
Please note that above method work with normal datframe but not with stream dataframe.
Is there a way to iterate through each row of the stream dataframe and assign the row value into the common list using scala and spark ?
That sounds like a very weird Use-Case as the stream could theoretically never end. Are you sure you are not just looking for common spark DataFrames?
if that is not the case what you can do is use Accumulators and sparks streaming foreachBatch sink. I used a simple socket connection to demonstrate this. You can start a simple socket server under e.g. ubuntu with nc -lp 3030 and just past message there to the stream, the resulting DataFrame will have a schema of [value: String]
val acc = spark.sparkContext.collectionAccumulator[String]
val stream = spark.readStream.format("socket").option("host", "localhost").option("port", "3030").load()
val query = stream.writeStream.foreachBatch((df: DataFrame, l: Long) => {
df.collect.foreach(v => acc.add(v(0).asInstanceOf[String]))
}).start()
...
// For some reason you are stopping the stream here
query.stop()
val myList = acc.value
Now one question you might have is why are we using Accumulators and not just an ArrayBuffer. ArrayBuffers would work locally but on a cluster the code in foreachBatch might be executed on a total different node. That means it would not have any effect and thats also the reason Accumulators exist in the first place (see https://spark.apache.org/docs/latest/rdd-programming-guide.html#accumulators)
I would like to print a list along with a string identifier like
list = [1, 2, 3]
IO.puts "list is ", list
This does not work. I have tried few variations like
# this prints only the list, not any strings
IO.inspect list
# using puts which also does not work
IO.puts "list is #{list}"
In javascript, I could simply do console.log("list is ", list). I'm confused how I could achieve the same in elixir.
Starting with Elixir 1.4, IO.inspect/2 accepts label option among others:
IO.inspect list, label: "The list is"
#⇒ The list is: [1, 2, 3]
Maybe there's a better way (I'm new to Elixir too) but this worked for me:
IO.puts(["list is ", Enum.join(list, " ")])
list is 1 2 3
Interpolation works too:
IO.puts("list is #{Enum.join(list, " ")}")
Edit: inspect seems to better than Enum.join for this use case:
IO.puts("list is #{inspect(list)}")
list is [1, 2, 3]
Intro
I'm trying to do something that sounds simple, but so far I'm not having luck finding the answer. I have 2 lists in a redis 2.6.4 standalone server(no cluster):
list1 = [4, 5 ,6]
list2 = [1, 2, 3]
The problem
And I need to concatenate the lists to produce something like this:
list3 = list1 + list2
list3 = [4, 5, 6, 1, 2, 3] <- I need to preserve order, list1 and then list 2
list4 = list2 + list1
list4 = [1, 2, 3, 4, 5, 6]
The question
Since redis use linked lists to store this lists I was expecting to have a straightforward way of doing this, does such a way exists? what's the usual way of doing this in redis?
Thanks in advance!
The easiest way to do this safely is to use LUA scripting, this way you have the guarantee that the resulting list is not missing any element (and you can easily preserve the order).
If LUA is not an option then you need to do this client side and use watch those keys for changes (see transaction in redis and WATCH command)
Here is the Redis command using Lua:
eval "for i,l in ipairs(ARGV) do for i,v in ipairs(redis.call('lrange',l,0,-1)) do redis.call('rpush',KEYS[1],v) end end" 1 list3 list1 list2
As an added bonus you can specify any number of lists to append into your master list by simply adding more list keys at the end
I have the following question for homework
Define a function append lists that
takes a list of lists and returns a
new list containing the sublist
values. For example, append lists([[1,
2], [3, 4], [5]]) should return the
list [1, 2, 3, 4, 5] and append
lists([[1, 2], [3], [[4, 5]]]) should
return the list [1, 2, 3, [4, 5]].
I've tried various ways of creating this function in order to append the list so it gives the desired output to no avail so I came here looking for some help. I've found a few other ways of going about this online, but they use extensive methods that we haven't even dabbled in as of yet in my CPSC 121 class. We're limited to the basics in what we've learned.
Any help would be much appreciated!
By now, it is likely that the assignment is gone, but here is a solution:
def append_lists(lists):
output = []
for l in lists:
for e in l:
output.append(e)
return output
This appends each element of each list to the output of the function, which eliminates exactly one level of nesting in the elements.