I have been playing with redis and am performing left pops on lists.
I wanted to rebuild the list so i can start over from the head again. I tried restarting the redis server and its still popping sub elements as if the data was never rebuilt. Am i missing something here?
Commands:
127.0.0.1:6379> lpop set:reviews
"review:547221"
127.0.0.1:6379> exit
root#development-1:~/redis-sandbox# /etc/init.d/redis-server restart
127.0.0.1:6379> lpop set:reviews
"review:547220"
Do I need to rebuild the redis data after a pop operation? Isn't the point to reuse the memory cache?
LPOP actually removes the left-most (head) element from the list. If your Redis is configured for persistency (which apparently it is), changes to the data, and specifically popping from lists, will be retained across restarts.
I'm not sure how and what for you're using Redis, but even caches benefit from having the ability to recover their data after failure.
Related
I'm running a node of an Ethereum side-chain. I started only getting "peer connected on snap without compatible eth support" error messages in the log a few days ago. It would not download any new blocks. The last block in my local chain was 5 days old. I thought maybe it has something to do with the merge.
The node runs inside a docker container and I don't know how to do anything with docker. My only options are interacting with the node.
First I tried using debug_setHead over RPC. I set the head back approx. 100k blocks before the last block in my chain. But when it reached that same block I would again only get those error messages. What's weird is the log message that came right before in both times (when it first happened and after setting back the head) was "Deep froze chain segment" and after that I only got "peer connected on snap without compatible eth support".
Because setting back the head didn't work, the next thing I tried was pruning the node. According to the documentation pruning should only take 1 to 2 hours for this side-chain (It's on an SSD). But even after running it overnight I would never get the log message "State pruning successful".
Not knowing what to do, I started my node and read the log.
The end of the log says:
WARNING!
The clean trie cache is not found. Please delete it by yourself after the pruning. Remember don't start the Geth without deleting the clean trie cache otherwise the entire database may be damaged!
Check the command description "geth snapshot prune-state --help" for more details.
INFO [09-16|18:14:45.182] Pruning state data nodes=1 size=115.00B elapsed=13m3.752s eta=14m13.881s
INFO [09-16|18:14:53.188] Pruning state data nodes=2,264,671 size=676.51MiB elapsed=13m11.758s eta=14m7.433s
INFO [09-16|18:15:01.198] Pruning state data nodes=4,284,801 size=1.25GiB elapsed=13m19.768s eta=14m2.59s
After that it would just stop logging. It never attempts to connect to the chain and download any blocks. I'm not sure if starting the node could have damaged the chain, because after all it never downloaded any new chain data. Also I have no idea how to delete the clean trie cache.
The last thing I tried was removing all docker containers. I ran docker system prune and it removed all containers, images and volumes. But after reinstalling the node nothing changed. I still get the same log as shown above (without downloading any blocks), because apparently it didn't delete any chain data.
Also the RPC endpoint does not work anymore when starting the node.
I'm completely lost. I don't know what caused this problem in the first place or how to fix it. What can I do to get my node up and running again?
UPDATE:
I have now also tried deleting chain data with geth removedb but I still get the exact same log warning and nothing happens after that. Maybe deleting the clean cache can help getting at least one step further, but I don't know how to do that in a docker container.
UPDATE 2:
While geth removedb did not delete the database, it must have deleted something, because after starting the node, the pruning was successfully completed. But as expected, it did not solve my original problem. I still get an endless stream of
ERROR[09-16|20:50:27.777] Snapshot extension registration failed peer=eec7c316 err="peer connected on snap without compatible eth support"
error logs. And my node is still stuck on the same old block. Mind you that this error stream only starts at a certain block and is not a general problem with my node. If I set the head to a prior block with debug_setHead, the node will successfully sync up to the block I'm stuck at.
You can try to run the command like this:
sudo -u eth1 geth --datadir /path/to/chaindata removedb
This makes the command run under the user "eth1" assigned to geth. Depending on your setup that is the only user that can access the chaindata.
The command then asks which data to remove. You can keep the ancient data. Then start geth again and it should be syncing now.
I am creating custom result files in my jobs and want to sync them from the worker nodes to the head nodes (To rsync them down to my local computer later on). I tried to write them all into the local_dir e.g. ~./ray_results but unfortunately it seems that ray tune is only synching the individual trial folders in the local_dir. Is this correct?
Yes; try writing them to the self.logdir for each trainable.
I would like to implement a concurrent priority queue in Redis, with multiple processes on different machines adding items (with scores) and multiple other processes popping these items, lowest score first.
A simple queue can be implemented with LPUSH and RPOP.
Using a ZSET, I can add the items using ZADD and pop them with ZRANGE and ZREM, as long as there is only one reader.
For multiple readers I think I need something like ZPOP which combines ZRANGE and ZREM in a single atomic operation. Otherwise two readers may get the same item from ZRANGE before either can ZREM it. Retrying if ZREM returns 0 would work but is not desirable.
Is there some way I can do this using the current Redis commands? Is there any reason this hasn't been added to Redis already? It seems like it would be a pretty simple command to implement.
You can guarantee atomicity if you use a Lua script that does the ZRANGE & ZREM or with a MULTI/EXEC block. This will prevent multiple workers from interfering with each other.
I assume that ZPOP wasn't put in in the first place because it isn't a common use case and, when needed, it can be easily scripted.
you can use redis command: watch
WATCH zset
element = ZRANGE zset 0 0
MULTI
ZREM zset element
EXEC
if exec fails (return a null reply), just repeat those commands.
From Redis 5.0.0 you can use ZPOPMIN and ZPOPMAX (and their blocking counterpart BZPOPMIN and BZPOPMAX).
Processes A and B both operate on a Redis resource R.
These processes may be executed in parallel, and I need both processes to be certain of the value of R at the moment they change it.
I'm therefore using Redis transactions with the WATCH command. From the docs: "we are asking Redis to perform the transaction only if no other client modified any of the WATCHed keys. Otherwise the transaction is not entered at all."
To retry in case of failure, the suggested way is looping the Watch/Multi-exec loop until it succeeds. However, I'm worried that both A and B might starting looping indefinitely (i.e.: livelock).
It this something to be worried about? Better yet, what to do about it? Would setting a random timeout on the retry solve the issue?
No need to worry because only A or B will succeed with their EXEC and change R (Redis is [mostly] single threaded). The one that fails will need to retry the transaction with the new R value.
I am using Redis' INCR to generate an ID for objects. And then use ZADD to add the object using the ID as key.
Do I need to worry about if there are multiple connections executing this same block of code? Say after id:12 if two connections connect at the same time and both add object using id:13, then one of them would be lost.
Since redis is single threaded, this can never happen - only one client can make a change to the database at a time.
As Jonatan Hedborg stated, Redis is single threaded so you never need to worry about two clients doing something at the same time. If, on the other hand, your worry is that you want to run the INCR and ZADD commands sequentially, and want to make sure no other commands are run in between them, you can use transactions, and be guaranteed your commands are run as a single unit with nothing in between.