I'm trying to use Sidekiq with my app, but for some reason I get a undefined local variable or method "worker" for #<SolutionController:0xb55bc358>
I have sidekiq, sinatra and slim installed for the Sidekiq web UI, and it doesn't register anything. I do have sidekiq running in another terminal window for now.
Worker call :
worker.perform_async(#user)
My Worker :
# app/workers/worker.rb
class worker
include Sidekiq::Worker
sidekiq_options retry: false
def perform(s_user)
user = $client.user_timeline(s_user, exclude_replies: 1,include_rts: 1 ).take(10)
user.each do |t|
array_list = $client.retweeters_of(t.id)
end
limited_list = array_list
array = []
hash = {}
limited_list.each do |g|
hash = {:key => g.followers_count, :value => g.profile_image_url.to_s}
array.push(hash)
end
array.sort_by! {|h| h[:key]}
array.reverse!
final = array.take(10)
$redis.set("#{hash[:value]}", "#{hash[:value]}")
end
end
Thanks
I recommend a bit more descriptive name, with proper case. try renaming the class to MyWorker , the file to my_worker.rb and calling MyWorker.perform_async(#user)
Related
Using akka typed: 2.6.10. My parent generates child actors to do some work as you can see below (note this is part of event sourced actor). Is there a way to acquire reference to internally created child actor using possibly name during testing time?
For example, below we have child actor provider_1 which is created at initialization time and I am hoping to acquire a reference to TestProbe using this name from outside. I am reluctant to change the way code is structured for sake of testing, for example in here there are some reference to passing in ref/factory or re-constructing parent in test in order to test this, which I would like to avoid.
def commandHandler(
ctx: ActorContext[Command]
): (State, Command) => Effect[Event, State] = { (state, cmd) =>
cmd match {
case Init =>
ctx.spawn(Provider(ctx.self), "provider_1")
Effect.none
}
}
If you're using the BehaviorTestKit to test the actor, the actor gets run with an alternative ActorContext implementation.
So the following should work (note that akka.actor.testkit.typed.Effect has little relation to Effect in persistence), using scalatest matchers:
import akka.actor.testkit.typed.Effect.Spawned
val testKit = BehaviorTestKit(behaviorUnderTest)
testKit.run(Init)
val effect = testKit.retrieveEffect : akka.actor.testkit.typed.Effect
val childActorRef =
effect match {
case s: Spawned if s.childName == "provider_1" => s.ref
case _ => fail()
}
val childInbox = testKit.childInbox(childActorRef)
testKit.run(SendMessageToYourChild("hello"))
childInbox.hasMessages shouldBe true
childInbox.receiveMessage shouldBe MessageFromParent("hello")
akka.actor.testkit.typed.scaladsl.TestInbox is intended to be the synchronous behavior testing analogue to the asynchronous TestProbe.
I'm not aware of an analogous method for the asynchronous ActorTestKit, where a child actor will actually be spawned.
I use this worker for process
class CreateOrUpdateContactWorker
include Sidekiq::Worker
sidekiq_options retry: 2, queue: 'contact_updater', concurrency: 1
sidekiq_retries_exhausted do |msg|
Airbrake.notify(error_message: "Contact update failed", session: { msg: msg })
end
def perform(user_id, changed_fields, update_address = false)
ContactUpdater.create_or_update_contact(user_id, changed_fields, update_address: update_address)
end
end
In the user model I have after_commit callback
def update_mautic_contact
CreateOrUpdateContactWorker.perform_async(id, previous_changes.keys, ship_address_changed || false)
end
The problem is when user updated twice at the same time, because to create_or_update_contact need some time. How can I limit threads only for specify user? That each task will be executed one by one for specify user_id.
I don't know if you have redis as part of your infrastructure but what you are describing is a race condition. To solve it you need mutex/locks to your critical path create_or_update_contact.
The race condition here is happening between two async workers/processes so you can't just use simple ruby mutex/lock. You need a distributed mutex that uses a central lock storage/keeper. This: https://github.com/kenn/redis-mutex should do it for you but you will need redis database.
Basically your code will something like:
class CreateOrUpdateContactWorker
include Sidekiq::Worker
sidekiq_options retry: 2, queue: 'contact_updater', concurrency: 1
sidekiq_retries_exhausted do |msg|
Airbrake.notify(error_message: "Contact update failed", session: { msg: msg })
end
def perform(user_id, changed_fields, update_address = false)
RedisMutex.with_lock("#{user_id}_create_or_update_contact") do
ContactUpdater.create_or_update_contact(user_id, changed_fields, update_address: update_address)
end
end
end
So if you have 2 user updates for user_id=1 at the same time the first to acquire the lock/mutex called 1_create_or_update_contact will execute first and will block the other call till it finishes and then the second call will start.
This will fix your problem :) I think redis is needed, useful and handful. I hardly can think of any of my rails projects without having to use redis.
I realised it with Redis, but without any gems. I used condition before execute worker:
def update_mautic_contact
if Rails.current.get("CreateOrUpdateContactWorkerIsRunning_#{id}")
Redis.current.set("CreateOrUpdateContactWorkerIsRunning_#{id}", true)
CreateOrUpdateContactWorker.perform_in(1.minutes, id, changed_fields)
else
Redis.current.set("CreateOrUpdateContactWorkerIsRunning_#{id}", true)
CreateOrUpdateContactWorker.perform_async(id, changed_fields)
end
end
and inside worker:
class CreateOrUpdateContactWorker
include Sidekiq::Worker
sidekiq_options retry: 2, queue: 'contact_updater', concurrency: 1
sidekiq_retries_exhausted do |msg|
Airbrake.notify(error_message: "Contact update failed", session: { msg: msg })
end
def perform(user_id, changed_fields, update_address = false)
ContactUpdater.create_or_update_contact(user_id, changed_fields, update_address: update_address)
Redis.current.del("CreateOrUpdateContactWorkerIsRunning_#{user_id}")
end
end
Writing a few small experiments to familiarise myself with the language, but have run into an issue which I'm guessing is elementary.
I have a simple supervisor, with 3 simple workers:
def init do
Supervisor.start_link(
[
worker(__MODULE__, [:"process-1"], [function: :test, id: :"p-1"]),
worker(__MODULE__, [:"process-2"], [function: :test, id: :"p-2"]),
worker(__MODULE__, [:"process-3"], [function: :test, id: :"p-3"])
],
strategy: :one_for_one
)
end
":test" looks like this:
def test(name) do
flag(:trap_exit, true)
IO.puts "Testing: #{name} == #{inspect self}"
register(self, name)
receive do
{ :death } ->
IO.puts("I WOZ MURDERED!")
exit(self, "Ex process...")
{ :life } ->
IO.puts("#{inspect self} is listening...")
__MODULE__.test(name)
{ :EXIT, pid, reason } ->
IO.puts "REASON: #{inspect reason} - PROCESS: #{inspect pid}"
end
end
This compiles, but it only ever spawns one process, and hangs/blocks iex.
In contrast, when I use a simple chain of 'spawn_link'ed' processes, all three (or however many) processes start concurrently and return control to the iex shell so I can send the registered processes messages from the command line.
My intention, for now, is to create an OTP supervisor, run and register three (or however many) workers processes and attach them to the supervisor, send a simple message to kill a given worker, and then have the supervisor restart it.
What am I doing wrong?
The problem is the function: you're giving as part of the worker specification doesn't do what OTP expects.
From http://www.erlang.org/doc/man/supervisor.html
The start function must create and link to the child process, and
should return {ok,Child} or {ok,Child,Info} where Child is the pid of
the child process and Info an arbitrary term which is ignored by the
supervisor.
Your code doesn't spawn a child but goes into a receive loop. You could perhaps use Elixir's Task module to do something similar to what it looks like you want:
worker(Task, [__MODULE__, :test, [:"process-1"]], id: :"p-1"),
worker(Task, [__MODULE__, :test, [:"process-2"]], id: :"p-2"),
worker(Task, [__MODULE__, :test, [:"process-3"]], id: :"p-3")
However if you're looking to learn more about what OTP does then having a go at implementing your own GenServer might be a better option.
I have an email helper extension I have written for my application. I use settings I have defined on the application like so:
set :mailgun_options, JSON.parse(File.open('config/mailer.json').read)[ENV['RACK_ENV']]
which reads configuration settings from a json file into the global Sinatra settings.
I then have my email helper which references these settings to create connections and what not.
require 'sinatra/base'
require 'faraday'
module Sinatra
module EmailHelper
def connect opts={}
connection = Faraday.new(:url => settings.mailgun_options['mailgun_url']) do |faraday|
if settings.mailgun_options['test']
faraday.adapter :test do |stubs|
stubs.post(settings.mailgun_options['domain'] + '/messages') {[ 200, {}, 'success' ]}
end
else
faraday.request :url_encoded
faraday.adapter Faraday.default_adapter
end
end
connection.basic_auth('api', settings.mailgun_options['key'])
return connection
end
def send_email params={}
connect.post((settings.mailgun_options['domain'] + '/messages'), params)
end
def send_password_reset_email email
template = File.open( File.expand_path( '../../helpers/email_helper/templates/password_reset.erb', __FILE__ ), 'r' ).read
send_email({
:from => 'REscour <noreply#' + settings.mailgun_options['domain'] + '>',
:to => email,
:subject => 'REscour.com Password Reset',
:text => ERB.new(template).result(binding)
})
end
end
end
So, when I am trying to write my specs to test the "connect" method, I get the following error:
undefined local variable or method `settings' for #<Class:0x007faaf9c9bd18>
My spec code is as follows:
module Sinatra
describe EmailHelper
let(:dummy_class) { Class.new { extend EmailHelper } }
it 'should connect to mailgun\'s api' do
app.set :mailgun_options, ::JSON.parse(File.open('config/mailer.json').read)['test']
puts dummy_class.connect
end
end
end
I'm not sure how to interact with the global Sinatra settings from my specs to have the local "settings" variable be defined during execution of the helper code.
I could have the architecture of my application wrong here for all I know. Does anyone have any experience in writing Sinatra extensions using the global settings and write specs for them?
The spec you've written seems to want to be a unit test, because you've not run the extension the way it would be used in an app. In the specs, try something like:
describe "connect" do
let(:app) { Sinatra.new do
helpers Sinatra::EmailHelper
configure do
:mailgun_options, ::JSON.parse(File.open('config/mailer.json')
end
get "/" do
connect
end
end
}
# now write specs for the `connect` helper
end
Using an anonymous class (Sinatra.new do…) lets you quickly set up an app that's convenient for a small test, but there are other ways too.
In the extension, you need to add:
helpers EmailHelper
as per the example in the docs.
I'm learnin about remote actors in Akka 2.1 and I tried to adapt the counter example provided by Typesafe.
I implemented a quick'n'dirty UI from the console to send ticks. And to quit with asking(and showing the result) the current count.
The idea is to start a master node that will run the Counter actor and some client node that will send messages to it through remoting. However I'd like to achieve this through configuration and minimal changes to code. So by changing the configuration local actors could be used.
I found this blog entry about similar problem where it was necessary that all API calls go through one actor even though there are many instances running.
I wrote similar configuration but I cant get it to work. My current code does use remoting but it creates a new actor on the master for each new node and I can't get it to connect to existing actor without explicitly giving it the path(and defying the point of configuration). However this is not what I want since state cannot be shared between JVMs this way.
Full runnable code available through a git repo
This is my config file
akka {
actor {
provider = "akka.remote.RemoteActorRefProvider"
deployment {
/counter {
remote = "akka://ticker#127.0.0.1:2552"
}
}
}
remote {
transport = "akka.remote.netty.NettyRemoteTransport"
log-sent-messages = on
netty {
hostname = "127.0.0.1"
}
}
}
And full source
import akka.actor._
import akka.pattern.ask
import scala.concurrent.duration._
import akka.util.Timeout
import scala.util._
case object Tick
case object Get
class Counter extends Actor {
var count = 0
val id = math.random.toString.substring(2)
println(s"\nmy name is $id\ni'm at ${self.path}\n")
def log(s: String) = println(s"$id: $s")
def receive = {
case Tick =>
count += 1
log(s"got a tick, now at $count")
case Get =>
sender ! count
log(s"asked for count, replied with $count")
}
}
object AkkaProjectInScala extends App {
val system = ActorSystem("ticker")
implicit val ec = system.dispatcher
val counter = system.actorOf(Props[Counter], "counter")
def step {
print("tick or quit? ")
readLine() match {
case "tick" => counter ! Tick
case "quit" => return
case _ =>
}
step
}
step
implicit val timeout = Timeout(5.seconds)
val f = counter ? Get
f onComplete {
case Failure(e) => throw e
case Success(count) => println("Count is " + count)
}
system.shutdown()
}
I used sbt run and in another window sbt run -Dakka.remote.netty.port=0 to run it.
I found out I can use some sort of pattern. Akka remote allows only for deploying on remote systems(can't find a way to make it look up on remote just through configuration..am I mistaken here?).
So I can deploy a "scout" that will pass back the ActorRef. Runnable code available on the original repo under branch "scout-hack". Because this feels like a hack. I will still appreciate configuration based solution.
The actor
case object Fetch
class Scout extends Actor{
def receive = {
case Fetch => sender ! AkkaProjectInScala._counter
}
}
Counter actor creating is now lazy
lazy val _counter = system.actorOf(Props[Counter], "counter")
So it only executes on the master(determined by the port) and can be fetched like this
val counter: ActorRef = {
val scout = system.actorOf(Props[Scout], "scout")
val ref = Await.result(scout ? Fetch, timeout.duration) match {
case r: ActorRef => r
}
scout ! PoisonPill
ref
}
And full config
akka {
actor {
provider = "akka.remote.RemoteActorRefProvider"
deployment {
/scout {
remote = "akka://ticker#127.0.0.1:2552"
}
}
}
remote {
transport = "akka.remote.netty.NettyRemoteTransport"
log-sent-messages = on
netty {
hostname = "127.0.0.1"
}
}
}
EDIT: I also found a clean-ish way: check configuration for "counterPath" anf if present actorFor(path) else create actor. Nice and you can inject the master when running and code is much cleaner than with the "scout" but it still has to decide weather to look up or create an actor. I guess this cannot be avoided.
I tried your git project and it actually works fine, aside from a compilation error, and that you must start the sbt session with -Dakka.remote.netty.port=0 parameter to the jvm, not as parameter to run.
You should also understand that you don't have to start the Counter actor in both processes. In this example it's intended to be created from the client and deployed on the server (port 2552). You don't have to start it on the server. It should be enough to create the actor system on the server for this example.