Cannot infer an appropriate lifetime when sharing self reference - concurrency

This is an experiment I'm doing while learning Rust and following Programming Rust.
Here's a link to the code in the playground.
I have a struct (Thing) with some inner state (xs). A Thing should be created with Thing::new and then started, after which the user should choose to call some other function like get_xs.
But! In start 2 threads are spawned which call other methods on the Thing instance that could mutate its inner state (say, add elements to xs), so they need a reference to self (hence the Arc). However, this causes a lifetime conflict:
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src/main.rs:18:30
|
18 | let self1 = Arc::new(self);
| ^^^^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined
on the method body at 17:5...
--> src/main.rs:17:5
|
17 | / fn start(&self) -> io::Result<Vec<JoinHandle<()>>> {
18 | | let self1 = Arc::new(self);
19 | | let self2 = self1.clone();
20 | |
... |
33 | | Ok(vec![handle1, handle2])
34 | | }
| |_____^
note: ...so that expression is assignable (expected &Thing, found &Thing)
--> src/main.rs:18:30
|
18 | let self1 = Arc::new(self);
| ^^^^
= note: but, the lifetime must be valid for the static lifetime...
note: ...so that the type `[closure#src/main.rs:23:20: 25:14
self1:std::sync::Arc<&Thing>]` will meet its required lifetime bounds
--> src/main.rs:23:14
|
23 | .spawn(move || loop {
| ^^^^^
Is there a way of spawning the state-mutating threads and still give back ownership of thing after running start to the code that's using it?
use std::io;
use std::sync::{Arc, LockResult, RwLock, RwLockReadGuard};
use std::thread::{Builder, JoinHandle};
struct Thing {
xs: RwLock<Vec<String>>
}
impl Thing {
fn new() -> Thing {
Thing {
xs: RwLock::new(Vec::new()),
}
}
fn start(&self) -> io::Result<Vec<JoinHandle<()>>> {
let self1 = Arc::new(self);
let self2 = self1.clone();
let handle1 = Builder::new()
.name("thread1".to_owned())
.spawn(move || loop {
self1.do_within_thread1();
})?;
let handle2 = Builder::new()
.name("thread2".to_owned())
.spawn(move || loop {
self2.do_within_thread2();
})?;
Ok(vec![handle1, handle2])
}
fn get_xs(&self) -> LockResult<RwLockReadGuard<Vec<String>>> {
return self.xs.read();
}
fn do_within_thread1(&self) {
// read and potentially mutate self.xs
}
fn do_within_thread2(&self) {
// read and potentially mutate self.xs
}
}
fn main() {
let thing = Thing::new();
let handles = match thing.start() {
Ok(hs) => hs,
_ => panic!("Error"),
};
thing.get_xs();
for handle in handles {
handle.join();
}
}

The error message says that the value passed to the Arc must live the 'static lifetime. This is because spawning a thread, be it with std::thread::spawn or std::thread::Builder, requires the passed closure to live this lifetime, thus enabling the thread to "live freely" beyond the scope of the spawning thread.
Let us expand the prototype of the start method:
fn start<'a>(&'a self: &'a Thing) -> io::Result<Vec<JoinHandle<()>>> { ... }
The attempt of putting a &'a self into an Arc creates an Arc<&'a Thing>, which is still constrained to the lifetime 'a, and so cannot be moved to a closure that needs to live longer than that. Since we cannot move out &self either, the solution is not to use &self for this method. Instead, we can make start accept an Arc directly:
fn start(thing: Arc<Self>) -> io::Result<Vec<JoinHandle<()>>> {
let self1 = thing.clone();
let self2 = thing;
let handle1 = Builder::new()
.name("thread1".to_owned())
.spawn(move || loop {
self1.do_within_thread1();
})?;
let handle2 = Builder::new()
.name("thread2".to_owned())
.spawn(move || loop {
self2.do_within_thread2();
})?;
Ok(vec![handle1, handle2])
}
And pass reference-counted pointers at the consumer's scope:
let thing = Arc::new(Thing::new());
let handles = Thing::start(thing.clone()).unwrap_or_else(|_| panic!("Error"));
thing.get_xs().unwrap();
for handle in handles {
handle.join().unwrap();
}
Playground. At this point the program will compile and run (although the workers are in an infinite loop, so the playground will kill the process after the timeout).

Related

How to have a connection with lazy static to S3 from Rust?

I am trying to replicate one of our Python lambda function to Rust and got stuck at the beginning with having a client to S3.
In Python it is simple, during the Lambda initialisation I declare a variable. However, when using Rust it is a bit more complicated.
As a newcomer to Rust dealing with async in a lazy_static setup is difficult.
use aws_config::meta::region::RegionProviderChain;
use aws_config::SdkConfig;
use aws_sdk_s3::Client as S3Client;
use lambda_http::Error as LambdaHttpError;
use lambda_http::{run, service_fn, Body, Error, Request, Response};
use lazy_static::lazy_static;
async fn connect_to_s3() -> S3Client {
let region_provider: RegionProviderChain =
RegionProviderChain::default_provider().or_else("eu-west-1");
let config = aws_config::load_from_env().await;
let client: S3Client = S3Client::new(&config);
client
}
lazy_static! {
static ref S3_CLIENT: S3Client = connect_to_s3();
}
This throws the following error:
error[E0308]: mismatched types
--> src/bin/lambda/rora.rs:20:38
|
20 | static ref S3_CLIENT: S3Client = connect_to_s3();
| -------- ^^^^^^^^^^^^^^^ expected struct `aws_sdk_s3::Client`, found opaque type
| |
| expected `aws_sdk_s3::Client` because of return type
|
note: while checking the return type of the `async fn`
--> src/bin/lambda/rora.rs:10:29
|
10 | async fn connect_to_s3() -> S3Client {
| ^^^^^^^^ checked the `Output` of this `async fn`, found opaque type
= note: expected struct `aws_sdk_s3::Client`
found opaque type `impl Future<Output = aws_sdk_s3::Client>`
help: consider `await`ing on the `Future`
|
20 | static ref S3_CLIENT: S3Client = connect_to_s3().await;
| ++++++
How can I initialize the connection to s3 during setup of the lambda?
After reviewing a bunch of code on github and talking to some of the Rust devs I know here is the current best option that I am aware of:
use async_once::AsyncOnce;
use lazy_static::lazy_static;
use aws_config::meta::region::RegionProviderChain;
use aws_sdk_glue::Client as GlueClient;
use aws_sdk_s3::Client as S3Client;
lazy_static! {
static ref S3_CLIENT: AsyncOnce<S3Client> = AsyncOnce::new(async {
let region_provider = RegionProviderChain::default_provider().or_else("eu-west-1");
let config = aws_config::from_env().region(region_provider).load().await;
S3Client::new(&config)
});
static ref GLUE_CLIENT: AsyncOnce<GlueClient> = AsyncOnce::new(async {
let region_provider = RegionProviderChain::default_provider().or_else("eu-west-1");
let config = aws_config::from_env().region(region_provider).load().await;
GlueClient::new(&config)
});
}

In a Rust Unit Test harness, how do I wait for a callback to be called?

Please consider the following function:
pub fn shiny_function(&mut self, cb: Arc<Mutex<dyn FnMut(usize) + Send>>) {
// Do stuff here...
}
Now, the question is, how do I write a Unit Test that checks that the callback (closure) parameter is equal to some value?
The obvious solution looks something like this:
#[test]
fn progress_cb() {
let cut = ... // cut stands for Class Under Test
cut.shiny_function(Arc::new(Mutex::new(move |percent| {
// Assert here maybe? I don't know.
})));
cut.shiny_function();
// Or maybe assert it somehow here? I don't know.
}
But the thing is the test finishes before the callback is even called. How can I tell the test harness to wait until the callback is called?
You can use the regular concurrency structs provided in the standard library to fix this issue. In this example, I use a barrier to ensure that the end of the closure is reached before the test function exits. I create the barrier with a value of 2 since wait must be called twice before the barrier is released on both threads. This behavior might not be desirable when calling shiny_function multiple times so you could also substitute another concurrency structure that only blocks in a single location.
use std::sync::{Arc, Barrier};
#[test]
fn progress_cb() {
let cut = ... // cut stands for Class Under Test
// Create a barrier for this thread and clone it to move into the closure
let barrier = Arc::new(Barrier::new(2));
let barrier_clone = barrier.clone();
cut.shiny_function(Arc::new(Mutex::new(move |percent| {
// Perform tests
assert_eq!(percent, foo);
// Once we finish we can trigger the barrier so the outer thread can continue
barrier_clone.wait();
})));
// Don't exit the function until the barrier has been resolved in the callback
barrier.wait();
}
Edit: Here is a struct you could use to if the barrier starts becoming an issue due to the closure blocking on every call and holding up later calls to shiny_function in a single test function.
use std::sync::{Arc, Mutex, Condvar};
pub struct SingleBlockingBarrier {
target: u32,
counter: Mutex<u32>,
lock: Condvar,
}
impl SingleBlockingBarrier {
pub fn new(target: u32) -> Arc<Self> {
Arc::new(SingleBlockingBarrier {
target,
counter: Mutex::new(0),
lock: Condvar::new(),
})
}
pub fn signal(&self) {
let mut guard = self.counter.lock().unwrap();
*guard += 1;
if *guard >= self.target {
self.lock.notify_all();
}
}
// Block until signal has been called the target number of times
pub fn block_until_finished(&self) {
let mut guard = self.counter.lock().unwrap();
loop {
if *guard >= self.target {
return;
}
guard = self.lock.wait(guard).unwrap();
}
}
}
#[test]
fn progress_cb() {
let cut = ... // cut stands for Class Under Test
// Create a barrier for this thread and clone it to move into the closure
let barrier = SingleBlockingBarrier::new(10);
for _ in 0..10 {
let barrier_clone = barrier.clone();
cut.shiny_function(Arc::new(Mutex::new(move |percent| {
// Perform tests
assert_eq!(percent, foo);
// Notify barrier that a worker has finished without blocking
barrier_clone.signal();
})));
}
// Block until all non-blocking barriers have been reached
barrier.block_until_finished();
}

different function type when using if statement in Rust

I'm new in rust and i need to make a small if statement on function option for example
use isahc::{
HttpClient,
config::{
RedirectPolicy,
VersionNegotiation,
SslOption},
prelude::*
};
use std::{
time::Duration
};
pub struct http {
pub timeout: u64
}
impl http {
pub fn send(&self) -> HttpClient {
let client =
HttpClient::builder()
.version_negotiation(VersionNegotiation::http11())
.redirect_policy(RedirectPolicy::None)
.timeout(Duration::from_secs(self.timeout));
.ssl_options(SslOption::DANGER_ACCEPT_INVALID_CERTS | SslOption::DANGER_ACCEPT_REVOKED_CERTS);
return client.build().unwrap();
}
}
fn main(){
let req = http{ timeout:"20".parse().unwrap()};
let test = req.send();
test.get("https://www.google.com");
}
now in my program the user will give me the options of the request (eg: follow redirects or not ) and this need if statement on these options so i tried to use it on this case but i always get a different function return type
impl http {
pub fn send(&self) -> HttpClient {
let client =
HttpClient::builder()
.version_negotiation(VersionNegotiation::http11())
.redirect_policy(RedirectPolicy::None)
.ssl_options(SslOption::DANGER_ACCEPT_INVALID_CERTS | SslOption::DANGER_ACCEPT_REVOKED_CERTS);
if 1 == 1 {
client.timeout(Duration::from_secs(self.timeout));
}
return client.build().unwrap();
}
}
Cargo Output
warning: type `http` should have an upper camel case name
--> src/sender.rs:14:12
|
14 | pub struct http {
| ^^^^ help: convert the identifier to upper camel case: `Http`
|
= note: `#[warn(non_camel_case_types)]` on by default
error[E0382]: use of moved value: `client`
--> src/sender.rs:32:16
|
24 | let client =
| ------ move occurs because `client` has type `HttpClientBuilder`, which does not implement the `Copy` trait
...
30 | client.timeout(Duration::from_secs(self.timeout));
| ------ value moved here
31 | }
32 | return client.build().unwrap();
| ^^^^^^ value used here after move
so what I'm doing wrong ?
i tired to change the function type exception but i can't use the function of the class client.get() for example
this a clear example in python for explain what i need to do
options : dict = {
"redirects": False,
"ssl_path":"~/cer.pem"
}
def send(opts):
# not real httplib !
r = httplib.get("http://stackoverflow.com")
if opts.get('redirects') == True:
r.redirects = True
if opts.get('cert_path',''):
r.ssl = opts.get('cert_path')
return r.send()
def main():
send(options)
Thanks
Since this is a builder pattern, each function call consumes the builder and returns it back. So you need to capture client from the return value of the timeout function in order to continue using it. Note that you also need to make client mutable.
Something like
impl http {
pub fn send(&self) -> HttpClient {
let mut client =
HttpClient::builder()
.version_negotiation(VersionNegotiation::http11())
.redirect_policy(RedirectPolicy::None)
.ssl_options(SslOption::DANGER_ACCEPT_INVALID_CERTS | SslOption::DANGER_ACCEPT_REVOKED_CERTS);
if 1 == 1 {
client = client.timeout(Duration::from_secs(self.timeout));
}
return client.build().unwrap();
}
}

Sharing mutable self between multiple threads

I have a server that accepts connections from multiple clients. Each client could send a message to the server, which is broadcast to all other clients. The problem is that the function that handles each connection should have a reference to the server. However, I want to handle the connections in separate threads, so I cannot use a reference directly.
Since scoped is deprecated, I tried wrapping self in an Arc, but more problems ensued. Below is my attempt:
struct Server {
listener: TcpListener,
clients: Vec<TcpStream>
}
impl Server {
fn new() -> Server {
Server {
listener : TcpListener::bind("127.0.0.1:8085").unwrap(),
clients : vec![]
}
}
fn handle(&self) {
println!("test");
}
fn start(mut self) {
let mut handles = vec![];
let a : Arc<Mutex<Server>> = Arc::new(Mutex::new(self));
let mut selfm = a.lock().unwrap();
// cannot borrow as mutable... ?
for stream in selfm.listener.incoming() {
match stream {
Ok(stream) => {
selfm.clients.push(stream);
let aa = a.clone();
handles.push(thread::spawn(move || {
aa.lock().unwrap().handle();
}));
},
Err(e) => { println!("{}", e); },
}
}
}
Rust Playground
I don't understand what to do anymore, and I fear deadlocks will arise with all these locks. Do you have any suggestions?
The error is pretty much unrelated to having multiple threads. The issue is, as the compiler says, that selfm is already borrowed in the line
for stream in selfm.listener.incoming() {
so it cannot be mutably borrowed in the line
selfm.clients.push(stream);
One way to fix this is to destructure selfm before the loop, so the borrows don't conflict. Your start method will then look as follows:
fn start(mut self) {
let mut handles = vec![];
let a : Arc<Mutex<Server>> = Arc::new(Mutex::new(self));
let mut selfm = a.lock().unwrap();
// destructure selfm here to get a reference to the listener and a mutable reference to the clients
let Server { ref listener, ref mut clients} = *selfm;
for stream in listener.incoming() { // listener can be used here
match stream {
Ok(stream) => {
clients.push(stream); // clients can be mutated here
let aa = a.clone();
handles.push(thread::spawn(move || {
aa.lock().unwrap().handle();
}));
},
Err(e) => { println!("{}", e); },
}
}
}
(That being said, you're right to be concerned about the locking, since the mutex will remain locked until selfm goes out of scope, i.e. only when start terminates, i.e. never. I would suggest an alternative design, but it's not really clear to me why you want the threads to have access to the server struct.)

Scala equivalent of C++ static variable in a function

I am quite new to Scala and stumbled across following problem:
what is Scala equivalent of function's static variable ?
void foo()
{
static int x = 5;
x++;
printf("%d", x);
}
EDIT:
What I want to achieve is a kind of function call counter - I want to check how many times my function has been executed and in the same time limit the visibility of this counter so that it can't be modified from outside.
Here is a block of code that has similar effect:
scala> object f extends Function0[Unit] {
| var x = 0;
| def apply = {
| x = x + 1;
| println(x);
| }
| }
defined module f
scala> f()
1
scala> f()
2
Although I must stress that this is a very bad practice since it kills referential transparency.
If you really need this behavior consider this:
type State = Int
def f(state: State) = {
val newState = state + 1
println(state);
newState;
}
Scala has no equivalent to the local static variables of C++. In Scala, scoping rules are more consistent than in C++ or Java - what is defined within a block, goes out of scope when the block is exited. As others noted, a local static variable would be a side effect, which is not desirable in functional programming.
Scala, being a hybrid OO/functional language, makes it possible to write in imperative style, but prefers and encourages functional style (e.g. by making immutable collections the default choice). Local static variables, apart from representing a side effect per se, are absent in Java too, which is one more reason not to provide them in Scala.
To get the equivalent of a C++ local static variable in Scala:
import scala.collection.parallel.mutable
import scala.reflect._
import scala.reflect.runtime.universe._
object StaticLocal {
private val classes = new mutable.ParHashSet[String]
private val variables = new mutable.ParHashMap[String, AnyVal]
}
import Numeric._
class StaticLocal[T <: AnyVal](value:T)(implicit tag: TypeTag[T], num: Numeric[T]){
val name = this.getClass + "." + tag.toString() ;
private var inited = false
if (!inited) {
inited = true
if (!StaticLocal.classes.contains(name)) {
StaticLocal.classes += name
StaticLocal.variables += name -> value.asInstanceOf[AnyVal]
}
}
def get():T = {StaticLocal.variables.get(name) match { case x:Some[Int] => (x.get).asInstanceOf[T] ; case None => throw new Exception("Not found:" + name) }}
def set(value:AnyVal) { StaticLocal.variables.put(name, value)}
def +(v:StaticLocal[T]):T = { num.plus(this.get, v.get) }
def +(v:T):T = { num.plus(this.get, v) }
def +=(v:T):Unit = { set(num.plus(this.get, v)) }
def +=(v:StaticLocal[T]):Unit = { set(num.plus(this.get, v.get)) }
override def toString() = { get.toString}
implicit def StaticLocalWrapper(s: StaticLocal[T]):T = s.get
}
Then in the method:
def foo():Unit
{
object x extends StaticLocal( 5 )
x += 1
println( x )
}
This will work just like in c++, including when the method or owning class instance goes out of scope (albeit yet with a performance penalty).
Not-thread-safe as it stands.