I'd like to share an evmap, a lock-free, eventually consistent, concurrent multi-value map, across all threads in a Rust program.
Naively, it would look like this:
#[macro_use]
extern crate lazy_static;
extern crate evmap;
use std::collections::hash_map::RandomState;
lazy_static! {
static ref MAP: (evmap::ReadHandle<u32, u32, (), RandomState>,
evmap::WriteHandle<u32, u32, (), RandomState>) = evmap::new();
}
fn main() {
println!("Hello, world!");
MAP.1.clear();
}
This gives:
error[E0277]: the trait bound `std::cell::Cell<()>: std::marker::Sync` is not satisfied in `(evmap::ReadHandle<u32, u32>, evmap::WriteHandle<u32, u32>)`
--> src/main.rs:8:1
|
8 | / lazy_static! {
9 | | static ref MAP: (evmap::ReadHandle<u32, u32, (), RandomState>,
10 | | evmap::WriteHandle<u32, u32, (), RandomState>) = evmap::new();
11 | | }
| |_^ `std::cell::Cell<()>` cannot be shared between threads safely
|
= help: within `(evmap::ReadHandle<u32, u32>, evmap::WriteHandle<u32, u32>)`, the trait `std::marker::Sync` is not implemented for `std::cell::Cell<()>`
= note: required because it appears within the type `std::marker::PhantomData<std::cell::Cell<()>>`
= note: required because it appears within the type `evmap::ReadHandle<u32, u32>`
= note: required because it appears within the type `(evmap::ReadHandle<u32, u32>, evmap::WriteHandle<u32, u32>)`
= note: required by `lazy_static::lazy::Lazy`
= note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
I think this is complaining about the () returned inside evmap::new():
pub fn new<K, V>(
) -> (ReadHandle<K, V, (), RandomState>, WriteHandle<K, V, (), RandomState>) where
K: Eq + Hash + Clone,
V: Eq + ShallowCopy,
Can it be done?
Can [placing a ReadHandle / WriteHandle directly in a lazy static variable] be done?
No. As the error message states:
std::cell::Cell<()> cannot be shared between threads safely
You are attempting to place a type that will fail when used in a multithreaded context in a static variable, which must be thread-safe.
Can [placing a ReadHandle / WriteHandle in a lazy static variable at all] be done?
Yes, but you have to use something to synchronize access, such as a Mutex or RwLock:
#[macro_use]
extern crate lazy_static;
extern crate evmap;
use std::collections::hash_map::RandomState;
use std::sync::Mutex;
type ReadHandle = evmap::ReadHandle<u32, u32, (), RandomState>;
type WriteHandle = evmap::WriteHandle<u32, u32, (), RandomState>;
lazy_static! {
static ref MAP: (Mutex<ReadHandle>, Mutex<WriteHandle>) = {
let (r, w) = evmap::new();
(Mutex::new(r), Mutex::new(w))
};
}
fn main() {
MAP.1.lock().unwrap().clear(1);
}
See also:
How do I create a global, mutable singleton?
How can I guarantee that a type that doesn't implement Sync can actually be safely shared between threads?
Related
I am using rust-aws-lambda project. My use case is with Lambda as an API Gateway Proxy. Additionally I am using closures as explained in this example, as I want to use shared resources such as clients and secret value (retrieved from AWS Secrets Manager) in my lambda code.
The issue I am struggling with for long time is how I can abstract the closures into another helper function. I want the function to return the outer closure in this case, for example. I have tryd but I cannot get Rust compiler to let me do this.
The both closures are within main function. Per the listed example, my inner closure is defined something like this:
let handler_func_closure = move |event: ApiProxyRequest| async move {
...
Ok::<_, Error>(success_resp)
};
These are my relevant imports in the code:
use lambda_http::{service_fn, Body, Error, IntoResponse, Request as ApiProxyRequest, RequestExt};
use lambda_http::tower::util::ServiceFn;
I have below closure which I have defined in async main function in main.rs, which is working for me so far.
let outer_closure = move |event: ApiProxyRequest| async move {
match handler_func_closure(event).await {
Ok(s) => Ok(s.into_response()),
Err(e) => Ok(http::Response::builder()
.header(http::header::CONTENT_TYPE, "application/json")
.status(400)
.body(
serde_json::to_string(&json!({"error": &err.to_string()}))
.expect("unable to serialize serde_json::Value")
.into(),
)
.expect("unable to build http::Response")),
}
};
Where into_response() is satisfied by a struct implementing from trait IntoResponse.
So basically what I'm trying to do is make a function to return closure that can be passed in to service_fn. Below is my attempt so far - but it's not satisfy Rust compiler currently.
pub fn get_service_fn<T, T2, F, F2, I: IntoResponse>(handler_func_closure: T) -> ServiceFn<T2>
where T: Fn(http::Request<Body>) -> F + Send + Sync,
T2: Fn(lambda_http::Request) -> F2,
F2: Future<Output=crate::Result<http::Response<Body>>>,
F: Future<Output=crate::Result<I>> + Send {
let outer_closure = move |event: ApiProxyRequest| async move {
match handler_func_closure(event).await {
Ok(s) => Ok(s.into_response()),
Err(e) => failure(Box::new(e), None),
}
};
service_fn(outer_closure)
};
I have seen other question asked on how can function return closure, but I cannot seem to get it to work in this case. I am curious to know if anyone able to get this to work where an outer closure returned by a function can be passed in to service_fn.
I feel like I've spent countless hours in trying to figure this out, but I've been unable to make it work. I'm not entirely sure whether this is possible in Rust to be honest, but I'd be curious if anyone has an idea how to approach this.
Edit: This is the compiler error I'm getting in case it's helpful:
error[E0308]: mismatched types
--> src/bin/my_lambda/main.rs:197:16
|
185 | pub fn get_service_fn<T, T2, F, F2, I: IntoResponse>(handler_func_closure: T) -> ServiceFn<T2>
| -- this type parameter
...
190 | let outer_closure = move |event: ApiProxyRequest| async move {
| __________________________-________________________________________-
| | _________________________|
| ||
191 | || match handler_func_closure(event).await {
192 | || Ok(s) => Ok(s.into_response()),
193 | || Err(e) => failure(Box::new(e), None),
194 | || }
195 | || };
| || -
| ||_____|
| |______the found closure
| the found `async` block
196 |
197 | service_fn(outer_closure)
| ^^^^^^^^^^^^^ expected type parameter `T2`, found closure
|
::: /Users/rnag/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/mod.rs:61:43
|
61 | pub const fn from_generator<T>(gen: T) -> impl Future<Output = T::Return>
| ------------------------------- the found opaque type
|
= note: expected type parameter `T2`
found closure `[closure#src/bin/my_lambda/main.rs:190:25: 195:6]`
= help: every closure has a distinct type and so could not always match the caller-chosen type of parameter `T2`
I am trying to add a simple u8 to my Substrate Runtime Module:
decl_storage! {
trait Store for Module<T: Trait> as TemplateModule {
MyByte: u8;
}
}
However, I get a compiler error that it does not implement Parity Codec's Encode or Decode:
error[E0277]: the trait bound `u8: _IMPL_DECODE_FOR_Event::_parity_codec::Encode` is not satisfied
--> /Users/shawntabrizi/Documents/GitHub/substrate-package/substrate-node-template/runtime/src/template.rs:23:1
|
23 | / decl_storage! {
24 | | trait Store for Module<T: Trait> as TemplateModule {
25 | | MyByte: u8;
26 | | }
27 | | }
| |_^ the trait `_IMPL_DECODE_FOR_Event::_parity_codec::Encode` is not implemented for `u8`
A similar problem occurs when I try to store a u8 in a Substrate Smart Contract using ink!:
contract! {
struct MyContract {
value: storage::Value<u8>,
}
...
}
Error:
error[E0277]: the trait bound `u8: parity_codec::codec::Encode` is not satisfied
--> src/lib.rs:26:1
|
26 | / contract! {
27 | | struct MyContract {
28 | | value: storage::Value<u8>,
29 | | }
... |
49 | | }
50 | | }
| |_^ the trait `parity_codec::codec::Encode` is not implemented for `u8`
Why is that, and what can I do to resolve the issue?
Today, the parity_codec does not support Encoding of u8 due to avoid a type collision, since Vec<u8> is a special case from Vec<T>.
See: https://github.com/paritytech/parity-codec/issues/47
gavofyork:
Because it would otherwise make the two encodings: Vec<u8> and Vec<T: Codec> clash.
It is possible this could be fixed in the future with additional Rust features, but for now, you will need to store your single bytes as [u8; 1] and work with that type.
Substrate Runtime Module
One hacky solution for a Substrate Runtime Module looks something like this:
use support::{decl_module, decl_storage, decl_event, StorageValue, dispatch::Result};
use system::ensure_signed;
pub trait Trait: system::Trait {
type Event: From<Event<Self>> + Into<<Self as system::Trait>::Event>;
}
type U8 = [u8; 1];
decl_storage! {
trait Store for Module<T: Trait> as TemplateModule {
MyByte get(my_byte): U8;
}
}
decl_module! {
pub struct Module<T: Trait> for enum Call where origin: T::Origin {
fn deposit_event<T>() = default;
pub fn set_my_byte(origin, input: U8) -> Result {
let who = ensure_signed(origin)?;
<MyByte<T>>::put(input);
Self::deposit_event(RawEvent::MyByteStored(input, who));
Ok(())
}
pub fn add_to_byte(origin, input: U8) -> Result {
let who = ensure_signed(origin)?;
let my_byte = Self::my_byte()[0];
let my_new_byte = my_byte.checked_add(input[0]).ok_or("Overflow")?;
<MyByte<T>>::put([my_new_byte]);
Self::deposit_event(RawEvent::MyByteStored([my_new_byte], who));
Ok(())
}
}
}
decl_event!(
pub enum Event<T> where AccountId = <T as system::Trait>::AccountId {
MyByteStored(U8, AccountId),
}
);
Where we assign a new type type U8 = [u8; 1];. The choice of our new type name is important since it will trick the Polkadot UI to treat this value simply as a u8 for any input/output fields it generates. If you try to use a custom type like type Byte = [u8; 1], the UI will ask you to import the definition of that custom type. If you try to use [u8; 1] directly, the Polkadot UI will not know how to render the input/output of that value.
Furthermore, as of the time of writing this post, the decl_event! macro has an issue depositing a [u8; 1] directly due to pattern matching.
Note that you will need to treat this type like an array when you use it. add_to_byte() shows an example of that. So ultimately, you need to extract the first item of the array to extract the byte, and you need to wrap your byte in an array to set a U8:
let my_byte = Self::my_byte()[0];
...
<MyByte<T>>::put([my_new_byte]);
Other solutions may involve using other types which are natively supported like Vec<u8> or u16, and doing the appropriate checks on your runtime that this is treated like a single u8, but the UI will not know better.
Substrate Smart Contracts
I have not found a great solution for ink! yet, but you should be able to use [u8; 1] directly in all of your code. Again, you will need to treat it like an array for getters and setters. But when generating the ABI you will need to manually change the instances of [u8; 1] to u8 to trick the UI to do what you want.
I used mocker to test my project. I have no idea what kind of type conversion is needed by say_hello_brother. The simplified code list below:
lib.rs
#![feature(plugin, custom_derive)]
#![plugin(mockers_macros)]
#[cfg(test)]
extern crate mockers;
use mockers::Scenario;
#[derive(Mock)]
trait SayHello {
fn hello(&self);
}
// assume `SayHello` is a service and worked on multiple threads
fn say_hello_brother<T: SayHello + Sync>(brother: &'static T) {
brother.hello()
}
#[test]
fn test_sya_hello() {
let scenario = Scenario::new();
let mock = scenario.create_mock_for::<SayHello>();
say_hello_brother(&mock)
}
Cargo.toml
[package]
name = "mock"
version = "0.1.0"
authors = ["llxxb"]
[dependencies]
mockers = "0.9.4"
mockers_macros = "0.9.4"
and error info:
error[E0277]: the trait bound `std::rc::Rc<std::cell::RefCell<mockers::ScenarioInternals>>: std::marker::Sync` is not satisfied in `SayHelloMock`
--> src\lib.rs:22:5
|
22 | say_hello_brother(&mock)
| ^^^^^^^^^^^^^^^^^ `std::rc::Rc<std::cell::RefCell<mockers::ScenarioInternals>>` cannot be shared between threads safely
|
= help: within `SayHelloMock`, the trait `std::marker::Sync` is not implemented for `std::rc::Rc<std::cell::RefCell<mockers::ScenarioInternals>>`
= note: required because it appears within the type `SayHelloMock`
UPDATE After some trying, I succeed to attach Sync to SayHelloMock. the new lib.rs:
#![feature(plugin, custom_derive)]
#![plugin(mockers_macros)]
#[cfg(test)]
extern crate mockers;
use mockers::Scenario;
trait SayHello {
fn hello(&self);
}
mock! {
SayHelloMock,
self,
trait SayHello {
fn hello(&self);
}
}
unsafe impl Sync for SayHelloMock {}
// assume `SayHello` is a service and worked on multiple threads
fn say_hello_brother<T: SayHello + Sync>(brother: &'static T) {
brother.hello()
}
#[test]
fn test_sya_hello() {
let scenario = Scenario::new();
// not work
// let mock = scenario.create_mock::<SayHelloMock>();
// static MOCK: SayHelloMock = || { mock };
// say_hello_brother(&MOCK)
// not work yet
let mock : &'static SayHelloMock = &(scenario.create_mock::<SayHelloMock>());
say_hello_brother(mock)
}
But I still can't convert it to 'static, error info:
error[E0597]: borrowed value does not live long enough
--> src\lib.rs:38:41
|
38 | let mock : &'static SayHelloMock = &(scenario.create_mock::<SayHelloMock>());
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ temporary value does not live long enough
39 | say_hello_brother(mock)
40 | }
| - temporary value only lives until here
|
= note: borrowed value must be valid for the static lifetime...
You cannot use the mocker library for this purpose because the mocks it generates are not thread safe.
That being said, nothing restricts you from creating your own mocks which are thread-safe:
use std::sync::atomic::{AtomicUsize, Ordering};
#[derive(Debug, Default)]
struct MyMock {
call_count: AtomicUsize,
}
impl SayHello for MyMock {
fn hello(&self) {
self.call_count.fetch_add(1, Ordering::SeqCst);
}
}
However, your signature requires a &'static T, which is really probably a terrible idea. You will need to use something like a lazy-static:
#[macro_use]
extern crate lazy_static;
#[test]
fn test_say_hello() {
lazy_static! {
static ref MOCK: MyMock = MyMock::default();
};
say_hello_brother(&*MOCK);
assert_eq!(MOCK.call_count.load(Ordering::SeqCst), 1);
}
The question is a little vague, but if I understood correctly, you want to know how you could pass that mock to function say_hello_brother. The problem is that that function expects an object that implements the Sync trait as well. For this reason, you cannot convert mock to some other type to get the code to compile.
According to the documentation of mocker, you could try mocking two traits at once. Here is some pseudo-code that illustrates the idea:
mock! {
SayHelloMock,
self,
trait SayHello {
// trait methods here
},
self,
trait Sync {
// trait methods here
}
}
Then in your test, you would create a mock like this:
let mut mock = scenario.create_mock::<SayHelloMock>();
This is the code I am trying to execute:
fn my_fn(arg1: &Option<Box<i32>>) -> i32 {
if arg1.is_none() {
return 0;
}
let integer = arg1.unwrap();
*integer
}
fn main() {
let integer = 42;
my_fn(&Some(Box::new(integer)));
}
(on the Rust playground)
I get the following error in previous versions of Rust:
error[E0507]: cannot move out of borrowed content
--> src/main.rs:5:19
|
5 | let integer = arg1.unwrap();
| ^^^^ cannot move out of borrowed content
And in more modern versions:
error[E0507]: cannot move out of `*arg1` which is behind a shared reference
--> src/main.rs:5:19
|
5 | let integer = arg1.unwrap();
| ^^^^
| |
| move occurs because `*arg1` has type `std::option::Option<std::boxed::Box<i32>>`, which does not implement the `Copy` trait
| help: consider borrowing the `Option`'s content: `arg1.as_ref()`
I see there is already a lot of documentation about borrow checker issues, but after reading it, I still can't figure out the problem.
Why is this an error and how do I solve it?
Option::unwrap() consumes the option, that is, it accepts the option by value. However, you don't have a value, you only have a reference to it. That's what the error is about.
Your code should idiomatically be written like this:
fn my_fn(arg1: &Option<Box<i32>>) -> i32 {
match arg1 {
Some(b) => **b,
None => 0,
}
}
fn main() {
let integer = 42;
my_fn(&Some(Box::new(integer)));
}
(on the Rust playground)
Or you can use Option combinators like Option::as_ref or Option::as_mut paired with Option::map_or, as Shepmaster has suggested:
fn my_fn(arg1: &Option<Box<i32>>) -> i32 {
arg1.as_ref().map_or(0, |n| **n)
}
This code uses the fact that i32 is automatically copyable. If the type inside the Box weren't Copy, then you wouldn't be able to obtain the inner value by value at all - you would only be able to clone it or to return a reference, for example, like here:
fn my_fn2(arg1: &Option<Box<i32>>) -> &i32 {
arg1.as_ref().map_or(&0, |n| n)
}
Since you only have an immutable reference to the option, you can only return an immutable reference to its contents. Rust is smart enough to promote the literal 0 into a static value to keep in order to be able to return it in case of absence of the input value.
Since Rust 1.40 there is Option::as_deref, so now you can do:
fn my_fn(arg1: &Option<Box<i32>>) -> i32 {
*arg1.as_deref().unwrap_or(&0)
}
I'm learning OCaml, and I'm a bit confused with the immutability of variables. According to the book I'm reading, variables are immutable. So far so good, but why on Earth can I do this:
let foo = 42
let foo = 4242
What am I missing??
I think the best way to explain is with an example. Consider this code (executed in the OCaml REPL):
# let foo = 42;;
val foo : int = 42
# let return_foo () = foo;;
val return_foo : unit -> int = <fun>
# let foo = 24;;
val foo : int = 24
# return_foo ();;
- : int = 42
The above code does the following:
Binds 42 to the name foo.
Creates a function return_foo () that returns the value bound to foo.
Binds 24 to the name foo (which hides the previous binding of foo).
Calls the return_foo () function, which returns 42.
Compare this with the behaviour of a mutable value (created using ref in OCaml):
# let foo = ref 42;;
val foo : int ref = {contents = 42}
# let return_foo () = !foo;;
val return_foo : unit -> int = <fun>
# foo := 24;;
- : unit = ()
# return_foo ();;
- : int = 24
which:
Creates a mutable reference containing 42 and binds it to the name foo.
Creates a function return_foo () that returns the value stored in the reference bound to foo.
Stores 24 in the reference bound to foo.
Calls the return_foo () function, which returns 24.
The name foo is first bound to an immutable value 42 and later it is rebound to another immutable value 4242. You can even bind the same name to variables of different types. In OCaml we are talking not about mutability of a variable, but about a mutability of a value. For example, if you bind foo to an array of values, this would be the same name, but bound to a mutable data, so that the value of a variable can change in time. Finally, each new binding just hides the previous one, so the original foo is still bound to 42, but it is invisible and will garbage collected.
Maybe a little example will clarify the idea:
let example () =
let foo = 42 in (* 1 *)
let foo = 4242 in (* 2 *)
let foo = [|42|] in (* 3 *)
foo.(0) <- 56 (* 4 *)
It might be easier to have the following mental model:
(*1*) +--------+
+----> | 42 |
+------------+ | +--------+
| +----+
| foo +----+ +--------+
| | +----> | 4242 |
+---------+--+ (*2*) +--------+
|
| (*3*) +--------+
+------------> |[| 42 |]|
(*4*) +--------+
On lines 1 and 2 we just bind a variable foo to two different values. On line 3 we bind it to an array that contains one element. On line 4, we change the value, and foo is still bound to the same value, but the value contains different datum.
I hope I didn't confuse you even more ;)
The usual form of let is the let ... in expression, where you define a variable binding, which only exists in the inside of the body of the let. The body of the let is a new scope.
let x = 42 in (* body here *)
Here the "body" of the let is a new scope that is different from the one outside, with all the variables from the outside with an additional local variable x that is only defined in the body of this let.
Now you are talking about lets at the top level of the file. These look a little different syntactically (there is no in), but really they are the same, with the "body" being the rest of the file. So here you can think of the rest of the file after the let as a new scope, with x being a local variable of this scope. So your code is equivalent to this:
let foo = 42 in (
let foo = 4242 in (
(* rest of file *)
)
)
Here your inner let binds a local variable that has the same name as a variable that already exists in the outer scope. That doesn't matter. You are binding a new variable in an inner scope. If it happens to have the same name as a variable in an outer scope, then code in the inner scope referencing that name will refer to the innermost binding. The two variables, however, are completely independent.
In a C-like language, it would be something like this:
{
const int foo = 42;
{
const int foo = 4242;
// rest of code here
}
}
See? There is no assignment to any variables here.