I am trying to identify a nasty bug where an object spontaneously gets corrupted while being inside a map, and after some hours of debugging, I think I might not have fully grasped the idea of std::shared_ptr.
Here is the Context:
Inside a method, I declare a std::shared_ptr and initialize it to point to a clone of the current object (created by new). Then - after some modifications to the object - I call the same method on that pointer (recursively).
Inside the next recursion of the method, the decision is made to insert this object into a std::unordered_map (which is a class-attribute, so it is available on all recursion levels).
This is some pseudo-code to illustrate what I mean:
class A
{
...
void DoSomething(void); // the recursive function
A* Clone(void) const { return new A(this); } // the clone method
...
static std::unordered_map<std::shared_ptr<A>,int> myMap{};
};
void A::DoSomething(void)
{
...
if (condition) myMap.insert({this,5}); // in a deeper recursive call, condition is true
...
std::shared_ptr<A> pA(Clone()); // make a copy
pA->... // modify it
pA->DoSomething(); // here is the recursive call
...
}
Problem:
Sometimes, the object behind the pointer inside the std::unordered_map is destroyed, and it seems like this happens when the original std::shared_ptr goes out of scope.
My (tentative) understanding: Calling a method of the object the std::shared_ptr points to does not increase the reference count - inside the called method, I have access to this, which is the ptr the std::shared_ptr points to, but what I do with this in there is not affecting the original std::shared_ptr.
To verify this, I added code to make an extra clone into an extra std::shared_ptr, right at the moment of insertion into the map, and then everything works fine (just slower, and uses double the memory, which is both an issue - class A has a lot of complex data).
Question: Is my understanding correct? If not, how would I call a method of a std::shared_ptr so that the this inside the method is still the 'std::shared_ptr'? Or is this not possible, and I have to use another design?
Regarding duplicates: Should we pass a shared_ptr by reference or by value? seems to point that way, but is about passing parameters by value or by reference, which is not a choice I have with the this pointer.
Your understanding is basically correct. This line is your problem:
if (condition) myMap.insert({this,5});
Because of the raw this a completely independent shared_ptr with its own independent reference count is created in this line. Later in the outer recursion level at the end of DoSomething() the original shared_ptr pA goes out of scope, its refcount drops to 0, the object is destroyed and the second shared_ptr in the map starts to dangle.
Solution 1
You can solve it with std::enable_shared_from_this:
class A : public std::enable_shared_from_this<A> { ... }
// Btw: Lose the void pseudo-parameter. This is not C. ;)
void A::DoSomething()
{
if (condition) {
myMap.insert({shared_from_this(), 5});
}
}
Potential Solution 2
From the code snippets you show I find it highly questionable that you need shared_ptr at all. Nothing of what you show indicates shared ownership. If that’s indeed the case, switch to unique_ptrs and std::move() them around. That gets rid of the problem, too.
Related
Suppose that we have a STL container with some objects, and these objects can post functions to a queue to be executed later. But before these functions get executed, the container gets modified in such a way that pointers pointing to that object are invalidated. Let me illustrate with an example:
#include <vector>
#include <functional>
class Class_A
{
public:
std::function<void()> getFunctionToRunLater()
{
return [this] () { somethingToDo(); moreThingsToDo(); };
// Returns a lambda function that captures the this pointer,
// so it can access the object's methods and variables.
}
void somethingToDo();
void moreThingsToDo();
}
int main()
{
std::vector<Class_A> vec;
vec.push_back(Class_A());
std::function<void()> pendingFunction = vec.back().getFunctionToRunLater();
// More code...
pendingFunction();
}
Everything fine, right? We get a function the object wants to run and, after some logic, we execute that function. This represents posting functions to a queue and them execute all functions in the queue. But now look at this one:
int main()
{
std::vector<Class_A> vec;
vec.push_back(Class_A());
std::function<void()> pendingFunction = vec.back().getFunctionToRunLater();
// More code...
vec.reserve(1000);
// This will surely reallocate the vector, invalidating all pointers.
pendingFunction();
// And now my program is going straight down to hell, right?
}
Is my assumption correct? What will happen if the lambda doesn't capture anything at all, will the program still be logically broken? And what about if the lambda doesn't capture the this pointer, but rather some other class field specifically?
The existing answer already mentions that the pointer can be invalidated. One way to avoid the problem is, as already mentioned, changing the ownership of *this by either shared_ptr, unique_ptr or a copy. However, this comes at extra cost (dynamic allocation or extra copy) and sometimes is simply not possible (non-copyable types).
Instead, I would suggest a design that doesn't lead to this problem in the first place, i.e. not making the this pointer part of the lambda's state. Take the object as a parameter:
std::function<void(Class_A&)> getFunctionToRunLater()
{
return [] (Class_A& obj) { obj.somethingToDo(); obj.moreThingsToDo(); };
}
If copying the object is a possibility, then you can capture *this by value: (requires C++17)
return [*this] { somethingToDo(); moreThingsToDo(); }
This copies the whole object into the closure to avoid out-of-lifetime access to the original object.
Yes this program is likely to have problems. C++ does not protect you from invalidating pointers, and as you've highlighted the objects in your vector will potentially move address when the vector resizes, which will cause problems if you try to run your lambda.
You will probably be unable to compile the program without capturing this. You will also end up with issues if you try to capture references or pointers to any part of your object without being sure the memory being pointed at will not move.
It pays to be cautious, as a program like this is not guaranteed to crash even if you have a bug, as the old data may still exist in memory even when your vector resizes. So if you try capturing this and don't see any issues at runtime it does not mean that your program is correct.
For a straight forward solution, I'd look at allocating your objects on the heap using one of the smart pointer types such as std::unique_ptr or std::shared_ptr.
I've a multithreaded C++ application that could call from any thread a function like the following, to get an Object from a list/vector.
class GlobalClass{
public:
MyObject* GlobalClass::getObject(int index) const
{
/* mutex lock & unlock */
if (m_list.hasValueAt(index))
return m_list[index];
else
return 0;
}
List<MyObject*> m_list;
};
//Thread function
MyObject* obj = globalClass->getObject(0);
if (!obj) return;
obj->doSomething();
Note: the scope here is to understand some best practice related to function returns by reference, value or pointer, so forgive some pseudo-code or missing declarations (I make use of lock/unlock, GlobalClass is a global singleton, etc...).
The issue here is that if the MyObject at that index in deleted inside GlobalClass, at a certain point I'm using a bad pointer (obj).
So I was thinking about returning a copy of the oject:
MyObject GlobalClass::getObject(int index) const
{
/* mutex lock & unlock */
if (m_list.hasValueAt(index))
return MyObject(*m_list[index]);
else
return MyObject();
}
The issue here is that the object (MyObject) being returned is a large enough object that returning a copy is not efficient.
Finally, I would like to return a reference to that object (better a const reference):
const MyObject& GlobalClass::getObject(int index) const
{
/* mutex lock & unlock */
if (m_list.hasValueAt(index))
return *m_list[index];
else{
MyObject* obj = new MyObject();
return *obj ;
}
}
Considering that my list couldn't cointain the object at that index, I'm introducing a memory leak.
What's the best solution to deal with this?
Must I fall back in returning a copy even if is less efficient or is there something I'm missing in returning a reference?
You have multiple choices:
Use a std::shared_ptr if "Get" pass the owning of the object to the caller. This way the object cannot get out of scope. Of course the caller is unaware when it happens.
Use a std::weak_ptr. This has the same meaning of 1., but the ptr can be reset. In this case the caller can detect if the object was deleted.
Use std::optional as suggested in a comment, and return a copy or a reference. The use of a reference type as argument of optional doesn't avoid the problem of the object being deleted so the reference can become invalid as well. A copy would avoid this, but it may be too expensive, as said.
Reading through the lines, you seems to suggest that the caller will use the pointer immediately after the call, and for a limited span of time. So 1. and 2. are equivalent and seems to fit your needs.
See this introduction to smart pointers for more details.
If you want to avoid copying the object, there are only two possible cases:
The m_list entry that is returned by getObject is/can be deleted concurrently by another thread. If you don't copy that object beforehand, there is nothing you can do within getObject to prevent another thread from suddenly having a reference/pointer dangle. However, you could make each entry of m_list be a std::shared_ptr<MyObject> and return that directly. The memory management will happen automatically (but beware of the potential overhead in the reference counting of shared_ptr, as well as the possibility of deadlocks).
You have (or add) some mechanism to ensure that objects can only be deleted from m_list if no other thread currently holds some pointer/reference to them. This very much depends on your algorithm, but it might e.g. be possible to mark objects for deletion only and then delete them later in a synchronous section.
Your issues seems to stem from the fact that your program is multithreaded - another way forward (and for raw pointer or the std::optional reference returning version: only way forward, perhaps short of a complete redesign), is that you need to expose the mutex to outside the function scope to accomplish what you need. This you can accomplish in multiple ways, however the most simple way to illustrate this is the following:
/*mutex lock*/
const MyObject& obj = globalClass.get(index);
/*do stuff with obj*/
/*mutex unlock*/
I'm trying to implement the Observer pattern, but i don't want observers being responsible for my program safety by maintaining the reference list in ObservableSubject.
Meaning that when the Observer object lifetime ends, I dont want to explicitly call ObservervableSubject::removeObserver(&object).
I have come up with an idea to use use pointer references in ObservableSubject.
My question are: Is implementation described above and attempted below possible?
And what is happening in my program, how do i prevent dereferencing trash?
Apriori excuse: This is an attempt at understanding C++, not something that should have actual use or be prefered over other implementations.
My solution attempt:
// Example program
#include <iostream>
#include <string>
#include <vector>
class ObserverInterface {
public:
virtual ~ObserverInterface() {};
virtual void handleMessage() = 0;
};
class ObservableSubject
{
std::vector<std::reference_wrapper<ObserverInterface*>> listeners;
public:
void addObserver(ObserverInterface* obs)
{
if (&obs)
{
// is this a reference to the copied ptr?
// still, why doesnt my guard in notify protect me
this->listeners.push_back(obs);
}
}
void removeObserver(ObserverInterface* obs)
{
// todo
}
void notify()
{
for (ObserverInterface* listener : this->listeners)
{
if (listener)
{
listener->handleMessage();
}
}
}
};
class ConcreteObserver : public ObserverInterface {
void handleMessage()
{
std::cout << "ConcreteObserver: I'm doing work..." << std::endl;
}
};
int main()
{
ObservableSubject o;
{
ConcreteObserver c;
o.addListener(&c);
}
o.notify();
std::cin.get();
}
Line in ObservableSubject::notify() : Listener->handleMessage() throws the following exception:
Exception thrown: read access violation.
listener->**** was 0xD8BF48B. occurred
Your program has undefined behavior.
ObservableSubject o;
{
ConcreteObserver c;
o.addListener(&c); // Problem
}
c gets destructed when the scope ends. You end up storing a stale pointer in the list of listeners of o.
You can resolve the problem by defining c in the same scope as o or by using dynamically allocated memory.
ObservableSubject o;
ConcreteObserver c;
o.addListener(&c);
or
ObservableSubject o;
{
ConcreteObserver* c = new ConcreteObserver;
o.addListener(c);
}
When you use dynamically allocated memory, the additional scope is not useful. You might as well not use it.
ObservableSubject o;
ConcreteObserver* c = new ConcreteObserver;
o.addListener(c);
If you choose to use the second approach, make sure to deallocate the memory. You need to add
delete c;
before the end of the function.
Update, in response to OP's comment
You said:
Maybe i wasn't clear. Solving the lifetime/stale pointer problem was the intention of my solution. I know i have no problems if i have properly managed lifetime, or if i add detachObserver option on Observer destruction. I want to somehow be able to tell from the ObservableSubject if his list of Observers was corrupted, without the Observer explicitly telling that.
Since dereferencing an invalid pointer is cause for undefined behavior, it is essential that you track the lifetime of observers and make sure to update the list of observers when necessary. Without that, you are courting undefined behavior.
Note, I don't recommend the following approach, but I think it meets your requirements. You have a duplicated observer list. One is under control of the Observers, and the other, using weak pointers, is handled by the Observable object.
Make the Observer constructors private and use an ObserverFactory (which is their friend) to obtain a std::shared_ptr<Observer>. The factory has a map from raw pointers to reference wrappers to the associated shared pointer.
The listeners list becomes std::vector<std::weak_ptr<Observer>>. On list traversal, you try to lock the weak_ptr; if it succeeds, handle the message; if it fails, that is, you get nullptr, remove the weak pointer from the list.
When the listener no longer wants to listen, it tells the Factory to do a reset on its shared pointer and remove from the map. This step is rather ugly, as it is just a fancy delete this, normally a code smell.
I believe you can also do this with std::shared_from_this.
The plan is you move the maintenance away from the ObservableSubject back into the Observers.
// is this a reference to the copied ptr?
Yes, it is. It invokes undefined behaviour because the obs pointer variable goes out of scope at the end of the function, resulting in a dangling reference.
The whole idea doesn’t gain you anything. Even if you make the ref-to-pointer approach work correctly, you are depending on one thing: That that exact pointer variable is set to nullptr once the object dies. Essentially that’s the same problem as ensuring that no dangling pointers are held in listeners.
For a heap object: How do you make sure that nobody deletes the object through a different pointer? Or forgets to null the registered pointer? It’s even worse for stack objects like in your example. The object goes out of scope and dies automatically. There is no opportunity to null anything unless you introduce an additional pointer variable that you’d have to manage manually.
You could consider two general alternatives to your approach:
Make the relation bidirectional. Then whoever dies first (observable or observer) can notify the other party abouts its death in the destructor.
If you don’t like the bidirectionality a central, all-knowing orchestrator that decouples oberservers and observables works, too. Of course that introduces some kind of global state.
Real-life implementations usually go in the general direction of leveraging C++ destructors for deregistration. Have a look at Qt’s signal/slot mechanism, for example.
I have some code that currently uses raw pointers, and I want to change to smart pointers. This helps cleanup the code in various ways. Anyway, I have factory methods that return objects and its the caller's responsibility to manager them. Ownership isn't shared and so I figure unique_ptr would be suitable. The objects I return generally all derive from a single base class, Object.
For example,
class Object { ... };
class Number : public Object { ... };
class String : public Object { ... };
std::unique_ptr<Number> State::NewNumber(double value)
{
return std::unique_ptr<Number>(new Number(this, value));
}
std::unique_ptr<String> State::NewString(const char* value)
{
return std::unique_ptr<String>(new String(this, value));
}
The objects returned quite often need to be passed to another function, which operates on objects of type Object (the base class). Without any smart pointers the code is like this.
void Push(const Object* object) { ... } // push simply pushes the value contained by object onto a stack, which makes a copy of the value
Number* number = NewNumber(5);
Push(number);
When converting this code to use unique_ptrs I've run into issues with polymorphism. Initially I decided to simply change the definition of Push to use unique_ptrs too, but this generates compile errors when trying to use derived types. I could allocate objects as the base type, like
std::unique_ptr<Object> number = NewNumber(5);
and pass those to Push - which of course works. However I often need to call methods on the derived type. In the end I decided to make Push operate on a pointer to the object stored by the unique_ptr.
void Push(const Object* object) { ... }
std::unique_ptr<Object> number = NewNumber(5);
Push(number.get());
Now, to the reason for posting. I'm wanting to know if this is the normal way to solve the problem I had? Is it better to have Push operate on the unique_ptr vs the object itself? If so how does one solve the polymorphism issues? I would assume that simply casting the ptrs wouldn't work. Is it common to need to get the underlying pointer from a smart pointer?
Thanks, sorry if the question isn't clear (just let me know).
edit: I think my Push function was a bit ambiguous. It makes a copy of the underlying value and doesn't actually modify, nor store, the input object.
Initially I decided to simply change the definition of Push to use
unique_ptrs too, but this generates compile errors when trying to use
derived types.
You likely did not correctly deal with uniqueness.
void push(std::unique_ptr<int>);
int main() {
std::unique_ptr<int> i;
push(i); // Illegal: tries to copy i.
}
If this compiled, it would trivially break the invariant of unique_ptr, that only one unique_ptr owns an object, because both i and the local argument in push would own that int, so it is illegal. unique_ptr is move only, it's not copyable. It has nothing to do with derived to base conversion, which unique_ptr handles completely correctly.
If push owns the object, then use std::move to move it there. If it doesn't, then use a raw pointer or reference, because that's what you use for a non-owning alias.
Well, if your functions operate on the (pointed to) object itself and don't need its address, neither take any ownership, and, as I guess, always need a valid object (fail when passed a nullptr), why do they take pointers at all?
Do it properly and make them take references:
void Push(const Object& object) { ... }
Then the calling code looks exactly the same for raw and smart pointers:
auto number = NewNumber(5);
Push(*number);
EDIT: But of course no matter if using references or pointers, don't make Push take a std::unique_ptr if it doesn't take ownership of the passed object (which would make it steal the ownership from the passed pointer). Or in general don't use owning pointers when the pointed to object is not to be owned, std::shared_ptr isn't anything different in this regard and is as worse a choice as a std::unique_ptr for Push's parameter if there is no ownership to be taken by Push.
If Push does not take owenrship, it should probably take reference instead of pointer. And most probably a const one. So you'll have
Push(*number);
Now that's obviously only valid if Push isn't going to keep the pointer anywhere past it's return. If it does I suspect you should try to rethink the ownership first.
Here's a polymorphism example using unique pointer:
vector<unique_ptr<ICreature>> creatures;
creatures.emplace_back(new Human);
creatures.emplace_back(new Fish);
unique_ptr<vector<string>> pLog(new vector<string>());
for each (auto& creature in creatures)
{
auto state = creature->Move(*pLog);
}
I have objects which create other child objects within their constructors, passing 'this' so the child can save a pointer back to its parent. I use boost::shared_ptr extensively in my programming as a safer alternative to std::auto_ptr or raw pointers. So the child would have code such as shared_ptr<Parent>, and boost provides the shared_from_this() method which the parent can give to the child.
My problem is that shared_from_this() cannot be used in a constructor, which isn't really a crime because 'this' should not be used in a constructor anyways unless you know what you're doing and don't mind the limitations.
Google's C++ Style Guide states that constructors should merely set member variables to their initial values. Any complex initialization should go in an explicit Init() method. This solves the 'this-in-constructor' problem as well as a few others as well.
What bothers me is that people using your code now must remember to call Init() every time they construct one of your objects. The only way I can think of to enforce this is by having an assertion that Init() has already been called at the top of every member function, but this is tedious to write and cumbersome to execute.
Are there any idioms out there that solve this problem at any step along the way?
Use a factory method to 2-phase construct & initialize your class, and then make the ctor & Init() function private. Then there's no way to create your object incorrectly. Just remember to keep the destructor public and to use a smart pointer:
#include <memory>
class BigObject
{
public:
static std::tr1::shared_ptr<BigObject> Create(int someParam)
{
std::tr1::shared_ptr<BigObject> ret(new BigObject(someParam));
ret->Init();
return ret;
}
private:
bool Init()
{
// do something to init
return true;
}
BigObject(int para)
{
}
BigObject() {}
};
int main()
{
std::tr1::shared_ptr<BigObject> obj = BigObject::Create(42);
return 0;
}
EDIT:
If you want to object to live on the stack, you can use a variant of the above pattern. As written this will create a temporary and use the copy ctor:
#include <memory>
class StackObject
{
public:
StackObject(const StackObject& rhs)
: n_(rhs.n_)
{
}
static StackObject Create(int val)
{
StackObject ret(val);
ret.Init();
return ret;
}
private:
int n_;
StackObject(int n = 0) : n_(n) {};
bool Init() { return true; }
};
int main()
{
StackObject sObj = StackObject::Create(42);
return 0;
}
Google's C++ programming guidelines have been criticized here and elsewhere again and again. And rightly so.
I use two-phase initialization only ever if it's hidden behind a wrapping class. If manually calling initialization functions would work, we'd still be programming in C and C++ with its constructors would never have been invented.
Depending on the situation, this may be a case where shared pointers don't add anything. They should be used anytime lifetime management is an issue. If the child objects lifetime is guaranteed to be shorter than that of the parent, I don't see a problem with using raw pointers. For instance, if the parent creates and deletes the child objects (and no one else does), there is no question over who should delete the child objects.
KeithB has a really good point that I would like to extend (in a sense that is not related to the question, but that will not fit in a comment):
In the specific case of the relation of an object with its subobjects the lifetimes are guaranteed: the parent object will always outlive the child object. In this case the child (member) object does not share the ownership of the parent (containing) object, and a shared_ptr should not be used. It should not be used for semantic reasons (no shared ownership at all) nor for practical reasons: you can introduce all sorts of problems: memory leaks and incorrect deletions.
To ease discussion I will use P to refer to the parent object and C to refer to the child or contained object.
If P lifetime is externally handled with a shared_ptr, then adding another shared_ptr in C to refer to P will have the effect of creating a cycle. Once you have a cycle in memory managed by reference counting you most probably have a memory leak: when the last external shared_ptr that refers to P goes out of scope, the pointer in C is still alive, so the reference count for P does not reach 0 and the object is not released, even if it is no longer accessible.
If P is handled by a different pointer then when the pointer gets deleted it will call the P destructor, that will cascade into calling the C destructor. The reference count for P in the shared_ptr that C has will reach 0 and it will trigger a double deletion.
If P has automatic storage duration, when it's destructor gets called (the object goes out of scope or the containing object destructor is called) then the shared_ptr will trigger the deletion of a block of memory that was not new-ed.
The common solution is breaking cycles with weak_ptrs, so that the child object would not keep a shared_ptr to the parent, but rather a weak_ptr. At this stage the problem is the same: to create a weak_ptr the object must already be managed by a shared_ptr, which during construction cannot happen.
Consider using either a raw pointer (handling ownership of a resource through a pointer is unsafe, but here ownership is handled externally so that is not an issue) or even a reference (which also is telling other programmers that you trust the referred object P to outlive the referring object C)
A object that requires complex construction sounds like a job for a factory.
Define an interface or an abstract class, one that cannot be constructed, plus a free-function that, possibly with parameters, returns a pointer to the interface, but behinds the scenes takes care of the complexity.
You have to think of design in terms of what the end user of your class has to do.
Do you really need to use the shared_ptr in this case? Can the child just have a pointer? After all, it's the child object, so it's owned by the parent, so couldn't it just have a normal pointer to it's parent?