Is there a usecase for nested classes? - c++

I've recently seen several people doing things like this here on Stackoverflow:
class A:
foo = 1
class B:
def blah(self):
pass
In other words, they have nested classes. This works (although people new to Python seem to run into problems because it doesn't behave like they thought it would), but I can't think of any reason to do this in any language at all, and certainly not in Python. Is there such a usecase? Why are people doing this? Searching for this it seems it's reasonably common in C++, is there a good reason there?

The main reason for putting one class in another is to avoid polluting the global namespace with things that are used only inside one class and therefore doesn't belong in the global namespace. This is applicable even to Python, with the global namespace being a namespace of a particular module. For example if you have SomeClass and OtherClass, and both of them need to read something in a specialized way, it is better to have SomeClass.Reader and OtherClass.Reader rather than SomeClassReader and OtherClassReader.
I have never encountered this in C++, though. It can be problematic to control access to the outer class' fields from a nested class. And it is also pretty common to have just one public class in a compilation unit defined in the header file and some utility classes defined in the CPP file (the Qt library is a great example of this). This way they aren't visible to "outsiders" which is good, so it doesn't make much sense to include them in the header. It also helps to increase binary compatibility which is otherwise a pain to maintain. Well, it's a pain anyway, but much less so.
A great example of a language where nested classes are really useful is Java. Nested classes there automatically have a pointer to the instance of the outer class that creates them (unless you declare the inner class as static). This way you don't need to pass "outer" to their constructors and you can address the outer class' fields just by their names.

It allows you to control the access of the nested class- for example, it's often used for implementation detail classes. In C++ it also has advantages in terms of when various things are parsed and what you can access without having to declare first.

I am not a big fan of python, but to me this type of decisions are more semantical than syntactical. If you are implementing a list, the class Node inside List is not a class in itself meant to be used from anywhere, but an implementation detail of the list. At the same time you can have a Node internal class inside Tree, or Graph. Whether the compiler/interpreter allows you to access the class or not is in a different thing. Programing is about writing specifications that the computer can follow and other programers can read, List.Node is more explicit in that Node is internal to List than having ListNode as a first level class.

In some languages, the nested class will have access to variables that are in scope within the outer class. (Similarly with functions, or with class-in-function nesting. Of course, function-in-class nesting just creates a method, which behaves fairly unsurprisingly. ;) )
In more technical terms, we create a closure.

Python lets you do a lot of things with functions (including lambdas) that in C++03 or Java you need a class for (although Java has anonymous inner classes, so a nested class doesn't always look like your example). Listeners, visitors, that kind of thing. A list comprehension is loosely a kind of visitor:
Python:
(foo(x) if x.f == target else bar(x) for x in bazes)
C++:
struct FooBar {
Sommat operator()(const Baz &x) const {
return (x.f == val) ? foo(x) : bar(x);
}
FooBar(int val) : val(val) {}
int val;
};
vector<Sommat> v(bazes.size());
std::transform(bazes.begin(), bazes.end(), v.begin(), FooBar(target));
The question that C++ and Java programmers then ask themselves is, "this little class that I'm writing: should it appear in the same scope as the big class that needs to use it, or should I confine it within the scope of the only class that uses it?"[*]
Since you don't want to publish the thing, or allow anyone else to rely on it, often the answer in these cases is a nested class. In Java, private classes can serve, and in C++ you can restrict classes to a TU, in which case you may no longer care too much what namespace scope the name appears in, so nested classes aren't actually required. It's just a style thing, plus Java provides some syntactic sugar.
As someone else said, another case is iterators in C++. Python can support iteration without an iterator class, but if you're writing a data structure in C++ or Java then you have to put the blighters somewhere. To follow the standard library container interface you'll have a nested typedef for it whether the class is nested or not, so it's fairly natural to think, "nested class".
[*] They also ask themselves, "should I just write a for loop?", but let's suppose a case where the answer to that is no...

In C++ at least, one major common use-case for nested classes is iterators in containers. For example, a hypothetical implementation might look something like this:
class list
{
public:
class iterator
{
// implementation code
};
class const_iterator
{
// implementation code
};
};
Another reason for nested classes in C++ would be private implementation details like node classes for maps, linked lists, etc.

"Nested classes" can mean two different things, which can be split into three different categories by intent. The first one is purely stylistic, the other two are used for practical purposes, and are highly dependent on the features language where they are used.
Nested class definitions for the sake of creating a new namespace and/or organizing your code better. For example, in Java this is accomplished through the use static nested classes, and it is suggested by the official documentation as a way to create more readable and maintainable code, and to logically group classes together. The Zen of Python, however, suggests that you nest code blocks less, thus discouraging this practice.
import this
In Python you'd much more often see the classes grouped in modules.
Putting a class inside another class as part of its interface (or the interface of the instances). First, this interface can be used by the implementation to aid subclassing, for example imagine a nested class HTML.Node which you can override in a subclass of HTML to alter the class used to create new node instances. Second, this interface might be used by the class/instance users, though this is not that useful unless you are in the third case described below.
In Python at least, you don't need to nest the definitions to achieve either of those, however, and it's probably very rare. Instead, you might see Node defined outside of the class and then node_factory = Node in the class definition (or a method dedicated to creating the nodes).
Nesting the namespace of the objects, or creating different contexts for different groups of objects. In Java, non-static nested classes (called inner classes) are bound to an instance of the outer class. This is very useful because it lets you have instances of the inner class that live inside different outer namespaces.
For Python, consider the decimal module. You can create different contexts, and have things like different precisions defined for each context. Each Decimal object can assigned a context on creation. This achieves the same as an inner class would, through a different mechanism. If Python supported inner classes, and Context and Decimal were nested, you'd have context.Decimal('3') instead of Decimal('3', context=context).
You could easily create a metaclass in Python that lets you create nested classes that live inside of an instance, you can even make it produce proper bound and unbound class proxies that support isinstance correctly through the use of __subclasscheck__ and __instancecheck__. However, it won't gain you anything over the other simpler ways to achieve the same (like an additional argument to __init__). It would only limit what you can do with it, and I have found inner classes in Java very confusing every time I had to use them.

In Python, a more useful pattern is declaration of a class inside a function or method. Declaration of a class in the body of another class, as people have noted in other answers, is of little use - yes, it does avoid pollution of the module namespace, but since there_is_ a module namespace at all, a few more names on it do not bother. Even if the extra classes are not intended to be instantiated directly by users of the module, putting then on the module root make their documentation more easily accessible to others.
However, a class inside a function is a completely different creature: It is "declared" and created each time the code containing the class body is run. This gives one the possibility of creating dynamic classes for various uses - in a very simple way. For example, each class created this way is in a different closure, and can have access to different instances of the variables on the containing function.
def call_count(func):
class Counter(object):
def __init__(self):
self.counter = 0
def __repr__(self):
return str(func)
def __call__(self, *args, **kw):
self.counter += 1
return func(*args, **kw)
return Counter()
And using it on the console:
>>> #call_count
... def noop(): pass
...
>>> noop()
>>> noop()
>>> noop.counter
2
>>> noop
<function noop at 0x7fc251b0b578>
So, a simple call_counter decorator could use a static "Counter" class, defined outside the function, and receiving func as a parameter to its constructor - but if you want to tweak other behaviors, like in this example, making repr(func) return the function representation, not the class representation, it is easier to be made this way.
.

Related

Best practices for a class with many members

Any opinions on best way to organize members of a class (esp. when there are many) in C++. In particular, a class has lots of user parameters, e.g. a class that optimizes some function and has number of parameters such as # of iterations, size of optimization step, specific method to use, optimization function weights etc etc. I've tried several general approaches and seem to always find something non-ideal with it. Just curious others experiences.
struct within the class
struct outside the class
public member variables
private member variables with Set() & Get() functions
To be more concrete, the code I'm working on tracks objects in a sequence of images. So one important aspect is that it needs to preserve state between frames (why I didn't just make a bunch of functions). Significant member functions include initTrack(), trackFromLastFrame(), isTrackValid(). And there are a bunch of user parameters (e.g. how many points to track per object tracked, how much a point can move between frames, tracking method used etc etc)
If your class is BIG, then your class is BAD.
A class should respect the Single Responsibility Principle , i.e. : A class should do only one thing, but should do it well. (Well "only one" thing is extreme, but it should have only one role, and it has to be implemented clearly).
Then you create classes that you enrich by composition with those single-role little classes, each one having a clear and simple role.
BIG functions and BIG classes are nest for bugs, and misunderstanding, and unwanted side effects, (especially during maintainance), because NO MAN can learn in minutes 700 lines of code.
So the policy for BIG classes is: Refactor, Composition with little classes targetting only at what they have do.
if i had to choose one of the four solutions you listed: private class within a class.
in reality: you probably have duplicate code which should be reused, and your class should be reorganized into smaller, more logical and reusable pieces. as GMan said: refactor your code
First, I'd partition the members into two sets: (1) those that are internal-only use, (2) those that the user will tweak to control the behavior of the class. The first set should just be private member variables.
If the second set is large (or growing and changing because you're still doing active development), then you might put them into a class or struct of their own. Your main class would then have a two methods, GetTrackingParameters and SetTrackingParameters. The constructor would establish the defaults. The user could then call GetTrackingParameters, make changes, and then call SetTrackingParameters. Now, as you add or remove parameters, your interface remains constant.
If the parameters are simple and orthogonal, then they could be wrapped in a struct with well-named public members. If there are constraints that must be enforced, especially combinations, then I'd implement the parameters as a class with getters and setters for each parameter.
ObjectTracker tracker; // invokes constructor which gets default params
TrackerParams params = tracker.GetTrackingParameters();
params.number_of_objects_to_track = 3;
params.other_tracking_option = kHighestPrecision;
tracker.SetTrackingParameters(params);
// Now start tracking.
If you later invent a new parameter, you just need to declare a new member in the TrackerParams and initialize it in ObjectTracker's constructor.
It all depends:
An internal struct would only be useful if you need to organize VERY many items. And if this is the case, you ought to reconsider your design.
An external struct would be useful if it will be shared with other instances of the same or different classes. (A model, or data object class/struct might be a good example)
Is only ever advisable for trivial, throw-away code.
This is the standard way of doing things but it all depends on how you'll be using the class.
Sounds like this could be a job for a template, the way you described the usage.
template class FunctionOptimizer <typename FUNCTION, typename METHOD,
typename PARAMS>
for example, where PARAMS encapsulates simple optimization run parameters (# of iterations etc) and METHOD contains the actual optimization code. FUNCTION describes the base function you are targeting for optimization.
The main point is not that this is the 'best' way to do it, but that if your class is very large there are likely smaller abstractions within it that lend themselves naturally to refactoring into a less monolithic structure.
However you handle this, you don't have to refactor all at once - do it piecewise, starting small, and make sure the code works at every step. You'll be surprised how much better you quickly feel about the code.
I don't see any benefit whatsoever to making a separate structure to hold the parameters. The class is already a struct - if it were appropriate to pass parameters by a struct, it would also be appropriate to make the class members public.
There's a tradeoff between public members and Set/Get functions. Public members are a lot less boilerplate, but they expose the internal workings of the class. If this is going to be called from code that you won't be able to refactor if you refactor the class, you'll almost certainly want to use Get and Set.
Assuming that the configuration options apply only to this class, use private variables that are manipulated by public functions with meaningful function names. SetMaxInteriorAngle() is much better than SetMIA() or SetParameter6(). Having getters and setters allows you to enforce consistency rules on the configuration, and can be used to compensate for certain amounts of change in the configuration interface.
If these are general settings, used by more than one class, then an external class would be best, with private members and appropriate functions.
Public data members are usually a bad idea, since they expose the class's implementation and make it impossible to have any guaranteed relation between them. Walling them off in a separate internal struct doesn't seem useful, although I would group them in the list of data members and set them off with comments.

Could C++ have not obviated the pimpl idiom?

As I understand, the pimpl idiom is exists only because C++ forces you to place all the private class members in the header. If the header were to contain only the public interface, theoretically, any change in class implementation would not have necessitated a recompile for the rest of the program.
What I want to know is why C++ is not designed to allow such a convenience. Why does it demand at all for the private parts of a class to be openly displayed in the header (no pun intended)?
This has to do with the size of the object. The h file is used, among other things, to determine the size of the object. If the private members are not given in it, then you would not know how large an object to new.
You can simulate, however, your desired behavior by the following:
class MyClass
{
public:
// public stuff
private:
#include "MyClassPrivate.h"
};
This does not enforce the behavior, but it gets the private stuff out of the .h file.
On the down side, this adds another file to maintain.
Also, in visual studio, the intellisense does not work for the private members - this could be a plus or a minus.
I think there is a confusion here. The problem is not about headers. Headers don't do anything (they are just ways to include common bits of source text among several source-code files).
The problem, as much as there is one, is that class declarations in C++ have to define everything, public and private, that an instance needs to have in order to work. (The same is true of Java, but the way reference to externally-compiled classes works makes the use of anything like shared headers unnecessary.)
It is in the nature of common Object-Oriented Technologies (not just the C++ one) that someone needs to know the concrete class that is used and how to use its constructor to deliver an implementation, even if you are using only the public parts. The device in (3, below) hides it. The practice in (1, below) separates the concerns, whether you do (3) or not.
Use abstract classes that define only the public parts, mainly methods, and let the implementation class inherit from that abstract class. So, using the usual convention for headers, there is an abstract.hpp that is shared around. There is also an implementation.hpp that declares the inherited class and that is only passed around to the modules that implement methods of the implementation. The implementation.hpp file will #include "abstract.hpp" for use in the class declaration it makes, so that there is a single maintenance point for the declaration of the abstracted interface.
Now, if you want to enforce hiding of the implementation class declaration, you need to have some way of requesting construction of a concrete instance without possessing the specific, complete class declaration: you can't use new and you can't use local instances. (You can delete though.) Introduction of helper functions (including methods on other classes that deliver references to class instances) is the substitute.
Along with or as part of the header file that is used as the shared definition for the abstract class/interface, include function signatures for external helper functions. These function should be implemented in modules that are part of the specific class implementations (so they see the full class declaration and can exercise the constructor). The signature of the helper function is probably much like that of the constructor, but it returns an instance reference as a result (This constructor proxy can return a NULL pointer and it can even throw exceptions if you like that sort of thing). The helper function constructs a particular implementation instance and returns it cast as a reference to an instance of the abstract class.
Mission accomplished.
Oh, and recompilation and relinking should work the way you want, avoiding recompilation of calling modules when only the implementation changes (since the calling module no longer does any storage allocations for the implementations).
You're all ignoring the point of the question -
Why must the developer type out the PIMPL code?
For me, the best answer I can come up with is that we don't have a good way to express C++ code that allows you to operate on it. For instance, compile-time (or pre-processor, or whatever) reflection or a code DOM.
C++ badly needs one or both of these to be available to a developer to do meta-programming.
Then you could write something like this in your public MyClass.h:
#pragma pimpl(MyClass_private.hpp)
And then write your own, really quite trivial wrapper generator.
Someone will have a much more verbose answer than I, but the quick response is two-fold: the compiler needs to know all the members of a struct to determine the storage space requirements, and the compiler needs to know the ordering of those members to generate offsets in a deterministic way.
The language is already fairly complicated; I think a mechanism to split the definitions of structured data across the code would be a bit of a calamity.
Typically, I've always seen policy classes used to define implementation behavior in a Pimpl-manner. I think there are some added benefits of using a policy pattern -- easier to interchange implementations, can easily combine multiple partial implementations into a single unit which allow you to break up the implementation code into functional, reusable units, etc.
May be because the size of the class is required when passing its instance by values, aggregating it in other classes, etc ?
If C++ did not support value semantics, it would have been fine, but it does.
Yes, but...
You need to read Stroustrup's "Design and Evolution of C++" book. It would have inhibited the uptake of C++.

Pros and cons of using nested C++ classes and enumerations?

What are the pros and cons of using nested public C++ classes and enumerations? For example, suppose you have a class called printer, and this class also stores information on output trays, you could have:
class printer
{
public:
std::string name_;
enum TYPE
{
TYPE_LOCAL,
TYPE_NETWORK,
};
class output_tray
{
...
};
...
};
printer prn;
printer::TYPE type;
printer::output_tray tray;
Alternatively:
class printer
{
public:
std::string name_;
...
};
enum PRINTER_TYPE
{
PRINTER_TYPE_LOCAL,
PRINTER_TYPE_NETWORK,
};
class output_tray
{
...
};
printer prn;
PRINTER_TYPE type;
output_tray tray;
I can see the benefits of nesting private enums/classes, but when it comes to public ones, the office is split - it seems to be more of a style choice.
So, which do you prefer and why?
Nested classes
There are several side effects to classes nested inside classes that I usually consider flaws (if not pure antipatterns).
Let's imagine the following code :
class A
{
public :
class B { /* etc. */ } ;
// etc.
} ;
Or even:
class A
{
public :
class B ;
// etc.
} ;
class A::B
{
public :
// etc.
} ;
So:
Privilegied Access: A::B has privilegied access to all members of A (methods, variables, symbols, etc.), which weakens encapsulation
A's scope is candidate for symbol lookup: code from inside B will see all symbols from A as possible candidates for a symbol lookup, which can confuse the code
forward-declaration: There is no way to forward-declare A::B without giving a full declaration of A
Extensibility: It is impossible to add another class A::C unless you are owner of A
Code verbosity: putting classes into classes only makes headers larger. You can still separate this into multiple declarations, but there's no way to use namespace-like aliases, imports or usings.
As a conclusion, unless exceptions (e.g. the nested class is an intimate part of the nesting class... And even then...), I see no point in nested classes in normal code, as the flaws outweights by magnitudes the perceived advantages.
Furthermore, it smells as a clumsy attempt to simulate namespacing without using C++ namespaces.
On the pro-side, you isolate this code, and if private, make it unusable but from the "outside" class...
Nested enums
Pros: Everything.
Con: Nothing.
The fact is enum items will pollute the global scope:
// collision
enum Value { empty = 7, undefined, defined } ;
enum Glass { empty = 42, half, full } ;
// empty is from Value or Glass?
Ony by putting each enum in a different namespace/class will enable you to avoid this collision:
namespace Value { enum type { empty = 7, undefined, defined } ; }
namespace Glass { enum type { empty = 42, half, full } ; }
// Value::type e = Value::empty ;
// Glass::type f = Glass::empty ;
Note that C++0x defined the class enum:
enum class Value { empty, undefined, defined } ;
enum class Glass { empty, half, full } ;
// Value e = Value::empty ;
// Glass f = Glass::empty ;
exactly for this kind of problems.
One con that can become a big deal for large projects is that it is impossible to make a forward declaration for nested classes or enums.
If you're never going to be using the dependent class for anything but working with the independent class's implementations, nested classes are fine, in my opinion.
It's when you want to be using the "internal" class as an object in its own right that things can start getting a little manky and you have to start writing extractor/inserter routines. Not a pretty situation.
It seems like you should be using namespaces instead of classes to group like things that are related to each other in this way. One con that I could see in doing nested classes is you end up with a really large source file that could be hard to grok when you are searching for a section.
There are no pros and cons per se of using nested public C++ classes. There are only facts. Those facts are mandated by the C++ standard. Whether a fact about nested public C++ classes is a pro or a con depends on the particular problem that you are trying to solve. The example you have given does not allow a judgement about whether nested classes are appropriate or not.
One fact about nested classes is, that they have privileged access to all members of the class that they belong to. This is a con, if the nested classes does not need such access. But if the nested class does not need such access, then it should not have been declared as a nested class. There are situations, when a class A wants to grant privileged access to certain other classes B. There are three solutions to this problem
Make B a friend of A
Make B a nested class of A
Make the methods and attributes, that B needs, public members of A.
In this situation, it's #3 that violates encapsulation, because A has control over his friends and over his nested classes, but not over classes that call his public methods or access his public attributes.
Another fact about nested classes is, that it is impossible to add another class A::C as a nested class of A unless you are owner of A. However, this is perfectly reasonable, because nested classes have privileged access. If it were possible to add A::C as a nested class of A, then A::C could trick A into granting access to privileged information; and that yould violate encapsulation. It's basically the same as with the friend declaration: the friend declaration does not grant you any special privileges, that your friend is hiding from others; it allows your friends to access information that you are hiding from your non-friends. In C++, calling someone a friend is an altruistic act, not an egoistic one. The same holds for allowing a class to be a nested class.
Som other facts about nested public classes:
A's scope is candidate for symbol lookup of B: If you don't want this, make B a friend of A instead of a nested class. However, there are cases where you want exactly this kind of symbol lookup.
A::B cannot be forward-declared: A and A::B are tightly coupled. Being able to use A::B without knowing A would only hide this fact.
To summarize this: if the tool does not fit your needs, don't blame the tool; blame yourself for using the tool; others might have different problems, for which the tool is perfect.
paercebal said everything I would say about nested enums.
WRT nested classes, my common and almost sole use case for them is when I have a class which is manipulating a specific type of resource, and I need a data class which represents something specific to that resource. In your case, output_tray might be a good example, but I don't generally use nested classes if the class is going to have any methods which are going to be called from outside the containing class, or is more than primarily a data class. I generally also don't nest data classes unless the contained class is not ever directly referenced outside the containing class.
So, for example, if I had a printer_manipulator class, it might have a contained class for printer manipulation errors, but printer itself would be a non-contained class.
Hope this helps. :)
Remember that you can always promote a nested class to a top-level one later, but you may not be able to do the opposite without breaking existing code. Therefore, my advice would be make it a nested class first, and if it starts to become a problem, make it a top-level class in the next version.
For me a big con to having it outside is that it becomes part of the global namespace. If the enum or related class only really applies to the class that it's in, then it makes sense. So in the printer case, everything that includes the printer will know about having full access to the enum PRINTER_TYPE, where it doesn't really need to know about it. I can't say i've ever used an internal class, but for an enum, this seems more logical to keep it inside. As another poster has pointed out, it's also a good idea to to use namespaces to group similar items, since clogging the global namespace can really be a bad thing. I have previously worked on projects which are massive and just bringing up an auto complete list on the global namespace takes 20 minutes. In my opinion nested enums and namespaced classes/structs are probably the cleanest approach.
I agree with the posts advocating for embedding your enum in a class but there are cases where it makes more sense to not do that (but please, at least put it in a namespace). If multiple classes are utilizing an enum defined within a different class, then those classes are directly dependent on that other concrete class (that owns the enum). That surely represents a design flaw since that class will be responsible for that enum as well as other responsibilities.
So, yeah, embed the enum in a class if other code only uses that enum to interface directly with that concrete class. Otherwise, find a better place to keep the enum such as a namespace.
If you put the enum into a class or a namespace, intellisense will be able to give you guidance when you're trying to remember the enum names. A small thing for sure, but sometimes the small things matter.
Visual Studio 2008 does not seem to be able to provide intellisense for nested classes, so I have switched to the PIMPL idiom in most cases where I used to have a nested class. I always put enums either in the class if it is used only by that class, or outside the class in the same namespace as the class when more than one class uses the enum.
I can see a con for nested classes, that one may better use generic programming.
If the little class is defined outside the big one, you can make the big class a class template and use any "little" class you may need in the future with the big class.
Generic programming is a powerful tool, and, IMHO, we should keep it in mind when developing extensible programs. Strange, that no one has mentioned this point.
Only problem with nested classes that I bumped into yet was that C++ does not let us refer to the object of the enclosing class, in the nested class functions. We cannot say "Enclosing::this"
(But maybe there's a way?)

Extending an existing class like a namespace (C++)?

I'm writing in second-person just because its easy, for you.
You are working with a game engine and really wish a particular engine class had a new method that does 'bla'. But you'd rather not spread your 'game' code into the 'engine' code.
So you could derive a new class from it with your one new method and put that code in your 'game' source directory, but maybe there's another option?
So this is probably completely illegal in the C++ language, but you thought at first, "perhaps I can add a new method to an existing class via my own header that includes the 'parent' header and some special syntax. This is possible when working with a namespace, for example..."
Assuming you can't declare methods of a class across multiple headers (and you are pretty darn sure you can't), what are the other options that support a clean divide between 'middleware/engine/library' and 'application', you wonder?
My only question to you is, "does your added functionality need to be a member function, or can it be a free function?" If what you want to do can be solved using the class's existing interface, then the only difference is the syntax, and you should use a free function (if you think that's "ugly", then... suck it up and move on, C++ wasn't designed for monkeypatching).
If you're trying to get at the internal guts of the class, it may be a sign that the original class is lacking in flexibility (it doesn't expose enough information for you to do what you want from the public interface). If that's the case, maybe the original class can be "completed", and you're back to putting a free function on top of it.
If absolutely none of that will work, and you just must have a member function (e.g. original class provided protected members you want to get at, and you don't have the freedom to modify the original interface)... only then resort to inheritance and member-function implementation.
For an in-depth discussion (and deconstruction of std::string'), check out this Guru of the Week "Monolith" class article.
Sounds like a 'acts upon' relationship, which would not fit in an inheritance (use sparingly!).
One option would be a composition utility class that acts upon a certain instance of the 'Engine' by being instantiated with a pointer to it.
Inheritance (as you pointed out), or
Use a function instead of a method, or
Alter the engine code itself, but isolate and manage the changes using a patch-manager like quilt or Mercurial/MQ
I don't see what's wrong with inheritance in this context though.
If the new method will be implemented using the existing public interface, then arguably it's more object oriented for it to be a separate function rather than a method. At least, Scott Meyers argues that it is.
Why? Because it gives better encapsulation. IIRC the argument goes that the class interface should define things that the object does. Helper-style functions are things that can be done with/to the object, not things that the object must do itself. So they don't belong in the class. If they are in the class, they can unnecessarily access private members and hence widen the hiding of that member and hence the number of lines of code that need to be touched if the private member changes in any way.
Of course if you want to access protected members then you must inherit. If your desired method requires per-instance state, but not access to protected members, then you can either inherit or composite according to taste - the former is usually more concise, but has certain disadvantages if the relationship isn't really "is a".
Sounds like you want Ruby mixins. Not sure there's anything close in C++. I think you have to do the inheritance.
Edit: You might be able to put a friend method in and use it like a mixin, but I think you'd start to break your encapsulation in a bad way.
You could do something COM-like, where the base class supports a QueryInterface() method which lets you ask for an interface that has that method on it. This is fairly trivial to implement in C++, you don't need COM per se.
You could also "pretend" to be a more dynamic language and have an array of callbacks as "methods" and gin up a way to call them using templates or macros and pushing 'this' onto the stack before the rest of the parameters. But it would be insane :)
Or Categories in Objective C.
There are conceptual approaches to extending class architectures (not single classes) in C++, but it's not a casual act, and requires planning ahead of time. Sorry.
Sounds like a classic inheritance problem to me. Except I would drop the code in an "Engine Enhancements" directory & include that concept in your architecture.

Should I use nested classes in this case?

I am working on a collection of classes used for video playback and recording. I have one main class which acts like the public interface, with methods like play(), stop(), pause(), record() etc... Then I have workhorse classes which do the video decoding and video encoding.
I just learned about the existence of nested classes in C++, and I'm curious to know what programmers think about using them. I am a little wary and not really sure what the benefits/drawbacks are, but they seem (according to the book I'm reading) to be used in cases such as mine.
The book suggests that in a scenario like mine, a good solution would be to nest the workhorse classes inside the interface class, so there are no separate files for classes the client is not meant to use, and to avoid any possible naming conflicts? I don't know about these justifications. Nested classes are a new concept to me. Just want to see what programmers think about the issue.
I would be a bit reluctant to use nested classes here. What if you created an abstract base class for a "multimedia driver" to handle the back-end stuff (workhorse), and a separate class for the front-end work? The front-end class could take a pointer/reference to an implemented driver class (for the appropriate media type and situation) and perform the abstract operations on the workhorse structure.
My philosophy would be to go ahead and make both structures accessible to the client in a polished way, just under the assumption they would be used in tandem.
I would reference something like a QTextDocument in Qt. You provide a direct interface to the bare metal data handling, but pass the authority along to an object like a QTextEdit to do the manipulation.
You would use a nested class to create a (small) helper class that's required to implement the main class. Or for example, to define an interface (a class with abstract methods).
In this case, the main disadvantage of nested classes is that this makes it harder to re-use them. Perhaps you'd like to use your VideoDecoder class in another project. If you make it a nested class of VideoPlayer, you can't do this in an elegant way.
Instead, put the other classes in separate .h/.cpp files, which you can then use in your VideoPlayer class. The client of VideoPlayer now only needs to include the file that declares VideoPlayer, and still doesn't need to know about how you implemented it.
One way of deciding whether or not to use nested classes is to think whether or not this class plays a supporting role or it's own part.
If it exists solely for the purpose of helping another class then I generally make it a nested class. There are a whole load of caveats to that, some of which seem contradictory but it all comes down to experience and gut-feeling.
sounds like a case where you could use the strategy pattern
Sometimes it's appropriate to hide the implementation classes from the user -- in these cases it's better to put them in an foo_internal.h than inside the public class definition. That way, readers of your foo.h will not see what you'd prefer they not be troubled with, but you can still write tests against each of the concrete implementations of your interface.
We hit an issue with a semi-old Sun C++ compiler and visibility of nested classes which behavior changed in the standard. This is not a reason to not do your nested class, of course, just something to be aware of if you plan on compiling your software on lots of platforms including old compilers.
Well, if you use pointers to your workhorse classes in your Interface class and don't expose them as parameters or return types in your interface methods, you will not need to include the definitions for those work horses in your interface header file (you just forward declare them instead). That way, users of your interface will not need to know about the classes in the background.
You definitely don't need to nest classes for this. In fact, separate class files will actually make your code a lot more readable and easier to manage as your project grows. it will also help you later on if you need to subclass (say for different content/codec types).
Here's more information on the PIMPL pattern (section 3.1.1).
You should use an inner class only when you cannot implement it as a separate class using the would-be outer class' public interface. Inner classes increase the size, complexity, and responsibility of a class so they should be used sparingly.
Your encoder/decoder class sounds like it better fits the Strategy Pattern
One reason to avoid nested classes is if you ever intend to wrap the code with swig (http://www.swig.org) for use with other languages. Swig currently has problems with nested classes, so interfacing with libraries that expose any nested classes becomes a real pain.
Another thing to keep in mind is whether you ever envision different implementations of your work functions (such as decoding and encoding). In that case, you would definitely want an abstract base class with different concrete classes which implement the functions. It would not really be appropriate to nest a separate subclass for each type of implementation.