I'm embedding Python into my game. The scripts will be used to define the character AI, how entities react to game events, etc — this means there's going to be a script for every type of entity in the game.
Each script will have a function like createEntity() or something which will return the constructed entity. What would be an efficient(ish) way of calling these functions (remember, there's one in every entity's script).
My initial thought was to do something like what you see below, however, I'm unsure as to how efficient this is. For example, what happens with the imported hero module after I run that string? Does it remain loaded in the main module? If that's the case, that's problematic since I'm going to be importing lots of scripts for all the entities I might need to add to the game world.
boost::python::handle<> result(
PyRun_String("import hero\n" "createEntity()\n",
Py_file_input, main_namespace.ptr(), main_namespace.ptr())
);
// Then extract the entity from `result`...
What suggestions do you have?
Your question does not specify whether space efficiency (i.e. memory), time efficiency, or labor efficiency is most important to you. Merely because you are considering a hybrid C++ / Python application, I assume that labor efficiency is a significant factor. Because you are developing a game, I assume that there will be some part of it that has a need for extremely tight execution speed.
Perhaps this approach will strike a balance. Make all the user interaction (input and output, including any networking) C++ based for minimum latency. You might place this in its own thread or process. Given a high level event from the model, like a character moving, this code very quickly updates the screen and the network. Given a user event, or an event from the network, it sends a message to the model.
The game model, which can be asynchronous to the view/control, would then be in Python for your convenience and so you can take advantage of functional programming etc. You could use shared memory or a similar IPC mechanism between the two and start them separately if actually embedding an interpreter is inconvenient.
While certain AI applications are CPU-intensive, and therefore it may be tempting to go back to C or C++ for them, I would advise doing that as a final step, in response to clear responsiveness issues in interpreted code, if you do it at all. You may even want to follow this line of thought with the graphics also, since nowadays most graphics processing is delegated to the video hardware, if you have a way to make the library calls you need from Python code.
Though I am not a game developer, I have been around and I have seen few situations in which the (nowadays) microseconds difference between single equivalent C and Python operations is perceptible to users. Perceptible performance problems are nearly always due to other factors, such as disk I/O, network latency or inefficient algorithm implementations.
Ousterhout on the role of scripting languages is an oldie but goodie. In the case of a game where you have said that the gameplay is to be scripted, the model (Python) is already organizing the flow as you have described it. If PyGame or similar library isn't up to the task of presenting the view, find or build a Python module that can.
Put another way, when I suggested you've got the integration backwards, it sounds like you are asking the view to call the model repeatedly to update itself. I cannot think of a way that putting the metaphoric cart before the horse will yield either efficiency or ease of development.
Related
I am tasked to maintain and update a library which allows a computer to send commands at a hardware device and then receive its response. Currently the code is setup in such a way that every single possible command the device can receive is sent via its own function. Code repetition is everywhere; a DRY advocate's worst nightmare.
Obviously there is much opportunity for improvement. The problem is each command has a different payload. Currently the data that is to be the payload is passed to each command function in the form of arguments. It's difficult to consolidate functionality without pushing the complexity to a level that calls the library.
When a response is received from the device its data is put into an object of a class solely responsible for holding this data, they do nothing else. There are hundreds of classes which do this. These objects are then used to access the returned data by the app layer.
My objectives:
Throughly reduce code repetition
Maintain similiar level of complexity at application layer
Make it easier to add new commands
My idea:
Have one function to send a command and one to receive (the receiving function is automatically called when a response from the device is detected). Have a struct holding all command/response data which will be passed to sending function and returned by receiving function. Since each command has a corresponding enum value, have a switch statement which sets up any command specific data for sending.
Is my idea the best way to do it? Is there a design pattern I could use here? I've looked and looked but nothing seems to fit my needs.
Thanks in advance! (Please let me know if clarification is necessary)
This reminds me of the REST vs. SOA debate, albeit on a smaller physical scale.
If I understand you correctly, right now you have calls like
device->DoThing();
device->DoOtherThing();
and then sometimes I get a callback like
callback->DoneThing(ThingResult&);
callback->DoneOtherTHing(OtherThingResult&)
I suggest that the user is the key component here. Do the current library users like the interface at the level it is designed? Is the interface consistent, even if it is large?
You seem to want to propose
device->Do(ThingAndOtherThingParameters&)
callback->Done(ThingAndOtherThingResult&)
so to have a single entry point with more complex data.
The downside from a library user perspective may that now I have to use a manual switch() or other type statement to tell what really happened. While the dispatching to the appropriate result callback used to be done for me, now you have made it a burden upon the library user.
Unless this bought me as a user some level of flexibility, that I as as user wanted I would consider this a step backwards.
For your part as an implementor, one suggestion would be to go to the generic form internally, and then offer both interfaces externally. Perhaps the old specific interface could even be auto-generated somehow.
Good Luck.
Well, your question implies that there is a balance between the library's complexity and the client's. When those are the only two choices, one almost always goes with making the client's life easier. However, those are rarely really the only two choices.
Now in the text you talk about a command processing architecture where each command has a different set of data associated with it. In the olden days, this would typically be implemented with a big honking case statement in a loop, where each case called a different routine with different parameters and perhaps some setup code. Grisly. McCabe complexity analysers hate this.
These days what you can do with an OO language is use dynamic dispatch. Create a base abstract "command" class with a standard "handle()" method, and have each different command inherit from it to add their own members (to represent the different "arguments" to the different commands). Then you create a big honking array of these at startup, usually indexed by the command ID. For languages like C++ or Ada it has to be an array of pointers to "command" objects, for the dynamic dispatch to work. Then you can just call the appropriate command object for the command ID you read from the client. The big honking case statement is now handled implicitly by the dynamic dispatch.
Where you can get the big savings in this scenario is in subclassing. Do you have several commands that use the exact same parameters? Make a subclass for them, and then derive all of those commands from that subclass. Do you have several commands that have to perform the same operation on one of the parameters? Make a subclass for them with that one method implemented for that operation, and then derive all those commands from that subclass.
Your first objective should be to produce a library that decouples higher software layers from the hardware. Users of your library shouldn't care that you have a hardware device that can execute a number of functions with a different payload. They should only care what the device does in a higher level. In this sense, it is in my opinion a good thing that every command is mapped to each one function.
My plan will be:
Identify the objects the higher data layers need to get the job done. Model the objects in C++ classes from their perspective, not from the perspective of the hardware
Define the interface of the library using the above objects
Start the implementation of the library. Perhaps an intermediate layer that maps software objects to hardware objects is necessary
There are many things you can do to reduce code repetition. You can use polymorphism. Define a class with the base functionality and extend it. You can also use utility classes, that implement functions needed for many commands.
Call main.lua script at each game loop iteration - is it good or bad design? How does it affect on the performance (relatively)?
Maintain game state from a. C++ host-program or b. from Lua scripts or c. from both and synchronise them?
(Previous question on the topic: Lua and C++: separation of duties)
(I vote for every answer. The best answer will be accepted.)
My basic rule for lua is - or any script language in a game -
Anything that happens on every frame: c++
asynchronous events - user input - lua
synchronous game engine events - lua
Basically, any code thats called at >33-100Hz (depending on frame rate) is C++
I try to invoke the script engine <10Hz.
Based on any kind of actual metric? not really. but it does put a dividing line in the design, with c++ and lua tasks clearly delineated - without the up front delineation the per frame lua tasks will grow until they are bogging processing per frame - and then theres no clear guideline on what to prune.
The best thing about lua is that it has a lightweight VM, and after the chunks get precompiled running them in the VM is actually quite fast, but still not as fast as a C++ code would be, and I don't think calling lua every rendered frame would be a good idea.
I'd put the game state in C++, and add functions in lua that can reach, and modify the state. An event based approach is almost better, where event registering should be done in lua (preferably only at the start of the game or at specific game events, but no more than a few times per minute), but the actual events should be fired by C++ code. User inputs are events too, and they don't usually happen every frame (except for maybe MouseMove but which should be used carefully because of this). The way you handle user input events (whether you handle everything (like which key was pressed, etc) in lua, or whether there are for example separate events for each keys on the keyboard (in an extreme case) depends on the game you're trying to make (a turn based game might have only one event handler for all events, an RTS should have more events, and an FPS should be dealt with care (mainly because moving the mouse will happen every frame)). Generally the more separate kinds of events you have, the less you have to code in lua (which will increase performance), but the more difficult it gets if a "real event" you need to handle is actually triggered by more separate "programming level events" (which might actually decrease performance, because the lua code needs to be more complex).
Alternatively if performance is really important you can actually improve the lua VM by adding new opcodes to it (I've seen some of the companies to do this, but mainly to make decompilation of the compiled lua chunks more harder), which is actually not a hard thing to do. If you have something that the lua code needs to do a lot of times (like event registering, event running, or changing the state of the game) you might want to implement them in the lua VM, so instead of multiple getglobal and setglobal opcodes they would only take one or two (for example you could make a SETSTATE opcode with a 0-255 and a 0-65535 parameter, where the first parameter descibes which state to modify, and the second desribes the new value of the state. Of course this only works if you have a maximum of 255 events, with a maximum of 2^16 values, but it might be enough in some cases. And the fact that this only takes one opcode means that the code will run faster). This would also make decompilation more harder if you intend to obscure your lua code (although not much to someone who knows the inner workings of lua). Running a few opcodes per frame (around 30-40 tops) won't hit your performance that badly. But 30-40 opcodes in the lua VM won't get you far if you need to do really complex things (a simple if-then-else can take up to 10-20 or more opcodes depending on the expression).
I don't like C++. But I do like games.
My approach might be a bit atypical: I do everything I can in Lua, and only the absolute minimum in C++. The game loop, the entities, etc are all done in Lua. I even have a QuadTree implementation done in Lua. C++ handles graphical and filesystem stuff, as well as interfacing with external libraries.
This is not a machine-based decision, but a programmer-based one; I output code much faster in Lua than In C++. So I spend my programmer cycles on new features rather than on saving computer cycles. My target machines (any laptop from the last 3 years) are able to cope with this amount of Lua very easily.
Lua is surprisingly low-footprint (take a look to luaJIT if you don't know it).
This said, if I ever find a bottleneck (I haven't yet) I'll profile the game in order to find the slow part, and I'll translate that part to C++ ... only if I can't find a way around it using Lua.
I am using Lua for the first time in a game I've been working on. The C++ side of my application actually holds pointers to instances of each game state. Some of the game states are implemented in C++ and some are implemented in Lua (such as the "game play" state).
The update and main application loop live on the C++ side of things. I have exposed functions that allow the Lua VM to add new game states to the application at runtime.
I have not yet had any problems with slowness, even running on hardware with limited resources (Atom processor with integrated video). Lua functions are called every frame. The most expensive (in terms of time) operation in my application is rendering.
The ability to create new states completely in Lua was one of the best decisions I made on the project, since it allows me to freely add portions of the game without recompiling the whole thing.
Edit: I'm using Luabind, which I have read performs slower in general than other binding frameworks and of course the Lua C API.
You probably don't want to execute the entire Lua script on every frame iteration, because any sufficiently complex game will have multiple game objects with their own behaviors. In other words, the advantages of Lua are lost unless you have multiple tiny scripts that handle a specific part of the behavior of a larger game. You can use the lua_call function to call any appropriate lua routine in your script, not just the entire file.
There's no ideal answer here, but the vast majority of your game state is traditionally stored in the game engine (i.e. C++). You reveal to Lua just enough for Lua to do the decision making that you've assigned to Lua.
You need to consider which language is appropriate for which behaviors. Lua is useful for high level controls and decisions, and C++ is useful for performance oriented code. Lua is particularly useful for the parts of your game that you need to tweak without recompiling. All magic constants and variables could go into Lua, for example. Don't try to shoehorn Lua where it does not belong, i.e. graphics or audio rendering.
IMHO Lua scripts are for specific behaviours, it's definitely going to hurt performance if you are calling a Lua script 60 times per second.
Lua scripts are often to separate stuff like Behaviour, and specific events from your Game Engine logic (GUI, Items, Dialogs, game engine events, etc...). A good usage of Lua for example would be when triggering an explosion (particle FX), if the Game Character walks somewhere, hard-coding the output of that event in your engine would be a very ugly choice. Though, making the engine trigger the correct script would be a better choice, decoupling that specific behavior off your engine.
I would recommend, to try to keep your Game State in one part, instead of upscaling the level of complexity of keeping states synchronized in two places (Lua and Engine), add threading to that, and you will end up having a very ugly mess. Keep it simple. (In my Designs I mostly keep Game State in C++)
Good luck with your Game!
I'd like to throw in my two cents since I strongly believe that there's some incorrect advice being given here. For context, I am using Lua in a large game that involves both intensive 3D rendering as well as intensive game logic simulation. I've become more familiar than I'd have liked to with Lua and performance...
Note that I'm going to talk specifically about LuaJIT, because you're going to want to use LuaJIT. It's plug-and-play, really, so if you can embed Lua you can embed LuaJIT. You'll want it, if not for the extra speed, then for the automagic foreign function interface module (require 'ffi') that will allow you to call your native code directly from Lua without ever having to touch the Lua C API (95%+ of cases).
It's perfectly fine to call Lua at 60hz (I call it at 90hz in VR..). The catch is that you are going to have to be careful to do it correctly. As someone else mentioned, it's critical that you load the script only once. You can then use the C API to get access to functions you defined in that script, or to run the script itself as a function. I recommend the former: for a relatively simple game, you can get by with defining functions like onUpdate (dt), onRender (), onKeyPressed (key), onMouseMoved (dx, dy), etc. You can call these at the appropriate time from your main loop in C++. Alternatively, you can actually have your entire main loop be in Lua, and instead invoke your C++ code for performance-critical routines (I do that). This is especially easy to do with the LuaJIT FFI.
This is the really hard question. It will depend on your needs. Can you quite easily hammer down the game state? Great, put it C++-side and access from LuaJIT FFI. Not sure what all will be in the game state / like to be able to prototype quickly? Nothing wrong with keeping it in Lua. That is, until you start talking about a complex game with 1000s of objects, each containing non-trivial state. In this case, hybrid is the way to go, but figuring out exactly how to split state between C++ and Lua, and how to martial said state between the two (especially in perf-critical routines) is something of an art. Let me know if you come up with a bulletproof technique :) As with everything else, the general rule of thumb is: data that goes through performance-critical pathways needs to be on the native side. For example, if your entities have positions and velocities that you update each frame, and you have thousands of said entities, you need to do this in C++. However, you can get away with layering an 'inventory' on top of these entities using Lua (an inventory doesn't need a per-frame update).
Now, a couple more notes I'd like to throw out both as general FYIs and in response to some of the other answers.
Event-based approaches are, in general, critical to the performance of any game, but that goes doubly for systems written in Lua. I said it's perfectly fine to call Lua # 60hz. But it's not perfectly fine to be running tight loops over lots of game objects each frame in said Lua. You might get away with wastefully calling update() on everything in the universe in C++ (though you shouldn't), but doing so in Lua will start eating up those precious milliseconds far too quickly. Instead, as others have mentioned, you need to be thinking of Lua logic as 'reactive' -- usually, this means handling an event. For example, don't check that one entity is in range of another each frame in Lua (I mean, this is fine for one entity, but in general when you're scaling up your game you need to not think like this). Instead, tell your C++ engine to notify you when the two entities get within a certain distance of one another. In this way, Lua becomes the high-level controller of game logic, dispatching high-level commands and responding to high-level events, not carrying out the low-level math grind of trivial game logic.
Be wary of the advice that 'mixed code' is slow. The Lua C API is lightweight and fast. Wrapping functions for use with Lua is, at worst, quite easy (and if you take a while to understand how Lua interfaces with C w.r.t. the virtual stack, etc, you will notice that it has been designed specifically to minimize call overhead), and at best, trivial (no wrapping required) and 100% as performant as native calls (thanks, LuaJIT FFI!) In most cases I find that a careful mixture of Lua and C (via the LJ FFI) is superior to pure Lua. Even vector operations, which I believe the LuaJIT manuals mention somewhere as an example where code should be kept to Lua, I have found to be faster when I execute via Lua->C calls. Ultimately, don't trust anyone or anything but your own (careful) performance measurements when it comes to pieces of code that are performance-sensitive.
Most small-ish games could get away just fine with game state, the core loop, input handling, etc. done entirely in Lua and running under LuaJIT. If you're building a small-ish game, consider the "move to C++ as needed" approach (lazy compilation!) Write in Lua, when you identify a clear bottleneck (and have measured it to make sure it's the culprit), move that bit to C++ and be on your way. If you're writing a big game with 1000s of complex objects with complex logic, advanced rendering, etc. then you'll benefit from more up-front designing.
Best of luck :)
About the performance of 1: if main.lua does not change, load it once with lua_loadfile or loadfile, save a reference to the returned function, and then call it when needed.
Most of the performance will be lost through the binding between Lua and C++. A function call will actually need to be wrapped, and re-wrapped, and like that a couple of time usually. Pure Lua or pure C++ code is usually faster than mixed code (for small operations).
Having said that, I personally didn't see any strong performance hit running a Lua script every frame.
Usually scripting is good at high level. Lua has been used in famous games for the Bots (Quake 3) and for the User Interface (World of Warcraft). Used at high level Lua micro-threads come handy: The coroutines can save a lot (compared to real threads). For example to run some Lua code only once in a while.
I have three legacy applications that share a lot of source code and data. Multiple instances of each of these applications be executed by a user at any time, e.g. a dozen mixed application executions can be active at a time. These applications currently communicate through shared memory and messaging techniques so that they can maintain common cursor positioning, etc. The applications are written primarily in C++, use Qt and run in total to about 5 million lines of code. Only some of the existing code is threadsafe.
I want to consolidate these three executables into a single executable and use multi-threading functionality to allow multiple instance of each of the three functionality branches to execute at the same time. It has been suggested that I look into some of the features provided by Boost, e.g. shared pointers, and use OpenMP to orchestrate overall execution of the multiple threads.
Any comments on how to proceed will be appreciated, particularly references on the best way to tackle this kind of a refactoring problem.
My suggestion to you would be design the desired solution first (firstly assuming that the requirements are the same) and then build a phased migration path from the existing code base basing it on the requirements that third party functionality may introduce.
Refactoring should be one small step at a time - but knowing where you are going.
The Working Effectively with Legacy Code (Robert C. Martin Series) will be a good read I'd suggest.
Trust me (I've got the t shirts) don't attempt to refactor unless you know how to prove functionality - automated verification tests will be your saviour.
I don't think anyone will be able to give you an informed answer. There are too many things to take into considerations. Only after careful analysis of the processes involved and the code behind would anyone (you included) be able to provide you with a project path.
That said, there are however a few things you should take into consideration.
Your applications and the way they seem to communicate with each other seems to me an already sound solution. This is so, especially because you have 5 million lines of code to tackle with, if you choose to get into refactoring your current system. This is a strong deterrent. If there is a need to provide threading support in order to optimize current application messaging, I'd perhaps suggest you consider the possibility of introducing MT into the shared memory and messaging routines, instead of merging applications.
The act of merging your applications could be seen at first glance as the "simple" act of gluing your current code and removing the current routines responsible for memory sharing and messaging, replacing them with normal function calls to the glued objects. My gut tells me it will not be that simple. You have to consider that almost certainly your current backbone code was designed for the current messaging solution. Your current abstractions were thought of with this in mind. You will almost certainly find that you will also need to make significant changes to existing backbone code. And again here you will find that wall of 5 million lines a huge problem. Because changes will quickly propagate across your entire system and leave you with 5 million lines of code to handle and god knows how many new bugs.
I'd suggest 1 above. But it may be that you do need for some reason to consolidate your applications. That being so, I'd still suggest something else before you try 2. Create an interface application responsible for firing up, displaying and maintaining the current 3 applications. Your users will know no better if you do this right. You'll still be able to apply 1 to your current system and join your applications under a common interface, despite them being in fact separate entities.
"...about 5 million lines of code..." Hmm... It is hard to say for sure without knowing your system, but since it is a "legacy" system, you can probably gain a lot by removing code duplication. Check out Simian and CPD.
5 million lines is a lot ot code. Of course, you may need it, but my guess is that you don't.
You definitely need a good reason to change a change a code-base like that to be multi-threaded, especially in C++. Doing it without cleaning it up completely first is a recipe for disaster.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions concerning problems with code you've written must describe the specific problem — and include valid code to reproduce it — in the question itself. See SSCCE.org for guidance.
Closed 9 years ago.
Improve this question
When developing an application which mostly interacts with a database, what is a good way to start? The application requires a lot of filtering based on user input, sorting and structuring.
The best way to start is by figuring out "user stories" (or "use cases" -- but the "story" approach tends to really work great and start dragging shareholder into the shared storytelling...!-); on top of that, designing the database schema as the best-normalized idea you can find to satisfy all data layer needs of the user stories.
Thirdly, you may sketch layers such as views on top of the schema; fourthly, and optionally, triggers and stored procedures that might live in the DB to ensure consistency and ease of use for higher layers (but, no matter how strongly DBAs will push you towards those, don't accept their assurances that they're a MUST: they aren't -- if your storage layer is well designed in terms of normalization and maybe useful views on top, non-storage-layer functionality CAN always reside elsewhere, it's an issue of convenience and performance, NOT logical consistency, completeness, correctness).
I think the business layer and user-experience layers should come after. I realize that's a controversial position, but my point is that the user stories (and implied business-rules that come with them) have ALREADY told you a LOT about the business and user layers -- so, "nailing down" (relatively speaking -- agility and "embrace change!" should always rule;-) the data storage layer is the next order of business, and refining ("drilling down") the higher layers can and should come after.
When you get to the database layer you'll want to handle the database access via stored procedures. This will help give you additional protection against SQL Injection attacks, and make it much easier to push logic changes to the database layer.
If it's mostly users interacting with data, you can design using a form perspective.
What forms are needed for user input?
What forms are needed for output reports?
Once you've determined that, the use of the forms will dictate the business logic needed to be coded behind the scenes. You'll take the inputs, create the set of procedures or methods to deal with them, and output what is necessary. Once you know the inputs and outputs, you will be able to easily design the necessary functions.
The scope of the question is very broad. You are expecting me to tell what to do. I can only do a good job of telling how to do things. Do investigate upon using Hibernate/Spring. Since most of your operations looks like querying db, hibernate should help. Make sure the tables are sufficiently indexed so your queries can run faster if filtered based on index fields. The challenging task is design your DB layer which will be the glue between your application and db. Design your db layer generic enough so that it can build queries based on the params that you pass to it. Then move on to develop the above presentation layer. Developing your application layer by layer helps since it will force you to decouple the db logic from the presentation logic. When you develop the db layer, assume that not just your presentation layer but any client can call it. This will help you to design applications that can be scalable and adaptable to new requirements.
So bottom line : Start with DB, DB integeration layer, Controller and last Presentation Layer.
For the purpose of discussion, I'm going to assume that you are working with a starting application that doesn't have a pre-existing database. If this is false, I'd probably move the order of steps around quite a bit.
1 - Understand the Universe
First, you've got to get a sense of what's around you so you can really understand the problem that you are trying to solve.
User stories or use cases are often a good starting point. Starting with what tasks the user will try to do, and evaluating how frequently they are likely to be is a great starting point. I like to start with screen mockups as well, with or without lots of hands on time with users, I find that having a screen gives our team something really finite to argue about.
What other tools exist in this sphere? These days, it seems to me that users never use just one tool, they swap around alot. You need to know two main things about the other tools you users use:
(1) - what will they be using as part of the process, along side your tool? Consider direct input/output needs - what might they want to cut/copy/paste from or to? What tools might you want to offer file upload/download for with specific formats, what tools are they using alongside your tool that you might want to share terminology, layout, color coding, icons or other GUI elements with. Focus especially on the edges of the tools - a real gotcha I hit in a recent project was emulating the databases of previous tools. It turned out that we had massive database shift, and we would likely have been better starting fresh.
(2) What (if anything) are you replacing or competing with? Steal the good stuff, dump and improve the bad stuff. Asking users is always best. If you can't at least understanding the management initiative is important - is this tool replacing a horrible legacy tool? It may be legacy, but there may be the One True Feature that has kept the tool in business all these years...
At this stage, I find that things are really mushy - there's some screen shots, some writing, some schemas or ICDs - but not a really gelled clue.
2 - Logical Entities
Or at least that's what the OO books call it.
I don't care much for all the writing I see on this task - but I find that any any given system, I have one true diagram that I draw over and over. It's usually about 3-10 boxes, and hopefully less than an exponentially large number of lines connecting them. W
The earlier you can get that diagram the better.
It doesn't matter to me if it's in UML, a database logical model, something older, or on the back of a napkin (as long as the napkin is shrouded in plastic and hung where everyone can see it).
The earlier you can make this diagram correctly, the better.
After the diagram is made, you can start working on the follow on work that may be more official.
I think it's a chicken and egg question on whether you start with your data or you start with your screens and business logic. I know that you certianly want to optimize for database sizing and searchability... but how do you know exactly what your database needs are without screens and interfaces giving you a sense for the data?
In practice, I think this is an ever-churning cycle. You do a little bit everywhere, and then you change it all.
Even if you don't get to do a formal agile lifecycle, I think you're best bet is to view design as agile -- it will take many repetitions and arguments before you really feel it's "right".
The most important thing to keep in mind is that your first, and most likely 2nd 3rd attempt at designing the database will be wrong in some way. That might sound negative, maybe even a little rash, (it's certainly more towards the 'agile' software design philosophy) but it's important thing to keep in mind.
You still need to do your analysis thoroughly of course, try to implement one feature at a time, but try to get all layers working first. That way you won't have to do to much rework when the specs change and you understand the issues better. One you have a lot of data loaded into a system, changing things becomes increasingly difficult.
The main benefit of this approach is you find out quickly where you design is broken, where you haven't separated you design layers correctly. One trick I find extremely useful is to do both a sqllite and a mysql version, so seamless switching between the two is possible. Because the two use a different accent of SQL it highlights where you have too tight a coupling between the layers.
A good start would be to get familiar with Multitier architecture
Then you design your presentation layer.
In your business logic layer implement all logic
And finally you implement your data access layer.
Try to setup a prototype with something that is more productive then C++ for example Ruby, Python and well maybe even PHP.
When the prototype works and you see your data model is okay and your queries are too slow then you can start using C++.
But as your questions suggests you have more options then data and in this case the speed of a scripting langauge should be enough.
I had an idea I was mulling over with some colleagues. None of us knew whether or not it exists currently.
The Basic Premise is to have a system that has 100% uptime but can become more efficient dynamically.
Here is the scenario: * So we hash out a system quickly to a
specified set of interfaces, it has
zero optimizations, yet we are
confident that it is 100% stable
though (dubious, but for the sake of
this scenario please play
along) * We then profile
the original classes, and start to
program replacements for the
bottlenecks.
* The original and the replacement are initiated simultaneously and
synchronized.
* An original is allowed to run to completion: if a replacement hasn´t
completed it is vetoed by the system
as a replacement for the
original.
* A replacement must always return the same value as the original, for a
specified number of times, and for a
specific range of values, before it is
adopted as a replacement for the
original.
* If exception occurs after a replacement is adopted, the system
automatically tries the same operation
with a class which was superseded by
it.
Have you seen a similar concept in practise? Critique Please ...
Below are comments written after the initial question in regards to
posts:
* The system demonstrates a Darwinian approach to system evolution.
* The original and replacement would run in parallel not in series.
* Race-conditions are an inherent issue to multi-threaded apps and I
acknowledge them.
I believe this idea to be an interesting theoretical debate, but not very practical for the following reasons:
To make sure the new version of the code works well, you need to have superb automatic tests, which is a goal that is very hard to achieve and one that many companies fail to develop. You can only go on with implementing the system after such automatic tests are in place.
The whole point of this system is performance tuning, that is - a specific version of the code is replaced by a version that supersedes it in performance. For most applications today, performance is of minor importance. Meaning, the overall performance of most applications is adequate - just think about it, you probably rarely find yourself complaining that "this application is excruciatingly slow", instead you usually find yourself complaining on the lack of specific feature, stability issues, UI issues etc. Even when you do complain about slowness, it's usually an overall slowness of your system and not just a specific applications (there are exceptions, of course).
For applications or modules where performance is a big issue, the way to improve them is usually to identify the bottlenecks, write a new version and test is independently of the system first, using some kind of benchmarking. Benchmarking the new version of the entire application might also be necessary of course, but in general I think this process would only take place a very small number of times (following the 20%-80% rule). Doing this process "manually" in these cases is probably easier and more cost-effective than the described system.
What happens when you add features, fix non-performance related bugs etc.? You don't get any benefit from the system.
Running the two versions in conjunction to compare their performance has far more problems than you might think - not only you might have race conditions, but if the input is not an appropriate benchmark, you might get the wrong result (e.g. if you get loads of small data packets and that is in 90% of the time the input is large data packets). Furthermore, it might just be impossible (for example, if the actual code changes the data, you can't run them in conjunction).
The only "environment" where this sounds useful and actually "a must" is a "genetic" system that generates new versions of the code by itself, but that's a whole different story and not really widely applicable...
A system that runs performance benchmarks while operating is going to be slower than one that doesn't. If the goal is to optimise speed, why wouldn't you benchmark independently and import the fastest routines once they are proven to be faster?
And your idea of starting routines simultaneously could introduce race conditions.
Also, if a goal is to ensure 100% uptime you would not want to introduce untested routines since they might generate uncatchable exceptions.
Perhaps your ideas have merit as a harness for benchmarking rather than an operational system?
Have I seen a similar concept in practice? No. But I'll propose an approach anyway.
It seems like most of your objectives would be meet by some sort of super source control system, which could be implemented with CruiseControl.
CruiseControl can run unit tests to ensure correctness of the new version.
You'd have to write a CruiseControl builder pluggin that would execute the new version of your system against a series of existing benchmarks to ensure that the new version is an improvement.
If the CruiseControl build loop passes, then the new version would be accepted. Such a process would take considerable effort to implement, but I think it feasible. The unit tests and benchmark builder would have to be pretty slick.
I think an Inversion of Control Container like OSGi or Spring could do most of what you are talking about. (dynamic loading by name)
You could build on top of their stuff. Then implement your code to
divide work units into discrete modules / classes (strategy pattern)
identify each module by unique name and associate a capability with it
when a module is requested it is requested by capability and at random one of the modules with that capability is used.
keep performance stats (get system tick before and after execution and store the result)
if an exception occurs mark that module as do not use and log the exception.
If the modules do their work by message passing you can store the message until the operation completes successfully and redo with another module if an exception occurs.
For design ideas for high availability systems, check out Erlang.
I don't think code will learn to be better, by itself. However, some runtime parameters can easily adjust onto optimal values, but that would be just regular programming, right?
About the on-the-fly change, I've shared the wondering and would be building it on top of Lua, or similar dynamic language. One could have parts that are loaded, and if they are replaced, reloaded into use. No rocket science in that, either. If the "old code" is still running, it's perfectly all right, since unlike with DLL's, the file is needed only when reading it in, not while executing code that came from there.
Usefulness? Naa...