question about reward in reinforcement learning (RL) - state

I have a question about reward in RL.
is this sentence true? and if it is why?
thank you in advance
"the reward each time (for the same action from the same state) needs not to be the same."

For deterministic perfect information game, it's true. Think of games like Go or Chess. But for other games, the reward of the same state and action mainly depends on the current internal state of the game.

Related

What are the principles involved for an Hierarchical State Machine, and how to implement a basic model?

So I'm attempting to make a game using C++, and I've read a ton of articles on Finite State Machines (FSM), and Hierarchical State Machines (HSM). However I will admit most of the stuff I've read is a bit dense and hard to understand, so I was hoping someone can simplify it for me. Is this answer an FSM or an HSM?
From what I would like to clear up:
How is an HSM different from a normal FSM, and why is it better for games?
Regarding C++, How do you implement a basic HSM following the state pattern? (I might be incorrect on this/using the wrong words.)
How exactly do you handle transitions? What is the on_exit and on_enter method I keep hearing a lot about?
Do I need one HSM for my entire game? (e.g. Handling all enemies, player actions, game menus) or do I use multiple HSMs?
When implementing player entities, would they all be a subset of an Entity state?
Lastly if someone could give some pseudo-code to help visualize these questions, I would appreciate it.
It's just about nesting. An HSM is basically an FSM, but where each state in turn can be a separate FSM.
For an example in a game, consider an NPC. It has multiple states:
Walk to point A
Wait a minute
Walk to point B
Wait a minute
Continue from 1
Fighting with PC
This FSM is simple, but all states needs to have a transition to state 6 (Fighting with PC) for when the NPC is attacked by a PC. This makes the FSM kind of ugly. So instead lets have this much more simple FSM:
Walking about
Fighting with PC
This FSM is very simple, there's only two transitions, and it's easy to understand. The major parts of state 1 is then a secondary FSM:
Walk to point A
Wait a minute
Walk to point B
Wait a minute
If there's an event which doesn't match the secondary FSM transitions, like a PC attacking, you go up a level to the top-level FSM to match the event and find a suitable transition.
You could in a way think about it as a stack, each state in a higher level could push a new lower-level FSM. If there's an event that doesn't match any possible transitions, pop the stack and go back up a level. Continue until there's a matching transition.
In short, it's a way to simplify an FSM.

How to put Amazon Mechanical Turkers in a waitlist?

I am a social psychology researcher. I have developed an online game that identifies players' specific behavioral factors. Each game requires a specific number of players to play simultaneously. In addition, all players should pass a screening phase through which we identify their skills and my program matches players with similar skills to play with each other.
My problem is how to make the players go through the screening phase and wait for others to pass the screening phase before starting the games? Is there anything on MTurk like a wait list? What is the average number of users who participate in a typical study at the same time? Is it possible to make them wait till we reach a specific number of players in the wait list?
Otree supports this functionality. As user Thomas pointed out, wait pages accomplish it.
In particular, when defining a View (same you would in Django) you just specify the variable wait_for_all_groups to be True. The app will not progress until all participants arrive. The definition of the wait page can be a simple as below, but you can also trigger methods like the reassignment of groups with wait pages.
class MyWait(WaitPage):
wait_for_all_groups=True
In the past, I have put such a wait page at the beginning of the instructions for the game.

How to process a very heavy downloadable multiplayer game

I am a programmer and I'd like to know how to process a very heavy downloadable multiplayer game. This game will have robots like in Armored Core with arms and legs which are changeable but it will have like 100 players in a certain area fighting aliens and each one fires like 10 bullets each second plus the enemy attacks.
That is like 2000 bullets each second flying in all directions, plus explosions and missiles and lasers and the environment too, and the AI of the aliens.
Is that very hard to process on today's computers? If it is heavy, how would I process that in a multiplayer scenario? Does every computer split up the job and do their own part?
Is that very hard to process on today's computers?
For a programmer that has to ask the question - yes. For a programmer capable of writting efficient code and using a modern high end computer (that would right now have 120 physical threads) - not really. THe AI may be a problem, but then that can run on a cluster of machines.
Does every computer split up the job and do their own part?
Do you TRUST your players not to cheat? If you do not - can you answer that question yourself?
Generally you hire someone who has experience writing distributed systems. Generally this is way over the head of "a programmer asking question" and the level architects with real time experience come into the game. All thos bullets may sound heavy, but I proces a data stream here doing sometimes in excess of 100.000 updates per second, so - it is doable.

Anti-hacking a game - best practices, suggestions

I recently made a simple game where user can submit his/her high scores online.
It is very easy to hack the game by increasing high score by using software such as Cheat Engine. High score is stored in an integer. Should I store encrypted high score instead of an integer and decrypt it to show in the game?
I was wondering what are the best practices since I'm new in these things (hacking).
This question over at GameDev SE has what you're looking for:
https://gamedev.stackexchange.com/questions/4181/how-can-i-prevent-cheating-on-global-highscore-tables
Another discussion on SO about the topic:
Suggestions for (semi) securing high-scores in Flash/PHP game
The summary is that while there are many methods to make cheating difficult, eventually somebody with enough time in their hands will bypass your security measures. The only way to make leader boards hacker proof is to run the game logic on the server.
Best practise would probably be to send the scores over an encrypted connection to your server using some kind of authentication. This is non-trivial and you would likely need to refer to your platform for any crypto/security functionality it makes available.
It is the essence of security research to be able to share a secret over the ether (net). Essentially both parties need to know how to encode/decode the messages but the method for doing that has to be kept secret from the "Man in The Middle".
I'd refer to Tanenbaum's book on Internetworks or have a look at "Trusted Computing Module".

Automated testing a game

Question
How would you go adding automated testing to a game?
I believe you can unit test a lot of the game engine's functionality (networking, object creation, memory management, etc), but is it possible to automate test the actual game itself?
I'm not talking about gameplay elements (like Protoss would beat Zerg in map X), but I'm talking about the interaction between the game and the engine.
Introduction
In game development, the engine is just a platform for the game. You could think of the game engine as an OS and the game as a software the OS would run. The game could be a collection of scripts or an actual subroutine inside the game engine.
Possible Answers
My idea is this:
You would need an engine that is deterministic. This means that given one set of input, the output would be exactly the same. This would inlude the random generator being seeded with the same input.
Then, create a bare-bone level which contains a couple of objects the avatar/user can interact with. Start small and then add objects into the level as more interactions are developed.
Create a script which follows a path (tests pathfinding) and interact with the different objects (store the result or expected behavior). This script would be your automated test. After a certain amount of time (say, one week), run the script along with your engine's unit tests.
This post at Games From Within might be relevant/interesting.
Riot Games has an article on using automated testing for League of Legends (LoL), a multiplayer online RTS game.
According to the developers, there are many changes to both the game code and game balance everyday. They built a Python test framework that is basically a simpler game client that sends commands to the Continuous Integration server that is running an instance of LoL's game server. The server then send the test framework the effect of the command, allowing the response to be tested.
The framework provides an event queue that records the events, data, and effect from a particular point in time. The article calls this a "snapshot".
The article described an example of a unittest for a spell:
Setup
1. Give a character the ability.
2. Spawn an enemy character in the midlane (a location on the map).
3. Spawn a creep in the midlane. (In the context of LoL, creeps are weak non-controllable characters that are part of each team's army. They are basically canon fodder and is a source of experience and gold for the enemy team. But if left unchecked, they can overwhelm the opposing team)
4. Teleport the character to the midlane.
Execute
1. Take a snapshot of all the variables (e.g. the current life from the player, enemy and normal characters).
2. Cast the spell.
3. Activate the spell's effects (for example, there are some spells that will proc on hit) on an enemy character.
4. Reset the spell's cooldown so it can be cast again immediately.
5. Cast the spell.
6. Activate the spell's effects on a creep (in the context of LoL, most spells have different calculations when used on creeps).
7. Take another snapshot.
Verify
Starting from the first snapshot, replay the events, and assert that the expected results (from a game designer's point of view) are correct. Examples of events that can be verified are: The damage is within the range of the spell's damage (LoL uses random numbers to give variance to attacks), Damage is properly resisted when compared with a player character and a creep, and spells are cast within its effective range.
The article shows that a video of the test can be extracted when the test server is viewed from a normal game client.
Values are so random within the gameplay aspects of development that it would be a far fetched idea to test for absolute values
But we can test deterministic values. For example, a unit test might have Guybrush Threepwood move toward a door (pathfinding), open the door (use command), fail because he doesn't have a key in his inventory (feedback), pick the door key (pathfinding + inventory management) and then finally opening the door.
All of these paths are deterministic. With this unit test, I can refactor the memory manager and if it somehow broke the inventory management routine, the unit test would fail.
This is just one idea for unit testing in games. I would love to know other ideas, hence, the motivation for this post.
I did something similar to your idea once and it was very successful, though I suspect it is really more of a system test than a unit test. As you suggest your random number generator must be seeded with the same value, and must produce an identical sequence each time.
The game ran on 50hz cycles, so timing was not an issue. I had a system that would record mouse clicks and locations, and used this to manually generate a 'script' which could be replayed to produce the same results. By removing the timing delays and turning off the graphic generation an hour of gameplay could be replicated in a few seconds.
The biggest problem was that changes to the game design would invalidate the script.
If your barebones room contained logic that was independent of the general game play then it could work very well. The engine could start up without any ui and start the script as soon as initialisation is complete. Testing for crashing along the way would be simple, but more complex tests such as leaving the characters in the correct positions would be more complex. If the recording of the scripts are simple enough, which they were in my system, then they can be updated very easily, and special scripts to test specialised behavior can be set up very quickly. My system had the added advantage that it could be used during game testing, and the exact sequence of events recorded to make bug fixing easier.
An article from Power of Two GamesGames From Within was mentioned in another answer already, but I suggest reading everything (or nearly everything) there, as they are all really well-written and apply directly to games development. The article on Assert is particularly good. You can also visit their previous website at Games From Within, which has a lot written about Test Driven Development, which is unit testing taken to the extreme.
The Power of Two guys are the ones who implemented UnitCpp, a pretty well-regarded unit testing framework. Personally, I prefer WinUnit.
If you are testing the rendering engine I guess you could render specific test scenes, do a screen captures and compare them to reference test renderings. That way you can detect if changes in the engine breaks anything, visually. You can write similar test for the sound engine, or even animation (by comparing a series of frames).
If you want to test game logic or scene progress you can do this by testing various conditions on the scripting variables (assuming you are using scripting to implement most of the scene and story aspects).
If you're using XNA (the idea could be extrapolated to other frameworks of course), you could use an in-game unit test framework that lets you access the game's state in the unit test. One such framework is Scurvy.Test :-)
http://flea.sourceforge.net/gameTestServer.pdf
This is an interesting discussion on implementing a full-blown functional tester in a game.
The term "unit testing" implies that a "unit" is being tested. This is one thing. If you're doing higher-level testing (e.g. several systems at once), usually this is called functional testing. It is possible to unit test much of a game, however you can't really test for fun.
Determinism isn't necessary, as long as your tests can be fuzzy. E.g. "did the character get hurt" as opposed to "did the character lose 14.7 hitpoints.
I have written a paper on that topic -
http://download.springer.com/static/pdf/722/art%253A10.7603%252Fs40601-013-0010-4.pdf?auth66=1407852969_87bc2e71ad5228b36738f0237084ebe5&ext=.pdf
This doesn't really answer your question but I was listening to a podcast on Pex from microsoft which does a similar thing to the solution you're proposing and when I was listening to it I remember thinking that it would be really interesting to see if it would be able to test games. I don't know if it would be able to help you specifically, but perhaps you could take a look at some of the ideas they use and apply it to your unit testing.