Is there a good tool out there that can model an application concurrency/locking scheme in a graphical way and that can simulate some of the aspects?
I know that Petri nets can be used for that more or less, but I don't know a good GUI tool that can design and simulate.
Is UML in any way usable for such purposes?
Any good links are very appreciated.
UML Activity Diagrams can be expressed as Petri nets (e.g. see this paper). Unfortunately I don't know any good industrial oriented tools for simulation of Petri Nets or Activity Diagram (but there are many academic projects which you can easily find).
Are you sure that you want to simulate your model (by simulation I mean that you actually want to sit and look how your Perti net is being executed)? Usually this type of analysis is applicable for small and simple algorithms. In real world situation you probably would like to do model checking of your algorithm rather than simulation. I would recommend you to check SPIN (used by many companies, e.g. Siemens). Also I have a positive experience with Alloy and Prism. But if your focus is on verifying parallel algorithms I would suggest you to consider SPIN first.
Edit: I checked some tools for simulation and I can advise looking at
1) http://sourceforge.net/projects/visual-petri/
2) http://www.renew.de/
3) http://www.winpesim.de/index.html
SPIN is a popular tool for verification of distributed systems but is command line only I think. But on the Spin webpage there is a link to a closely related GUI tool called GOAL
I'm doubt this is what you are looking for, but I'll throw in my two cents:
At my university, in our class on concurrent software systems, we use a tool called Labelled Transition System Analyser (LTSA). It's actually a language that you can use to model the behavior of a system.
The "code" is turned into a state diagram and a transition table.
Here is an interactive Java applet which can design and run a Petri net.
It's been a long time since I've looked at it, but it sounds like Ptolemy would be a good fit.
You can check Petri Net Sim, to simulate common/timed/colored Petrinets, it comes with a nice GUI that displays Petri Net execution in real time.
Try using the concurrency tool, LTSA. Java program. (Labelled Transition System Analyser), to simulate programs. You can download it from:
http://www.doc.ic.ac.uk/ltsa/
But you have to be patient while using it, it can take a couple of hours to learn how to use it. Probably works best while modeling Java programs.
And it's always good to use UML models of course :)
Related
I want to write an algorithm (a bunch of machine learning algorithms) in C/C++ or maybe in Java, possibly in Python. The language doesn't really matter to me - I'm familiar with all of the above.
What matters to me is the testing. I want to train my models using training data. So I have the test input and I know what the output should be and I compare it to the model's output. What kind of test is this? Is it a unit test? How do I approach the problem? I can see that I can write some code to check what I need checking but I want to separate testing from main code. Testing is a well developed field and I've seen this done before but I don't know the name and type of this particular kind of testing so that I can read up on it and not create a mess. I'd be grateful if you could let me know what this testing method is called.
Your best bet is watch the psychology of testing videos from the tetsing God http://misko.hevery.com/
Link of Misko videos:
http://misko.hevery.com/presentations/
And read this Google testing guide http://misko.hevery.com/code-reviewers-guide/
Edited:
Anyone can write tests, they are really simple and there is no magic to write a test, you can simply do something like:
var sut = new MyObject();
var res = sut.IsValid();
if(res != true)
{
throw new ApplicationException("message");
}
That is the theory of course these days we have tools to simplify the tests and we can write something like this:
new MyObject().IsValid().Should().BeTrue();
But what you should do is focus on writing testable code, that's the magic key
Just follow the psychology of testing videos from Misko to get you started
This sounds a lot like Test-Driven Development (TDD), where you create unit-tests ahead of the production code. There are many detailed answers around this site on both topics. I've linked to a couple of pertinent questions to get you started.
If your inputs/outputs are at the external interfaces of your full program, that's black box system testing. If you are going inside your program to zoom in on a particular function, e.g., a search function, providing inputs directly into the function and observing the behavior, that's unit testing. This could be done at function level and/or module level.
If you're writing a machine learning project, the testing and training process isn't really Test-Driven Development. Have you ever heard of co-evolution? You have a set puzzles for your learning system that are, themselves, evolving. Their fitness is determined by how much they confound your cases.
For example, I want to evolve a sorting network. My learning system is the programs that produce networks. My co-evolution system generates inputs that are difficult to sort. The sorting networks are rewarded for producing correct sorts and the co-evolutionary systems are rewarded for how many failures they trigger in the sorting networks.
I've done this with genetic programming projects and it worked quite well.
Probably back testing, which means you have some historical inputs and run your algorithm over them to evaluate the performance of your algorithm. The term you used yourself - training data - is more general and you could search for that to find some useful links.
Its Unit testing. the controllers are tested and the code is checked in and out without really messing up your development code. This process is also called a Test Driven Development(TDD) where your every development cycle is tested before going into the next software iteration or phase.
Although this is a very old post, my 2 cents :)
Once you've decided which algorithmic method to use (your "evaluation protocol", so to say) and tested your algorithm on unitary edge cases, you might be interested in ways to run your algorithm on several datasets and assert that the results are above a certain threshold (individually, or on average, etc.)
This tutorial explains how to do it within the pytest framework, that is the most popular testing framework within python. It is based on an example (comparing polynomial fitting algorithms on several datasets).
(I'm the author, feel free to provide feedback on the github page!)
I'm on a project that among other video related tasks should eventually be capable of extracting the audio of a video and apply some kind of speech recognition to it and get a transcribed text of what's said on the video. Ideally it should output some kind of subtitle format so that the text is linked to a certain point on the video.
I was thinking of using the Microsoft Speech API (aka SAPI). But from what I could see it is rather difficult to use. The very few examples that I found for speech recognition (most are for Text-To-Speech which mush easier) didn't perform very well (they don't recognize a thing). For example this one: http://msdn.microsoft.com/en-us/library/ms717071%28v=vs.85%29.aspx
Some examples use something called grammar files that are supposed to define the words that the recognizer is waiting for but since I haven't trained the Windows Speech Recognition thoroughly I think that might be adulterating the results.
So my question is... what's the best tool for something like this? Could you provide both paid and free options? Well the best "free" (as it comes with Windows) option I believe it's SAPI, all the rest should be paid but if they are really good it might be worth it. Also if you have any good tutorials for using SAPI (or other API) on a context similar to this it would be great.
On the whole this is a big ask!
The issue with any speech recognition system is that it functions best after training. It needs context (what words to expect) and some kind of audio benchmark (what does each voice sound like). This might be possible in some cases, such as a TV series if you wanted to churn through hours of speech -separated for each character- to train it. There's a lot of work there though. For something like a film there's probably no hope of training a recogniser unless you can get hold of the actors.
Most film and TV production companies just hire media companies to transcribe the subtitles based on either direct transcription using a human operator, or converting the script. The fact that they still need humans in the loop for these huge operations suggests that automated systems just aren't up to it yet.
In video you have a plethora of things that make you life difficult, pretty much spanning huge swathes of current speech technology research:
-> Multiple speakers -> "Speaker Identification" (can you tell characters apart? Also, subtitles normally have different coloured text for different speakers)
-> Multiple simultaneous speakers -> The "cocktail party problem" - can you separate the two voice components and transcribe both?
-> Background noise -> Can you pick the speech out from any soundtrack/foley/exploding helicopters.
The speech algorithm will need to be extremely robust as different characters can have different gender/accents/emotion. From what I understand of the current state of recognition you might be able to get a single speaker after some training, but asking a single program to nail all of them might be tough!
--
There is no "subtitle" format that I'm aware of. I would suggest saving an image of the text using a font like Tiresias Screenfont that's specifically designed for legibility in these circumstances, and use a lookup table to cross-reference images against video timecode (remembering NTSC/PAL/Cinema use different timing formats).
--
There's a bunch of proprietary speech recognition systems out there. If you want the best you'll probably want to license a solution off one of the big boys like Nuance. If you want to keep things free the universities of RWTH and CMU have put some solutions together. I have no idea how good they are or how well they might be suited to the problem.
--
The only solution I can think of similar to what you're aiming at is the subtitling you can get on news channels here in the UK "Live Closed Captioning". Since it's live, I assume they use some kind of speech recognition system trained to the reader (although it might not be trained, I'm not sure). It's got better over the past few years, but on the whole it's still pretty poor. The biggest thing it seems to struggle with is speed. Dialogue is normally really fast, so live subtitling has the extra issue of getting everything done in time. Live closed captions quite frequently get left behind and have to miss a lot of content out to catch up.
Whether you have to deal with this depends on whether you'll be subtitling "live" video or if you can pre-process it. To deal with all the additional complications above I assume you'll need to pre-process it.
--
As much as I hate citing the big W there's a goldmine of useful links here!
Good luck :)
This falls into the category of dictation, which is a very large vocabulary task. Products like Dragon Naturally Speaking are amazingly good and that has a SAPI interface for developers. But it's not so simple of a problem.
Normally a dictation product is meant to be single speaker and the best products adapt automatically to that speaker, thereby improving the underlying acoustic model. They also have sophisticated language modeling which serves to constrain the problem at any given moment by limiting what is known as the perplexity of the vocabulary. That's a fancy way of saying the system is figuring out what you're talking about and therefore what types of words and phrases are likely or not likely to come next.
It would be interesting though to apply a really good dictation system to your recordings and see how well it does. My suggestion for a paid system would be to get Dragon Naturally Speaking from Nuance and get the developer API. I believe that provides a SAPI interface, which has the benefit of allowing you to swap in the Microsoft speech or any other ASR engine that supports SAPI. IBM would be another vendor to look at but I don't think you will do much better than Dragon.
But it won't work well! After all the work of integrating the ASR engine, what you will probably find is that you get a pretty high error rate (maybe half). That would be due to a few major challenges in this task:
1) multiple speakers, which will degrade the acoustic model and adaptation.
2) background music and sound effects.
3) mixed speech - people talking over each other.
4) lack of a good language model for the task.
For 1) if you had a way of separating each actor on a separate track that would be ideal. But there's no reliable way of separating speakers automatically in a way that would be good enough for a speech recognizer. If each speaker were at a distinctly different pitch, you could try pitch detection (some free software out there for that) and separate based on that, but this is a sophisticated and error prone task.) The best thing would be hand editing the speakers apart, but you might as well just manually transcribe the speech at that point! If you could get the actors on separate tracks, you would need to run the ASR using different user profiles.
For music (2) you'd either have to hope for the best or try to filter it out. Speech is more bandlimited than music so you could try a bandpass filter that attenuates everything except the voice band. You would want to experiment with the cutoffs but I would guess 100Hz to 2-3KHz would keep the speech intelligible.
For (3), there's no solution. The ASR engine should return confidence scores so at best I would say if you can tag low scores, you could then go back and manually transcribe those bits of speech.
(4) is a sophisticated task for a speech scientist. Your best bet would be to search for an existing language model made for the topic of the movie. Talk to Nuance or IBM, actually. Maybe they could point you in the right direction.
Hope this helps.
I have started learning Data Mining and wish to create a small project in C++/Java that allows me to utilize a database, say from twitter and then publish a particular set of results (for eg. all the news items on a feed). I want to know how to go about it? Where should I start?
This is a really broad question, so it's hard to answer. Here are some things to consider:
Where are you going to get the data? You mention twitter, but you'll still need to collect the data in some way. There are probably libraries out there for listening to twitter streams, or you could probably buy the data if someone is selling it.
Where are you going to store the data? Depending on how much you'll have and what you plan to do with it, a traditional relational database may or may not be the best fit. You may be better off with something that supports running mapreduce jobs out-of-the box.
Based on the answers to those questions, the choice of programming languages and libraries will be easier to make.
If you're really set on Java, then I think a Hadoop cluster is probably what you want to start out with. It supports writing mapreduce jobs in Java, and works as an effective platform for other systems such as HBase, a column-oriented datastore.
If your data are going to be fairly regular (that is, not much variation in structure from one record to the next), maybe Hive would be a better fit. With Hive, you can write SQL-like queries, given only data files as input. I've never used Mahout, but I understand that its machine learning capabilities are suited for data mining tasks.
These are just some ideas that come to mind. There are lots of options out there and choosing between them has as much to do with the particular problem you're trying to solve and your own personal tastes as anything else.
If you just want to start learning about Data Mining there are two books that I particularly really enjoy:
Pattern Recognition and Machine Learning. Christopher M. Bishop. Springer.
And this one, which is for free:
http://infolab.stanford.edu/~ullman/mmds.html
Good references for you are
AI course taught by people who actually know the subject,Weka website, Machine Learning datasets, Even more datasets, Framework for supporting the mining of larger datasets.
The first link is a good introduction on AI taught by Peter Norvig and Sebastian Thrun, Google's Research Director, and Stanley's creator (the autonomous car), respectively.
The second link you get you to Weka website. Download the software - which is pretty intuitive - and get the book. Make sure you understand all the concepts: what's data mining, what's machine learning, what are the most common tasks, and what are the rationales behind them. Play a lot with the examples - the software package bundles some datasets - until you understand what generated the results.
Next, go to real datasets and play with them. When tackling massive datasets, you may face several performance issues with Weka - which is more of a learning tool as far as my experience can tell. Thus I recommend you to take a look at the fifth link, which will get you to Apache Mahout website.
It's far from being a simple topic, however, it's quite interesting.
I can tell you how I did it.
1) I got the data using twitter4j.
2) I analyzed the data using JUNG.
You have to define a class representing edges and a class representing vertices.
These classes will contain the attributes of the edges and vertices.
3) Then, there is a simple function to add an edge g.addedge(V1,V2,edgeFromV1ToV2) or to add a vertex g.addVertex(V).
The class that defines edges or vertices is easy to create. As an example :
`public class MyEdge {
int Id;
}`
The same is done for vertices.
Today I would do it with R, but if you don't want to learn a new programming language, just import jung which is a java library.
Data mining is broad fields with many different techniques; classification, clustering, association and pattern mining, outlier detection, etc.
You should first decide what you want to do and then decide wich algorithm you need.
If you are new to data mining, I would recommend to read some books like Introduction to Data Mining by Tan, Steinbach and Kumar.
I would like to suggest you to use python or R for data mining process. Doing work with java or c , it bit difficult in the sense you need to do a lot coding
I've been working in C++ in embedded environments for a number of years, developing navigation applications. There is a gaming company in my hometown that I like the look of, but I don't have game development experience. You could consider a navigation app as a type of game, depending on who you are running from.
My question is, what steps should I take to enter the industry? Is it a bad idea to enter the industry at this stage (I'm 30)?
Being 30 doesn't really matter, you can enter the games industry at any age assuming you have the drive and ability.
Start reading about gaming topics, and game development websites (gamedev, gamasutra etc.)
Start writing games. Clones of games you like, your own original ideas, tech demos, anything that you can point to and say "I wrote that, and along the way I learned these things, and solved these problems."
If there is a specific area of interest to you, AI, Rendering, Frontend, Tools & Pipelines, Audio, focus on building game/demo/sample projects that challenge you in that area. "Yeah, I've done that" sounds a lot better in an interview than "yeah I've heard of that".
Get to know people in the industry if you can, through online forums, friends of friends, etc... One good contact can do more for your chances than weeks of demo coding or months of sending resumes out. Game companies may have open houses or job fairs.
The "entry level" jobs in game development are likely to be Frontend or Tools. If you've done navigation apps, sounds like that might be a decent fit for you. If that has included more low level work and optimization on embedded platforms, you might also look at Systems roles.
I'd suggest you start trying to write some games in your spare time. Having some demos is always a good start when you go to an interview and it'll give you some insights into what your job is going to be.
Gamedev.net has an excellent set of tutorials to work through to get a grip of a lot of game-coding concepts.
Do they have any job offerings? If so, look at what they ask for in the CV and start educating yourself in those concepts / technologies.
Contacting them and asking if they have any jobs for an excellent software engineer can't hurt either :)
I see you already accepted an answer, but I'll throw in my two cents:
If the company does console (e.g. Xbox360, PS3) or handheld (e.g. DS, iPhone) games, you should definitely emphasize the embedded aspect of your resume. A few anecdotes about how you optimized the memory layout of a class, or sped up some code by taking advantage of an obscure feature of the chipset will show that you can think like a console programmer. Also, if you did any sort of AI for the navigation apps (e.g. A*, Djikstra), it's good to mention that.
A few people recommended writing games - that's not a bad long term plan if you know you want to get into the industry, but I don't think you should let that stop you from applying to this particular company in the meantime. However you should definitely pick up a copy of one of their recent games, play it for a few hours over the weekend, and be able to say what you liked about it.
As for websites, I second the Gamasutra recommendation, along with Kotaku.
Good luck!
"game industry" is a broad question. There are:
AI programming
Graphic programming.
Sound programming.
Tool programming.
Scripting.
Physics programming.
Network programming.
You probably already can deal with #7, #5 and #4.
As for the rest - mostly it is a dealing with some kind of API, plus you need a very good understanding of 3D math (unless you make 2D game, that is).
For 3d math I cannot help you. I picked info in various non-english sources, and most of them aren't available anymore. However, I think this resource might contain info of interest.
For general 3d graphic info you need to study DirectX SDK and NVidia SDKs (both DirectX and OpenGL), plus there are OpenGL books you HAVE to read:
1. Francis s Hill, "Computer Graphics using OpenGL".
2. OpenGL programming guide aka "Red Book"
3. OpenGL shading language (aka "Orange Book")
4. And you might want to take a look at OpenGL reference manucal ("Blue Book")
I'm talking about OpenGL because while it doesn't offer same level of control for hardware resources, it is easier to get started with than DirectX, and available on larger selection of platforms and have a same power as DirectX. Plus GLSL isn't that different from HLSL (except that GLSL doesn't have remnants of assembly shader programming like HLSL), close enough to C++, so it is relatively easy to get started.
One important thing - if you seriously want to deal with 3D, you have to be able to easily imagine 3d operations in your mind. I.e. how to rotate object, scale object, move object, what matrix means, what is reflection vectors, how to cut polygon with planes, how to find intersection of two meshes, etc, and you should have at least basic understanding of more complex thing like boolean operations on polygonal meshes. I have no idea how to develop this skill (it is very close to "mechanical drawing"), but you'll get a lot of difficulties without it.
Just putting "experienced C++ dev" on your CV will probably get you in the door. The (UK at least) games industry is dominated by graduates and inexperienced programmers - the older ones either burn out or get promoted into management.
A lot of games programming is just programming - the skills are entirely transferable. And your navigation software experience probably puts you in for an AI-related role.
If someone with your background applied to me, I'd certainly give them an interview.
Well I started at 16 with (paid) game development. Search for jobs on websites. Make your own low-budget games and then publish them in a way or another.
If you are good people will search for you, otherwise you have to struggle a bit.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
One of the first things I do when I'm on a new project is design a database model. To visualize the model I use a 7 year old version of Smartdraw. Maybe it's time for something new. What is the ultimate program to make a drawing of a database model. Smartdraw is for Windows only. Is there something that can be used on unix as well?
The best tool is pencil and paper.
Perhaps not the answer you are looking for, but sometimes the most simple solution is the best. :-)
A whiteboard (and camera to take a picture afterwards)
Your brain is the best drawing tool.
I prefer to develop a database schema in a simple text file.
At first it contains just the table names, attributes and foreign keys:
company:
company_name
...
employee:
name
age
company_name -> company
...
...:
The syntax is not important.
It just needs to be clearly arranged and easy to change.
Later I add types and CHECK() constraints,
so the text file gradually transforms into valid SQL code.
Using a drawing tool in that early stage is just distracting,
because it encourages to waste your time with moving reactangles.
Instead, let your brain build a picture of the schema
in the same way it creates fantasy pictures while reading a book.
As the schema grows,
it becomes necessary to support the brain by creating an overview.
Here, however, pencil and paper are faster than any drawing software.
Also note that there is no need to clutter an overview with unimportant details.
Just sketch the table names and the most important relationships (foreign keys).
Any further details will decrease the value of this overview.
However, if you really feel more comfortable with extremely detailed graphics than with text files and overview graphics, you might want to try
DBDesigner4
or
WWW SQL Designer.
I'm happy using Sybase PowerDesigner for years now.
Did you take a look at Visio 2007 SQL Server Add-In?
You can find it here: http://dbalink.wordpress.com/2008/04/24/microsoft-office-visio-2007-professional-sql-server-add-in/
WWW SQL Designer is one of the best that I've seen, which is pretty amazing since it is all javascript. It can also import and export xml and sql code of everything you draw. And they've added nifty bezier curves since I last used it.
I don't know if it's the "ultimate" program for drawing database models, but I use Visio. Unfortunately, it only runs in Windows.
On the upside, I can create my own shapes, or modify existing ones, and save them in collections called stencils. I can also make my shapes "smart" by programming them to do various things when I double click on them.
I use a whiteboard and a camera as well. I second the pencil and paper. I keep a pen(cil) and pad of paper with me almost always because I am forever designing something in my head and need to jot it down. I like Visio as well but my favorite software program to use has always been ERWin. The price of that thing is just way out line, but it is great.
I like ERWin. Not Cheap, but it can reverse engineer or do initial design + generate CRUD and manipulate db structure. Viso is pretty good for this too, but its not as complete and of course as strong MS SQL leanings is capabilities.
I used to use Visio, but if your database server is MySQL, try MySQL Workbench. It has a linux version as well as a win32 version. Like their other GUI products it has its fair share of quirks, but it works quite well and has the ability to create a diagram of a schema and a schema from a diagram.
I also agree that a pencil and paper, or whiteboard and camera is a great way to sketch things out, but I do like the GUI tools for putting down an idea which is a bit more well formed or complicated.
I like to use the open-source mind-mapping program Freemind. It's similar to designing with pencil and paper in that there's not a lot of surface complexity to interrupt the design process.
But it has two huge advantages over paper/pencil:
node folding
easy drag-and-drop re-arranging
It is very easy to navigate the interface without using the mouse within about five minutes. You can add as many or as few details as you like and can always fold up the details to de-clutter your view. Here's a sample screenshot:
The circles at the ends of the Departments, Employees, and Hours tables indicate that there are more nodes that are folded up. You can go crazy with different fonts, background colors, and even HTML formatting. I just did a [Ctrl]-[B] to make my table names Bold.
FreeMind--and mind-mapping software in general--provides its biggest benefit by staying out of the way of the creative process. It's the first tool I turn to when starting a new project from scratch.
NOTE: I've only ever used the program on Windows, but it is available on Linux.
DIA is not bad, and there are tools to actually generate some code from some types of models. If you are using PostgreSQL, there is even a tool for going the other way, pg-autodoc.
DIA is available for Unix, and I believe Windows as well.
I used to use Viso but now, as I'm more Mac based I use Omnigraffle.
I do have to admit though, as andyUK does, I do a rough sketch on paper.
It also depends on what Database you're using. If it's MySQL then there are quite a few visual development tools available, just have a google
There may be more technical programs, but I use SmartDraw. I would also like to note that 'ultimate' is up to your circumstances. Find what works best for you or you and your company. What works for you is the 'ultimate'.
Brainstorm/sketch the database on paper/whiteboard first, and then go with a diagram tool.
Which tool depends on your target database. We use SQL Server and thus the designer in SQL Server Management Studio works great for us, since we create the database itself at the same time.
For linux I use umbrello
If you're looking for a cheap solution/tool without any reverse/forward engineering capabilities, MS Visio might be your best bet. However, if you're planning to invest some money toward ERD tool, spend it on ER/Studio.
I used to use and be champion of ERWIN. As soon as I started to use ER/Studio, I became fan of it. I'm an enterprise data architect at one of the Fortune 100 company and I don't know how to get my work done without it.
PS: I don't have no affiliations with any of those products and companies.
OpenOffice has a vector drawing tool, and Inkscape is a another good one.
Otherwise, you can use Graphviz (dot language) to generate such diagram out of a textual description.
There are also some tools to generate such diagram out of an existing database (I first thought it was what you were asking).
If you don't need much take Dia.
I used Power Designer. It's powerful but rather complex.