I have some information I want to try to integrate into a project I'm working on. It would be "publically maintained" by volunteers in the community I would be working with.
The idea I had was to keep things simple for the people doing the maintenence: I would give them access to a shared google docs spreadsheet and let them do their thing when stuff needed to be updated. I would update the database using this spreadsheet on some automated daily or weekly interval.
I've never worked with databases before. My project is in C++ and my searching around for a database to use seemed to point to sqlite. The data wouldn't be anything complex, just a unique id with a name. maybe an additional piece of information but unlikely.
I'm mostly concerned about getting the data to be an actual database. I looked around for some conversion tool(I assumed spreadsheet -> database would be trivial?). I'm up for any ideas on how to make this work. Thanks
Using google docs with output as tsv/csv and sticking that in sqlite will work fine for me, thanks.
Related
I need to store and load some data in a C++ application. This data is basically going to end up as a set of tables as per a relational database.
I write the data to tables using something like csv format, then parse them myself and apply the database logic I need in my C++ code. But it seems stupid to reinvent the wheel with this and end up effectively writing my own database engine.
However, using something like a MySQL database seems like massive overkill for what is going to be a single user local system. I have tried setting up a MySQL daemon on my Windows system and I found it rather complex and possibly even impossible without admin privileges. It would be a serious obstacle to deployment as it would need each user's system to have MySQL set up and running.
Is there a reasonable middle ground solution? Something that can provide me with a simple database, accessible from C++, without all the complexities of setting up a full MySQL install?
NB. I have edited this question such that I hope it satisfies those who have voted to close the question. I am not asking for a recommendation for a tool, or someone's favourite tool or the best tools. That would be asking which database engine should I use. I am asking for what tools and design patterns are available to solve a specific programming problem - i.e. how can I get access to database like functionality from a C++ program, without writing my own database engine, nor setting up a full database server. This is conceptually no different to asking e.g. How do I print out the contents of a vector? - it's just a bigger problem. I have described the problem and what has been done so far to solve it. My understanding from the On Topic Page is that this is within scope.
You can try sqlite.
Here are some simple code examples: https://www.tutorialspoint.com/sqlite/sqlite_c_cpp.htm
Is there anything specifically that I should do or refrain from doing? The only thing I have found so far is that I should create the tables initially using Django's ORM.
I haven't found any resources specific to this topic which makes me think the answer is either very obvious or that this isn't done much, which probably means it's a bad idea.
Details: I have a standalone C# program that reads a file and exports the data to a PostgreSQL database. I would like to make a website where users can upload these types of files and query the data generated by the program. I really like python and I have done a number of projects with it (but not Django), and my experience with C# and .NET so far has been fairly negative. I have also done the Django tutorial, some of Tango, and I'm reading Two Scoops of Django, all of which make me think I will like working with Django.
From last few day , I was searching for static code analysis tool for ColdFusion. I have not got a good one till now. I found two.
YASCA
https://code.google.com/p/cf-metrics/
From YASCA I was getting only XSS alerts and some alerts for session mgmt, nothing more than that. I have tried with my entire project.
I am not even able to properly install cf-metrics using ColdFusion10 , After putting the required jar file in the lib folder I was not able to access any one my IIS site
because of some isapi redirect isse.
Any other tools available?
If you're still looking for a ColdFusion Linter, I would recommend CFLint. It's hosted on GitHub and Maven. The parser was updated to use ANTLR4, so it's much faster than previous editions. We're also making it easier to customize than JSLint.
I've looked at this a couple of times in the past, as I maintain a large CF application.
Each time I looked I was unable to find anything suitable. I spent a while looking into using the Railo CFML parser (because it's open source) to build something ourselves and concluded at the time that it was possible, but was no small task.
You may be able to re-examine the Railo approach, but feed the AST from Railo into an existing code analysis tool. I never got that far, but it may be possible to an extent.
I'd love to hear different, but the short answer is that there's not much out there.
I am currently faced with the task of importing around 200K items from a custom CMS implementation into Sitecore. I have created a simple import page which connects to an external SQL database using Entity Framework and I have created all the required data templates.
During a test import of about 5K items I realized that I needed to find a way to make the import run a lot faster so I set about to find some information about optimizing Sitecore for this purpose. I have concluded that there is not much specific information out there so I'd like to share what I've found and open the floor for others to contribute further optimizations. My aim is to create some kind of maintenance mode for Sitecore that can be used when importing large columes of data.
The most useful information I found was on Mark Cassidy's blogpost http://intothecore.cassidy.dk/2009/04/migrating-data-into-sitecore.html. At the bottom of this post he provides a few tips for when you are running an import.
If migrating large quantities of data, try and disable as many Sitecore event handlers and whatever else you can get away with.
Use BulkUpdateContext()
Don't forget your target language
If you can, make the fields shared and unversioned. This should help migration execution speed.
The first thing I noticed out of this list was the BulkUpdateContext class as I had never heard of it. I quickly understood why as a search on the SND forum and in the PDF documentation returned no hits. So imagine my surprise when i actually tested it out and found that it improves item creation/deletes by at least ten fold!
The next thing I looked at was the first point where he basically suggests creating a version of web config that only has the bare essentials needed to perform the import. So far I have removed all events related to creating, saving and deleting items and versions. I have also removed the history engine and system index declarations from the master database element in web config as well as any custom events, schedules and search configurations. I expect that there are a lot of other things I could look to remove/disable in order to increase performance. Pipelines? Schedules?
What optimization tips do you have?
Incidentally, BulkUpdateContext() is a very misleading name - as it really improves item creation speed, not item updating speed. But as you also point out, it improves your import speed massively :-)
Since I wrote that post, I've added a few new things to my normal routines when doing imports.
Regularly shrink your databases. They tend to grow large and bulky. To do this; first go to Sitecore Control Panel -> Database and select "Clean Up Database". After this, do a regular ShrinkDB on your SQL server
Disable indexes, especially if importing into the "master" database. For reference, see http://intothecore.cassidy.dk/2010/09/disabling-lucene-indexes.html
Try not to import into "master" however.. you will usually find that imports into "web" is a lot faster, mostly because this database isn't (by default) connected to the HistoryManager or other gadgets
And if you're really adventureous, there's a thing you could try that I'd been considering trying out myself, but never got around to. They might work, but I can't guarantee that they will :-)
Try removing all your field types from App_Config/FieldTypes.config. The theory here is, that this should essentially disable all of Sitecore's special handling of the content of these fields (like updating the LinkDatabase and so on). You would need to manually trigger a rebuild of the LinkDatabase when done with the import, but that's a relatively small price to pay
Hope this helps a bit :-)
I'm guessing you've already hit this, but putting the code inside a SecurityDisabler() block may speed things up also.
I'd be a lot more worried about how Sitecore performs with this much data... assuming you only do the import once, who cares how long that process takes. Is this going to be a regular occurrence?
I'm trying to code a small program for my friend's company. They build metal cabinets and every cabinet is made out of several parts. So when a customer tells them they need a new cabinet, they tell them the single part numbers. My friend needs now a little tool/database where he can look after those part numbers and if there is an entry, he can download the corresponding blueprint (saved as PDF). Of course the program needs also a function to create a new entry and uploading a PDF file with this entry.
The program needs only to be installed local on one windows machine.
Now I need to know if there is maybe a special way to solve this. It would be helpful if someone could give me some keywords, so I can google it and figure out how to begin :)
I have basic skills in C++ and Java and willing to learn new stuff :)
thanks!
Have used sqlite for database applications and found it to have a lot of functionality and speed. It lacks all the advanced database admin stuff but for single user/embedded databases its ideal. I use it over MySQL because of a significant performance improvement.
It has a Java interface via java.sql.Connection.
I think this is a good opportunity to try something else than C++/Java. In your case I would go for Ruby on Rails and for example the PaperClip extension (http://thoughtbot.com/community/) (haven't used PaperClip myself, though). Ruby on Rails is kind of made for this type of problem.