I am currently running too many sites on a server and I don't think the template cache can handle it. But, what really seems to be the biggest drag is when I load my UDF library per site. I say this because whenever I run Fusion Reactor to see where the holdup is, the stacktrace is always sitting on the template that loads the UDF's.
Is the only cure for this more RAM and a higher template cache, or is there a better way?
Maybe I am wrong as well, could there be another issue?
Before increasing the heap and template cache available, look at a few things.
First, do you actually have more templates in the system than you have template cache? If not, increasing it certainly won't help. Even if you do, if they aren't called often, it probably won't help, but that's harder to measure.
Second, examine whether the server is having difficulty actually loading the UDFs, or if the page is having a problem executing a UDF. Are the functions included on the same template that calls them?
Third, find out why it take so long to load this UDF library. Is it really that big? Can it be split into smaller libraries? Is there one (or more) particular UDF that seems to hang the compile process?
Finally, if there is a large UDF library that must be loaded for each request, I would look at using the Application scope to store it. Include the librar onApplicationStart(), then reference functions as application.myFunction(). This prevents CF from needing to load (and possibly compile) the file at each request.
Related
Description
For a hybrid C/C++ (C++ wrapped in extern "C") application I'm writing I am trying to decide if it is better for me to include some static definitions my program needs to run as global variables or in an external datastore that needs to be read in every time it is run (i.e. a .csv file or sql database).
My static definitions, when laid out in a csv file, take appx 15 columns each with a maximum of 40 definitions (less than that to begin, but up to 40 due to feature scaling).
Problem
I'm torn because it feels wrong to me to include so much data as global variables that get loaded with the program at compile time. However, the overhead of reading from a datastore every time I run the program after compiling seems unnecessary.
Question
What are the best practices here? My code needs to be portable enough for someone else to understand and I do not want to obfuscate it.
It might be appropriate to generate a seperate C file from the CSV using a high level language, e.g. Python. Then either #include the generated file (only if used in a single module using static), or as a seperate compilation unit.
That way you can easily change the values in the CSV/spreadsheet program of your choice, while still having all data available. The code generating program can be called by the build system, so no manual fiddling.
For speed reasons and code clarity staticly define these variables. Lay out your definitions nicely and comment generously to help future viewers of your code. In a file, you can't comment to inform future editors what everything is. It's just slower.
The very best practice is to implement both options with flexibility to switch between implementations based on memory, computation speeds, and other load conditions.
If the application will be run server-side with generous allocation for memory/cpu, then design to those conditions. Why "feel wrong" as you put it?
Your end goal is not clearly defined. So obfuscation is not an issue yet. Obfuscation comes when you willfully redirect your coding to hide your tracks. But making a complete solution, if that is what is needed, is not obfuscation.
currently i'm developing a native-C nodejs Addon to wrap a 3rdparty closed source c-lib and expose it to nodejs.
So far so good. My solution works and native code can be called and worked with.
One problem arises, when calls to those functions are being made subsequently.
The 3rdParty library seems to alocate always the same struct(?; speak: same place in memory) for it's runtime object.
Can I "sandbox" this 3rdparty-lib somehow?
Is there a pattern to solve this? Maybe specific to nodejs-addon development, 'cause nodejs is single threaded, single process-application.
The general suggestion in such cases seems to run the 3rdParty lib in a seperate Process like stated here for example: Isolating and multiply instantiating a C library in-process
I'm not quite sure how to adapt this pattern to nodejs except creating own nodejs processes as "workers" and comunicate to it via rpc somehow. But this seems a little awkward to me and i don't want to reinvent the wheel.
IMHO spawning different node-processes just for this seems to be the 'ugliest' solution.
But correct me if i'm wrong.
Long story, short question. Thanks for adivce.
This is a very very difficult problem, the only solution I have found is to create many copies of the shared object in a temporary directory (with different names) where your program loads each of them once. Most linkers will not realize they are the same and allow you to load them into different address spaces. However this does rely on the code being position independent AFAIK so may not work. It is also nearly as ugly as using IPC so it is up to you what to do.
Is there any way in C++ or Java or Python that would allow me to save the state of my program, no questions asked? For example, I've spent an hour learning how to save a tree-like structure into a file. Very educative but I feel I could just do:
saveState(file);
And the "file" would contain whole memory my program uses. Just like operating system's "hibernate" or "suspend-to-disk" feature. I know about boost serialization, this is probably not what I'm looking for.
What you most likely want is what we call serialization or object marshalling. There are a whole butt load of academic problems with data/object serialization that you can easily google.
That being said given the right library (probably very native) you could do a true snapshot of your running program similarly what "OS specific hibernate" does. Here is an SO answer for doing that on Linux: https://stackoverflow.com/a/12190830/318174
To do the above snapshot-ing though you will most likely need an external process from the process you want to save. I highly recommend you don't that. Instead read/lookup in your language of choice (btw welcome to SO, don't tag every language... that pisses people off) how to do serialization or object marshalling... hint... most people these days pick JSON.
I think that what you describe would be a feature that few people would actually want to use for a real system. Usually you want to save something so it can be transmitted, or so you can stop running the program, or guard against the possibility that the program quits (or power fails).
In most production systems one wants to make the writes to disk small and incremental so that the system can remain responsive, and writing inconsistent data can be avoided. Writing ALL memory to disk on a regular basis would probably result in lots of non-responsive time. You would need to lock the entire system to avoid inconsistent state.
Writing your own persistence is tedious and error prone however so you may find this SO question of interest: Persisting graph data (Java)
There are a couple of frameworks around this. Check out Google Protocol Buffers if you need support for Java, Python, and C++ https://developers.google.com/protocol-buffers/ I've used it in some projects and it works well.
There's also Thrift (orginally from Facebook) http://thrift.apache.org/ I don't have any experience with it though.
Another option is what #QuentinUK suggests. Use a class that inherits from something streamable and/or make streamable operators/functions.
I'd use a framework.
Here's your problem:
http://en.wikipedia.org/wiki/Address_space_layout_randomization
Back in ancient history (16-bit DOS programs with extenders), compilers used to support "based" pointers which stored relative addresses. These were safe to serialize en masse. And applications did so, saving both code and data, the serialized modules were called "overlays".
Today, you'd need based pointer support in your toolchain (resulting in every pointer access requiring an extra adjustment), or else to go through all the data, distinguishing the pointers from the other data (how?) and adjusting them to their new storage location, in case the OS already loaded some library at the same address your old program had used for its heap. In modern "managed" environments, where pointers already have to be identified for the garbage collector, this is feasible even if not commonly done. In native code, it's very difficult, although that metadata is created to enable relocation of shared libraries.
So instead people end up walking their entire data structures manually, and converting object links (pointers) into something that can be restored on the other end, even though the object has a new address (again, because the old address may have been used for a shared library).
Note that many processors have features to support based addressing... and that since based addressing is no longer common, compilers went ahead and used those pointer arithmetic features to speed up user code.
Yes, derive objects from a streamable class and add the streaming functions. Then you can stream everything to disk. You will need a library for this such as MFC.
my problem is pretty complicated and potentially impossible but here we go:
Using C++,
I'm currently working on an universal server engine for a game project of mine. Universal, because every part of the engine will be loaded dynamically after startup. Now, also game objects will inherit from a base object and have overloaded "Simulate" functions. In that way, every object would have it's specific behavior and I can do something I call "C++ Scripting" which is alot faster than interpreted lua script files. Also it's more dynamic.
(Please no solutions which would kill the c++ "scripting" part, like "forget the dynamic linking, that's insane". This performance boost is totally necessary, since I'm working with large voxel maps)
My Problem:
That are indeed alot of .dll/.so files and I wanted to pack those into a simple archive so I can use zlib on said source code and maybe pack everything together with textures and sounds in little "object packages".
Now the Windows DLL API and the Linux SO API won't allow me to load a dll/so file from a memory address, which is a shame.(Am I right there, or can I bypass that? :) ) I don't want to unzip and temp save those files on the filesystem because there are hundreds to thousands of them and that would increase the loading time alot.
Also I'm not interested in more external dependencies like boost.
So here are my Questions:
Is there a cross platform-method to create virtual files IN memory with a real path?
That way I could bypass the slow IO speeds of HDDs.
Or is it really not such a big deal to use temp files, because the file buffers of modern operating systems are fast/intelligent enough to NOT write all those files to disc?
(Actually Linux supports virtual file systems, but windows does not...)
I hope you guys can help me there :)
Not with winapi, that's for sure, but you can do it manually. You can load it into the memory, fill it's import table and call exported functions (after you called DllMain). I saw a program, where someone actually created a new process with that method ... See the PE documentation for details, but it works.
Also it's relatively easy to do, since you only need to find the PE import tables, and do what the dynamic linker does, fill it with jumps and addresses. Dlls contains position independent code, so no relocation needed.
It sould be the same on linux (only using the elf structure), but if you have a better solution with virtual file systems, you should use that.
As the question says, I want to load a DLL from a location in memory instead of a file, similarly to LoadLibrary(Ex). I'm no expert in WinAPI, so googled a little and found this article together with MemoryModule library that pretty much meets my needs.
On the other hand the info there is quite old and the library hasn't been updated for a while too. So I wanted to know if there are different, newer and better ways to do it. Also if somebody has used the library mentioned in the article, could they provide insight on what I might be facing when using it?
Just for the curious ones, I'm exploring the concept of encrypting some plug-ins for applications without storing the decrypted version on disk.
Implementing your own DLL loader can get really hairy really fast. Reading this article it's easy to miss what kind of crazy edge cases you can get yourself into. I strongly recommend against it.
Just for a taste - consider you can't use any conventional debugging tools for the code in the DLL you're loading since the code you're executing is not listed in the region of any DLL known by the OS.
Another serious issue is dealing with DEP in windows.
Well, you can create a RAM Drive according to these instructions, then copy the DLL you can in memory to a file there and the use LoadLibrary().
Of course this is not very practical if you plan to deploy this as some kind of product because people are going to notice a driver being installed, a reboot after the installation and a new drive letter under My Computer. Also, this does nothing to actually hide the DLL since its just sitting there in the RAM Drive for everybody to watch.
Another thing I'm interested about is Why you actually want to do this? Perhaps your end result can be achieved by some other means other than Loading the DLL from memory. For instance when using a binary packer such as UPX, the DLL that you have on disk is different from the one that is eventually executed. Immediately after the DLL is loaded normally with LoadLibrary, The unpacker kicks in and rewrites the memory which the DLL is loaded to with the uncompressed binary (the DLL header makes sure that there is enough space allocated)
Similar question was raised in here:
Load native C++ .dll from RAM in debugger friendly manner
One of the answers proposes dllloader sample application shown in github:
https://github.com/tapika/dllloader
It supports .dll debugging out of box.