I'm pretty new to AppFabric and what I'm trying to understand is how to stipulate that I want data to go into the Distributed cache as well as the Local Cache
I read the post here which is doing this based on config. I am not using any XML config but rather creating my objects with configuration programmatically. I am playing around with the following code:-
// Declare array for cache host(s).
List<DataCacheServerEndpoint> servers = new List<DataCacheServerEndpoint>();
servers.Add(new DataCacheServerEndpoint("SERVER1", 10023));
servers.Add(new DataCacheServerEndpoint("SERVER2", 10023));
servers.Add(new DataCacheServerEndpoint("SERVER3", 10023));
DataCacheLocalCacheProperties localCacheConfig;
TimeSpan localTimeout = new TimeSpan(0, 5, 0);
localCacheConfig = new DataCacheLocalCacheProperties(10000, localTimeout, DataCacheLocalCacheInvalidationPolicy.TimeoutBased);
// Setup the DataCacheFactory configuration.
DataCacheFactoryConfiguration factoryConfig = new DataCacheFactoryConfiguration();
factoryConfig.Servers = servers;
factoryConfig.SecurityProperties = new DataCacheSecurity(DataCacheSecurityMode.None, DataCacheProtectionLevel.None);
factoryConfig.LocalCacheProperties = localCacheConfig;
DataCacheFactory factory = DataCacheFactoryExtensions.Create(factoryConfig);
DataCache dataCache = factory.GetCache("MyCache");
dataCache.Put("myKey", "MyValue");
Am I right to assume that because I have added the local cache config to the factoryConfig object that my cached item will be automatically added to local cache as well as the distributed cache?
And therefore if I want items only cached to distributed cache do I just need to drop off adding the local cache config to the factoryConfig object?
Or do I need two separate factory config objects - one for each cache?
You can see here that, yes, the object will be stored in the local cache, if the local cache is enabled:
When local cache is enabled, the cache client stores a reference to the object locally.
The instructions for "enabling the local cache" are exactly as you've done -- basically just using the DataCacheLocalCacheProperties (although the local cache can also be enabled using app.config settings instead).
So it's exactly as you say -- to use the distributed cache only, without the local, then use a DataCache object taken from a DataCacheFactory that does not use DataCacheLocalCacheProperties.
Note also that items in the local cache can be evicted depending on the policies configured:
The lifetime of an object in the local cache is dependent on several factors, such as the maximum number of objects in the local cache and the invalidation policy.
Related
(Node.js API)
I am trying to do the following:
Generate file path like /uploads/${uuid.v4()}.extension
Write the file.
This is the code:
const path = `/uploads/${uuidv4()}.${extname(fileName)}`;
const file = bucket.file(path);
await new Promise((resolve, reject) =>
data
.pipe(file.createWriteStream({ contentType }))
.once('error', reject)
.once('finish', resolve),
);
It works fine. But bothers me to no end that there is that miniscule probability that same UUID will be generated. It is not a practical concern.
How can I upload data to Cloud Storage but get an error if there's a clash? I can check if the file exists beforehand but there is still a race condition technically...
The chance of a collision is not just miniscule: it's astronomically low for UUIDs of significant size. Putting effort into solving the problem of such a collision is not likely to be worth the effort.
That said, if you still want to, you won't be able to do it with Cloud Storage APIs alone, since there is no transactional, atomic API to interact with. If you want a "hard" guarantee that there is no collision, you will need to interact with an entirely different Cloud service that does allow you to effectively "lock" some unique string (e.g. a file path) as a flag for all other processes to check so that they don't collide. Since you are working in Google Cloud, you might want to consider using a database (like any SQL database, or Firestore) with atomic transactional operations to "reserve" the path so that only one process can use it (assuming they all correctly observe this reservation and cooperate as such).
Isn't this exactly what preconditions are for?
Copied from the docs: https://cloud.google.com/storage/docs/uploading-objects#storage-upload-object-nodejs
const options = {
destination: destFileName,
// Optional:
// Set a generation-match precondition to avoid potential race conditions
// and data corruptions. The request to upload is aborted if the object's
// generation number does not match your precondition. For a destination
// object that does not yet exist, set the ifGenerationMatch precondition to 0
// If the destination object already exists in your bucket, set instead a
// generation-match precondition using its generation number.
preconditionOpts: {ifGenerationMatch: generationMatchPrecondition},
};
await storage.bucket(bucketName).upload(filePath, options);
console.log(`${filePath} uploaded to ${bucketName}`);
While there are quite a few questions and answers on here already about the life of different variables within python I am looking for how they translate into the django environment in terms of application scope and endpoint scopes. Here is a simple version of what I am making and I want to ensure that it will behave the way I am expecting it to
my_cache/models/GlobalCache.py:
# This class should be global to the entire application and only
# load when the server is started.
class GlobalCacheobject):
_cache = {}
#classmethod
def fetch(cls):
return cls._cache
#classmethod
def flush(cls):
cls._cache = {}
#classmethod
def load_cache(cls, files_to_load_data_from):
for file in files_to_load_from:
cls._cache[file] = <load file and process its data into an entry>
my_cache/models/InstanceCache.py:
from .GlobalCache import GlobalCache
# This class will contain a reference to the global cache and use it to look
# up entries.
class InstanceCache(object):
def __init__(self, name=None):
self._name = name
self._cache = GlobalCache.fetch()
def fetch_file_data(self, file_name):
cache_entry = self._cache.get(file_name, None)
if cache_entry is None:
raise EntryNotFoundException()
return ReadOnlyInterfaceObject(cache_entry)
The intent is to have GlobalCache have a cls._cache value that will persist as long as the server is running. Calling GlobalCache.flush() will drop its global reference to the data it was tracking and calling GlobalCache.load(files_to_load_from) will populate a new instance of its data from.
The InstanceCache object is then intended to hold a reference to the current version of the data and return read-only objects for the different data sets identified by their original file name.
From my testing this seems to work, though I do not really have the InstanceCache object per se. I can load the global cache, retrieve read only objects to it and then flush the global, load it with new data. The original read only objects still return the values they were originally loaded with, new requests will use the new data values.
What I want to confirm is that GlobalCache will exist as long as the server is running and only alter its data with direct calls to flush() and load_cache(). And that when I hit an endpoint and create an InstanceCache it will keep a reference to the original data only as long as it exists. When the execution on the end point is done I would expect it to go out of scope removing the reference to the global cache and if that was the last one, it goes away and only the new/current data is kept. If it matters I am running Python 2.7.6 and django 1.5.12. Solutions that require an upgrade may be useful as well but it is not an immediate option for me.
The answer here is a maybe, and it also depends a lot on which app server you are using to run django (if you are running multi-process).
So, generally speaking, yes, the GlobalCache will retain its cached contents for the lifetime of the process it is in after it has been initialized.
But InstanceCache, on the other hand, is only guaranteed to be garbage collected at some time after there are no more references to it. Garbage collection is a deep field and there are often teams of people that work on the algorithms so going into exact scenarios is probably outside the scope of an answer on SO. A popular implementation of python is pypy, and you can read more about the garbage collection used in pypy here.
That said, please remember that most app servers are multi-process. Both uwsgi and gunicorn spin up child processes to serve requests. So even though GlobalCache is a singleton in its process, there may be several processes, each with its own GlobalCache. And, this GlobalCache will ultimately be garbage collected/cleaned up when the process exits. Both uwsgi and gunicorn will usually kill child processes after the child services some number of HTTP requests.
I am building a simple CEF3 based browser. I want the cache to be removed/deleted after the user closes/ends all the sessions. First I have tried to store the cache on the hard drive by using Cefsettings.cache_path but the folder is empty.
Here is my code:
CefSettings settings;
const char* path = "E:\\test\\Cefclient\\cache";
//store cache on hdd
CefString(&settings.cache_path).FromASCII(path);
The cache folder is empty, also this path: C:\Users\user\AppData\Local\CEF\User Data, which was generated before I changed the path is empty. What could be the problem? And what method do I use to clear/delete this cache?
you can delete the cache path after you called CefShutDown,
you can also confirm you cache path by "chrome://version"
Basically, the hash on the cache busting file is not updating.
class S3PipelineStorage(PipelineMixin, CachedFilesMixin, S3BotoStorage):
pass
PIPELINE_JS = {
'main.js': {
'output_filename': 'js/main.min.js',
'source_filenames': [
'js/external/underscore.js',
'js/external/backbone-1.0.0.js',
'js/external/bootstrap-2.2.0.min.js',
]
}
}
When I first ran the collectstatic command, it properly created a cache busting file named "main.min.d25bdd71759d.js
Now when I run the command, however, it is failing to overwrite that cached file (and update the hash) during the post process phase.
It keeps updating "main.min.js", such that main.min.js is current with my filesystem. A new cached file, however is not created. It keeps the same old hash even though the underlying main.min.js file has changed.
When I manually delete the cached file on AWS, I get the following message from running collectstatic with verbosity set to 3:
Post-processed 'js/main.min.js' as 'js/main.min.d25bdd71759d.js
settings.DEBUG is set to False
Why won't the hash update?
Try using the manifest storage instead:
class S3PipelineManifestStorage(PipelineMixin, ManifestFilesMixin, S3BotoStorage):
pass
According to the django docs here https://docs.djangoproject.com/en/1.11/ref/contrib/staticfiles/#cachedstaticfilesstorage it's not recommended to use CachedStaticFilesStorage.
Your files names for your static files are probably getting cached. So use the manifest one.
CachedStaticFilesStorage isn’t recommended – in almost all cases ManifestStaticFilesStorage is a better choice. There are several performance penalties when using CachedStaticFilesStorage since a cache miss requires hashing files at runtime. Remote file storage require several round-trips to hash a file on a cache miss, as several file accesses are required to ensure that the file hash is correct in the case of nested file paths.
Note this is also documented at django-pipelines http://django-pipeline.readthedocs.io/en/latest/storages.html#using-with-other-storages
I am trying to develop a tool (in Visual Studio 2010, C#) which can read all the items present in an Appfabric cache and store them in a Table. I don't have to use powershell.
First I thought that If I can get all the regions present in the cache, I can make use of the DataCache.GetObjectsInRegion Method to complete my task. But I was not able to get all the region names from the cache as it does not shows the user defined region names but only the default ones, so now I am giving up on this approach.
Can anyone please guide me here, my main goal is to read all the items present in a cache.
There is no built-in method to list all items in the cache.
You're correct, it's possible to list all items using GetObjectsInRegion for a named cache. You have to know first all regions names (if used) or call GetSystemRegions to get all (default) system regions. A simple foreach will allow you to list all items. When you put something into the cache without region name, it will be added to a system region.
Here is a basic example
// Declare array for cache host(s).
DataCacheServerEndpoint[] servers = new DataCacheServerEndpoint[1];
servers[0] = new DataCacheServerEndpoint("YOURSERVERHERE", 22233);
// Setup the DataCacheFactory configuration.
DataCacheFactoryConfiguration factoryConfig = new DataCacheFactoryConfiguration();
factoryConfig.Servers = servers;
factoryConfig.SecurityProperties = new DataCacheSecurity(DataCacheSecurityMode.None, DataCacheProtectionLevel.None);
// Create a configured DataCacheFactory object.
DataCacheFactory mycacheFactory = new DataCacheFactory(factoryConfig);
// Get a cache client for the default cache
DataCache myCache = mycacheFactory.GetDefaultCache(); //or change to mycacheFactory.GetCache(myNamedCache);
//inserty dummytest data
myCache.Put("key1", "myobject1");
myCache.Put("key2", "myobject2");
myCache.Put("key3", "myobject3");
Random random = new Random();
//list all items in the cache : important part
foreach (string region in myCache.GetSystemRegions())
{
foreach (var kvp in myCache.GetObjectsInRegion(region))
{
Console.WriteLine("data item ('{0}','{1}') in region {2} of cache {3}", kvp.Key, kvp.Value.ToString(), region, "default");
}
}