I have a bucket with more than ten thousand files. I am getting the following error while doing 'ls -l' folder but working fine for less than 1000 files in the folder. Any suggestions?
ls: reading directory ./: Input/output error
I notice that each listing of a file in folder makes an http call to S3, any chance this # can be increased?
Thanks!
I would like to recommend to take a look at the new project RioFS (Userspace S3 filesystem): https://github.com/skoobe/riofs.
This project is “s3fs” alternative, the main advantages comparing to “s3fs” are: simplicity, the speed of operations and bugs-free code. Currently the project is in the “testing” state, but it's been running on several high-loaded fileservers for quite some time.
We are seeking for more people to join our project and help with the testing. From our side we offer quick bugs fix and will listen to your requests to add new features.
RioFS should work fine with such great number of files per directory, but please try to increase the directory caching timeout in the configuration file (see dir_cache_max_time description in riofs.conf.xml)
Hope it helps you and we are looking forward to seeing you joined our community !
Related
I'm working on a long-term project for some time now. After moving some folders around, correctly refactoring different parts of the code, Webstorm's Dart analyser is showing NO errors, but running either pub serve or pub build fails.
Pub or Dart2JS doesn't seems to be traversing folders correctly.
This is the first error I get:
[Error from Dart2JS on CS_Game_Engine|web/main.dart]:
web/UIElements/NodeSettingsItems/TextVariableEditor.dart:4:1:
Can't read 'file:///Data/ownCloud/cybersecgame/CS%20Game%20Engine/web/UIElements/Data/Variables/Variable.dart' (Could not find asset CS_Game_Engine|web/UIElements/Data/Variables/Variable.dart.).
import '../../Data/Variables/Variable.dart';
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This is interesting, because the file TextVariableEditor.dart, as you see in the error, is located in the web/UIElements/NodeSettingsItems/ directory.
The file I am trying to access (Variables.dart) is located in web/Data/Variables/Variable.dart.
The import statement moves 2 folders up with the ../../, but Pub or Dart2JS is not going 2 folders up, its only going 1 folder up. See the error again:
Can't read 'file:///Data/ownCloud/cybersecgame/CS%20Game%20Engine/web/UIElements/Data/Variables/Variable.dart'
It's looking for the file in the wrong place. I'm not sure what to do to fix this, and I've looked around and can't really find anything. Again, Webstorm's Dart Analyzer shows no issues, but the pub build or pub serve both fail, unable to find the file.
I've used the ../../ many times before in my code, but this is the first time it is causing issues. There are other errors similar to this, but having the same issue of not reading the ../../ correctly.
Update 1:
I've done more testing, and adding an additional ../ to the path's beginning fixes the error during the pub build. However, now Webstorm and it's Dart Analyzer is showing lots of errors since the path is obviously wrong (1 too many ../). This is a workaround, but not a fix. Does anybody know why pub or the Dart2JS is reading the path incorrectly? Somehow skipping one of the ../?
Code within the web directory should only access files in two places:
Things in the web directory, via relative paths
Things in packages via package: imports
#2 is interesting. This can include files within your own project, you just need to put those files in the lib directory and access them via a package:my_proj URI.
See https://github.com/dart-lang/sample-pop_pop_win/ as an example
See also https://www.dartlang.org/tools/pub/package-layout
I've installed Drupal8 locally using Acquia Dev Desktop interface.
I've got error as below in Reports/Status Report:
Configuration directories Not present
Your sites/drupal-8-2-6.dd/settings.php file must define the $config_directories variable as an array containing the names of directories in which configuration files can be found. It must contain a sync key.
I also could not push to Cloud Dev(internet interface) because it shows error as below:
Requirement Problem:
Configuration directories Not present
Your /settings.php file must define the $config_directories variable as an array containing the names of directories in which configuration files can be found. It must contain a sync key.
My setting.php is under ~/Documents/DRUPAL/drupal-8.2.6/sites/default/settings.php
Should I make changings in this file?
What changes need to make to get rid from error?
thanks.
Precise detailed steps for the user6781412 solution at:
https://www.drupal.org/node/2891394
===
#user6781412 : Thanks
I detailed your solution in precise detail for newbies like myself, but this bloody forum keeps regecting it on the grounds that "Your post appears to contain code, blah, blah, blah."
I gave up after about an additional half-hour trying to format my solution for this site, and have instead posted it at the above URL at drupal.org.
You da' one. Peace out.
"All the best; intended."
Christopher James Francis Rodgers
I've solved the problem and fixed error by creating sync dir. outside of my root. So in this case I've created sync.dir under ~/Documents/DRUPAL and left it empty.
Then I've created code that pointing to this dir as below:
$config_directories = array(
// REPLACE THIS PATH for your path
CONFIG_SYNC_DIRECTORY => '~/Documents/DRUPAL/drupal_sync',
);
This fixed error.
Short history of sync directory:
"Sync directory contains configuration files that help move your site from your local development machines to the production servers. In the past, all configuration was contained in the database and moving from development to production was very troublesome".
I'm using boost 1.63 for logging with the following line of code to use Boost config file for logging:
Logger::initFromConfig(logConfigName);
I decided so because I want it to be configurable and this way I could save myself from understanding Boost log setup API. Config file contents are:
[Core]
DisableLogging="false"
[Sinks.SYSLF]
Destination="TextFile"
Asynchronous="true"
AutoFlush="true"
Format="[%TimeStamp(format=\"%Y-%m-%d %H:%M:%S.%f\")%][%Severity%] %Message%"
Target="logs"
FileName="dsmip_%N.log"
RotationTimePoint="00:00:00"
ScanForFiles="Matching"
MaxSize="10000000"
Filter="%Severity% >= info"
[Sinks.Console]
Destination="Console"
AutoFlush="true"
Format="[%TimeStamp(format=\"%Y-%m-%d %H:%M:%S.%f\")%][%Severity%] %Message%"
Filter="%Severity% >= debug"
My problem is autoflush does not work for files. On stdout I get every event instantly, but the files are written first when I exit the application. This is quite embarassing for logging...
First I used 1.58, the situation was the same. What do I do wrong?
Thank you in advance.
Best regards: Balazs Bamer
Google was my friend, I have found the solution here. Boost really operates such that it creates the log file in the application directory, and only copies it to the destination when it is closed (log rotates or the program exists). So, in order to get all the log files appear in a specific directory, one has to include the directory name in the filename pattern as well as in the target.
[Sinks.SYSLF]
...
Target="logs"
FileName="logs/dsmip_%N.log"
Best regards: Balazs
first, I'd like to apologize for my english, it's not my 1st language !
I'm a total n00b in the wonderful world of TFS Build (2010), and I've got a problem.
I'll try to explain it to you using a simple example (but my actual situation is much more complicated).
I have a project with a console application "MyApp1", its location on my computer is "D:\MyProjets\MyApp1".
I have another project "Res" which contains only resources, including a file named emailTemplate.html.
My project "MyApp1" uses this file. Therefore, in the "App.config" file there's a key that stores the path of this resource : "D:\MyProjets\Res\emailTemplate.html"
Finally, I have a test for that application "MyApp1". This test checks if an e-mail has been sent. To send the e-mail "MyApp1" will need the file "emailTemplate.html", and will use the key in the configuration file to find it.
When I run the test on my computer : everything's ok.
But if I build the solution with TFS Build, when the tests are run I have a problem with this resource. During the build, the source files are copied in a directory (for example "D:\Build\1\My build projet\Sources\MyProjets\Res", and therefore "MyApp1" will look for "emailTemplate.html" in "D:\MyProjets\Res\emailTemplate.html" (configuration file) and of course won't be able to find it.
How should I do ?
I already know that my project shouldn't work with resources this way, but it's almost impossible for us to change that now, since it's the way we've been working in my company for a loooong time...
I thought about modifying the BuildProcessTemplate to force the Build server to run a getLatest on the Res projects exactly where I want. But I don't know if it's a good idea, or if it's even possible...
Thanks a lot for your help ! :)
Edit your build definition to include the "Res" project directory in the workspace as well. It should be automatically download/updated at each build (if you use any of the default process templates), and as long as you use relative paths in your tests you should be fine.
I recently wrote a few scripts for Javascript which I am trying to run as a native app on my iPad in offline mode.
The scripts do nothing complex, simply run off a bunch of prompts and generate a document.write() based on the information the user has entered.
So I set up my own little website to host the scripts on and went about setting up an application cache. I am new to this sort of thing however from what I gather, I have followed instructions correctly. To set up my app cache, I did the following:
Added the following two lines to my .htaccess file:
AddType text/cache-manifest .manifest
AddType text/manifest .manifest
Made a new file called "generator.manifest", which has the following lines:
CACHE MANIFEST
generator.html
answergenerator.js
And added in to my generator.html file:
Apart from a few functions used to prompt some iOS features, all I have in my generator.html file is this:
along with the other stock standard html tags.
That is my appcache setup. The generator.manifest file is in the exact same directory as the generator.html file.
When I enter Firefox and go to my site, I can run the script, allow it to cache files for offline usage (Firefox comes up with a prompt), and then run the script in offline mode.
HOWEVER, whenever I try to do this on my iPad (4.2.1), I simply get the error: "Generator could not be opened because it is not connected to the Internet."
Where have I gone wrong? Can someone help me fix this please?
A huge thanks in advance,
Haz.
You might look at this answer. iOS does seem to have some issues. I found that I had to power cycle my iPad (in addition to clearing open Safari's) before it would recognize the appcache.