salt.states.file.managed takes source_hash as an argument to verify a downloaded file. This blocks me from using file.managed for a file on an online server I don't have control over. The file also changes regularly. My configuration looks like this.
download_stuff:
file.managed:
- name: localfile.tar.gz
- source: http://someserver.net/onlinefile.tar.gz
- source_hash: ???
I don't want to use cmd.run with Curl or wget because this would always download the file, even when it's already on the local machine.
I would like the know if one of the options below is possible/exists:
online md5 calculation service. Is there any way of getting an md5 hash of the file, using a free web service? I'm thinking of something like http://md5service.net?url={url-to-file}.
salt-internal conversion or workaround. Is it possible to handle this in Salt? Maybe by leaving out source_hash somehow?
alternative state. Is there another state in Salt for doing something like this, without losing the benefit of only downloading the file when needed?
If you can't control the other server, please make sure that you can trust it to download its content. Not using a hash will prevent you from detecting partial or corrupted downloads. There's also no way to work with a file that has changed on the remote server.
Nevertheless you could use a state like this to circumvent the hashcode. The creates part will prevent a second download once the file has been downloaded:
bootstrap:
cmd.run:
- name: curl -L https://bootstrap.saltstack.com -o /etc/salt/cloud.deploy.d/bootstrap-salt.sh
- creates: /etc/salt/cloud.deploy.d/bootstrap-salt.sh
Downloading a file with file.managed can be done since version 2016.3.0., even if you don't have access to the hash, by adding skip_verify: True. For the example given, it would be:
download_stuff:
file.managed:
- name: localfile.tar.gz
- source: http://someserver.net/onlinefile.tar.gz
- skip_verify: True
From the docs:
If True, hash verification of remote file sources (http://, https://, ftp://) will be skipped, and the source_hash argument will be ignored.
Related
Summary of the problem
I've setup a repository with the checks I developed, designed to be used exclusively in an another repository.
The problem I've been facing is that when I run pre-commit run -a -v the checks don't go through every file, and time to time they even change!
A little more details
I've run the [identity][1] check and it prints every file in the repo, meaning the files are read by pre-commit (per my understanding)
When I execute pre-commit run the file in staging are correctly parsed
Why do i think that the checks don't run on every file?
I always print something to the standard output for every check, and when I run it verbosely it only prints 5 to 7 lines, meaning it checked only those.
If I try and edit a file (breaking a check) whose not on the short list, the check still passes through
Some code
The structure of the folder containing the files to be checked
./
articles/
some_name/
file.md
file.png
and_so_on.jpeg
team/
some_other_name/
file.md
propic.jpg
Summary of the .pre-commit-hooks.yaml
- id: team-check-name
name: blabla
description: blabla
entry: team-checkname
files: 'team/.*/'
types: [markdown]
language: python
- id: article-check-name
name: blabla
description: blabla
entry: article-checkname
files: 'articles/.*/'
types: [markdown]
language: python
Essentially the checks supposed to run on articles/.*/ start with article- and they all have this property: files: 'articles/.*/'.
Similarly every check supposed to run on team/.*/ starts with team- and they all have files: 'team/.*/'
Summary of the .pre-commit-config.yaml
repos:
- repo: https://github.com/repourl
rev: commit_hash
hooks:
- id: team-check-name
- id: article-check-name
But as you can see, I don't override any settings so it shouldn't interfere
the hook tools you've written are incorrect -- they only process sys.argv[1] rather than positional arguments
your code uses a lot of global variables so it's not a straightforward refactor -- typically you'd either use argparse to collect nargs='*' or loop over sys.argv[1:] (if you don't have any options)
as to why this is the convention -- it's very wasteful to start a linter process over and over to lint a single file (often times executable startup cost dwarfs the actual linting / formatting process)
disclaimer: I wrote pre-commit
I have a container that is built to run selenium-chromedriver with python to download an excel(.xlsx) file from a website.
I am Using SAM to build & deploy this image to be run in AWS Lambda.
When I build the container and invoke it locally, the program executes as expected: The download occurs and I can see the file placed in the root directory of the container.
The problem is: when I deploy this image to AWS and invoke my lambda function I get no errors, however, my download is never executed. The file never appears in my root directory.
My first thought was that maybe I didn't allocate enough memory to the lambda instance. I gave it 512 MB, and the logs said it was using 416MB. Maybe there wasn't enough room to fit another file inside? So I have increased the memory provided to 1024 MB, but still no luck.
My next thought was that maybe the download was just taking a long time, so I also allowed the program to wait for 5 minutes after clicking the download to ensure that the download is given time to complete. Still no luck.
I have also tried setting the following options for chromedriver (full list of chromedriver options posted at bottom):
options.add_argument(f"--user-data-dir={'/tmp'}"),
options.add_argument(f"--data-path={'/tmp'}"),
options.add_argument(f"--disk-cache-dir={'/tmp'}")
and also setting tempfolder = mkdtemp() and passing that into the chrome options as above in place of /tmp. Still no luck.
Since this applicaton is in a container, it should run the same locally as it does on AWS. So I am wondering if it is part of the config outside of the container that is blocking my ability to download a file? Maybe the request is going out but the response is not being allowed back in?
Please let me know if there is anything I need to clarify -- Any help on this issue is greatly appreciated!
Full list of Chromedriver options
options.binary_location = '/opt/chrome/chrome'
options.headless = True
options.add_argument('--disable-extensions')
options.add_argument('--no-first-run')
options.add_argument('--ignore-certificate-errors')
options.add_argument('--disable-client-side-phishing-detection')
options.add_argument('--allow-running-insecure-content')
options.add_argument('--disable-web-security')
options.add_argument('--lang=' + random.choice(language_list))
options.add_argument('--user-agent=' + fake_user_agent.user_agent())
options.add_argument('--no-sandbox')
options.add_argument("--window-size=1920x1080")
options.add_argument("--single-process")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--disable-dev-tools")
options.add_argument("--no-zygote")
options.add_argument(f"--user-data-dir={'/tmp'}")
options.add_argument(f"--data-path={'/tmp'}")
options.add_argument(f"--disk-cache-dir={'/tmp'}")
options.add_argument("--remote-debugging-port=9222")
options.add_argument("start-maximized")
options.add_argument("enable-automation")
options.add_argument("--headless")
options.add_argument("--disable-browser-side-navigation")
options.add_argument("--disable-gpu")
driver = webdriver.Chrome("/opt/chromedriver", options=options)```
Just in case anybody stumbles across this queston in future, adding the following to chrome options solved my issue:
prefs = {
"profile.default_content_settings.popups": 0,
"download.default_directory": r"/tmp",
"directory_upgrade": True
}
options.add_experimental_option("prefs", prefs)
I am trying to mount ecryptfs from within a C++ program. I can definitely mount it without it asking questions by issuing this command at the prompt:
sudo mount -t ecryptfs -o "rw,key=passphrase:passphrase_passwd=geoff,ecryptfs_cipher=aes,ecryptfs_key_bytes=32,ecryptfs_passthrough=n,ecryptfs_enable_filename_crypto=n,no_sig_cache" ~/source/ ~/target/
Note that in reality, I am passing a full canonical path in case that matters.
But from within the program I get failure with errno=EINVAL after trying by using the mount() function with the same arguments:
mount("~/source/", "~/target/", "ecryptfs", MS_NODEV, "rw,key=passphrase:passphrase_passwd=geoff,ecryptfs_cipher=aes,ecryptfs_key_bytes=32,ecryptfs_passthrough=n,ecryptfs_enable_filename_crypto=n,no_sig_cache")
The program does launch with root privileges and I have checked that I have CAP_SYS_ADMIN.
The mount() function returns -1 and sets errno to EINVAL.
Have I got the arguments correct? Is this maybe a privileges issue?
EDIT: I got it to work by executing mount externally via system(), but would still like to use the function because of reasons.
I believe this is because mount -t ecryptfs is actually calling the helper executable mount.ecryptfs, and it's processing some of the options (in particular, key=) itself. What's actually passed to the kernel is different (you can see this by looking at /proc/mounts afterward).
If you look closely at https://manpages.ubuntu.com/manpages/kinetic/en/man7/ecryptfs.7.html, key= and ecryptfs_enable_filename_crypto= are listed under "MOUNT HELPER OPTIONS" - the actual kernel module's options are ecryptfs_sig=(fekek_sig) and ecryptfs_fnek_sig=(fnek_sig).
So, if you want to bypass the helper and do the mount directly, you'd need to load the tokens into the kernel's keyring with https://man7.org/linux/man-pages/man2/keyctl.2.html and replace key= with the resulting token signatures, like mount.ecryptfs did.
It does appear that there is a libecrytpfs with functions in ecryptfs.h like ecryptfs_add_passphrase_key_to_keyring which you can (presumably, not tested) use to do this in a way matching the mount.ecryptfs
At the moment I have to hard code the names of servers on my vimrc in order to either make it different on the remote machine. This is done by conditional statement using hostname() function in vim. I want to make the conditional to be based on the status of remote connection and not on the hostname. So...
The first possible solution I found was using the following bash command in system():
cat /proc/$PPID/status | head -1 | cut -f2
This does not work because I use GNU screen and this will not detect my connection status properly.
The second possible solution I am exploring right now is using who am i This reliably shows whether or not remote connection has been made from which client, but I have trouble getting it working with system()
if substitute(system('who am i'), "theclient", ????, "") == ""
...
How could I get ???? to extract my client name somehow??
Even if the second solution works, allowing me to use .vimrc for many different remote machines, it is still tied to one client. I want the conditional to work in all remote session, regardless of the client name. So I am wondering, is this possible?
The following line allows me to create a variable that detects the remote connection status:
let g:remoteSession = ($STY == "")
Now you can surround the lines that you want to be ignored in the remote connection via:
if g:remoteSession
...
endif
On a side note, I do not know how expensive it is look up the environment variable compared to the global variable, but I am guessing the difference is negligible. The system call in an environment like cygwin where fork() is inefficient, it is worth doing the optimization.
Instead of adding conditional logic to a shared ~/.vimrc, you could alternatively source system-local settings. I use the following:
" Source system-specific .vimrc first.
if filereadable(expand('~/local/.vimrc'))
source ~/local/.vimrc
endif
" Stop sourcing if inclusion guard exists.
if exists('g:loaded_vimrc')
finish
endif
" Common settings of .vimrc here...
I find this more scalable than trying to maintain an ever-changing list of hostnames in a central location.
I've spent all night researching this without a solution.
I'm trying to verify the digital signature of a file in the drives folder (C:\Windows\System32\drivers*.sys) pick whatever one you want. I know that the code is correct because if you move the file from that folder to C:\ the test works.
WinVerifyTrust gives error 80092003
http://pastebin.com/nLR7rvZe
CryptQueryObject gives error 80092009
http://pastebin.com/45Ra6eL4
What's the deal?
0x80092003 = CRYPT_E_FILE_ERROR = An error occurred while reading or writing to the file.
0x80092009 = CRYPT_E_NO_MATCH = No match when trying to find the object.
I'm guessing you're running on a 64-bit machine and WOW64 file system redirection is redirecting you to syswow64\drivers, which is empty. You can disable redirection with Wow64DisableWow64FsRedirection().
if you right click and view properties of file can you see a digital signature? most likely your file is part of a catalogue and you need to use the catalogue API to extract the cert from cert DB and verify it.