I have setup a cluster using Ambari that includes 3 nodes .
Now I want to access a file in a HDFS using my client application.
I can find all node URIs under Data Nodes in Amabari.
What is the URI + Port I need to use to access a file ? I have used the default installation process.
Default port is "8020".
You can access the "hdfs" paths in 3 different ways.
Simply use "/" as the root path
For e.g.
E:\HadoopTests\target>hadoop fs -ls /
Found 6 items
drwxrwxrwt - hadoop hdfs 0 2015-08-17 18:43 /app-logs
drwxr-xr-x - mballur hdfs 0 2015-11-24 15:36 /tmp
drwxrwxr-x - mballur hdfs 0 2015-10-20 15:27 /user
Use "hdfs:///"
For e.g.
E:\HadoopTests\target>hadoop fs -ls hdfs:///
Found 6 items
drwxrwxrwt - hadoop hdfs 0 2015-08-17 18:43 hdfs:///app-logs
drwxr-xr-x - mballur hdfs 0 2015-11-24 15:36 hdfs:///tmp
drwxrwxr-x - mballur hdfs 0 2015-10-20 15:27 hdfs:///user
Use "hdfs://{NameNodeHost}:8020/"
For e.g.
E:\HadoopTests\target>hadoop fs -ls hdfs://MBALLUR:8020/
Found 6 items
drwxrwxrwt - hadoop hdfs 0 2015-08-17 18:43 hdfs://MBALLUR:8020/app-logs
drwxr-xr-x - mballur hdfs 0 2015-11-24 15:36 hdfs://MBALLUR:8020/tmp
drwxrwxr-x - mballur hdfs 0 2015-10-20 15:27 hdfs://MBALLUR:8020/user
In this case, "MBALLUR" is the name of my Name Node host.
Related
My repo:
/
dbt-action/
action.yml
Dockerfile
entrypoint.sh
dbt/
profiles.yml
My workflow step:
- name: Run DBT
uses: ./dbt-action
My Dockerfile:
FROM ghcr.io/dbt-labs/dbt-redshift:1.3.latest
COPY dbt .dbt
COPY entrypoint.sh /entrypoint.sh
My entrypoint:
!/bin/bash
pwd
ls -la
Outputs the following:
drwxr-xr-x 6 1001 123 4096 Jan 7 13:06 .
drwxr-xr-x 6 root root 4096 Jan 7 13:06 ..
drwxr-xr-x 8 1001 123 4096 Jan 7 13:06 .git
drwxr-xr-x 3 1001 123 4096 Jan 7 13:06 .github
drwxr-xr-x 3 1001 123 4096 Jan 7 13:06 blah
-rw-r--r-- 1 1001 123 1744 Jan 7 13:06 README.md
drwxr-xr-x 3 1001 123 4096 Jan 7 13:06 dbt-action
Expected output:
Same as above but with additional directory .dbt coming from COPY dbt .dbt in my Dockerfile.
Why don't I see dir .dbt when I ls -la in my entrypoint?
Seems like you are executing your ‘Docker build’ from the wrong working directory, since the ‘dbt-action’ folder is present, but not it contents. Can you double check the PWD before you build?
I have a problem with the privileges of [files/folders] created by Nautilus. As a test I set the UMASK of my specific user to 0000:
[simone#MYPC:~] >cat /mnt/home/simone/.bashrc | grep mask
# User Umask Override
umask 0000
[simone#MYPC:~] >cat /mnt/home/simone/.bashrc
# $HOME/.bashrc
umask 0000
[simone#MYPC:~] >umask
000
When I write a file or folder passing from terminal I can do it with the required privileges:
[simone#MYPC:~/Desktop] >touch file1.txt
[simone#MYPC:~/Desktop] >mkdir Folder1
[simone#MYPC:~/Desktop] >ls -la
-rw-rw-rw-. 1 simone home 0 Mar 21 2022 file1.txt
drwxrwxrwx. 2 simone home 4096 Mar 21 2022 Folder1
But when I create a file (TextEditor) or folder through Nautilus, I get different privileges:
[simone#MYPC:~] >ls -lart /tmp
drwxr-xr-x. 1 simone home 0 Mar 18 10:04 FolderByNautilus
[simone#MYPC:~] > ls -la /tmp/fileNautilus.txt
-rw-r--r--. 1 simone home 5 Mar 11 2022 /tmp/fileNautilus.txt
I would like Nautilus or text editor to write with the mask 0000 leaving the same privileges with which my terminal writes
I am using gcp vm machine instance N1-standard 8V-30GB and N1-standard 4V-15GB
os-Debian
version - Debian GNU/Linux 10(buster)
this issue i am facing from last 1 month.
public access permission denied is one of message i am seeing while trying to access from cloud shell
I had run command chmod 777 <home directory> earlier.
I've tried to reproduce your steps and was able to solve this issue.
Please have a look at my steps below:
create VM instances:
gcloud compute instances create instance-1 --zone=europe-west3-a --machine-type=e2-medium --image=ubuntu-1804-bionic-v20200701 --image-project=ubuntu-os-cloud
gcloud compute instances create instance-2 --zone=europe-west3-a --machine-type=e2-medium --image=ubuntu-1804-bionic-v20200701 --image-project=ubuntu-os-cloud
change permissions recursively on my home directory at the VM instance instance-1:
instance-1:~$ chmod -R 777 ~
instance-1:~$ ls -la
...
drwxrwxrwx 2 username username 4096 Jul 15 07:50 .ssh
create snapshot of the VM instance instance-1 boot disk:
gcloud compute disks snapshot instance-1 --snapshot-names instance-1-snapshot --zone=europe-west3-a
create a new disk with the snapshot:
gcloud compute disks create instance-1-snapshot-disk --zone=europe-west3-a --source-snapshot=instance-1-snapshot
attach created disk instance-1-snapshot-disk to the VM instance instance-2:
instance-2:~$ ls -l /dev/ | grep sd
brw-rw---- 1 root disk 8, 0 Jul 15 07:39 sda
brw-rw---- 1 root disk 8, 1 Jul 15 07:39 sda1
brw-rw---- 1 root disk 8, 14 Jul 15 07:39 sda14
brw-rw---- 1 root disk 8, 15 Jul 15 07:39 sda15
instance-2:~$ mount | grep sda
/dev/sda1 on / type ext4 (rw,relatime)
/dev/sda15 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
then
gcloud compute instances attach-disk instance-2 --disk=instance-1-snapshot-disk --zone=europe-west3-a
after that
instance-2:~$ ls -l /dev/ | grep sd
brw-rw---- 1 root disk 8, 0 Jul 15 07:39 sda
brw-rw---- 1 root disk 8, 1 Jul 15 07:39 sda1
brw-rw---- 1 root disk 8, 14 Jul 15 07:39 sda14
brw-rw---- 1 root disk 8, 15 Jul 15 07:39 sda15
brw-rw---- 1 root disk 8, 16 Jul 15 08:04 sdb
brw-rw---- 1 root disk 8, 17 Jul 15 08:04 sdb1
brw-rw---- 1 root disk 8, 30 Jul 15 08:04 sdb14
brw-rw---- 1 root disk 8, 31 Jul 15 08:04 sdb15
instance-2:~$ sudo mkdir /mnt/instance-1-snapshot-disk
instance-2:~$ sudo mount /dev/sdb1 /mnt/instance-1-snapshot-disk
instance-2:~$ ls -la /mnt/instance-1-snapshot-disk
total 104
drwxr-xr-x 23 root root 4096 Jul 15 07:56 .
drwxr-xr-x 3 root root 4096 Jul 15 08:05 ..
drwxr-xr-x 2 root root 4096 Jul 1 19:14 bin
drwxr-xr-x 4 root root 4096 Jul 1 19:19 boot
drwxr-xr-x 4 root root 4096 Jul 1 19:11 dev
drwxr-xr-x 93 root root 4096 Jul 15 07:55 etc
drwxr-xr-x 4 root root 4096 Jul 15 07:50 home
lrwxrwxrwx 1 root root 30 Jul 1 19:18 initrd.img -> boot/initrd.img-5.3.0-1030-gcp
lrwxrwxrwx 1 root root 30 Jul 1 19:18 initrd.img.old -> boot/initrd.img-5.3.0-1030-gcp
drwxr-xr-x 22 root root 4096 Jul 1 19:17 lib
drwxr-xr-x 2 root root 4096 Jul 1 19:01 lib64
drwx------ 2 root root 16384 Jul 1 19:13 lost+found
drwxr-xr-x 2 root root 4096 Jul 1 19:01 media
drwxr-xr-x 2 root root 4096 Jul 1 19:01 mnt
drwxr-xr-x 2 root root 4096 Jul 1 19:01 opt
drwxr-xr-x 2 root root 4096 Apr 24 2018 proc
drwx------ 3 root root 4096 Jul 15 07:36 root
drwxr-xr-x 4 root root 4096 Jul 1 19:19 run
drwxr-xr-x 2 root root 4096 Jul 1 19:17 sbin
drwxr-xr-x 6 root root 4096 Jul 15 07:36 snap
drwxr-xr-x 2 root root 4096 Jul 1 19:01 srv
drwxr-xr-x 2 root root 4096 Apr 24 2018 sys
drwxrwxrwt 7 root root 4096 Jul 15 07:56 tmp
drwxr-xr-x 10 root root 4096 Jul 1 19:01 usr
drwxr-xr-x 13 root root 4096 Jul 1 19:12 var
lrwxrwxrwx 1 root root 27 Jul 1 19:18 vmlinuz -> boot/vmlinuz-5.3.0-1030-gcp
lrwxrwxrwx 1 root root 27 Jul 1 19:18 vmlinuz.old -> boot/vmlinuz-5.3.0-1030-gcp
change permissions:
.ssh directory: 700 drwx------
public key (.pub file): 644 -rw-r--r--
private key (id_rsa): 600 -rw-------
lastly your home directory should not be writeable by the group or others: 755 drwxr-xr-x
instance-2:~$ chmod -R 755 /mnt/instance-1-snapshot-disk/home/username/
instance-2:~$ chmod -R 700 /mnt/instance-1-snapshot-disk/home/username/.ssh/
instance-2:~$ chmod 644 /mnt/instance-1-snapshot-disk/home/username/.ssh/authorized_keys
unmount the disk when you finish:
instance-2:~$ sudo umount /mnt/instance-1-snapshot-disk/
detach disk instance-1-snapshot-disk from the VM instance instance-2:
gcloud compute instances detach-disk instance-2 --disk=instance-1-snapshot-disk --zone=europe-west3-a
create a new instance from the repaired disk:
gcloud compute instances create instance-3 --zone=europe-west3-a --machine-type=e2-medium --disk=name=instance-1-snapshot-disk
check SSH connection to at the VM instance instance-1.
In addition, please have a look at the documentation Troubleshooting SSH section Inspect the VM instance without shutting it down to find more details.
From owner's account i tried to access instance-1 but owner is also not able to connect to the instance-1.
owner of project got this pop-up on ssh window
[1]: https://i.stack.imgur.com/y2fzC.jpg
I observe that in fresh new created instance if i add add some file like git clone repo, after that if i restart it then i am able to connect SSH again.
If I have a folder with the following files:
hello-version-1-090.txt
hello-awesome-well-091.txt
goodday-087.txt
hellooo-874.txt
hello_476.txt
hello_094.txt
How can I search for a file which has the term: 'hello' and '091' in it using tcl.
Possible solution:
taking the output of an ls -l in a folder, splitting it with '\n' and then running a foreach on each line and using regexp to match the criteria. But how do I run an ls -l in a folder and record its save its contents (the file names) using tcl?
With glob, you can apply the pattern and can get the list of file names matching our criteria.
puts [ exec ls -l ]; #Just printing the 'ls -l' output
set myfiles [ glob -nocomplain hello*091*.txt ]
if {[llength $myfiles]!=0} {
puts "Following files matched your pattern : "
foreach fname $myfiles {
puts $fname
}
} else {
puts "No files matched your pattern"
}
The reason for using -nocomplain is to allow an empty list to be returned without error if there is no file matched with our search pattern.
Output
sh-4.2# tclsh main.tcl
total 4
-rw-r--r-- 1 root root 0 Mar 4 15:23 goodday-087.txt
-rw-r--r-- 1 root root 0 Mar 4 15:23 hello-awesome-well-091.txt
-rw-r--r-- 1 root root 0 Mar 4 15:23 hello-version-1-090.txt
-rw-r--r-- 1 root root 0 Mar 4 15:23 hello_094.txt
-rw-r--r-- 1 root root 0 Mar 4 15:23 hello_476.txt
-rw-r--r-- 1 root root 0 Mar 4 15:23 hellooo-874.txt
-rw-r--r-- 1 root root 262 Mar 4 15:24 main.tcl
Following files matched your pattern :
hello-awesome-well-091.txt
By the way, with respect to your query on how to save the ls -l output, simply save the output to a variable.
set result [ exec ls -l ]
Then with the result variable, you can apply regexp by means of looping line by line just as you have mentioned.
But, I hope that using glob would be a better approach.
Reference : glob
This seems so simple, yet I cannot find a single source that provides a working solution.
I have a project like this:
src
-main
-resources
-bpmns
-nnn.bpmn
and I want gradle to rename all files that end with bpmn to bpmn20.xml, leaving all files that already have bpmn20.xml extensions untouched.
This is my approach:
task renameBPMN(type: Copy) {
outputs.upToDateWhen { false } //to pevent gradle from not executing this task
from('src/main/resources/bpmn/')
into('src/main/resources/bpmn/')
rename ('/^.*\\.(bpmn)$/', '$1.bpmn20.xml')
}
According to some regex testers online, the regex should match nnn.bpmn, gradle however does nothing at all.
Any ideas anyone?
This is what he needs or actually this script would do it.
Here it'll change a file: filenamebpmn or filename.bpmn to filenamebpmn20.xml or filename.bpmn20.xml respectively. It'll ignore all other files whether it's a filenamebpmn.xml or filename.bpmn.xml or any other file.
$ cat build.gradle
task renameBPMN() << {
fileTree("bpmnfolder").matching { include "*bpmn" }.each { fileName ->
println "Will be changed to new name --- " + fileName
file("${fileName}").renameTo("${fileName}20.xml")
}
}
$ ls -l bpmnfolder
total 3
-rw-r--r-- 1 c400093 Domain Users 3 Aug 6 15:14 1bpmn
-rw-r--r-- 1 c400093 Domain Users 3 Aug 6 14:50 2bpmn
-rw-r--r-- 1 c400093 Domain Users 6 Aug 6 14:50 3bpmn.xml
$ /cygdrive/c/gradle-2.0/bin/gradle renameBPMN
:renameBPMN
Will be changed to new name --- C:\cygwin\home\c400093\giga\goga\giga\bpmnfolder\1bpmn
Will be changed to new name --- C:\cygwin\home\c400093\giga\goga\giga\bpmnfolder\2bpmn
BUILD SUCCESSFUL
Total time: 3.153 secs
$ ls -l bpmnfolder
total 3
-rw-r--r-- 1 c400093 Domain Users 3 Aug 6 15:14 1bpmn20.xml
-rw-r--r-- 1 c400093 Domain Users 3 Aug 6 14:50 2bpmn20.xml
-rw-r--r-- 1 c400093 Domain Users 6 Aug 6 14:50 3bpmn.xml
$
AND
If you want to include filenames which end with bpmn as extension i.e. filename.bpmn or filenamebpmn, then use the following code.
$ ls -l bpmnfolder
total 5
-rw-r--r-- 1 c400093 Domain Users 3 Aug 6 15:39 1bpmn
-rw-r--r-- 1 c400093 Domain Users 3 Aug 6 15:39 2bpmn
-rw-r--r-- 1 c400093 Domain Users 6 Aug 6 15:39 3bpmn.xml
-rw-r--r-- 1 c400093 Domain Users 8 Aug 6 15:39 4.bpmn
-rw-r--r-- 1 c400093 Domain Users 3 Aug 6 15:39 5.bpmn.xml
$ cat build.gradle
task renameBPMN() << {
fileTree("bpmnfolder").matching { include "*bpmn" include "*.bpmn" }.each { fileName ->
println "Will be changed to new name --- " + fileName
file("${fileName}").renameTo("${fileName}20.xml")
}
}
$ /cygdrive/c/gradle-2.0/bin/gradle renameBPMN
:renameBPMN
Will be changed to new name --- C:\cygwin\home\c400093\giga\goga\giga\bpmnfolder\1bpmn
Will be changed to new name --- C:\cygwin\home\c400093\giga\goga\giga\bpmnfolder\2bpmn
Will be changed to new name --- C:\cygwin\home\c400093\giga\goga\giga\bpmnfolder\4.bpmn
BUILD SUCCESSFUL
Total time: 4.271 secs
$ ls -l bpmnfolder
total 5
-rw-r--r-- 1 c400093 Domain Users 3 Aug 6 15:39 1bpmn20.xml
-rw-r--r-- 1 c400093 Domain Users 3 Aug 6 15:39 2bpmn20.xml
-rw-r--r-- 1 c400093 Domain Users 6 Aug 6 15:39 3bpmn.xml
-rw-r--r-- 1 c400093 Domain Users 8 Aug 6 15:39 4.bpmn20.xml
-rw-r--r-- 1 c400093 Domain Users 3 Aug 6 15:39 5.bpmn.xml
$
Modify your task to:
task rename(type: Copy) {
outputs.upToDateWhen { false } //to pevent gradle from not executing this task
from('src/main/resources/bpmn/')
into('src/main/resources/bpmn/')
exclude { resource -> !resource.file.name.endsWith('.bpmn')}
rename '(.*).bpmn', '$1.bpmn20.xml'
}
According to the gradle documentation, AbstractCopyTask::rename()'s regular expression does not need delimiters:
rename '(.*)_OEM_BLUE_(.*)', '$1$2'
Try replacing:
rename ('/^.*\\.(bpmn)$/', '$1.bpmn20.xml')
With:
rename ('^.*\\.(bpmn)$', '$1.bpmn20.xml')