hadoop 3.3.1
I am learning Hadoop,and find a command
hadoop fs -ls <args>
What is the possible value of <args>?
OK,I know why.
by default,only '/' directory exists.
Related
Is there any way to push files from local compute to HDFS.
I have tried to send GET request to port 50070 but It always shows
Authentication required.
Please help me ! I am quite new to HDFS
To create new folder
hadoop fs -mkdir your-folder-path-on-HDFS
Example
hadoop fs -mkdir /folder1 /folder2
hadoop fs -mkdir hdfs://:/folder1
Uploading your file
bin/hdfs dfs –put ~/#youruploadfolderpath# /#yourhadoopfoldername#
I am trying to copy a text file on my Mac Desktop to hdfs, for that purpose I am using this code
hadoop fs -copyFromLocal Users/Vishnu/Desktop/deckofcards.txt /user/gsaikiran/cards1
But it is throwing an Error
copyFromLocal: `deckofcards.txt': No such file or directory
It sure exists on the desktop
Your command is missing a slash / at the source file path. It should be:
hadoop fs -copyFromLocal /Users/Vishnu/Desktop/deckofcards.txt /user/gsaikiran/cards1
more correctly/efficiently,
hdfs dfs -put /Users/Vishnu/Desktop/deckofcards.txt /user/gsaikiran/cards1
Also, if you are dealing with HDFS specifically, better to use hdfs dfs syntax instead of hadoop fs [1]. (It doesn't change the output in your case, but hdfs dfs command is designed for interacting with HDFS whereas hadoop fs is the deprecated one)
File exists in HDFS:
hdfs dfs -ls hdfs://nameservice/user/user123/workflow.xml
-rw-r--r-- 3 user123 user123 662 2016-10-24 11:25 hdfs://nameservice/user/user123/workflow.xml
Try the same with oozie:
oozie validate hdfs://nameservice/user/user123/workflow.xml
oozie can't find it:
App definition [File does not exist, hdfs://nameservice/user/user123/workflow.xml] does not exist
Same error if I try to submit the workflow using oozie run.
What are the possible causes/things to check?
You need to provide the local file system path not the HDFS location. This is client side validation.
oozie validate <ARGS> : validate a workflow XML file
I have a crontab that fires a PHP script that runs the AWS CLI command "aws ec2 create-snapshot".
When I run the script via the command line the php script completes successfully with the aws command returning a JSON string to PHP. But when I setup a crontab to run the php script the aws command doesn't return anything.
The crontab is running as the same user as when I run the PHP script on the command line myself, so I am a bit stumped?
I had the same problem with running a ruby script (ruby script.rb).
I replace ruby by its full path (/sources/ruby-2.0.0-p195/ruby) and it worked.
in you case, replace "aws" by its full path. to find it:
find / -name "aws"
The reason it's necessary to specify the full path to the aws command is because cron by default runs with a very limited environment. I ran into this problem as well, and debugged it by adding this to the cron script:
set | sort > /tmp/environment.txt
I then ran the script via cron and via command line (renaming the environment file between runs) and compared them. This led me to see that I needed to set both the PATH and the AWS_DEFAULT_REGION environment variables. After doing this the script worked just fine.
I just want to access the hdfs from web with django,so I use the hadoopy,I just write
def list(reqeust):
return HttpResponse(hadoopy.ls("."))
in views.py,but something is wrong,there are some informations about the error:"IOError at /list/ Ran[hadoop fs -ls .]: /bin/sh: 1: hadoop: not found",I think the "hadoop" command can't be resolved by shell,but I don't know what to do
The hadoopy library you're attempting to use is simply acting as a wrapper over the existing Apache Hadoop bash command scripts (hadoop, hdfs, mapred, etc. commands) and thereby requires those to be installed and available on your OS's or Application's PATH env-var, so it may call a hadoop fs -ls <path> shell command when you attempt to do hadoopy.ls(…).