I am trying to edit file in EC2 remotely, I spend a while to setup the config.json but I still got timeout error.
I am using mac and I already chmod 400 to .pem file
{
"type": "sftp",
"sync_down_on_open": true,
"host": "xxx.xx.xx.xxx",
"user": "ubuntu",
"remote_path": "/home/ubuntu/",
"connect_timeout": 30,
"sftp_flags": ["-o IdentityFile=/Users/kevinzhang/Desktop/zhang435_ec2.pem"],
}
I figure it out, Just in case anyone also have the same problem
I am use MAC OS
installed ubuntu
the config file is have is looks like
{
// The tab key will cycle through the settings when first created
// Visit http://wbond.net/sublime_packages/sftp/settings for help
// sftp, ftp or ftps
"type": "sftp",
// "save_before_upload": true,
"upload_on_save": true,
"sync_down_on_open": true,
"sync_skip_deletes": false,
"sync_same_age": true,
"confirm_downloads": false,
"confirm_sync": true,
"confirm_overwrite_newer": false,
"host": "xxxx.compute.amazonaws.com",
"user": "ubuntu",
//"password": "password",
"port": "22",
"remote_path": "/home/ubuntu/",
"ignore_regexes": [
"\\.sublime-(project|workspace)", "sftp-config(-alt\\d?)?\\.json",
"sftp-settings\\.json", "/venv/", "\\.svn/", "\\.hg/", "\\.git/",
"\\.bzr", "_darcs", "CVS", "\\.DS_Store", "Thumbs\\.db", "desktop\\.ini"
],
//"file_permissions": "664",
//"dir_permissions": "775",
//"extra_list_connections": 0,
"connect_timeout": 30,
//"keepalive": 120,
//"ftp_passive_mode": true,
//"ftp_obey_passive_host": false,
"ssh_key_file": "~/.ssh/id_rsa",
"sftp_flags": ["-o IdentityFile=<YOUR.PEM FILE path>"],
//"preserve_modification_times": false,
//"remote_time_offset_in_hours": 0,
//"remote_encoding": "utf-8",
//"remote_locale": "C",
//"allow_config_upload": false,
}
If you have permission problem :
chmod -R 0777 /home/ubuntu/YOURFILE/
this just enable read and write for all user
You may want to create a new user if above not working for you:
https://habd.as/sftp-to-ubuntu-server-sublime-text/
I do not know if this makes different , But looks like it start working for me for both user once Icreate a new user
Related
I'm trying to create a Image Label Adjustment Job in Ground Truth and I'm having some trouble. The thing is that I have a dataset of images, in which there are pre-made bounding boxes. I have an external python script that creates the "dataset.manifest" file with the json's of each image. Here are the first four lines of that manifest file:
{"source-ref": "s3://automatic-defect-detection/LM-WNB1-M-0000126254-camera_2_0022.jpg", "bounding-box": {"image_size": [{"width": 2048, "height": 1536, "depth": 3}], "annotations": [{"class_id": 0, "width": 80, "height": 80, "top": 747, "left": 840}]}, "bounding-box-metadata": {"class-map": {"0": "KK"}, "type": "groundtruth/object-detection", "human-annotated": "yes"}}
{"source-ref": "s3://automatic-defect-detection/LM-WNB1-M-0000126259-camera_2_0028.jpg", "bounding-box": {"image_size": [{"width": 2048, "height": 1536, "depth": 3}], "annotations": [{"class_id": 0, "width": 80, "height": 80, "top": 1359, "left": 527}]}, "bounding-box-metadata": {"class-map": {"0": "KK"}, "type": "groundtruth/object-detection", "human-annotated": "yes"}}
{"source-ref": "s3://automatic-defect-detection/LM-WNB1-M-0000126256-camera_3_0006.jpg", "bounding-box": {"image_size": [{"width": 2048, "height": 1536, "depth": 3}], "annotations": [{"class_id": 3, "width": 80, "height": 80, "top": 322, "left": 1154}, {"class_id": 3, "width": 80, "height": 80, "top": 633, "left": 968}]}, "bounding-box-metadata": {"class-map": {"3": "FF"}, "type": "groundtruth/object-detection", "human-annotated": "yes"}}
{"source-ref": "s3://automatic-defect-detection/LM-WNB1-M-0000126253-camera_2_0019.jpg", "bounding-box": {"image_size": [{"width": 2048, "height": 1536, "depth": 3}], "annotations": [{"class_id": 2, "width": 80, "height": 80, "top": 428, "left": 1058}]}, "bounding-box-metadata": {"class-map": {"2": "DD"}, "type": "groundtruth/object-detection", "human-annotated": "yes"}}
Now the problem is that I'm creating private jobs in Amazon Sagemaker to try it out. I have the manifest file and the images in a S3 bucket, and it actually kinda works. So I select the input manifest, activate the "Existing-labels display options". The existing labels for the bounding boxes do not appear automatically, so I have to enter them manually (don't know why), but if I do that and try the preview before creating the adjustment job, the bounding boxes appear perfectly and I can adjust them. The thing is that, me being the only worker invited for the job, the job never apears to start working on it, and it just auto-completes. I can see later that the images are there with my pre-made bounding boxes, but the job never appears to adjust those boxes. I don't have the "Automated data labeling" option activated. Is there something missing in my manifest file?
There can be multiple reasons for this. First of all, the automated labeling option is not support for label adjustment and verification tasks. so thats ruled out.
It looks like you have not setup the adjustment job properly. Some things to check for:
Have you specified the Task timeout and Task expiration time? If these values are practically low, then the tasks would be expired even before somebody can pick them.
Have you checked the "I want to display existing labels from the dataset for this job." box? It should be checked for your case.
Does your existing label are fetched properly? If this is not fetched correctly, either you need to review your manifest file or you need to manually provide the label values(which i guess you are doing)
Since you are the only worker in the workforce. Do you have correct permissions to access the labeling task?
How many images you have? Have you set any minimum batch size while setting the label adjustment job?
I'm trying to modify a script to automate lightsail snapshots, and I am having trouble modifying the jq query.
I'm trying to parse the output of aws lightsail get-instance-snapshots
This is the original line from the script:
aws lightsail get-instance-snapshots | jq '.[] | sort_by(.createdAt) | select(.[0].fromInstanceName == "WordPress-Test-Instance") | .[].name'
which returns a list of snapshot names with one per line.
I need to modify the query so that is does not return all snapshots, but rather only ones where the name start with 'autosnap'. i'm doing this as the script rotates snapshots, but I don't want it to delete snapshots I manually create (which will not start with 'autosnap').
Here is a redacted sample output from aws lightsail get-instance-snapshots
{
"instanceSnapshots": [
{
"location": {
"availabilityZone": "all",
"regionName": "*****"
},
"arn": "*****",
"fromBlueprintId": "wordpress_4_9_2_1",
"name": "autosnap-WordPress-Test-Instance-2018-04-16_01.46",
"fromInstanceName": "WordPress-Test-Instance",
"fromBundleId": "nano_1_2",
"supportCode": "*****",
"sizeInGb": 20,
"createdAt": 1523843190.117,
"fromAttachedDisks": [],
"fromInstanceArn": "*****",
"resourceType": "InstanceSnapshot",
"state": "available"
},
{
"location": {
"availabilityZone": "all",
"regionName": "*****"
},
"arn": "*****",
"fromBlueprintId": "wordpress_4_9_2_1",
"name": "Premanent-WordPress-Test-Instance-2018-04-16_01.40",
"fromInstanceName": "WordPress-Test-Instance",
"fromBundleId": "nano_1_2",
"supportCode": "*****",
"sizeInGb": 20,
"createdAt": 1523842851.69,
"fromAttachedDisks": [],
"fromInstanceArn": "*****",
"resourceType": "InstanceSnapshot",
"state": "available"
}
]
}
I would have thought something like this would work, but I'm not having any luck after many attempts...
aws lightsail get-instance-snapshots | jq '.[] | sort_by(.createdAt) | select(.[0].fromInstanceName == "WordPress-Test-Instance") | select(.[0].name | test("autosnap")) |.[].name'
Any help would be greatly appreciated!
The basic query for making the selection you describe would be:
.instanceSnapshots | map(select(.name|startswith("autosnap")))
(If you didn't need to preserve the array structure, you could go with:
.instanceSnapshots[] | select(.name|startswith("autosnap"))
)
You could then perform additional filtering by extending the pipeline.
If you were to use test/1, the appropriate invocation would be test("^autosnap") or perhaps test("^autosnap-").
Example
.instanceSnapshots
| map(select(.name|startswith("autosnap")))
| map(select(.fromInstanceName == "WordPress-Test-Instance"))
| sort_by(.createdAt)
| .[].name
The two successive selects could of course be compacted into one. For efficiency, the sorting should be done as late as possible.
Postscript
Although you might indeed be able to get away with commencing the pipeline with .[] instead of .instanceSnapshots, the latter is advisable in case the JSON schema changes. In a sense, the whole point of data formats like JSON is to make it easy to write queries that are robust with respect to (sane) schema-evolution.
I'm using bash but I do not get a bash rematch... Every online regex check tool worked fine for this string and regex.
#!/bin/bash
set -x
regex='hd_profile_pic_url_info": {"url": "([0-9a-zA-Z._:\/\-_]*)"'
str='{"user": {"pk": 12345, "username": "dummy", "full_name": "dummy", "is_private": true, "profile_pic_url": "censored", "profile_pic_id": "censored", "is_verified": false, "has_anonymous_profile_picture": false, "media_count": 0, "follower_count": 71114, "following_count": 11111, "biography": "", "external_url": "", "usertags_count": 0, "hd_profile_pic_versions": [{"width": 320, "height": 320, "url": "censored"}, {"width": 640, "height": 640, "url": "censored"}], "hd_profile_pic_url_info": {"url": "https://scontent-frt3-2.cdninstagram.com/vp/censored/censored_a.jpg", "width": 930, "height": 930}, "has_highlight_reels": false, "auto_expand_chaining": false}, "status": "ok"}'
[[ $str =~ $regex ]] && echo ${BASH_REMATCH}
Parsing json with bash it's not a good idea, as others said, jq is the right tool for the job.
Having said that, I think
regex='hd_profile_pic_url_info": {"url": "[0-9a-zA-Z._:\/_-]*"'
would work. Notice the '-' as the last char in the set, to avoid being interpreted as a range.
You have to remove the duplicate _ at the end of your regex :
regex='"hd_profile_pic_url_info": {"url": "([0-9a-zA-Z._:\/\-]*)"'
I am using UBUNTU 14.04.
I have started to explore about querying HDFS using apache drill, installed it my local system and configured the Storage plugin to point remote HDFS. Below is the configuration setup:
{
"type": "file",
"enabled": true,
"connection": "hdfs://devlpmnt.mycrop.kom:8020",
"workspaces": {
"root": {
"location": "/",
"writable": false,
"defaultInputFormat": null
}
},
"formats": {
"json": {
"type": "json"
}
}
}
After creating a json file "rest.json", I passed the query:
select * from hdfs.`/tmp/rest.json` limit 1
I am getting following error:
org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: From line 1, column 15 to line 1, column 18: Table 'hdfs./tmp/rest.json' not found
I would appreciate if someone tries to help me figure out what is wrong.
Thanks in advance!!
I want to get an url to play file media from a dropbox I tested the shared link from the user account with cvlc but it won't work .
how can i generate an url to play media file.
Metadatas don't provide an url to play file it only provide :
https://api.dropbox.com/1/metadata/auto/
{
"size": "225.4KB",
"rev": "35e97029684fe",
"thumb_exists": false,
"bytes": 230783,
"modified": "Tue, 19 Jul 2011 21:55:38 +0000",
"client_mtime": "Mon, 18 Jul 2011 18:04:35 +0000",
"path": "/Getting_Started.pdf",
"is_dir": false,
"icon": "page_white_acrobat",
"root": "dropbox",
"mime_type": "application/pdf",
"revision": 220823
}
I think you're looking for /media, which returns a direct link to the file contents that works for 4 hours.