Fail to upload file to SFTP host with golang - amazon-web-services

Question

I have the following golang function to upload a file to SFTP:
func uploadObjectToDestination(sshConfig SSHConnectionConfig, destinationPath string, srcFile io.Reader) {
// Connect to destination host via SSH
conn, err := ssh.Dial("tcp", sshConfig.sftpHost+sshConfig.sftpPort, sshConfig.authConfig)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
// create new SFTP client
client, err := sftp.NewClient(conn)
if err != nil {
log.Fatal(err)
}
defer client.Close()
log.Printf("Opening file on destination server under path %s", destinationPath)
// create destination file
dstFile, err := client.OpenFile(destinationPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC)
if err != nil {
log.Fatal(err)
}
defer dstFile.Close()
log.Printf("Copying file to %s", destinationPath)
// copy source file to destination file
bytes, err := io.Copy(dstFile, srcFile)
if err != nil {
log.Fatal(err)
}
log.Printf("%s - Total %d bytes copied\n", dstFile.Name(), bytes)
}
The code above works 95% of the cases but fails for some files. The only relation between this files which are failing is the size (3-4kb). The other files which succeed are smaller (0.5-3kb). In some cases files with size 2-3kb are failing as well.
I was able to reproduce the same issue with different SFTP servers.
When changing the failing code (io.Copy) with sftp.Write I can see the same behavior, except that the process does not return an error, instead I see that 0 bytes were copied, which seems to be the same like failing with io.Copy.
Btw, when using io.Copy, the error I receive is Context cancelled, unexpected EOF.
The code is running from AWS lambda and there is no memory or time limit issue.

Answer 1

After few hours of digging, it turns out, my code was the source of the issue.
Here is the answer for future reference:
There was another function not in the original question which downloads the object(s) from S3:
func getObjectFromS3(svc *s3.S3, bucket, key string) io.Reader {
var timeout = time.Second * 30
ctx := context.Background()
var cancelFn func()
ctx, cancelFn = context.WithTimeout(ctx, timeout)
defer cancelFn()
var input = &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
}
o, err := svc.GetObjectWithContext(ctx, input)
if err != nil {
if aerr, ok := err.(awserr.Error); ok && aerr.Code() == request.CanceledErrorCode {
log.Fatal("Download canceled due to timeout", err)
} else {
log.Fatal("Failed to download object", err)
}
}
// Load S3 file into memory, assuming small files
return o.Body
}
The code above is using context and for some reason, the object returned object size was wrong.
Since I don't use contexts here I simply converted my code to use GetObject(input) which fixed the issue.
func getObjectFromS3(svc *s3.S3, bucket, key string) io.Reader {
var input = &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
}
o, err := svc.GetObject(input)
if err != nil {
if aerr, ok := err.(awserr.Error); ok {
switch aerr.Code() {
case s3.ErrCodeNoSuchKey:
log.Fatal(s3.ErrCodeNoSuchKey, aerr.Error())
default:
log.Fatal(aerr.Error())
}
} else {
// Print the error, cast err to awserr.Error to get the Code and
// Message from an error.
log.Fatal(err.Error())
}
}
// Load S3 file into memory, assuming small files
return o.Body
}