I'd like to upload files to my s3 bucket via the aws golang sdk. I have a web server listening to POST requests and I'm expecting to receive multiple files of any type.
Using the sdk, the s3 struct PutObjectInput expects Body to be of type io.ReadSeeker and I'm not sure how to extract the content from the files uploaded and in turn satisfy the io.ReadSeeker interface.
images := r.MultipartForm.File
for _, files := range images {
for _, f := range files {
# In my handler, I can loop over the files
# and see the content
fmt.Println(f.Header)
_, err = svc.PutObjectWithContext(ctx, &s3.PutObjectInput{
Bucket: aws.String("bucket"),
Key: aws.String("key"),
Body: FILE_CONTENT_HERE,
})
}
}
Use the FileHeader.Open method to get an io.ReadSeeker.
f, err := f.Open()
if err != nil {
// handle error
}
_, err = svc.PutObjectWithContext(ctx, &s3.PutObjectInput{
Bucket: aws.String("bucket"),
Key: aws.String("key"),
Body: f,
})
Open returns a File. This type satisfies the io.ReadSeeker interface.
Use the S3 Manager's Uploader.Upload method, http://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/#Uploader.Upload. We have an example at http://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-example-basic-bucket-operations.html#s3-examples-bucket-ops-upload-file-to-bucket.
Related
I am trying to upload base64 encoded string to aws s3 bucket in the form of PNG image. I had tried following code:
import (
awsconfig "github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
signature:= "iVBORw0KGgoAAAANSUhEUgAAAfQAAAC0CAYAAABi+d5SAAAAAXNSR0IArs4c6QAAHANJREFUeF7tnWusdkdZhm+kVUSxYg+2klirFWMrrdD+wPKjEg+QCtQayw8PqYApQUPaRhJDhBQUEyJqDyHGeihtQI1oLI02WklBYrWgrWBaRKxQEKgoLWjVWA5ac/utRd9v9333embeWWvNrHVNsrP39+1nZs1cM+++1zwz88zjRIIABCAAAQhAoHkCj2u+BTQAAhCAAAQgAAEh6AwCCEAAAhCAwAIIIOgL6ESaAAEIQAACEEDQGQMQgAAEIACBBRBA0BfQiTQBAhCAAAQggKAzBiAAAQhAAAILIICgL6ATaQIEIAABCEAAQWcMQAACEIAABBZAAEFfQCfSBAhAAAIQgACCzhiAAAQgAAEILIAAgr6ATqQJEIAABCAAAQSdMQABCEAAAhBYAAEEvb1O/HZJx0n6zq7q/vdXdz/7u7++YaNZ/ybJX04Hvx9svfMeL+mzkh6W9MABgz+T9GFJ90u6rT101BgCEIDAcgkg6PX2rcX17E64LdoWaX+vIT0i6c8lvUbSO2uoEHWAAAQgsHYCCHo9I+D8TrA98+4FvJ7a7a7J6yS9uoWKUkcIQAACSyaAoM/Tu559P0fSj0k6uaKZdy6NmyT9QG5m8kEAAhCAwP4EEPT9GUZK2HSff/8CBHxbm6+RdHkEBjYQgAAEIFCeAIJenmlfol3oFu/ehT7ek+op+XRJH6qnOtQEAhCAwHoIIOhl+9pr35d0rvR+53nZJ9Rd2lsk/WjdVaR2EIAABJZJAEHfv18t3BZxu5s3j4vtX3J7JTzUHalrr+bUGAIQgEDjBBD0/A60kF/WCfkaZ+O7yPko22vzsZITAhCAAARyCCDoOdSOzMgtXGufkW+jd4ek8/KwkgsCEIAABHIJIOhp5Czgb9qI0paWu4z1u7qIb+/b+O6SHQXO/xdNm9HlNvO4jR+V9BFJZ0g6qfvlyyVdFCz8BEkPBm0xgwAEIACBAgQQ9DhEu9c9K5/SvW7xtkg75KoFNkWw4y2LWZ4j6c6YqZ7d1TlojhkEIAABCOxLAEEfJmgBd+CUPnb6cI48C8dOv7UTQgv4nOK9qwXXSvJMfSg5YM6NQ0b8HgIQgAAEyhFA0A9n6XPkdrGPMSu3W9vCfY+kd0u6vVy3jlaSPRRXBkr3pjjbkiAAAQhAYCICCPp20BZwC1fJyGf/LultGzNwu9BbSwh6az1GfSEAgdUQQNAf29UODuNZeYmbzTwLt4jfUKkLPXWgI+ipxLCHAAQgMBEBBP1o0F4jfpmkY/bkf7Okqxe4MeyqoNfiiq79e2IkOwQgAAEIRAkg6EdI2cXuTVwviILbYecyPItt0Z0eabrvPo9sDmSXe4QmNhCAAAQKEkDQjwSH8S72fVzsPl7m9fYad6YXHC56b5DTWZLuLvlgyoIABCAAgcMJrF3QPdu0mOfuYvdGN8/I7V5fQ7pf0imBhq59XAUQYQIBCECgLIE1/+H1WWlvfstNf9tdj7pU9/o2Lo8EYH1qI7pcwBwTCEAAAhAoQWCtgr6vmK/1nHVE0D0u1zquSnwmKQMCEIBAFoE1/uH1Wrd3a+cku9gdbMYBYdaYIoLOFaprHBm0GQIQmJ3A2gR9n5n52yW9sLsEZfaOm6kCX5D0+IFn/6ekJ81UPx4LAQhAYLUE1iTo3gDnY1c56Xck/VBOxoXliczQ/1vSExfWbpoDAQhAoHoCaxF072K/L2M3+yckPW8Fx9GiAzUi6C5rLeMqyg07CEAAAqMTWMsfXodfvTCRpnexe1bve8ZJRwgg6IwECEAAApUSWIOg56ybO+Kb85GOJhANLHPagqPlMSYgAAEIVElg6YLuKHAWoZTAMYj57qEaDf16rqS7qhzxVAoCEIDAQgksXdBTXe2I+eED3YF4Ip6L50v6o4V+ZmgWBCAAgSoJLFnQfV7cYV2jCTEfJnWdpEuHzf4/HK6D75AgAAEIQGAiAksVdLvY7Wq3yz2S2AAXoXQkbv2VAdO1RtILoMEEAhCAwDgElirovizlsiAyR3/zbval35QWxHGoGYJegiJlQAACEBiBwBIF3degenYeTcwmo6SOuNvtdh9KMB0ixO8hAAEIFCawREH3TPvsICe72ve5Bz34mMWYMUNfTFfSEAhAYGkEliboUcHp+/HpuNqThvQ7JD07kMM2a73AJoAHEwhAAALlCSxJ0C+QdEsComsk+eY1UpyA7zo/IWCOoAcgYQIBCECgJIElCbpDtB4XhPPRztVOWNcgsM7sc5KODWSx6D8YsMMEAhCAAAQKEViKoKfsajc6ZpDpA8hHAH3BzVD6rKQnDBnxewhAAAIQKEtgCYKeei3qzZIcdIaURiAaqOceSU9LKxprCEAAAhDYl0Drgp56LarPnHumias9feRENxxyZC2dLTkgAAEI7E2gdUGPXhbSg7pIkuO7k9IJROPis5yRzpYcEIAABPYm0LKgR2eMPaQPSDpjb2LrLeAjkk4NNJ+rUwOQMIEABCBQmkCrgp66bv4ZSd+Iq32v4fNIILeXNFKuqg0UiQkEIAABCEQItCjoqevm5oAbODIadttEX6De1cXF3+9p5IYABCAAgWQCLQq647SnhGtlk1bysHhMBt+B7rvQhxLBeoYI8XsIQAACIxFoTdAtKhaXaCJWe5TU4XbRc/5XSLItCQIQgAAEJibQkqBHZ4k9Qq/neibvzVyk/Qg4Lvv5gSJY2ghAwgQCEIDAGARaEXQLs4+opWy4epGkG8aAtsIyozvcn8zGwxWODpoMAQhUQaAFQbeIe93cAWGi6cZE13y03DXamb9PCQwldrgPEeL3EIAABEYk0IKgpwaPYd287IBZ0g53vxTuOkvvFxd/9VEEt0UT9KU+LOGUHV+UBgEIFCJQu6CnBo9h3bzQwNgoJrp3oeQO915c/f3gDXr+Pwvz5vKLf+7j81uI/dXbpCzTpNLzcx6W9HlJj5f0QUm+YvZEST7C90/d122pBWMPAQhAIJVAzYLudXO72lMSm7JSaMVsoy9Vu3a495vp3J+9uG7+vE2gYzVrw8oBee6V9BOSEPY2+oxaQqBJArUKuv/I29Wect6cI1PjDMHoDvc3SvrCRp/ZVU96lIDZ/IKknwEKBCAAgTEI1Cro0XPPPRM2wY0xOo6UabfyQbf3eE9bfskvkXT98ptJCyEAgakJ1Cjoqa52b4LzbJArUcuMHrvIz5X0PEmnSPqWMsVSygYBb8zz+joJAhCAQDECNQp6yq52b4KzmL+vGJF1FNSva5udlzf8ErW5rr0OCvO10ssTL5/v8TwZAhBYIoHaBD26o7rvC4LHbB+VFme7yXuRPvh9iWO5tTZ9TfB8f2vtor4QgMBMBGoT9PsSAsisfd3cs2wfzdr8YpY90wcp47Fe1rgrIx9ZIAABCGwlUJOgp8zO7Wq3kK1h3dwu8bO7pYXeNZ4SNY+hXyeBX5P00jqrRq0gAIEWCdQk6CnXol4k6W0tAg/U2WJ9Yecu9xo34h2AdoiJX/784rctwpv/f9f+i557H+Rm8xH9ccp9dv/7bPpT92sauSEAAQg8SqAWQXeUr5uCHeMIXEs742wBd5vMYe0CbgG2+B70vvj/Dorywf/7tKSHurxTe29Ok3S8pKclHEur5fMX/OhhBgEI1Eyglj8onm1b1CJpKdHgPMu7rBPxMcOTRpiOZdPHPt+cCW+KcD9znlp8x2qvyz1D0vsDD7hf0lMCdphAAAIQCBGoQdCjt3m5Qa3Pzpco4vdIelCSI8r1Yr1tNh0akAswil5m8x5Jz1xAe2kCBCBQCYEaBP1ySVcFebQ4O/cLyyXdda4poWyDSGY1a/0Fawx4b5V0caDgX5L0ioAdJhCAAARCBGoQ9OhmuNauRfVMrRfyUGdUYOT1a8+4nxWsC/HzHwvK6/hPDvB7oaTfC9hhAgEIQCBEYG5BTwnz2koQGYu4vQ61zsYt2t7Z3bvF/bPXsO0yd0rxmHgjGPeDH/1R81WqxwQ+fSd0SxUBU0wgAAEIDBOYW9BTxMOznlo3T3ln+hskfU8lF5nYFe5kke43pB12RGtzpFjgfe59KLXmMRlqT4nfnyPpzkBBvnnt2IAdJhCAAATCBOYW9GhkuFqjwtmt7rXQZ4SJlzO0aPez7H523X/PfcoSPSa5LHLyOT77tYGMd0g6L2CHCQQgAIEwgTkFPWV3e22BZCzkV054Ht6zYYu1v3p3ebiTEwxv6Nb9I1lq9phE6j+GzZu6zY9DZb9S0uuHjPg9BCAAgRQCcwp61N3us8y1BFtxPbwj3wFgxkxus8Xb5/N7t/mYz3PZfsGyxyRyJr5Wj8nYjIbKj3qcniPpT4cK4/cQgAAEUgjMKejRa1JrEQ+/gHhWHhG8lD7obS3iFnDPkue4DvY1Xfsida/NYxKp89g20YAyrsfJkv5l7ApRPgQgsC4Ccwr6/0j6kgDuucXDAu6wtGOEm+1F/OoKdos/0IUuHeqSmjwmQ3Wd8vc/GDyG9o+SvnnKivEsCEBgHQTmEvSUzVdz1dEjwPW0J6H0rNxeB8/E993EVmqU/rikXw8Wxtnz7aB+V5LPlg8lblkbIsTvIQCBLAJziWX0qlQHOfFlF3Mkz8g9My8l5t7Y5pm43eq1Hb+7W9K3BSD7DLtfcjh7/lhY/ysp8nn6EUm/FWCNCQQgAIEkApE/QEkFBo0tbL6YZCi9WtLrhoxG+H2pmbkF0ALu9s6xLh5Bk+ItqWU/Q6RdU9o8V9IfBx/49ZI+FrTFDAIQgECYwFyCblfz+QO1/Jykr51hNpuy23tXExw4xOfTf7WB2WzKUbWnV/xiEh70Ixj+Q3Bd3OPe9xGQIAABCBQnMJegPxJoyc0THA/bVo1opLRteT/RzcZ/MdC+GkxSYgF8UtIpNVS6sjpEg8m42hckzOQraybVgQAEaicwh6D7LLfP6w6lOXa3pxzd2qy/18ed1+71llJ06cNtYu13e8+6788Kdvocn7dg1TCDAARaJzDHH5jofdFTRyJLWUvu+91r5BZyC2NrKWV2zlG17b3rl5w3Bzsed3sQFGYQgEAegTkEPRIhbo6LP1Jd7a6jI8a1uuM7ZXbOUbXtn6+/k/StwY+e185rOaYYrDJmEIBASwTmEPSIW3tqAYnU6aCL3Z6G2o6fRcdeyuzcXggvk7Ta1iiTVLuop8nlMjtPpYs9BCCQTGAOQfc684UDNZ1yN7XF6r0J5809M29ZzI0+ZXb+2m5ZIXlwLTzDrZK+N9hGZudBUJhBAAL5BOYQ9MiRtSnrFXnB6AkvYbaa+gIz9V6G/NE8Xc5Lukh/kSf+a3f8MmKbYvOVks6V9BWSHHb2uyR5fH55F1K5v9DIS0K+e92hln26ZNeGVJfzdQG7zTra03OCJB/T9Gd2s+yTJDm+/Wb6+BZbvxz7b4JD4vqUyB9IcqAjEgQgkEhgSuHsqza0Vu17vseIm74NjdfAHQ0umubYeR+tW9Qu8kLVl3WNJO95IB1NIHqrmnO9KEH8t3H+UklPkXRq5xF4pqQTg5H9Wuy3/+iuoLWwkyAAgQQCcwi612KPO6SOU7l4Pbuwqz16Netc5+ITunPQNPUFhtn5Y5FeJ+nSQdJHDLxp7sygbW/m0xbfJ+n53ezb+ef4nCZWu6i5PQmOi//7RUulMAgsnMAcfyiGgspMtSEuZR15CTHMUyPgTfVi1dJH7BWS3pBQ4V1r5+6Ls7uXSb9QWsT77wnFL97UkSK9ZEGCAAQCBGoU9Ck2EDli1y0BPr3JVC8ZCVVKNh1a6tgscAl7BZIBDWQ4R9KdCYV6b4aXaJy8hORQxxbuXrwTilqtqe9x8H0OJAhAIEBgrYI+5PbfRDfHmfhA1yWZpMRrd8H7rvsmVa4R47/uNqFFq+tY/n4JmGo/SLRerdnZm+EXTBIEIDBAoEZBH3vdNvXM+ZRH6MYYsNGravtnL+EFpjRHR4NzVDjS9AQcVpdd79Nz54kNEqhR0MesU+qRrdZ3eadugvMQbv0FpvTH8FWSfq50oZQXJvDDkn47bI0hBFZMYEzx3IV1yN09Zp1SXM//3J2jbTVCWs6d7myEO3rUet2bcK3z/oFk+Wde/jy9IQJjiucuDEPnoMeqU/SWt77eLZ85zxFzXO1Hj9jjJb0j4Sa1hj72TVXVY9ljkwQBCAwQGEs8D3vsXIKeMjv/kKTTGx09qefr3cwlHMsr2V0O3nJHyQJHKstBmOxBcl19I957uudsXhhkL4OTj399oPM6OTCNA7g8aUu9bPdAF7zGdruS8z/c2d3f1cP2fR7//m8OZHZdbHuapD8JMnF5rg8JAhCoUNCHQq2O8ZKRchmJ/0j5Bq0Wb1FzO9/ZHY1KGfwteyNS2hmx9YvcvRHDCW08Ji3Wfhn+cBci9bYJn1/6UdGNmn8p6VmlH055EFgqgTHEc4jV0C5zv72XFtPoHxDXveV15KGXpW1903J7h8Za6u8de/w3JH1HasaC9nYve/w7bsAHJX1M0u0Fy6+hKEfac8S9oeSXGHtLSBCAQIBAjYI+RmCZqNC1HFAlZUmhHxo3dnGzA0Nl8SYWDs8Ip/hMeJxZsP3Vi7e/l36RrbXTvHP9LYHK/ZckX0JDggAEAgSm+ON1sBpDR6nGEPShnfV9HVudraaEse3b6vVX90Wru/gDwzvJZCgkcVJhB4wt4HaX918W8jWnp3behwgDe/T8uSRBAAIDBOYQdEfO8jrvrlRa0L1L1pewRNLYQW0idUi1eb2kn07MtIQ73RObfKj5X0g6r2SBkj4v6Vc6EbeHiHQ0gc9I8p6PSPKmOF+O5AA/Hru8hEaoYbM6AnMI+pDAlp4lR9fPW7xNbejlaNuA9m5o9wF/FB+lU3J27stbHNnM4kPaTcCcfNlNTvLYPbhk0S9j5JRHHggsgsAcgm5wh/0BLS3oQ5vw+o4s/dyxB0jOWXP/0fNLwNpdvpt984Ju9rdPf/klyXsYvKHu4/sUtKK8Hr9egjjsKuVcHCX3IzxB0smS/N2eAn/5hWLzhfjgv3PrnZJv07vR/7zt/6Jl9u08pmvb2o4K+m+ilyGb9qatQdCHzr33A760qz/6Qcqx8x/DmxLucvczEPPtpKMbtHb1k18E/VVylp8zJlrMk7P3o8V2Uud2CGzekthOrbuaziXofoM+dQet0vHTD3vWZhXGOC43xoDIOWtuMffSQ9Nvn2PAlJSyQWuzCl6isSARGja/Y07qThZ8U34R5IRAcQKteWu/CGAuQT9s1lx6LTs6c5qLRcpozBFzl0887MMp35fo7bCb/g9TOg7bnQR+UtIb4QOBygi0MsE7CttcInaYoHsdo9Qd0tFNYyWfOda4zBXz6yW9ZKxKLaTcSJz/uyS9XdIrF9Lmmprxy5KuqKlC1GX1BJqMnrl0QY/ucK89wEqumN8q6bmr/2jGAHxZF1/cUeL8c598TMr7FX6WdfIYyEyrt0q6ODMv2SBQmkCTXs2lC/oSdrjn7Gb34G7B61D6Q1iqPLvbvMeD9fFSRGPlvFjST3UXyMRyYAWBcQi0tEn6iwSWLuh2kz4j0N81dp5n5ZdJ8ktJauIq1FRi2NdC4ExJ/rJHxBsW5/obVQsP6jE9gWb/fs71YZlqDd3RunyucijVFiHOSwVXJUTS2mxfy/Hoh/qJ36+PgPc32EvlL++J8b93nZBZHx1aXJpA08d75xL0wy5LKeUqjm6Ie1DSCaVHRWZ5rvOVe2wKbHowZjIj2/oI2Htlgbe4+6vfRIvYr28slGyxtceTqWYvSVqyoFsYI+7qv+/uPy85MFLLOqu7TnKfqyIR81Tq2C+ZgAU/Git+8w+4o771+fpocJuR2CJl2qb/6hlH8u3qj74efUS6fcM2b8vv//MLUc/CP/fP69tyMJ///zBec46vvl6b3w/Wp2+P27qI/TJLFnTvTPZtYkPp2m6teshujN/7A+EXD5/FPXaPByDme8AjKwQgAIElEJhL0KdYQ4/e5vTdkm6boTPtPfCmt33e3F1txHyGzuOREIAABGojsGRBrzFCnMX7km4pYF8h91jybkyv+XDZSm2fLOoDAQhAYGICNQp6qSAvNQm61/M8G/cSQAkh9zBxzHvP8vddT5t4yPE4CEAAAhAYgwCCPt45V4v4+d0M2j+XSr6q83IuWimFk3IgAAEILIPAXIJ+2LG1Ujfd3Cvp9EA3vVuSw33umzzztoB7Ft6fl923zIP5mZWXJkp5EIAABBZCYC5B33VG3Bu8PJstcQ7wOkmXBvsph0Mv4G6Lv0rOwg9W2+cjPStnrTzYoZhBAAIQWBuBHCErxcgi6DVgC6HXgS1W/ncp0YoGlnF7/Ezf9jR0FtEz+Zd1F56cWArEIeV8urspjXvMJ4DNIyAAAQi0TGBOQZ+CW+o913dL+pSkn5f0ye7nM7oLIyzmU0WUe1jSb0p6FZvephgmPAMCEIBA+wSWLui3SLqgoW7yksPV3Re71xvqOKoKAQhAYG4CSxd0h1T1jWuRC1rm7AsL+Q2dkJfYPzBnW3g2BCAAAQjMQGDpgm6kvqj++hnYRh6JkEcoYQMBCEAAAoME1iDohvB+SV4LryX1rnXPypmR19Ir1AMCEIBAwwTWIuhnSrqngn66uQsI413rrJFX0CFUAQIQgMBSCKxF0N1fc4i6Y637KFz/hYgv5ZNDOyAAAQhURmBNgm70Pnrm8+YXj9QPDstq8fYM3N8R8JFAUywEIAABCBxNYG2C3rfeN5T51jMHn9knPdTdeOZNdxZw1sP3oUleCEAAAhDIJrBWQe+BOTSs472/VNJXBSh6xu0wrN7M5p+HIssFisQEAhCAAAQgsD+BtQv6JsEXS3I411MknS3pJElP7GbdFvG/knS7JM/KSRCAAAQgAIGqCCDoVXUHlYEABCAAAQjkEUDQ87iRCwIQgAAEIFAVAQS9qu6gMhCAAAQgAIE8Agh6HjdyQQACEIAABKoigKBX1R1UBgIQgAAEIJBHAEHP40YuCEAAAhCAQFUEEPSquoPKQAACEIAABPIIIOh53MgFAQhAAAIQqIoAgl5Vd1AZCEAAAhCAQB4BBD2PG7kgAAEIQAACVRFA0KvqDioDAQhAAAIQyCOAoOdxIxcEIAABCECgKgIIelXdQWUgAAEIQAACeQQQ9Dxu5IIABCAAAQhURQBBr6o7qAwEIAABCEAgjwCCnseNXBCAAAQgAIGqCCDoVXUHlYEABCAAAQjkEUDQ87iRCwIQgAAEIFAVAQS9qu6gMhCAAAQgAIE8Agh6HjdyQQACEIAABKoigKBX1R1UBgIQgAAEIJBHAEHP40YuCEAAAhCAQFUEEPSquoPKQAACEIAABPIIIOh53MgFAQhAAAIQqIoAgl5Vd1AZCEAAAhCAQB4BBD2PG7kgAAEIQAACVRFA0KvqDioDAQhAAAIQyCOAoOdxIxcEIAABCECgKgIIelXdQWUgAAEIQAACeQQQ9Dxu5IIABCAAAQhURQBBr6o7qAwEIAABCEAgjwCCnseNXBCAAAQgAIGqCCDoVXUHlYEABCAAAQjkEUDQ87iRCwIQgAAEIFAVAQS9qu6gMhCAAAQgAIE8Agh6HjdyQQACEIAABKoigKBX1R1UBgIQgAAEIJBH4P8AbePq4jPqwCcAAAAASUVORK5CYII="
decodedSignature, decodeErr := base64.StdEncoding.DecodeString(signature)
if decodeErr != nil {
// handle error here
}
r := bytes.NewReader(decodedSignature)
bucketName := "test"
filepath := "uploads"
contentType := "image/png"
cfg, err := awsconfig.LoadDefaultConfig(context.TODO(),
awsconfig.WithRegion(config.AwsRegion),
)
client := s3.NewFromConfig(cfg)
// Create an uploader with the client and options
uploader := manager.NewUploader(client)
cacheExpireTime := time.Now().AddDate(1, 0, 0)
cacheMaxAgeSecs := "31556926" // seconds in 1 year
// Upload the file to S3.
uploadResp, err := uploader.Upload(context.TODO(), &s3.PutObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(filePath),
Body: r,
ContentType: aws.String(contentType),
Expires: &cacheExpireTime,
CacheControl: aws.String(cacheMaxAgeSecs),
})
But when I try to load the s3 url on which the image is uploaded, then there is black screen as shown in the screenshot:
The Body parameter in the uploader.Upload function accepts io.Reader objects as body, meaning that you either need to use os.Open to open a png you have saved, or in this case, use bytes.NewReader(data) to make your data conform to the io.Reader interface (have the .Read() method).
I found many questions related to uploading image to S3, but I haven't found a post that can help me fully. So, I decided to ask my problem.
I have the case to upload image from the URL to S3, so what I did is just download the image from URL first, and then upload to S3.
// download image form url
resp, _ := http.Get("https://sample-videos.com/img/Sample-jpg-image-50kb.jpg")
byteImage, _ = ioutil.ReadAll(resp.Body)
// upload to s3 (using s3manager)
_, err = s3ManagerClient.Upload(&s3manager.UploadInput{
Bucket: aws.String(awsS3Bucket),
Key: aws.String(awsS3Destination),
Body: bytes.NewReader(byteImage),
})
But, there is a problem with my code, the upload process must wait until the download process is completed. My goal is to upload image to s3 without waiting for the download process to complete, meaning that the download and upload process occurs simultaneously.
Here is my new code (handling error is removed for more readable)
resp, _ := http.Get("https://sample-videos.com/img/Sample-jpg-image-50kb.jpg")
// init s3
sess := session.Must(session.NewSession(&aws.Config{
Region: aws.String(awsRegion),
Credentials: credentials.NewStaticCredentials(awsS3Key, awsS3Secret, ""),
}))
s3ManagerClient := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
u.PartSize = int64(DefaultS3OptionPartSizeInMB)
u.Concurrency = DefaultS3OptionConcurrency
})
buf := make([]byte, 1024) // per kb
for {
n, _ := resp.Body.Read(buf)
if n == 0 {
break
}
_, err = s3ManagerClient.Upload(&s3manager.UploadInput{
Bucket: aws.String(awsS3Bucket),
Key: aws.String(awsS3Destination),
Body: bytes.NewReader(buf[:n]), // upload image per part
})
}
This code successfully uploads the image to S3, but the file is broken, I suspect that the image is not fully uploaded (there is something wrong with the process).
Your problem is that the for-loop will upload the image in chunks of bytes onto the awsS3Destination. What you really are doing is you are replacing the contents of the image with individual chunks of bytes, one at a time.
You can try instead to pass the response body into the s3 uploader:
resp, err := http.Get("https://sample-videos.com/img/Sample-jpg-image-50kb.jpg")
if err != nil {
// handle error
}
_, err = s3ManagerClient.Upload(&s3manager.UploadInput{
Bucket: aws.String(awsS3Bucket),
Key: aws.String(awsS3Destination),
Body: resp.Body, // upload image
})
I am using Amazon S3 bucket to upload files (using GO SDK). I have a requirement to charge client when their directory size exceeds 2GB.
The directory hierarchy inside bucket is like: /BUCKET/uploads/CLIENTID/yyyy/mm/dd
For this, I have searched a lot about it. But could not find anything.
How can I get the directory size inside a bucket using SDK ?
First of all, /uploads/CLIENTID/yyyy/mm/dd is not a directory in S3 bucket, but a prefix. The S3 management UI in AWS Console may trick you to think a bucket has subdirectories, just like your computer file system, but they are prefixes.
Your question really is: How can I get the total size of all objects inside a bucket, with a given prefix?
Hope this code snippet can clear your doubts.
package main
import (
"context"
"fmt"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
// iterate all objects in a given S3 bucket and prefix, sum up objects' total size in bytes
// use: size, err := S3ObjectsSize("example-bucket-name", "/a/b/c")
func S3ObjectsSize(bucket string, prefix string, s3client S3Client) (int64, error) {
output, err := s3client.ListObjectsV2(context.TODO(), &s3.ListObjectsV2Input{
Bucket: aws.String(bucket),
Prefix: aws.String(prefix),
})
if err != nil {
return -1, fmt.Errorf("cannot ListObjectsV2 in %s/%s: %s", bucket, prefix, err.Error())
}
var size int64
for _, object := range output.Contents {
size += object.Size
}
return size, nil
}
// stub of s3.Client for dependency injection
type S3Client interface {
ListObjectsV2(ctx context.Context, params *s3.ListObjectsV2Input, optFns ...func(*s3.Options)) (*s3.ListObjectsV2Output, error)
}
Is there an API to read aws s3 file in Go, I only find the API to download file to local machine, and then read the local downloaded file, but I need to read file in stream (like reading a local file).
I want to be able to read the file in real time, like read 100 bytes, do something to the 100 bytes, and read the last file.
I only find the Go aws s3 API to download the entire file to local machine, and the handle the downloaded local file.
My current test code is this
func main() {
bucket := "private bucket"
item := "private item"
file, err := os.Create("local path")
if err != nil {
exitErrorf("Unable to open file %q, %v", item, err)
}
defer file.Close()
sess, _ := session.NewSession(&aws.Config{
Region: aws.String(" ")},
)
downloader := s3manager.NewDownloader(sess)
numBytes, err := downloader.Download(file,
&s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(item),
})
// Handle the downloaded file
scanner := bufio.NewScanner(file)
for scanner.Scan() {
// Do something
}
}
I will download the file from s3 to local machine and then open the downloaded file and handle each byte.
I wonder can i directly read each line of the file(or read each 100 bytes of the file) from s3
Download() takes a WriterAt, but you want a Reader to read from. You can achieve this in four steps:
Create a fake WriterAt to wrap a Writer:
type FakeWriterAt struct {
w io.Writer
}
func (fw FakeWriterAt) WriteAt(p []byte, offset int64) (n int, err error) {
return fw.w.Write(p)
}
Create an io.Pipe to have the ability to read what is written to a writer:
r, w := io.Pipe()
Set concurrency to one so the download will be sequential:
downloader.Concurrency = 1
Wrap the writer created with io.Pipe() with the FakeWriterAt created in the first step. Use the Download function to write to the wrapped Writer:
go func() {
defer w.Close()
downloader.Download(FakeWriterAt{w},
&s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
})
}()
You can now use the reader from the io.Pipe to read from S3.
The minimum part size is 5 MB according to the documentaiton.
Reference: https://dev.to/flowup/using-io-reader-io-writer-in-go-to-stream-data-3i7b
As far as i understand, you probably need a Range request to get file chunk by chunk.
Here is some pseudo-code:
// Setup input
input := &s3.GetObjectInput{
Bucket: aws.String(BucketName),
Key: aws.String(Path),
}
// calculate position
input.Range = aws.String(fmt.Sprintf("bytes=%d-%d", Position, Offset))
// Get particular chunk of object
result, err := o.Service().GetObject(input)
if err != nil {
return nil, err
}
defer result.Body.Close()
// Read the chunk
b, err := ioutil.ReadAll(result.Body)
Or, if you in some case need a file at once (i can't recommend it), just omit Range and that's it.
I need to get public permanent (not signed) URL of a resource using golang and official aws go sdk. In Java AWS S3 SDK there's a method called getResourceUrl() what's the equivalent in go?
This is how you get presigned URLs using the go sdk:
func GetFileLink(key string) (string, error) {
svc := s3.New(some params)
params := &s3.GetObjectInput{
Bucket: aws.String(a bucket name),
Key: aws.String(key),
}
req, _ := svc.GetObjectRequest(params)
url, err := req.Presign(15 * time.Minute) // Set link expiration time
if err != nil {
global.Log("[AWS GET LINK]:", params, err)
}
return url, err
}
If what you want is just the URL of a public access object you can build the URL yourself:
https://<region>.amazonaws.com/<bucket-name>/<key>
Where <region> is something like us-east-2. So using go it will be something like:
url := "https://%s.amazonaws.com/%s/%s"
url = fmt.Sprintf(url, "us-east-2", "my-bucket-name", "some-file.txt")
Here is a list of all the available regions for S3.
Looks almost clean:
import "github.com/aws/aws-sdk-go/private/protocol/rest"
...
params := &s3.GetObjectInput{
Bucket: aws.String(a bucket name),
Key: aws.String(key),
}
req, _ := svc.GetObjectRequest(params)
rest.Build(req) // aws method to build URL in request object
url = req.HTTPRequest.URL.String() // complete URL to resource