I am running the webserver in Go on an EC2:
package main
import (
"fmt"
"net/http"
"github.com/gorilla/websocket"
)
var upgrader = websocket.Upgrader{
ReadBufferSize: 1024,
WriteBufferSize: 1024,
CheckOrigin: func(r *http.Request) bool { return true },
}
func echo(w http.ResponseWriter, r *http.Request) {
conn, err := upgrader.Upgrade(w, r, nil)
if err != nil {
fmt.Println(err)
return
}
defer conn.Close()
for {
msgType, msg, err := conn.ReadMessage()
if err != nil {
break
}
conn.WriteMessage(msgType, msg)
}
}
func main() {
fmt.Println("listening")
http.HandleFunc("/echo", echo)
http.ListenAndServe(":8918", nil)
}
And sending a request to it using a Python script:
import websocket
ws = websocket.WebSocket()
ws.connect("ws://myEIP:8918/echo") // redacted elastic IP
ws.send("Hello, WebSocket!")
ws.close()
However this blocks for ~30 seconds before raising an exception:
Exception has occurred: TimeoutError
[Errno 60] Operation timed out
I have my NACL set up to allow everything on port 8918, on both inbound at outbound.
I have the security group set up to allow everything on port 8918 - also outbound, but I believe security group access is stateful, so outbound isn't necessary.
Related
I wrote a lambda function in Go that would handle a form submission and send an email with the captured form data to a configured email address.
I am using AWS SES, or trying to.
My emailService.go has the following code:
package aj
import (
"bytes"
"context"
"encoding/base64"
"fmt"
"io/ioutil"
"mime/multipart"
"net/http"
"net/smtp"
"os"
"path/filepath"
"strings"
"go.uber.org/zap"
)
var (
host = os.Getenv("EMAIL_HOST")
username = os.Getenv("EMAIL_USERNAME")
password = os.Getenv("EMAIL_PASSWORD")
portNumber = os.Getenv("EMAIL_PORT")
from = os.Getenv("SOURCE_EMAIL")
)
type EmailService struct {
auth smtp.Auth
}
type EmailSender interface {
SendEmail(logger *zap.Logger, ctx context.Context, m *Message) error
}
func NewEmailService() *EmailService {
auth := smtp.PlainAuth("", username, password, host)
return &EmailService{auth}
}
func (s *EmailService) SendEmail(logger *zap.Logger, ctx context.Context, m *Message) error {
return smtp.SendMail(fmt.Sprintf("%s:%s", host, portNumber), s.auth, from, m.To, m.ToBytes())
}
func NewMessage(s, b string) *Message {
return &Message{Subject: s, Body: b, Attachments: make(map[string][]byte)}
}
func (m *Message) AttachFile(src string) error {
b, err := ioutil.ReadFile(src)
if err != nil {
return err
}
_, fileName := filepath.Split(src)
m.Attachments[fileName] = b
return nil
}
func (m *Message) ToBytes() []byte {
buf := bytes.NewBuffer(nil)
withAttachments := len(m.Attachments) > 0
buf.WriteString(fmt.Sprintf("Subject: %s\n", m.Subject))
buf.WriteString(fmt.Sprintf("To: %s\n", strings.Join(m.To, ",")))
if len(m.CC) > 0 {
buf.WriteString(fmt.Sprintf("Cc: %s\n", strings.Join(m.CC, ",")))
}
if len(m.BCC) > 0 {
buf.WriteString(fmt.Sprintf("Bcc: %s\n", strings.Join(m.BCC, ",")))
}
buf.WriteString("MIME-Version: 1.0\n")
writer := multipart.NewWriter(buf)
boundary := writer.Boundary()
if withAttachments {
buf.WriteString(fmt.Sprintf("Content-Type: multipart/mixed; boundary=%s\n", boundary))
buf.WriteString(fmt.Sprintf("--%s\n", boundary))
} else {
buf.WriteString("Content-Type: text/plain; charset=utf-8\n")
}
buf.WriteString(m.Body)
if withAttachments {
for k, v := range m.Attachments {
buf.WriteString(fmt.Sprintf("\n\n--%s\n", boundary))
buf.WriteString(fmt.Sprintf("Content-Type: %s\n", http.DetectContentType(v)))
buf.WriteString("Content-Transfer-Encoding: base64\n")
buf.WriteString(fmt.Sprintf("Content-Disposition: attachment; filename=%s\n", k))
b := make([]byte, base64.StdEncoding.EncodedLen(len(v)))
base64.StdEncoding.Encode(b, v)
buf.Write(b)
buf.WriteString(fmt.Sprintf("\n--%s", boundary))
}
buf.WriteString("--")
}
return buf.Bytes()
}
As you can see, I am using the standard library net/smtp package. Is that Ok to do when you want to send an email through SES? Of course, I am passing my SES SMTP credentials for the necessary/required environment variables.
Right now, I'm not getting any errors, but I am also not receiving emails even though the count on "emails sent" on the SES Dashboard (within the AWS Console) increments each time I send a request to my lambda function.
Am I going about this the wrong way? It is not working because I must use the SES package within the AWS SDK?
In the HTTP handler, I'm doing:
err = r.ParseMultipartForm(10 << 20)
if err != nil {
logger.Error("error parsing form data", zap.Error(err))
http.Error(w, "error parsing form data", http.StatusBadRequest)
return
}
I believe there's a limit of 10Mb on SES emails, and I believe the 10 << 20 ensure I don't go over? Correct me if I'm wrong please.
This bounty has ended. Answers to this question are eligible for a +50 reputation bounty. Bounty grace period ends in 21 hours.
Abhisek Roy wants to draw more attention to this question:
Looking for a workaround to the current issue.
I was implementing RDS IAM authentication from my Golang service and came across an issue. I am using the init function to call create a function where I have created a sync for every 10 mins where every 10 mins the token should get refreshed, it is creating the new token but it's not passing the same to the main function. When the existing thread that was initiated between DB and the service gets killed it is unable to re-authenticate the connection for the second time. Some observations that I have noticed were-
When initially deploying the application it is creating the connection to RDS successfully but after some time as soon as the initial thread between service and DB gets killed it's unable to authenticate the reason which I checked was- the address where the token is getting stored. While making an API call the service picks the token from a static address where the initial token gets stored at the time of service deployment. However, while the token gets refreshed every 10 mins its getting stored on dynamic addresses from where the service is unable to pick up the token.
Here is the go file where actually I am calling and creating the DB function-
package db
import (
"fmt"
"log"
"os"
"time"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/rds/rdsutils"
"github.com/go-co-op/gocron"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database/postgres"
_ "github.com/golang-migrate/migrate/v4/source/file"
"github.com/jinzhu/gorm"
_ "github.com/lib/pq"
)
const DatabaseVersion = 1
const Success = "Database connection successful"
const database = "postgres"
// Service ...
type Service interface {
InitDatabaseConnection(db **gorm.DB)
CreateDatabaseConnection(db **gorm.DB)
Migrate(db *gorm.DB) error
}
type service struct {
}
// NewService ...
func NewService() Service {
return &service{}
}
func (s *service) InitDatabaseConnection(db **gorm.DB) {
log.Print("Creating Db Connection")
s.CreateDatabaseConnection(db)
//todo: add if; if env for iam run the below code
// todo: change 1 min to 10 mins
startTime := time.Now().Local().Add(time.Minute + time.Duration(1))
sch := gocron.NewScheduler(time.UTC)
sch.Every(600).Seconds().StartAt(startTime).Do(func() {
log.Printf("refreshing rds creds")
s.CreateDatabaseConnection(db)
})
sch.StartAsync()
}
func (s *service) CreateDatabaseConnection(db **gorm.DB) {
configuration, _ := common.New()
var dbURI = ConnectDataBase(configuration.UserName, configuration.Password,
configuration.Port, configuration.DbName, configuration.Host, configuration.SearchPath)
var err error
*db, err = gorm.Open(database, dbURI)
if err != nil {
panic(err)
}
print("\n \n", dbURI)
print("\n \n in server.go \n \n ", &db)
if err != nil {
log.Println(err.Error())
os.Exit(3)
} else {
log.Println(Success)
}
(*db).DB().SetMaxOpenConns(6)
(*db).DB().SetConnMaxLifetime(600 * time.Second)
(*db).DB().SetMaxIdleConns(2)
(*db).DB().SetConnMaxIdleTime(100 * time.Second)
}
func ConnectDataBase(username string, password string, port int, dbName string, host string, searchPath string) string {
var dbURI string
if searchPath == "" {
searchPath = "public"
}
dbEndpoint := fmt.Sprintf("%s:%d", host, port)
sess := session.Must(session.NewSession())
creds := sess.Config.Credentials
authToken, err := rdsutils.BuildAuthToken(dbEndpoint, "us-east-1", username, creds)
print("\n \n", authToken)
if err != nil {
panic(err)
}
dbURI = fmt.Sprintf("host=%s port=%d user=%s dbname=%s sslmode=require password=%s search_path=%s",
host, port, username, dbName, authToken, searchPath) //Build connection string
// if password != "" && username != "" {
// dbURI = fmt.Sprintf("host=%s port=%d user=%s dbname=%s sslmode=require password=%s search_path=%s",
// host, port, username, dbName, authToken, searchPath) //Build connection string
// } else {
// dbURI = fmt.Sprintf("host=%s port=%d dbname=%s sslmode=disable ",
// host, port, dbName) //Build connection string
// }
return dbURI
}
func (s *service) Migrate(db *gorm.DB) error {
migrationSourceURL := "file://../resources/db/migrations/"
database := db.DB()
row := db.Table("schema_migrations").Limit(1).Row()
var version int8
var dirty bool
err := row.Scan(&version, &dirty)
if err == nil {
log.Printf("database is currently on version : %v \n", version)
}
log.Println("Migrating database to version ", DatabaseVersion)
driver, err := postgres.WithInstance(database, &postgres.Config{})
if err != nil {
return err
}
m, err := migrate.NewWithDatabaseInstance(migrationSourceURL, "postgres", driver)
if err != nil {
return err
}
m.Log = LogService{}
err = m.Migrate(uint(DatabaseVersion))
if err != nil {
log.Println(err.Error())
return err
}
return nil
}
The common property above is from this piece of code-
package common
import (
"fmt"
"log"
"os"
"github.com/spf13/viper"
)
type ApiResponse struct {
Code int `json:"code"`
Message string `json:"message"`
}
type Constants struct {
Port int `json:"port"`
Host string `json:"host"`
DbName string `json:"dbName"`
UserName string `json:"userName"`
Password string `json:"password"`
AuthUserName string `json:"authUserName"`
AuthPassword string `json:"authPassword"`
SearchPath string `json:"searchPath"`
AwsRegion string `json:"awsRegion"`
}
func New() (*Constants, error) {
fmt.Println("reached constants")
config := Constants{}
constants, err := initViper()
config = constants
if err != nil {
return &config, err
}
return &config, nil
}
func initViper() (Constants, error) {
// temporary, will be removed later when auto deploy is available
env := os.Getenv("CS_ENV")
var configName string
if env == "develop" {
configName = "config-develop"
} else {
configName = "config-local"
}
viper.SetConfigName(configName) // Constants fileName without the .TOML or .YAML extension
viper.AddConfigPath("../resources/config") // Search the root directory for the configuration file
err := viper.ReadInConfig() // Find and read the config file
if err != nil {
log.Println(err.Error()) // Handle errors reading the config file
return Constants{}, err
}
if err = viper.ReadInConfig(); err != nil {
log.Panicf("Error reading config file, %s", err)
}
var constants Constants
err = viper.Unmarshal(&constants)
return constants, err
}
Also attaching the main function go file where I am calling the DB function for connecting the service to DB-
package main
import (
"bufio"
"encoding/base64"
"encoding/json"
"fmt"
"log"
"net/http"
"os"
"strings"
"time"
gokitlog "github.com/go-kit/kit/log"
"github.com/go-kit/kit/log/level"
"github.com/gorilla/mux"
"github.com/jinzhu/gorm"
_ "github.com/lib/pq"
)
func runServer(config *common.Constants) {
var logger gokitlog.Logger
logger = gokitlog.NewLogfmtLogger(gokitlog.NewSyncWriter(os.Stderr))
logger = gokitlog.With(logger, "ts", gokitlog.DefaultTimestampUTC)
dbService := db2.NewService()
var db *gorm.DB
dbService.InitDatabaseConnection(&db)
err := dbService.Migrate(db)
if err != nil {
fmt.Println("Migrations scripts did not run :", err.Error())
if err.Error() != "no change" {
fmt.Println("failed to run migrations hence exiting", err.Error())
os.Exit(1)
}
}
level.Info(logger).Log("Starting Darwin ==========>")
repo, err := allocation.NewUserAllocationRepository(db, logger)
if err != nil {
level.Error(logger).Log("exit", err)
os.Exit(3)
}
userAllocationService := allocation.NewUserAllocationService(repo, logger)
endpoints := allocation.MakeEndpoints(userAllocationService)
router := mux.NewRouter()
sh := http.StripPrefix("/swaggerui/", http.FileServer(http.Dir("../resources/swaggerui/")))
router.PathPrefix("/swaggerui/").Handler(sh)
httpLogger := gokitlog.With(logger, "component", "http")
subRouter := router.PathPrefix("/api/v1").Subrouter()
subRouter.NotFoundHandler = http.HandlerFunc(notFound)
subRouter.Use(checkBasicAuth(config))
subRouter.HandleFunc("/healthCheck", healthCheck).Methods("GET")
allocation.MakeHandler(subRouter, httpLogger, endpoints)
http.Handle("/", subRouter)
log.Print("Running server on port 8080")
f, err := os.OpenFile("../performance.log", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
log.Printf("error opening file: %v", err)
log.Fatal(http.ListenAndServe(":8080", router))
} else {
by := bufio.NewWriter(f)
defer f.Close()
log.Fatal(http.ListenAndServe(":8080", Logger(by, router)))
}
}
func healthCheck(w http.ResponseWriter, r *http.Request) {
resp := map[string]string{
"Status": "Success",
}
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(resp)
}
func notFound(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json;")
w.WriteHeader(http.StatusNotFound)
json.NewEncoder(w).Encode(map[string]string{"error": "api not found"})
}
func authHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json;")
w.WriteHeader(http.StatusUnauthorized)
json.NewEncoder(w).Encode(map[string]string{"error": "Not authorized"})
}
func checkBasicAuth(config *common.Constants) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.RequestURI() != "/api/v1/healthCheck" {
w.Header().Set("WWW-Authenticate", `Basic realm="Restricted"`)
w.Header().Set("Content-Type", "application/json;")
s := strings.SplitN(r.Header.Get("Authorization"), " ", 2)
if len(s) != 2 {
authHandler(w, r)
return
}
b, err := base64.StdEncoding.DecodeString(s[1])
if err != nil {
authHandler(w, r)
return
}
pair := strings.SplitN(string(b), ":", 2)
if len(pair) != 2 {
authHandler(w, r)
return
}
if pair[0] != config.AuthUserName || pair[1] != config.AuthPassword {
authHandler(w, r)
return
}
}
next.ServeHTTP(w, r)
})
}
}
func Logger(out *bufio.Writer, h http.Handler) http.Handler {
logger := log.New(out, "", 0)
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
o := &responseObserver{ResponseWriter: w, status: http.StatusOK}
start := time.Now()
h.ServeHTTP(o, r)
endTime := time.Now()
if !strings.Contains(r.URL.String(), "/health") && strings.Contains(r.URL.String(), "/api") {
logger.Printf("Method: %s Path: %s Status: %d ExecutionTime: %d ms", r.Method, r.URL, o.status, endTime.Sub(start).Milliseconds())
out.Flush()
}
})
}
type responseObserver struct {
http.ResponseWriter
status int
}
func (r *responseObserver) WriteHeader(status int) {
r.status = status
r.ResponseWriter.WriteHeader(status)
}
I am building a dApp using #cosmjs.
I want to make a transaction on the COSMOS chain on my dApp running on the JUNO chain.
When I tried to get the signingClient, I got the following running error.
TypeError: Failed to fetch
at http (http://localhost:3000/static/js/bundle.js:53443:12)
at HttpClient.execute (http://localhost:3000/static/js/bundle.js:53471:65)
at Tendermint34Client.detectVersion (http://localhost:3000/static/js/bundle.js:55135:35)
at Tendermint34Client.create (http://localhost:3000/static/js/bundle.js:55128:33)
at Tendermint34Client.connect (http://localhost:3000/static/js/bundle.js:55115:33)
at SigningCosmWasmClient.connectWithSigner (http://localhost:3000/static/js/bundle.js:23403:64)
at http://localhost:3000/static/js/bundle.js:127476:51
at Generator.next (<anonymous>)
at fulfilled (http://localhost:3000/static/js/bundle.js:462877:24)
Here is my code;
import { SigningCosmWasmClient } from "#cosmjs/cosmwasm-stargate";
import { GasPrice }from "#cosmjs/stargate";
.....
const config = {
chainName: "Cosmos Hub",
chainId: "cosmoshub-4",
rpcEndpoint: "https://rpc-cosmoshub.whispernode.com",
restEndpoint: "",
faucetEndpoint: "",
addressPrefix: "cosmos",
microDenom: "uatom",
coinDecimals: "6",
gasPrice: "0.025",
}
.....
await window.keplr?.enable(config.chainId);
const offlineSigner= window.getOfflineSigner?.(
config.chainId
);
const account = await offlineSigner?.getAccounts();
let wasmChainClient = null;
if (offlineSigner) {
try {
wasmChainClient = await SigningCosmWasmClient.connectWithSigner(
config.rpcEndpoint,
offlineSigner,
{
gasPrice: GasPrice.fromString(
`${config.gasPrice}${config.microDenom}`
),
}
);
} catch (e) {
console.error("wallets", e);
}
}
const result= {
account: account?.[0],
client: wasmChainClient,
};
console.log(result)
Is this the problem of rpc endpoint?
I have tried other several rpc endpoints but all of them failed.
I really don't know why this happens.
I would be very thankful if anyone could help me with solving this issue.
This happens because your browser blocks request with different origin. Your origin is http://localhost:3000, and you requested https://rpc-cosmoshub.keplr.app. So, your browser blocks the response of the requested origin if there is no access-control-allow-origin header in the response. You can learn more here (https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors).
I have a workaround program that adds necessary headers to a response.
You have to have golang to run it.
package main
import (
"flag"
"fmt"
"io/ioutil"
"log"
"net/http"
)
var hostname string
var port int
func main() {
// flags declaration using flag package
flag.StringVar(&hostname, "H", "https://rpc-cosmoshub.keplr.app", "Specify hostname")
flag.IntVar(&port, "p", 8081, "Specify port")
flag.Parse() // after declaring flags we
http.HandleFunc("/", serveCorsProxy)
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", port), nil))
}
// Serve a reverse proxy for a given url
func serveCorsProxy(res http.ResponseWriter, req *http.Request) {
proxyRequest, err := http.NewRequest(req.Method, hostname, req.Body)
proxyRequest.URL.Path = req.URL.Path
proxyRequest.URL.RawQuery = req.URL.RawQuery
if err != nil {
fmt.Printf("create request error: %v", err)
return
}
response, err := http.DefaultClient.Do(proxyRequest)
if err != nil {
fmt.Printf("proxy request error: %v", err)
return
}
setHeaders(response, &res)
body, err := ioutil.ReadAll(response.Body)
if err != nil {
fmt.Printf("response read error: %v", err)
return
}
res.WriteHeader(response.StatusCode)
_, _ = res.Write(body)
}
func setHeaders(src *http.Response, dest *http.ResponseWriter) {
header := (*dest).Header()
for name, values := range (*src).Header {
for _, value := range values {
header.Set(name, value)
}
}
header.Set("access-control-allow-headers", "Accept,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With")
header.Set("access-control-allow-methods", "GET, POST, OPTIONS")
header.Set("access-control-allow-origin", "*")
header.Set("access-control-expose-headers", "Content-Length,Content-Range")
header.Set("access-control-max-age", "1728000")
}
You have to save it to file main.go and run by go run main.go -H https://rpc-cosmoshub.keplr.app -p 3001.
After that you can access the RPC on localhost:3001
How to mock redis cluster server in Golang ?
Simple redis server can be mocked using 'github.com/alicebob/miniredis' package but it does not support cluster mocking.
Received the following error:
ERR unknown command cluster, with args beginning with: slots``
My program uses 'github.com/go-redis/redis' package for redis implementation.
For example - create mock server for the following client
redisCache := redis.NewClusterClient(redisConfig)
_, err := redisCache.Ping().Result()
if err != nil {
log.Fatalf("fatal error Not able to connect using redis client: %s", err)
}
Maybe using "github.com/alicebob/miniredis/v2" instead of "github.com/alicebob/miniredis" would solve the problem.
Here's an example.
package redis_cluster_test
import (
"log"
"testing"
"time"
"github.com/alicebob/miniredis/v2"
"github.com/go-redis/redis"
)
func TestRedisCluster(t *testing.T) {
mock, err := miniredis.Run()
if err != nil {
panic(err)
}
redisConfig := redis.ClusterOptions{
Addrs: []string{mock.Addr()},
ReadTimeout: 1 * time.Second,
WriteTimeout: 1 * time.Second,
PoolSize: 6500,
PoolTimeout: 30 * time.Second}
redisCache := redis.NewClusterClient(&redisConfig)
_, err = redisCache.Ping().Result()
if err != nil {
t.Errorf("fatal error Not able to connect using redis client: %s", err)
log.Fatalf("fatal error Not able to connect using redis client: %s", err)
}
}
I am trying to create websockets using golang with gorilla on aws (without docker) with load balancer, I have a certificate connected to the load balancer.
I managed to make the code work with http, but when I try to use the code over https it does not work.
What am I doing wrong?
When moving from http to https I changed the client request to wss instead of ws, and added the certificate to the load balancer.
Here is my code.
This is my main.go
http.HandleFunc("/wss", serveWs)
fmt.Printf("Listening on port %s\n\n", port)
if err := http.ListenAndServe(":"+port, nil); err != nil {
fmt.Printf(err.Error())
}
This is the serveWs:
func serveWs(w http.ResponseWriter, r *http.Request) {
upgrader.CheckOrigin = func(r *http.Request) bool {
// allow all connections by default
return true
}
ws, err := upgrader.Upgrade(w, r, nil)
if err != nil {
fmt.Println(err)
if _, ok := err.(websocket.HandshakeError); !ok {
log.Println(err)
}
return
}
var lastMod time.Time
if n, err := strconv.ParseInt(r.FormValue("lastMod"), 16, 64); err != nil {
lastMod = time.Unix(0, n)
}
go writer(ws, lastMod, w, r)
reader(ws)
}
This is the request:
var conn = new WebSocket("wss://https://www.weldpad.com/wss?"
When I send a request from the client i get the folloing error:
failed: Error during WebSocket handshake: Unexpected response code: 400
Looking at the server log, I see.
"could not find upgrade header with token 'websocket"
This is my load balancer configuration:
Load Balancing Configuration
I believe it should be wss://www.weldpad.com/wss? not wss://https://www.weldpad.com/wss?