In C++ there's a freopen func, which is very useful to r/w files with just stdin/out(cin/cout). So I decided to find similar solution in Go, but found only
import "os"
os.Stdin, err = os.OpenFile("input.txt",
os.RDONLY | os.O_CREATE, 0666)
os.Stdout, err = os.OpenFile("output.txt",
os.O_WRONLY | os.O_CREATE | os.O_TRUNC, 0666)
, which is not working anymore! Am I wrong?
So, do you know other way?
While Jeff Allen provided a good answer, there's a minor low-level "catch" to the approach presented there: the os.File values representing new destinations for the standard output streams will refer to file descriptors different from those of stdout and stderr as the OS sees them.1
The thing is, when a process starts on a POSIX-compatible system, it has its three standard streams open to file descriptors 0, 1 and 2 for stdin, stdout and stderr, correspondingly.
Hence in an obscure case of some bit of the code relying on the fact the standard streams being connected to the standard file descriptors, the code provided by Jeff Allen will not be quite correct.
To make it 100% correct, we may rely on another POSIX property which reuses the lowest free file descriptor when opening a new file. Hence if we close the file representing one standard stream and immediately open another one,—that new file will be open using the file descriptor of the just closed standard stream. To guarantee that no file is open between the sequence of steps just presented, we must run our code before any goroutine except the main one starts running—that is, in main() or any init().
Here's the code to demonstrate the idea:
package main
import (
"os"
)
func init() {
err := os.Stdout.Close()
if err != nil {
panic(err)
}
fd, err := os.OpenFile("output.txt",
os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0666)
if err != nil {
panic(err)
}
os.Stdout = fd
}
func main() {
myfd := os.NewFile(1, "")
_, err := myfd.Write([]byte("Direct output to FD 1\n"))
if err != nil {
panic(err)
}
_, err = os.Stdout.Write([]byte("Write to redirected os.Stdout\n"))
if err != nil {
panic(err)
}
os.Exit(1)
}
…and here is how it works:
$ go build
$ ./freopen
$ cat output.txt
Direct output to FD 1
Write to redirected os.Stdout
It might seem like nitpicking but I think it worth explaning the "full stack" of what's going on.
Oh, and redirecting this way will also provide sensible view to the process for outside observers: say, on Linux, inspecting the file descriptors opened by the process via something like
$ vdir /proc/<pid>/fd
will provide sensible results.
1 …and everything else which does not know about Go—for instance, a bit of linked in C code which calls something like write(1, "OK\n", 3);
You cannot declare and assign to a variable in another package (os.Stdin, for example).
However this works:
package main
import (
"fmt"
"log"
"os"
)
func main() {
stdin, err := os.OpenFile("input.txt",
os.O_RDONLY|os.O_CREATE, 0666)
if err != nil {
log.Fatal(err)
}
os.Stdin = stdin
stdout, err := os.OpenFile("output.txt",
os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0666)
if err != nil {
log.Fatal(err)
}
os.Stdout = stdout
fmt.Println("out")
return
}
Related
This question already has an answer here:
Does ReadString() discard bytes following newline?
(1 answer)
Closed 2 years ago.
I've have a process that spits out data to a TCP port in bursts with a few minutes pause between files. I've tried the code below that I've seen on multiple different posts however a large amount (multiple lines worth) of data is lost from the output. I've also tried writing similar code in C++ with the same result. The only reliable way I've found to get all output is to just listen using nc but I would like to do this programmatically so that I can use the downtime between bursts in order to separate output into multiple files. Has anyone ran into this issue before? I don't see any pattern to the missing data, just as if some random lines are getting skipped. I even tried to just send the data to a go chan, to see if the print statement was slowing down execution somehow. Any help would be appreciated!
package main
import (
"bufio"
"fmt"
"net"
"os"
)
func main() {
arguments := os.Args
if len(arguments) == 1 {
fmt.Println("Please provide host:port.")
return
}
CONNECT := arguments[1]
c, err := net.Dial("tcp", CONNECT)
if err != nil {
fmt.Println(err)
return
}
for {
message, _ := bufio.NewReader(c).ReadString('\n')
fmt.Print(message)
}
}
Unfortunately I can not mark comments as answers, however as Cerise Limón pointed out- using a bufio.Scanner() was the solution. The functional code is below. Thank you!
package main
import (
"bufio"
"fmt"
"net"
"os"
)
func main() {
arguments := os.Args
if len(arguments) == 1 {
fmt.Println("Please provide host:port.")
return
}
CONNECT := arguments[1]
c, err := net.Dial("tcp", CONNECT)
if err != nil {
fmt.Println(err)
return
}
scanner := bufio.NewScanner(c) // Declare outside of the loop
for scanner.Scan() {
fmt.Println(scanner.Text())
}
}
I wrote an SSH client in Go and I would like to write some tests. The problem is that I've never really written proper unit tests before, and most tutorials seem to focus on writing tests for a function that adds two numbers or some other toy problem. I've read about mocking, using interfaces, and other techniques, but I'm having trouble applying them. Also, my client is going to be used concurrently to allow fast configuration of multiple devices at a time. Not sure if that would change the way I write my tests or would add additional tests. Any help is appreciated.
Here is my code. Basically, a Device has 4 main functions: Connect, Send, Output/Err and Close for connecting to a device, sending it a set of configuration commands, capturing the output of the session, and closing the client, respectively.
package device
import (
"bufio"
"fmt"
"golang.org/x/crypto/ssh"
"io"
"net"
"time"
)
// A Device represents a remote network device.
type Device struct {
Host string // the device's hostname or IP address
client *ssh.Client // the client connection
session *ssh.Session // the connection to the remote shell
stdin io.WriteCloser // the remote shell's standard input
stdout io.Reader // the remote shell's standard output
stderr io.Reader // the remote shell's standard error
}
// Connect establishes an SSH connection to a device and sets up the session IO.
func (d *Device) Connect(user, password string) error {
// Create a client connection
client, err := ssh.Dial("tcp", net.JoinHostPort(d.Host, "22"), configureClient(user, password))
if err != nil {
return err
}
d.client = client
// Create a session
session, err := client.NewSession()
if err != nil {
return err
}
d.session = session
return nil
}
// configureClient sets up the client configuration for login
func configureClient(user, password string) *ssh.ClientConfig {
var sshConfig ssh.Config
sshConfig.SetDefaults()
sshConfig.Ciphers = append(sshConfig.Ciphers, "aes128-cbc", "aes256-cbc", "3des-cbc", "des-cbc", "aes192-cbc")
config := &ssh.ClientConfig{
Config: sshConfig,
User: user,
Auth: []ssh.AuthMethod{ssh.Password(password)},
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
Timeout: time.Second * 5,
}
return config
}
// setupIO creates the pipes connected to the remote shell's standard input, output, and error
func (d *Device) setupIO() error {
// Setup standard input pipe
stdin, err := d.session.StdinPipe()
if err != nil {
return err
}
d.stdin = stdin
// Setup standard output pipe
stdout, err := d.session.StdoutPipe()
if err != nil {
return err
}
d.stdout = stdout
// Setup standard error pipe
stderr, err := d.session.StderrPipe()
if err != nil {
return err
}
d.stderr = stderr
return nil
}
// Send sends cmd(s) to the device's standard input. A device only accepts one call
// to Send, as it closes the session and its standard input pipe.
func (d *Device) Send(cmds ...string) error {
if d.session == nil {
return fmt.Errorf("device: session is closed")
}
defer d.session.Close()
// Start the shell
if err := d.startShell(); err != nil {
return err
}
// Send commands
for _, cmd := range cmds {
if _, err := d.stdin.Write([]byte(cmd + "\r")); err != nil {
return err
}
}
defer d.stdin.Close()
// Wait for the commands to exit
d.session.Wait()
return nil
}
// startShell requests a pseudo terminal (VT100) and starts the remote shell.
func (d *Device) startShell() error {
modes := ssh.TerminalModes{
ssh.ECHO: 0, // disable echoing
ssh.OCRNL: 0,
ssh.TTY_OP_ISPEED: 14400,
ssh.TTY_OP_OSPEED: 14400,
}
err := d.session.RequestPty("vt100", 0, 0, modes)
if err != nil {
return err
}
if err := d.session.Shell(); err != nil {
return err
}
return nil
}
// Output returns the remote device's standard output output.
func (d *Device) Output() ([]string, error) {
return readPipe(d.stdout)
}
// Err returns the remote device's standard error output.
func (d *Device) Err() ([]string, error) {
return readPipe(d.stdout)
}
// reapPipe reads an io.Reader line by line
func readPipe(r io.Reader) ([]string, error) {
var lines []string
scanner := bufio.NewScanner(r)
for scanner.Scan() {
lines = append(lines, scanner.Text())
}
if err := scanner.Err(); err != nil {
return nil, err
}
return lines, nil
}
// Close closes the client connection.
func (d *Device) Close() error {
return d.client.Close()
}
// String returns the string representation of a `Device`.
func (d *Device) String() string {
return fmt.Sprintf("%s", d.Host)
}
You make a good point about unit test tutorials nearly always being toy problems (why is it always Fibonacci?), when what we have is databases and http servers. The big realization that helped me is that you can only unit test things where you can control the input and output of the unit. configureClient or readPipe (give it a strings.Reader) would be good candidates. Start there.
Anything that leaves your program by talking directly to the disk, the network, stdout, etc, like the Connect method you would consider part of the external interface of your program. You don't unit test those. You integration test them.
Change Device to be an interface rather than a struct, and make a MockDevice that implements it. The real device is now maybe SSHDevice. You can unit test the rest of your program (which uses Device interface) by inserting a MockDevice, to isolate yourself from the network.
The SSHDevice will get tested in your integration tests. Start a real ssh server (maybe a test one you write in Go using crypto/ssh package, but any sshd would work). Start your program with an SSHDevice, make them talk to each other, and check outputs. You'll be using the os/exec package a lot. Integration tests are even more fun to write than unit tests!
I'm using Go, the line causing the segfault is:
shader := gl.CreateShader(shaderType)
I'm basing my code on this example from go-gl repo, which was working, so gl seems to be fine.
Most of my googling pointed to C++ questions to which the answer was to enable glewExperimental, but the Go bindings for OpenGL were generated with GLOW, which apparently circumvents GLEW.
I am initializing my glfw contetxt:
if err := glfw.Init(); err != nil {
log.Fatalln("failed to initialize glfw:", err)
}
and the gl library:
// Initialize Glow
if err := gl.Init(); err != nil {
panic(err)
}
Is there anything I'm missing? Are there any principles I can use to track this down, i.e. "glCreateShader() Segfaults when XYZ occurs"?
I'm creating a micro service to handle some attachments uploads to Amazon S3, What I'm trying to achieve is accept a file and then store it directly to my Amazon S3 bucket, my current function :
func upload_handler(w http.ResponseWriter, r *http.Request) {
file, header, err := r.FormFile("attachment")
if err != nil {
fmt.Fprintln(w, err)
return
}
defer file.Close()
fileSize, err := file.Seek(0, 2) //2 = from end
if err != nil {
panic(err)
}
fmt.Println("File size : ", fileSize)
bytes := make([]byte, fileSize)
// read into buffer
buffer := bufio.NewReader(file)
_, err = buffer.Read(bytes)
auth := aws.Auth{
AccessKey: "XXXXXXXXXXX",
SecretKey: "SECRET_KEY_HERE",
}
client := s3.New(auth, aws.EUWest)
bucket := client.Bucket("attachments")
err = bucket.Put(header.Filename, bytes, header.Header.Get("Content-Type"), s3.ACL("public-read"))
if err != nil {
fmt.Println(err)
os.Exit(1)
}
}
The problem is that the files stored in S3 are all corrupted, After a small verification it seems that the file payload is not read as bytes
How to convert the file to bytes and store it correctly to S3 ?
Use ioutil.ReadAll:
bs, err := ioutil.ReadAll(file)
// ...
err = bucket.Put(
header.Filename,
bs,
header.Header.Get("Content-Type"),
s3.ACL("public-read"),
)
Read is a lower-level function which has subtle behavior:
Read reads data into p. It returns the number of bytes read into p. It calls Read at most once on the underlying Reader, hence n may be less than len(p). At EOF, the count will be zero and err will be io.EOF.
So what was probably happening was some subset of the file data was being written to S3 along with a bunch of 0s.
ioutil.ReadAll works by calling Read over and over again filling a dynamically expanding buffer until it reaches the end of the file. (so there's no need for the bufio.Reader either)
Also the Put function will have issues with large files (using ReadAll means the entire file must fit in memory) so you may want to use PutReader instead:
bucket.PutReader(
header.Filename,
file,
fileSize,
header.Header.Get("Content-Type"),
s3.ACL("public-read"),
)
I'm currently looking into creating some unit tests for net.Conn interface in Go, as well as other functions that build up on top of that functionality, and I'm wondering what is the best way to unit test that in Google Go? My code looks like:
conn, _:=net.Dial("tcp", "127.0.0.1:8080")
...
fmt.Fprintf(conn, "test")
...
buffer:=make([]byte, 100)
conn.Read(buffer)
Is the most efficient way of testing this code and the code that uses these functions to spin up a separate goroutine to act like the server, use net.http.httptest package, or something else?
You might be able to do what you need with net.Pipe which basically gives you both ends of a connection (think, after .Accept())
server, client := net.Pipe()
go func() {
// Do some stuff
server.Close()
}()
// Do some stuff
client.Close()
Although it will depend on the implementation details of your particular case, the general approach will be to start a server (in a separate goroutine, as you already hinted), and listen to the incoming connections.
For example, let's spin up a server and verify that the content we are reading from the connection is indeed the one we send over from the client:
func TestConn(t *testing.T) {
message := "Hi there!\n"
go func() {
conn, err := net.Dial("tcp", ":3000")
if err != nil {
t.Fatal(err)
}
defer conn.Close()
if _, err := fmt.Fprintf(conn, message); err != nil {
t.Fatal(err)
}
}()
l, err := net.Listen("tcp", ":3000")
if err != nil {
t.Fatal(err)
}
defer l.Close()
for {
conn, err := l.Accept()
if err != nil {
return
}
defer conn.Close()
buf, err := ioutil.ReadAll(conn)
if err != nil {
t.Fatal(err)
}
fmt.Println(string(buf[:]))
if msg := string(buf[:]); msg != message {
t.Fatalf("Unexpected message:\nGot:\t\t%s\nExpected:\t%s\n", msg, message)
}
return // Done
}
}
Note that here I'm not starting the server in the goroutine, as otherwise the test case is likely to be finished before the listener has run the test.
Another option is the counterfeiter package which lets you create mocks from interfaces and then you can stub out whatever calls you need. I have used it with great success to stub out net.Conn instances where I am testing out a protobuf client for Geode.
For example - https://github.com/gemfire/geode-go-client/blob/master/connector/protobuf_test.go