How can a Unix program display output on screen even when stdout and stderr are redirected? - c++

I was running a program (valgrind, actually) on my Ubuntu machine, and had redirected both stdout and stderr to different files. I was surprised to see a short message appear on the screen -- how is that possible? How could I do that myself in a C++ program?
EDIT: Here's the command I used, and the output:
$ valgrind ./myprogram > val.out 2> val.err
*** stack smashing detected ***: ./myprogram terminated
EDIT2: Playing with it a little more, it turns out that myprogram, not valgrind, is causing the message to be printed, and as answered below, it looks like gcc stack smashing detection code is printing to /dev/tty

It is not written by valgrind but rather glibc and your ./myprogram is using glibc:
#define _PATH_TTY "/dev/tty"
/* Open a descriptor for /dev/tty unless the user explicitly
requests errors on standard error. */
const char *on_2 = __libc_secure_getenv ("LIBC_FATAL_STDERR_");
if (on_2 == NULL || *on_2 == '\0')
fd = open_not_cancel_2 (_PATH_TTY, O_RDWR | O_NOCTTY | O_NDELAY);
if (fd == -1)
fd = STDERR_FILENO;
...
written = WRITEV_FOR_FATAL (fd, iov, nlist, total);
Below are some relevant parts of glibc:
void
__attribute__ ((noreturn))
__stack_chk_fail (void)
{
__fortify_fail ("stack smashing detected");
}
void
__attribute__ ((noreturn))
__fortify_fail (msg)
const char *msg;
{
/* The loop is added only to keep gcc happy. */
while (1)
__libc_message (2, "*** %s ***: %s terminated\n",
msg, __libc_argv[0] ?: "<unknown>");
}
/* Abort with an error message. */
void
__libc_message (int do_abort, const char *fmt, ...)
{
va_list ap;
int fd = -1;
va_start (ap, fmt);
#ifdef FATAL_PREPARE
FATAL_PREPARE;
#endif
/* Open a descriptor for /dev/tty unless the user explicitly
requests errors on standard error. */
const char *on_2 = __libc_secure_getenv ("LIBC_FATAL_STDERR_");
if (on_2 == NULL || *on_2 == '\0')
fd = open_not_cancel_2 (_PATH_TTY, O_RDWR | O_NOCTTY | O_NDELAY);
if (fd == -1)
fd = STDERR_FILENO;
...
written = WRITEV_FOR_FATAL (fd, iov, nlist, total);

The message is most probably from GCC's stack protector feature or from glib itself. If it's from GCC, it is output using the fail() function, which directly opens /dev/tty:
fd = open (_PATH_TTY, O_WRONLY);
_PATH_TTY is not really standard, but SingleUnix actually demands that /dev/tty exists.

Here is some sample code that does exactly what was asked (thanks to earlier answers pointing me in the right direction). Both are compiled with g++, and will print a message to the screen even when stdout and stderr are redirected.
For Linux (Ubuntu 14):
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <string.h>
int main( int, char *[]) {
printf("This goes to stdout\n");
fprintf(stderr, "This goes to stderr\n");
int ttyfd = open("/dev/tty", O_RDWR);
const char *msg = "This goes to screen\n";
write(ttyfd, msg, strlen(msg));
}
For Windows 7, using MinGW:
#include <stdio.h>
#include <fcntl.h>
#include <string.h>
#include <conio.h>
void writeConsole( const char *s) {
while( *s) {
putch(*(s++));
}
}
int main( int, char *[]) {
printf("This goes to stdout\n");
fprintf(stderr, "This goes to stderr\n");
writeConsole( "This goes to screen\n");
}

Related

Why can't I run some programs after using unshare(CLONE_NEWUSER)

I'm working on adding some restrictions to my build process - to detect cycles, specifically. To achieve this I've been experimenting with user namespaces.
Here's my 'hello world' program:
#include <sched.h>
#include <unistd.h>
int main()
{
if( unshare(CLONE_NEWUSER) != 0)
{
return -1;
}
execl("/bin/sh", "/bin/sh", "-e", "-c", "make", NULL);
return 0;
}
Here is the makefile being run by make, namespace_test.cpp is the name of the file above:
namespace_test: namespace_test.cpp
g++ namespace_test.cpp -o ./namespace_test
When everything is up to date (as determined by make) the exec'd program works as expected:
make: 'namespace_test' is up to date.
But if make actually runs the g++ invocation I get an opaque error:
g++ namespace_test.cpp -o ./namespace_test
make: g++: Invalid argument
make: *** [Makefile:2: namespace_test] Error 127
What is the reason for this behavior?
This error was due to my failure to set up the uid_map and gid_map. I have not produced a satisfactory, explicit, minimal example of the error, but I have written a working minimal solution, that I will share here. Notice that int main() is identical, except before exec'ing the target command we first set up the uid_map and then the gid_map (granting ourselves permission via setgroups).
On my terminal $ id informs me that my real uid and gid are both 1000, so I have hardcoded that in the maps. It is more correct to query for the original id at the start of the process, see this excellent blog post. Also instrumental in this solution is this man page.
#include <cstdio>
#include <cstring>
#include <fcntl.h>
#include <sched.h>
#include <stdlib.h>
#include <unistd.h>
#define fatal_error(...) \
do { \
fprintf(stderr, "namespace_test \033[1;31merror:\033[0m "); \
fprintf(stderr, __VA_ARGS__ ); \
fprintf(stderr, "\n"); \
exit(EXIT_FAILURE); \
} while(0)
void write_string_to_file(const char* filename, const char* str, size_t str_len)
{
int fd = open(filename, O_RDWR);
if(fd == -1)
{
fatal_error("Failed to open %s: %m", filename);
}
if( write(fd, str, str_len) != str_len )
{
fatal_error("Failed to write %s: %m", filename);
}
close(fd);
}
void write_uid_mapping()
{
const char* mapping = "0 1000 1";
write_string_to_file("/proc/self/uid_map", mapping, strlen(mapping));
}
void write_set_groups()
{
const char* deny = "deny\n";
write_string_to_file("/proc/self/setgroups", deny, strlen(deny));
}
void write_gid_mapping()
{
write_set_groups();
const char* mapping = "0 1000 1";
write_string_to_file("/proc/self/gid_map", mapping, strlen(mapping));
}
int main()
{
if(unshare(CLONE_NEWUSER) != 0)
{
fatal_error("Failed to move into new user namespace");
}
write_uid_mapping();
write_gid_mapping();
execl("/bin/sh", "/bin/sh", "-e", "-c", "make", NULL);
return 0;
}

Linux - segmentation fault only sometimes - how to debug

I have a Linux program, that from time to time ends with a segmentation fault. The program is running periodically every hour, but the segmentation fault occurs only sometimes.
I have a problem to debug this, because if I run the program again with the same input, no error is reported and all is OK.
Is there a way, how to "report" in which part of the code error occured or what caused the problem?
The usual way is to have the crashing program generate a corefile and analyze this after the crash. Make sure, that:
the maximum corefile-size is big enough (i.e. unlimited) by calling ulimit -c unlimited in the shell, which starts the process.
The cwd is writable by the segfaulting process.
Then you can analyze the file with
gdb <exe> <corefile>
Since your code not crashing every time, you can use backtrace as well. Using this you can see the function call stack at the time of crash. There are many examples available. In my projects I normally use the following code for backtracing.
/*
* call reg_usr2 function from main
* gcc -rdynamic myfile.c -o output
*/
#include <stdio.h>
#include <stdarg.h>
#include <signal.h>
#include <unistd.h>
#include <stdlib.h>
#include <execinfo.h>
#define FILE_NAME "/tmp/debug"
#define MODE 0xFFFF
void dbgprint(int flag, char* fmt, ...)
{
if(flag & MODE) {
char buf[100];
va_list vlist;
FILE *fp = fopen(FILE_NAME,"a");
va_start(vlist, fmt);
vsnprintf( buf, sizeof( buf), fmt, vlist);
va_end( vlist);
fprintf(fp,"[%x]->%s\n", flag, buf);
fclose(fp);
}
}
/** Here is the code to print backtrace **/
void print_stack_trace ()
{
void *array[20];
size_t size;
char **strings;
size_t i;
size = backtrace (array, 20);
strings = backtrace_symbols (array, size);
dbgprint(0xFFFF, "Obtained %zd stack frames.", size);
dbgprint(0xFFFF, "-------------------------");
dbgprint(0xFFFF, "---------Backtrace-------");
for (i = 0; i < size; i++)
dbgprint (0xFFFF, "%s", strings[i]);
dbgprint(0xFFFF, "-------------------------");
free (strings);
}
void sig_handler(int signo)
{
FILE *fp = fopen(FILE_NAME,"a");
if (signo == SIGUSR2){
dbgprint(0xFFFF, "received SIGUSR2");
dbgprint(0xFFFF, "----------------");
}
print_stack_trace();
exit(0);
}
void reg_usr2()
{
if (signal(SIGUSR2, sig_handler) == SIG_ERR)
printf("\ncan't catch SIGUSR2\n");
}
int main()
{
reg_usr2(); //should be first line of main after variables
//Code.....
return 0;
}
You can generate backtrace by catching SIGSEGV signal, and see where your application throw an invalid access.
see https://stackoverflow.com/a/77336/4490542
But there is more easier solution, try running your application with catchsegv
catchsegv './program args'
and better alternative, valgrind
valgrind --tool=none ./program args

Cannot compile demo program using fork() and pipes() in C++

All, the first part of my homework assignment is simply a demo program that I need to compile, and then modify. It was provided by the teacher, however I simply cannot get it to compile using g++. I will be creating a make file at the end of the assignment, but for the moment I am simply trying to test it out, and am having no luck. I've tried the most basic g++ command: g++ -o main TwoPipesTwoChildren.cpp . Can someone please help? I can't even get started on this until I can get this working.
// description: This program will execute "ls -ltr | grep 3376"
// by using a parent and child process
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
int main(int argc, char **argv){
printf("TEST");
int status;
int childpid;
char *cat_args[] = {"ls", "-ltr", NULL};
char *grep_args[] = {"grep", "3376", NULL};
// create one pipe to send the output of "ls" process to "grep" process
int pipes[2];
pipe(pipes);
// fork the first child (to execute cat)
if((childpid = fork()) == -1)
{
perror("Error creating a child process");
exit(1);
}
// replace cat's stdout with write part of 1st pipe
if (childpid == 0)
{
dup2(pipes[1], 1);
printf("AFTER FORK CHILD");
//close all pipes (very important!); end we're using was safely copied
close(pipes[0]);
close(pipes[1]);
execvp(*cat_args, cat_args);
exit(0);
}
else
{
// replace grep's stdin with read end of 1st pipe
dup2(pipes[0], 0);
close(pipes[0]);
close(pipes[1]);
execvp(*grep_args, grep_args);
}
return (0);
}

clang: error: linker command failed with exit code 1 (use -v to see invocation) MINIX3

I am trying to run a C/C++ application on MINIX3 which is supposed to send a messages between two processes using msgsnd() and msgget() using msg.h.
This is the error I am getting:
send.cpp:(.text+0x7f): undefined reference to `msgget'
send.cpp:(.text+0x1c1): undefined reference to `msgsnd'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
I am using clang++ to compile the code:
clang++ send.cpp -o send.out
This is the send.cpp code:
#include <lib.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/msg.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#define MSGSZ 128
/*
* Declare the message structure.
*/
typedef struct msgbufer {
long mtype;
char mtext[MSGSZ];
} message_buf;
int main()
{
int msqid;
int msgflg = IPC_CREAT | 0666;
key_t key;
message_buf sbuf;
size_t buf_length;
/*
* Get the message queue id for the
* "name" 1234, which was created by
* the server.
*/
key = 1234;
(void)fprintf(stderr, "\nmsgget: Calling msgget(%i,\
%#o)\n",
key, msgflg);
if ((msqid = msgget(key, msgflg)) < 0) {
perror("msgget");
exit(1);
}
else
(void)fprintf(stderr, "msgget: msgget succeeded: msqid = %d\n", msqid);
/*
* We'll send message type 1
*/
sbuf.mtype = 1;
(void)fprintf(stderr, "msgget: msgget succeeded: msqid = %d\n", msqid);
(void)strcpy(sbuf.mtext, "Hello other process 2.");
(void)fprintf(stderr, "msgget: msgget succeeded: msqid = %d\n", msqid);
buf_length = strlen(sbuf.mtext) + 1;
/*
* Send a message.
*/
if (msgsnd(msqid, &sbuf, buf_length, IPC_NOWAIT) < 0) {
printf("%d, %li, %s, %lu\n", msqid, sbuf.mtype, sbuf.mtext, buf_length);
perror("msgsnd");
exit(1);
}
else
printf("Message: \"%s\" Sent\n", sbuf.mtext);
exit(0);
}
You aren't linking with the library that contains the msgsnd and msgget functions, so your linker step fails. I'm not familiar with Minix so I'm not sure where the library is stored or what it is called. Basically, you need to a -l<msg> flag to your linking step. Where <msg> is the name of the library that contains the implementation.

Is it possible to set a gdb watchpoint programmatically?

I want to set a watchpoint (break on hardware write) temporarily in my C++ program to find memory corruption.
I've seen all the ways to do it manually through gdb, but I would like to actually set the watchpoint via some method in my code so I don't have to break into gdb, find out the address, set the watchpoint and then continue.
Something like:
#define SET_WATCHPOINT(addr) asm ("set break on hardware write %addr")
Set hardware watchpoint from child process.
#include <signal.h>
#include <syscall.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <stddef.h>
#include <sys/ptrace.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <linux/user.h>
enum {
DR7_BREAK_ON_EXEC = 0,
DR7_BREAK_ON_WRITE = 1,
DR7_BREAK_ON_RW = 3,
};
enum {
DR7_LEN_1 = 0,
DR7_LEN_2 = 1,
DR7_LEN_4 = 3,
};
typedef struct {
char l0:1;
char g0:1;
char l1:1;
char g1:1;
char l2:1;
char g2:1;
char l3:1;
char g3:1;
char le:1;
char ge:1;
char pad1:3;
char gd:1;
char pad2:2;
char rw0:2;
char len0:2;
char rw1:2;
char len1:2;
char rw2:2;
char len2:2;
char rw3:2;
char len3:2;
} dr7_t;
typedef void sighandler_t(int, siginfo_t*, void*);
int watchpoint(void* addr, sighandler_t handler)
{
pid_t child;
pid_t parent = getpid();
struct sigaction trap_action;
int child_stat = 0;
sigaction(SIGTRAP, NULL, &trap_action);
trap_action.sa_sigaction = handler;
trap_action.sa_flags = SA_SIGINFO | SA_RESTART | SA_NODEFER;
sigaction(SIGTRAP, &trap_action, NULL);
if ((child = fork()) == 0)
{
int retval = EXIT_SUCCESS;
dr7_t dr7 = {0};
dr7.l0 = 1;
dr7.rw0 = DR7_BREAK_ON_WRITE;
dr7.len0 = DR7_LEN_4;
if (ptrace(PTRACE_ATTACH, parent, NULL, NULL))
{
exit(EXIT_FAILURE);
}
sleep(1);
if (ptrace(PTRACE_POKEUSER, parent, offsetof(struct user, u_debugreg[0]), addr))
{
retval = EXIT_FAILURE;
}
if (ptrace(PTRACE_POKEUSER, parent, offsetof(struct user, u_debugreg[7]), dr7))
{
retval = EXIT_FAILURE;
}
if (ptrace(PTRACE_DETACH, parent, NULL, NULL))
{
retval = EXIT_FAILURE;
}
exit(retval);
}
waitpid(child, &child_stat, 0);
if (WEXITSTATUS(child_stat))
{
printf("child exit !0\n");
return 1;
}
return 0;
}
int var;
void trap(int sig, siginfo_t* info, void* context)
{
printf("new value: %d\n", var);
}
int main(int argc, char * argv[])
{
int i;
printf("init value: %d\n", var);
watchpoint(&var, trap);
for (i = 0; i < 100; i++) {
var++;
sleep(1);
}
return 0;
}
Based on user512106's great answer, I coded up a little "library" that someone might find useful:
It's on github at https://github.com/whh8b/hwbp_lib. I wish I could have commented directly on his answer, but I don't have enough rep yet.
Based on feedback from the community, I am going to copy/paste the relevant code here:
#include <stdio.h>
#include <stddef.h>
#include <signal.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/ptrace.h>
#include <sys/user.h>
#include <sys/prctl.h>
#include <stdint.h>
#include <errno.h>
#include <stdbool.h>
extern int errno;
enum {
BREAK_EXEC = 0x0,
BREAK_WRITE = 0x1,
BREAK_READWRITE = 0x3,
};
enum {
BREAK_ONE = 0x0,
BREAK_TWO = 0x1,
BREAK_FOUR = 0x3,
BREAK_EIGHT = 0x2,
};
#define ENABLE_BREAKPOINT(x) (0x1<<(x*2))
#define ENABLE_BREAK_EXEC(x) (BREAK_EXEC<<(16+(x*4)))
#define ENABLE_BREAK_WRITE(x) (BREAK_WRITE<<(16+(x*4)))
#define ENABLE_BREAK_READWRITE(x) (BREAK_READWRITE<<(16+(x*4)))
/*
* This function fork()s a child that will use
* ptrace to set a hardware breakpoint for
* memory r/w at _addr_. When the breakpoint is
* hit, then _handler_ is invoked in a signal-
* handling context.
*/
bool install_breakpoint(void *addr, int bpno, void (*handler)(int)) {
pid_t child = 0;
uint32_t enable_breakpoint = ENABLE_BREAKPOINT(bpno);
uint32_t enable_breakwrite = ENABLE_BREAK_WRITE(bpno);
pid_t parent = getpid();
int child_status = 0;
if (!(child = fork()))
{
int parent_status = 0;
if (ptrace(PTRACE_ATTACH, parent, NULL, NULL))
_exit(1);
while (!WIFSTOPPED(parent_status))
waitpid(parent, &parent_status, 0);
/*
* set the breakpoint address.
*/
if (ptrace(PTRACE_POKEUSER,
parent,
offsetof(struct user, u_debugreg[bpno]),
addr))
_exit(1);
/*
* set parameters for when the breakpoint should be triggered.
*/
if (ptrace(PTRACE_POKEUSER,
parent,
offsetof(struct user, u_debugreg[7]),
enable_breakwrite | enable_breakpoint))
_exit(1);
if (ptrace(PTRACE_DETACH, parent, NULL, NULL))
_exit(1);
_exit(0);
}
waitpid(child, &child_status, 0);
signal(SIGTRAP, handler);
if (WIFEXITED(child_status) && !WEXITSTATUS(child_status))
return true;
return false;
}
/*
* This function will disable a breakpoint by
* invoking install_breakpoint is a 0x0 _addr_
* and no handler function. See comments above
* for implementation details.
*/
bool disable_breakpoint(int bpno)
{
return install_breakpoint(0x0, bpno, NULL);
}
/*
* Example of how to use this /library/.
*/
int handled = 0;
void handle(int s) {
handled = 1;
return;
}
int main(int argc, char **argv) {
int a = 0;
if (!install_breakpoint(&a, 0, handle))
printf("failed to set the breakpoint!\n");
a = 1;
printf("handled: %d\n", handled);
if (!disable_breakpoint(0))
printf("failed to disable the breakpoint!\n");
return 1;
}
I hope that this helps someone!
Will
In GDB, there are two types of watchpoints, hardware and software.
you can't implement easily software watchpoints: (cf. GDB Internals)
Software watchpoints are very slow, since gdb needs to single-step the program being debugged and test the value of the watched expression(s) after each instruction.
EDIT:
I'm still trying to understand what are hardware watchpoint.
for hardware breakpoints, this article gives some technics:
We want to watch reading from or writing into 1 qword at address 100005120h (address range 100005120h-100005127h)
lea rax, [100005120h]
mov dr0, rax
mov rax, dr7
and eax, not ((1111b shl 16) + 11b) ; mask off all
or eax, (1011b shl 16) + 1 ; prepare to set what we want
mov
dr7, rax ; set it finally
Done, now we can wait until code falls into the trap! After accessing any byte at memory range 100005120h-100005127h, int 1 will occur and DR6.B0 bit will be set to 1.
You can also take a look at GDB low-end files (eg, amd64-linux-nat.c) but it (certainly) involves 2 processes: 1/ the one you want to watch 2/a lightweight debugger who attaches to the first one with ptrace, and uses:
ptrace (PTRACE_POKEUSER, tid, __regnum__offset__, address);
to set and handle the watchpoint.
The program itself can supply commands to the GDB. You'll need a special shell script to run GDB though.
Copy this code into the file named untee, and execute chmod 755 untee
#!/bin/bash
if [ -z "$1" ]; then
echo "Usage: $0 PIPE | COMMAND"
echo "This script will read the input from both stdin and PIPE, and supply it to the COMMAND."
echo "If PIPE does not exist it will be created with mkfifo command."
exit 0
fi
PIPE="$1"
if [ \! -e "$PIPE" ]; then
mkfifo "$PIPE"
fi
if [ \! -p "$PIPE" ]; then
echo "File $PIPE does not exist or is not a named pipe" > /dev/stderr
exit 1
fi
# Open the pipe as a FD 3
echo "Waiting for $PIPE to be opened by another process" > /dev/stderr
exec 3<"$PIPE"
echo "$PIPE opened" > /dev/stderr
OPENED=true
while true; do
read -t 1 INPUT
RET=$?
if [ "$RET" = 0 ]; then
echo "$INPUT"
elif [ "$RET" -lt 128 ]; then
echo "stdin closed, exiting" > /dev/stderr
break
fi
if $OPENED; then
while read -t 1 -u 3 INPUT; do
RET=$?
if [ "$RET" = 0 ]; then
echo "$INPUT"
else
if [ "$RET" -lt 128 ]; then
echo "$PIPE closed, ignoring" > /dev/stderr
OPENED=false
fi
break
fi
done
fi
done
And now the C code:
#include <stdio.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <signal.h>
#include <unistd.h>
void gdbCommand(const char *c)
{
static FILE * dbgpipe = NULL;
static const char * dbgpath = "/tmp/dbgpipe";
struct stat st;
if( !dbgpipe && stat(dbgpath, &st) == 0 && S_ISFIFO(st.st_mode) )
dbgpipe = fopen(dbgpath, "w");
if( !dbgpipe )
return;
fprintf(dbgpipe, "%s\n", c);
fflush(dbgpipe);
}
void gdbSetWatchpoint(const char *var)
{
char buf[256];
snprintf(buf, sizeof(buf), "watch %s", var);
gdbCommand("up"); /* Go up the stack from the kill() system call - this may vary by the OS, you may need to walk the stack more times */
gdbCommand("up"); /* Go up the stack from the gdbSetWatchpoint() function */
gdbCommand(buf);
gdbCommand("continue");
kill(getpid(), SIGINT); /* Make GDB pause our process and execute commands */
}
int subfunc(int *v)
{
*v += 5; /* GDB should pause after this line, and let you explore stack etc */
return v;
}
int func()
{
int i = 10;
printf("Adding GDB watch for var 'i'\n");
gdbSetWatchpoint("i");
subfunc(&i);
return i;
}
int func2()
{
int j = 20;
return j + func();
}
int main(int argc, char ** argv)
{
func();
func2();
return 0;
}
Copy that to the file named test.c, compile with command gcc test.c -O0 -g -o test then execute ./untee /tmp/dbgpipe | gdb -ex "run" ./test
This works on my 64-bit Ubuntu, with GDB 7.3 (older GDB versions might refuse to read commands from non-terminal)
If you happen to be using Xcode, you can achieve the required effect (automatic setting of watchpoints) by using an action on another breakpoint to set your watchpoint:
Set up a breakpoint somewhere where the variable you want to watch will be in scope that will be hit before you need to start watching the variable,
Right-click on the breakpoint and select Edit Breakpoint...,
Click on Add Action and add a Debugger Command with an LLDB command like: watchpoint set variable <variablename> (or if you're using GDB1, a command like: watch <variablename>),
Check the Automatically continue after evaluating actions checkbox.
1: GDB is no longer supported in more recent versions of Xcode, but I believe it is still possible to set it up manually.