Same tshark version miss http2 decodings in other machine - virtualbox

During decoding the same pcap file, with the same tshark version, I get different results in different machines. I thought that tshark decodes offline a pcap archive, then, I don't understand this behaviour for the same tshark version and pcap library, etc.. The part missed is the http2 decodings.
When I do
tshark -r capture.pcap -T json
I miss all the source layers 'http2' in one machine.
This is easy to see with this:
tshark -V -r capture.pcap -Y 'http2' -T json
Because, the machine which miss http2, shows nothing for that (obviously).
I could attach the capture, but I think it is not really neccessary to ask the question. It is just a pcap capture for http2 traffic.
The thing is to know what could affect the different behaviour once, every thing related to the tshark version is exactly the same ?
Note: one machine is a host, the other is a vagrant machine via VirtualBox. Both are Ubuntu Bionic.
HOST MACHINE:
tshark --version
TShark (Wireshark) 2.6.10 (Git v2.6.10 packaged as 2.6.10-1~ubuntu18.04.0)
Copyright 1998-2019 Gerald Combs <gerald#wireshark.org> and contributors.
License GPLv2+: GNU GPL version 2 or later <http://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Compiled (64-bit) with libpcap, with POSIX capabilities (Linux), with libnl 3,
with GLib 2.56.4, with zlib 1.2.11, with SMI 0.4.8, with c-ares 1.14.0, with Lua
5.2.4, with GnuTLS 3.5.18, with Gcrypt 1.8.1, with MIT Kerberos, with MaxMind DB
resolver, with nghttp2 1.30.0, with LZ4, with Snappy, with libxml2 2.9.4.
Running on Linux 4.15.0-70-generic, with Intel(R) Core(TM) i7-8650U CPU #
1.90GHz (with SSE4.2), with 32017 MB of physical memory, with locale
LC_CTYPE=en_US.UTF-8, LC_NUMERIC=es_ES.UTF-8, LC_TIME=es_ES.UTF-8,
LC_COLLATE=en_US.UTF-8, LC_MONETARY=es_ES.UTF-8, LC_MESSAGES=en_US.UTF-8,
LC_PAPER=es_ES.UTF-8, LC_NAME=es_ES.UTF-8, LC_ADDRESS=es_ES.UTF-8,
LC_TELEPHONE=es_ES.UTF-8, LC_MEASUREMENT=es_ES.UTF-8,
LC_IDENTIFICATION=es_ES.UTF-8, with libpcap version 1.8.1, with GnuTLS 3.5.18,
with Gcrypt 1.8.1, with zlib 1.2.11, binary plugins supported (13 loaded).
Built using gcc 7.4.0.
VAGRANT MACHINE:
tshark --version
TShark (Wireshark) 2.6.10 (Git v2.6.10 packaged as 2.6.10-1~ubuntu18.04.0)
Copyright 1998-2019 Gerald Combs <gerald#wireshark.org> and contributors.
License GPLv2+: GNU GPL version 2 or later <http://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Compiled (64-bit) with libpcap, with POSIX capabilities (Linux), with libnl 3,
with GLib 2.56.4, with zlib 1.2.11, with SMI 0.4.8, with c-ares 1.14.0, with Lua
5.2.4, with GnuTLS 3.5.18, with Gcrypt 1.8.1, with MIT Kerberos, with MaxMind DB
resolver, with nghttp2 1.30.0, with LZ4, with Snappy, with libxml2 2.9.4.
Running on Linux 4.15.0-65-generic, with Intel(R) Core(TM) i7-8650U CPU #
1.90GHz (with SSE4.2), with 7976 MB of physical memory, with locale C, with
libpcap version 1.8.1, with GnuTLS 3.5.18, with Gcrypt 1.8.1, with zlib 1.2.11,
binary plugins supported (13 loaded).
Built using gcc 7.4.0.
The fourth packet in my capture starts to differ. It is the first packet with HTTP2:
At host:
tshark -r capture.pcap -T json -c 4 > host.capture1-4.json
At vagrant machine:
tshark -r capture.pcap -T json -c 4 > vagrant.capture1-4.json
I put together both files:
diff host.capture1-4.json vagrant.capture1-4.json
426c426
< "frame.protocols": "sll:ethertype:ip:tcp:http2"
---
> "frame.protocols": "sll:ethertype:ip:tcp:data"
524,527c524
< "tcp.payload": "50:52:49:20:2a:20:48:54:54:50:2f:32:2e:30:0d:0a:0d:0a:53:4d:0d:0a:0d:0a:00:00:12:04:00:00:00:00:00:00:02:00:00:00:00:00:04:00:40:00:00:00:06:00:a0:00:00:00:00:04:08:00:00:00:00:00:40:00:00:00",
< "tcp.pdu.size": "24",
< "tcp.pdu.size": "27",
< "tcp.pdu.size": "13"
---
> "tcp.payload": "50:52:49:20:2a:20:48:54:54:50:2f:32:2e:30:0d:0a:0d:0a:53:4d:0d:0a:0d:0a:00:00:12:04:00:00:00:00:00:00:02:00:00:00:00:00:04:00:40:00:00:00:06:00:a0:00:00:00:00:04:08:00:00:00:00:00:40:00:00:00"
529,567c526,528
< "http2": {
< "http2.stream": {
< "http2.magic": "PRI * HTTP\/2.0\r\n\r\nSM\r\n\r\n"
< },
< "http2.stream": {
< "http2.length": "18",
< "http2.type": "4",
< "http2.flags": "0x00000000",
< "http2.flags_tree": {
< "http2.flags.ack.settings": "0",
< "http2.flags.unused_settings": "0x00000000"
< },
< "http2.r": "0x00000000",
< "http2.streamid": "0",
< "http2.settings": {
< "http2.settings.id": "2",
< "http2.settings.enable_push": "0"
< },
< "http2.settings": {
< "http2.settings.id": "4",
< "http2.settings.initial_window_size": "4194304"
< },
< "http2.settings": {
< "http2.settings.id": "6",
< "http2.settings.max_header_list_size": "10485760"
< }
< },
< "http2.stream": {
< "http2.length": "4",
< "http2.type": "8",
< "http2.flags": "0x00000000",
< "http2.flags_tree": {
< "http2.flags.unused": "0x00000000"
< },
< "http2.r": "0x00000000",
< "http2.streamid": "0",
< "http2.window_update.r": "0x00000000",
< "http2.window_update.window_size_increment": "1073741824"
< }
---
> "data": {
> "data.data": "50:52:49:20:2a:20:48:54:54:50:2f:32:2e:30:0d:0a:0d:0a:53:4d:0d:0a:0d:0a:00:00:12:04:00:00:00:00:00:00:02:00:00:00:00:00:04:00:40:00:00:00:06:00:a0:00:00:00:00:04:08:00:00:00:00:00:40:00:00:00",
> "data.len": "64"
I can't post the whole capture (it is too much and also have sensible data). I'm a newbie here but think that stack overflow has not file hosting service.

Related

gRPC(C Base) polling engine is built with 'epollex' despite being under linux kernel version 4.5

I have a question about gRPC Core(C Base).
I have been implementing servers with gRPC(C++) async API in CentOS7. I have been analyzing the gRPC internal code(tag: v1.35.0) for my curiosity. I have recently read a document about poller engine. This document explains as follows.
epollex (default but requires kernel version >= 4.5),
epoll1 (If epollex is not available and glibc version >= 2.9)
poll (If kernel does not have epoll support)
However, as far as I have confirmed, this is not true.
I built gRPC in linux kernel version 3.10, then I thought this gRPC poller engine have to be built based on epoll1, but this gRPC poller engine was built into epollex.
I checked gRPC code in detail to find the reason. In the code, the kernel version has not been checked by the macro condition. (only check GRPC_LINUX_EPOLL_CREATE1, then, it is true in my system because the macro variable is satisfied with glibc version above 2.8)
Also, I checked g_factories and GPR_GLOBAL_CONFIG_DEFINE_STRING.
static event_engine_factory g_factories[] = {
{ENGINE_HEAD_CUSTOM, nullptr}, {ENGINE_HEAD_CUSTOM, nullptr},
{ENGINE_HEAD_CUSTOM, nullptr}, {ENGINE_HEAD_CUSTOM, nullptr},
{"epollex", grpc_init_epollex_linux}, {"epoll1", grpc_init_epoll1_linux},
{"poll", grpc_init_poll_posix}, {"none", init_non_polling},
{ENGINE_TAIL_CUSTOM, nullptr}, {ENGINE_TAIL_CUSTOM, nullptr},
{ENGINE_TAIL_CUSTOM, nullptr}, {ENGINE_TAIL_CUSTOM, nullptr},
};
static bool is(const char* want, const char* have) {
return 0 == strcmp(want, "all") || 0 == strcmp(want, have);
}
static void try_engine(const char* engine) {
for (size_t i = 0; i < GPR_ARRAY_SIZE(g_factories); i++) {
if (g_factories[i].factory != nullptr && is(engine, g_factories[i].name)) {
if ((g_event_engine = g_factories[i].factory(
0 == strcmp(engine, g_factories[i].name)))) {
g_poll_strategy_name = g_factories[i].name;
gpr_log(GPR_DEBUG, "Using polling engine: %s", g_factories[i].name);
return;
}
}
}
}
GPR_GLOBAL_CONFIG_DEFINE_STRING(
grpc_poll_strategy, "all",
"Declares which polling engines to try when starting gRPC. "
"This is a comma-separated list of engines, which are tried in priority "
"order first -> last.")
epollex is used in my system on the above conditions
epollex uses EPOLLEXCLUSIVE which can avoid thundering herd problems.
When a wakeup event occurs and multiple epoll file descriptors are attached to the same target file using EPOLLEXCLUSIVE, one or more of the epoll file descriptors will receive an event with epoll_wait(2).
https://man7.org/linux/man-pages/man2/epoll_ctl.2.html
EPOLLEXCLUSIVE is supported by the 4.5 or higher kernel version. Therefore, epollex cannot avoid the above problem in my system. Then, I think it could be inefficient to run into epollex that has a mechanism.
Thus, I would like to ask two questions.
Is this the intended behavior?
epollex (default but requires kernel version >= 4.5),
Then, Is this wrong information?
TL;DR
RHEL7/CentOS7's kernel 3.10.x may have EPOLLEXCLUSIVE.
epollex engine does NOT exist in gRPC source anymore.
Details
CentOS, or RHEL seems to have EPOLLEXCLUSIVE backported into its kernel 3.10.x, which is available in release >= 7.3.
https://bugzilla.redhat.com/show_bug.cgi?id=1426133
gRPC has kernel feature availability check code which actually tries epoll system call with EPOLLEXCLUSIVE flag on. It does not depends on actual version of linux kernel.
https://github.com/grpc/grpc/blob/77e2827f3d70650182474624b4de22e053ac01f6/src/core/lib/iomgr/is_epollexclusive_available.cc#L63-L95
/* This polling engine is only relevant on linux kernels supporting epoll() */
bool grpc_is_epollexclusive_available(void) {
...
struct epoll_event ev;
/* choose events that should cause an error on
EPOLLEXCLUSIVE enabled kernels - specifically the combination of
EPOLLONESHOT and EPOLLEXCLUSIVE */
ev.events =
static_cast<uint32_t>(EPOLLET | EPOLLIN | EPOLLEXCLUSIVE | EPOLLONESHOT);
ev.data.ptr = nullptr;
if (epoll_ctl(fd, EPOLL_CTL_ADD, evfd, &ev) != 0) {
if (errno != EINVAL) {
if (!logged_why_not) {
gpr_log(
GPR_ERROR,
"epoll_ctl with EPOLLEXCLUSIVE | EPOLLONESHOT failed with error: "
"%d. Not using epollex polling engine.",
errno);
logged_why_not = true;
}
close(fd);
close(evfd);
return false;
}
...
BTW epollex polling engine is now removed from gRPC repository for some unknown reason.
https://github.com/grpc/grpc/pull/29160
https://github.com/grpc/grpc/issues/30328#issuecomment-1189477119
I believe some of the document is out-of-date, such as ev_epoll1_posix.cc and other files are moved. But I don't think the description about choosing engine is wrong.
I used the same environment(kernel v3.10.107-1) but I found the epoll1 was used. I'm doubting how you got the conclusion.
Check gRPC version(master branch):
> grep "Version()" ./src/cpp/common/version_cc.cc
std::string Version() { return "1.40.0-dev"; }
Check libc version:
> ldd --version
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
Check kernel version:
> uname -r
3.10.107-1-tlinux2_kvm_guest-0050
Check bazel version:
> bazel version
INFO: Running bazel wrapper (see //tools/bazel for details), bazel version 3.7.1 will be used instead of system-wide bazel installation.
Build label: 3.7.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Nov 24 17:38:30 2020 (1606239510)
Build timestamp: 1606239510
Build timestamp as int: 1606239510
Check the engine(Using polling engine: epoll1):
> GRPC_VERBOSITY=DEBUG CC=`which gcc` bazel run //examples/cpp/helloworld:greeter_server 2>&1 | grep polling
D0810 16:27:41.592830634 2111 is_epollexclusive_available.cc:86] epoll_ctl with EPOLLEXCLUSIVE | EPOLLONESHOT succeeded. This is evidence of no EPOLLEXCLUSIVE support. Not using epollex polling engine.
D0810 16:27:41.593064805 2111 ev_posix.cc:173] Using polling engine: epoll1

C++ gRPC TCP error Protocol not available

I successfully compiled the grpc example helloworld, but when I run the following error appears(then the server starts anyway):
E0625 21080 socket_utils_common_posix.cc:223] check for SO_REUSEPORT:
{
"created":"#1593068359.950045200",
"description":"Protocol not available",
"errno":92,
"file":"/mnt/.../grpc/src/core/lib/iomgr/socket_utils_common_posix.cc",
"file_line":201,"os_error":"Protocol not available",
"syscall":"getsockopt(SO_REUSEPORT)"
}
E0625 21080 socket_utils_common_posix.cc:327] setsockopt(TCP_USER_TIMEOUT) Protocol not available
Server listening on 0.0.0.0:50051
So the server does compile and run, but unfortunately it does not communicate with the client.
It's worth to mention that I am using WSL for windows:
me#com: $ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
I have not found anything related to this in google. What might be the problem and how can I move towards a solution?

openocd fails to load board/ti_cc26x0_launchpad.cfg

Has anyone got openocd to work with the TI cc2640r2 launchpad? I built the latest openocd source but it fails to initialise.
OS is Ubuntu 18.04.1 LTS and openocd was built with
configure --enable-xds110 --enable-cmsis-dap
make
make install
Running
openocd -f board/ti_cc26x0_launchpad.cfg
gets the output
Open On-Chip Debugger 0.10.0+dev-00676-g346ce2f1 (2019-02-05-00:53)
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
adapter speed: 2500 kHz
Error: The 'jtag configure' command must be used after 'init'.
placing 'debug level 3' statements inside the script files show that it is failing within target/ti_cc26x0.cfg at line 25, which is
jtag configure $_CHIPNAME.cpu -event tap-enable "icepick_c_tapenable $_CHIPNAME.jrc 0"
The scripts must have worked (at least once) as they are part of the source distribution.
I use the zephyr folk of open-ocd:
git clone https://github.com/zephyrproject-rtos/openocd.git
cd open-ocd
configure
make
make install
I also needed to reduce the JTAG clock speed:
diff --git a/tcl/board/ti_cc26x0_launchpad.cfg b/tcl/board/ti_cc26x0_launchpad.cfg
index 3613a47f7..2580faa52 100644
--- a/tcl/board/ti_cc26x0_launchpad.cfg
+++ b/tcl/board/ti_cc26x0_launchpad.cfg
## -2,6 +2,6 ##
# TI CC26x0 LaunchPad Evaluation Kit
#
source [find interface/xds110.cfg]
-adapter_khz 2500
+adapter_khz 1500
transport select jtag
source [find target/ti_cc26x0.cfg]

Compiling on Vortex86: "Illegal instruction"

I'm using an embedded PC which has a Vortex86-SG CPU, Ubuntu 10.04 w/ kernel 2.6.34.10-vortex86-sg. Unfortunately we can't compile a new kernel, cause we don't have any source code, not even drivers or patches.
I have to run a small project written in C++ with OpenFrameworks. The framework compiles right each script in of_v0071_linux_release/scripts/linux/ubuntu/install_*.sh.
I noticed that in order to compile against Vortex86/Ubuntu 10.04, the following options must be added in every config.make file:
USER_CFLAGS = -march=i486
USER_LDFLAGS = -lGLEW
In effects, it compiles without errors, but the generated binary doesn't start at all:
root#jb:~/openframeworks/of_v0071_linux_release/apps/myApps/emptyExample/bin# ./emptyExample
Illegal instruction
root#jb:~/openframeworks/of_v0071_linux_release/apps/myApps/emptyExample/bin# echo $?
132
Strace last lines:
munmap(0xb77c3000, 4096) = 0
rt_sigprocmask(SIG_BLOCK, [PIPE], NULL, 8) = 0
--- SIGILL (Illegal instruction) # 0 (0) ---
+++ killed by SIGILL +++
Illegal instruction
root#jb:~/openframeworks/of_v0071_linux_release/apps/myApps/emptyExample/bin#
Any idea to solve this problem?
I know I am a bit late on this but I recently had my own issues trying to compile the kernel for the vortex86dx. I finally was able to build the kernel as well. Use these steps at your own risk as I am not a Linux guru and some settings you may have to change to your own preference/hardware:
Download and use a Linux distribution that runs on a similar kernel version that you plan on compiling. Since I will be compiling Linux 2.6.34.14, I downloaded and installed Debian 6 on virtual box with adequate ram and processor allocations. You could potentially compile on the Vortex86DX itself, but that would likely take forever.
Made sure I hade decencies: #apt-get install ncurses-dev kernel-package
Download kernel from kernel.org (I grabbed Linux-2.6.34.14.tar.xz). Extract files from package.
Grab Config file from dmp ftp site: ftp://vxmx:gc301#ftp.dmp.com.tw/Linux/Source/config-2.6.34-vortex86-sg-r1.zip. Please note vxmx user name. Copy the config file to freshly extracted Linux source folder.
Grab Patch and at ftp://vxdx:gc301#ftp.dmp.com.tw/Driver/Linux/config%26patch/patch-2.6.34-hda.zip. Please note vxdx user name. Copy to kernel source folder.
Patch Kernel: #patch -p1 < patchfilename
configure kernel with #make menuconfig
Load Alternate Configuration File
Enable generic x86 support
Enable Math Emulation
I disabled generic IDE support because I will using legacy mode(selectable in bios)
Under Device Drivers -> Ethernet (10 or 100Mbit) -> Make sure RDC R6040 Fast Ethernet Adapter Support is selected
USB support -> Select Support for Host-side USB, EHCI HCD (USB 2.0) support, OHCI HCD support
safe config as .config
check serial ports: edit .config manually make sure CONFIG_SERIAL_8250_NR_UARTS = 4 (or more if you have additional), CONFIG_SERIAL_8250_RUNTIME_UARTS = 4(or more if you have additional). If you are to use more that 4 serial ports make use config_serail_8250_MANY_PORTs is set.
compile kernel headers and source: #make-kpkg --initrd kernel_image kernel_source kernel_headers modules_image

C++ apt-get working on stdout of popen

This might sound like a stupid question, I am new to C++.
On debian I try to invoke apt-get install with popen. I need to parse the output of this programm. Unfortunately I am not able to read the full output of apt-get.
At some point apt-get might ask for user input (asking if dependencies should be installed). My programm is not able to output this line.
Whis is the last line (see examples below, the line missing in my program is: "Do you want to continue [Y/n]?") not outputted?
When I run the apt-get command manually, the console output looks like this:
$ sudo apt-get install python-wxtools
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
python-wxgtk2.8 python-wxversion
Suggested packages:
wx2.8-doc wx2.8-examples ruby wish tk8.5 tcsh csh octave3.0 mksh pdksh
python-xml editra
The following NEW packages will be installed:
python-wxgtk2.8 python-wxtools python-wxversion
0 upgraded, 3 newly installed, 0 to remove and 8 not upgraded.
Need to get 5,942kB of archives.
After this operation, 25.0MB of additional disk space will be used.
Do you want to continue [Y/n]?
NOTE: no newline at the end of the last line.
when I use my own program, the last line is missing (output)
$ sudo ./test/popen
g++ -Wall -o test/popen test/popen.cpp
test/popen.cpp: In function ‘int main(int, char**, char**)’:
test/popen.cpp:22: warning: comparison between signed and unsigned integer expressions
apt-get install python-wxtools
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
python-wxgtk2.8 python-wxversion
Suggested packages:
wx2.8-doc wx2.8-examples ruby wish tk8.5 tcsh csh octave3.0 mksh pdksh
python-xml editra
The following NEW packages will be installed:
python-wxgtk2.8 python-wxtools python-wxversion
0 upgraded, 3 newly installed, 0 to remove and 8 not upgraded.
Need to get 5,942kB of archives.
After this operation, 25.0MB of additional disk space will be used.
NOTE: newline at the end of output
My reference implementation of popen looks like this in c++:
// $Id: popen.cpp 126 2011-04-25 18:48:02Z wus $
#include <iostream>
#include <stdio.h>
using namespace std;
/**
* run debians apt-get and check output
*/
int main(int argc, char **argv, char **envp) {
FILE *fp;
char buffer[9];
// must use a package which asks for dependencies
char command[255] = "apt-get install python-wxtools";
cout << command << endl;
// Execute command, open /dev/stdout for reading
fp = popen(command, "r");
// read output character by character
while (fread(buffer, 1, 1, fp) != EOF) {
cout << buffer;
}
// close
pclose(fp);
}
I try this to do on a Linux System
$ uname -a
Linux shell1 2.6.35-28-server #50-Ubuntu SMP Fri Mar 18 18:59:25 UTC 2011 x86_64 GNU/Linux
$ gcc --version
gcc (Ubuntu/Linaro 4.4.4-14ubuntu5) 4.4.5
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Try adding the -y option to apt-get. This will make it assume the user says Yes to everything, which should make it "just work." I think it's failing now because it wants to prompt the user but it realizes there is no keyboard input available (possibly by calling isatty(3)) so it gives up.
The problem was not actually input buffering but output buffering of cout. Manual flushes solved the problem:
while (fread(buffer, 1, 1, fp) != EOF) {
cout << buffer << flush;
}