Clean Hex dump of ASCII code - regex

I've got a simple PDF file in Hex dump format, lines looking like this:
00000000: 25 50 44 46 2d 31 2e 34 0d 0a 25 e2 e3 cf d3 0d %PDF-1.4..%.....
00000010: 0a 31 20 30 20 6f 62 6a 0d 0a 3c 3c 20 2f 46 69 .1 0 obj..<< /Fi
00000020: 6c 74 65 72 20 2f 46 6c 61 74 65 44 65 63 6f 64 lter /FlateDecod
00000030: 65 20 2f 4c 65 6e 67 74 68 20 31 38 34 30 36 20 e /Length 18406
00000040: 3e 3e 0d 0a 73 74 72 65 61 6d 0d 0a 2b 2c 9c 77 >>..stream..+,.w
How can I clean up the file so I'm left with the Hex values only?
I'm guessing regular expressions but I fail to apply it successfully in an editor.
Am running on a Windows machine.

Notepad++
Ctrl+F -> Replace
- Tick Regular Expression in the bottom left
Find: ^(.{73}).*$
Replace With: \1
Or Alt+Drag for column selection & copy/pasta.

Related

libevent gives data that I'm not expecting?

So I've been trying to find out why I receive information through my buffer that I know for a fact that aren't being sent by the client. Does anyone else where the problem lies?
I've tried googling and ofcourse stackoverflow but still couldn't find an answer to my question.
char tmpBuffer[SIZE_OF_BUFFER];
struct evbuffer* bufferInput = bufferevent_get_input(bufferEvent);
int size = evbuffer_remove(bufferInput, tmpBuffer, sizeof(tmpBuffer));
dump_hex(tmpBuffer, size);
0x000000: 0d e2 00 00 00 03 01 26 .......&
0x000008: 01 00 02 65 6e 00 08 75 ...en..u
0x000010: 73 65 72 6e 61 6d 65 00 sername.
0x000018: 08 70 61 73 73 77 6f 72 .passwor
0x000020: 64 00 00 00 e7 53 65 72 d....Ser
0x000028: 76 65 72 20 63 6f 6e 6e ver conn
0x000030: 65 63 74 69 6f 6e 20 73 ection s
0x000038: 74 61 74 75 73 3a 0a 20 tatus:.
0x000040: 20 43 6f 6e 6e 65 63 74 Connect
0x000048: 65 64 3a 20 20 20 20 20 ed:
0x000050: 20 20 59 65 73 0a 20 20 Yes.
0x000058: 43 6f 6e 6e 65 63 74 65 Connecte
0x000060: 64 20 74 6f 3a 20 20 20 d to:
0x000068: 20 31 32 37 2e 30 2e 30 127.0.0
0x000070: 2e 31 3a 35 35 35 35 0a .1:6666.
0x000078: 20 20 52 61 77 20 70 61 Raw pa
0x000080: 72 73 65 72 3a 20 20 20 rser:
0x000088: 20 20 20 5b 6f 62 6a 65 [obje
0x000090: 63 74 20 4d 65 73 73 61 ct Messa
0x000098: 67 65 52 65 63 65 69 76 geReceiv
0x0000a0: 65 72 5d 0a 20 20 4d 65 er]. Me
0x0000a8: 73 73 61 67 65 20 68 61 ssage ha
0x0000b0: 6e 64 6c 65 72 3a 20 5b ndler: [
0x0000b8: 6f 62 6a 65 63 74 20 4d object M
0x0000c0: 65 73 73 61 67 65 57 61 essageWa
0x0000c8: 74 63 68 65 72 5d 0a 20 tcher].
0x0000d0: 20 4f 75 74 70 75 74 20 Output
0x0000d8: 62 75 66 66 65 72 3a 20 buffer:
0x0000e0: 20 20 30 20 6d 65 73 73 0 mess
0x0000e8: 61 67 65 28 73 29 0a 20 age(s).
0x0000f0: 20 49 6e 70 75 74 20 62 Input b
0x0000f8: 75 66 66 65 72 3a 20 20 uffer:
0x000100: 20 20 30 20 62 79 74 65 0 byte
0x000108: 28 73 29 0a 00 20 30 38 (s).. 08
0x000110: 37 33 33 30 38 36 64 39 733086d9
0x000118: 32 63 38 33 64 62 37 31 2c83db71
0x000120: 39 33 36 39 32 61 62 30 93692ab0
0x000128: 31 36 64 36 31 64 16d61d
Is the data that I receive, but the only data that I know that is being sent by the client is in the first part. You can see the username and password there.

APNs Provider API HTTP/2 using c++, openssl

I try to send header frame and data frame with HPACK, and I received goaway frame with COMPRESSION_ERROR (0x9) from APNS.
my step:
send setting frame
recv setting frame
send ACK
recv ACK
send header frame
send data frame
requset:
HEADERS
- END_STREAM
+ END_HEADERS
:method = POST
:scheme = https
:path = /3/device/ad92823c9e42e37241b6cdf6f583c6cdf5dcc21289c57fb6641213abeff8095d
host = api.development.push.apple.com
DATA
+ END_STREAM
{ "aps" : { "alert" : "Hello" } }
Hex:
00 00 77 01 04 00 00 00 01 HEADER FRAME
83 :method = POST
86 :scheme = https
04 :path = (Literal Header Field without Indexing)
4a length(74)
2f 33 2f 64 65 76 69 63 /3/device/ad92823c9e42e37241b6cdf6f583c6cdf5dcc21289c57fb6641213abeff8095d
65 2f 61 64 39 32 38 32
33 63 39 65 34 32 65 33
37 32 34 31 62 36 63 64
66 36 66 35 38 33 63 36
63 64 66 35 64 63 63 32
31 32 38 39 63 35 37 66
62 36 36 34 31 32 31 33
61 62 65 66 66 38 30 39
35 64
66 host =
1e length(30)
61 70 69 2e 64 65 76 65 api.development.psuh.apple.com
6c 6f 70 6d 65 6e 74 2e
70 75 73 68 2e 61 70 70
6c 65 2e 63 6f 6d
00 00 21 00 01 00 00 00 01 DATA FRAME
7b 20 22 61 70 73 22 20 { "aps" : { "alert" : "hello" } }
3a 20 7b 20 22 61 6c 65
72 74 22 20 3a 20 22 68
65 6c 6c 6f 22 20 7d 20
7d
any tips?
thanks

POP3 Authentication

I am trying to write a simple POP3 server and I have a problem with authentication.
I run the server and try to connect to it with help of Icedove. When the client connects to the server, the following occurs:
Server:+OK POP3 server ready.\r\n
Client:CAPA\r\n
Server:-ERR incorrect operation\r\n
Client:USER sti#blabla.ru\r\n
Server:+OK please send PASS command\r\n
Client:PASS sadsd\r\n
Server:+OK sti#blabla.ru is welcome here\r\n
Client:QUIT\r\n
socket closed
I can't understand why client close connection after "+OK sti#blabla.ru is welcome here\r\n" and just says me that the user name or pass incorrect.
I am trying to experiment so new log is slightly different.
Wireshark log:
00000000 2b 4f 4b 20 50 4f 50 33 20 73 65 72 76 65 72 20 +OK POP3 server
00000010 72 65 61 64 79 2e 0d 0a ready...
00000000 41 55 54 48 0d 0a AUTH..
00000018 2d 45 52 52 20 69 6e 63 6f 72 72 65 63 74 20 6f -ERR inc orrect o
00000028 70 65 72 61 74 69 6f 0d 0a peratio. .
00000006 43 41 50 41 0d 0a CAPA..
00000031 2d 45 52 52 20 69 6e 63 6f 72 72 65 63 74 20 6f -ERR inc orrect o
00000041 70 65 72 61 74 69 6f 0d 0a peratio. .
0000000C 55 53 45 52 20 73 74 69 0d 0a USER sti ..
0000004A 2b 4f 4b 20 70 6c 65 61 73 65 20 73 65 6e 64 20 +OK plea se send
0000005A 50 41 53 53 20 63 6f 6d 6d 61 6e 64 0d 0a PASS com mand..
00000016 50 41 53 53 20 73 61 64 73 64 0d 0a PASS sad sd..
00000068 2b 4f 4b 20 73 74 69 27 73 20 6d 61 69 6c 64 72 +OK sti' s maildr
00000078 6f 70 20 68 61 73 20 35 20 6d 65 73 73 61 67 65 op has 5 message
00000088 73 20 28 35 35 20 6f 63 74 65 74 73 29 0d 0a s (55 oc tets)..
00000022 51 55 49 54 0d 0a QUIT..

how to decrypt xxtea?

I am trying to se if I can use XXTEA in my application.
The code that I found on Wikipedia code is only for encryption and there is no code for decryption.
How can I decrypt with XXTEA?
I am developing c++ in windows (visual studio 2012)
The code snippet in the article apparently uses negative values of n to mean decoding.
#include <stdio.h>
#include <stdint.h>
void btea(uint32_t *v, int n, uint32_t const key[4]); /* ... */
int main(void) {
char s[128] = "hello world bla bla bla bla bla bla";
uint32_t key[4] = {1,2,3,4};
int n = 128 / sizeof(uint32_t);
printf("%s\n",s);
btea((uint32_t*)s, n, key);
printf("%s\n",s);
btea((uint32_t*)s, -n, key);
printf("%s\n",s);
}
prints:
$ ./a.out | hexdump -C
00000000 68 65 6c 6c 6f 20 77 6f 72 6c 64 20 62 6c 61 20 |hello world bla |
00000010 62 6c 61 20 62 6c 61 20 62 6c 61 20 62 6c 61 20 |bla bla bla bla |
00000020 62 6c 61 0a 2f 44 86 75 d5 16 83 bd 5d 20 af f3 |bla./D.u....] ..|
00000030 a7 dd bf 9f 3a cd a0 13 ef 2b 89 48 2e f6 89 20 |....:....+.H... |
00000040 e2 ba e4 9f ed 38 d1 86 43 82 9e a6 47 6c e4 6d |.....8..C...Gl.m|
00000050 a8 82 22 9e cb 5b d1 a1 18 14 ef 18 ca 23 26 cc |.."..[.......#&.|
00000060 18 1d 4f ba 1b d5 f5 d0 45 72 1c 69 b9 22 a3 08 |..O.....Er.i."..|
00000070 44 71 1f 3b 8f a9 9d 5b 14 93 7b 59 b4 6b e8 1b |Dq.;...[..{Y.k..|
00000080 18 97 1b 74 61 d6 e9 e9 60 96 8e 4c 26 be 21 fc |...ta...`..L&.!.|
00000090 75 9f 4a 18 67 46 f0 95 2b ca 90 d9 f4 ce 3f 2f |u.J.gF..+.....?/|
000000a0 44 82 56 44 c0 0a 39 57 ff 7f 0a 68 65 6c 6c 6f |D.VD..9W...hello|
000000b0 20 77 6f 72 6c 64 20 62 6c 61 20 62 6c 61 20 62 | world bla bla b|
000000c0 6c 61 20 62 6c 61 20 62 6c 61 20 62 6c 61 0a |la bla bla bla.|
000000cf

FFMPEG: directly decode packets after encoding

using FFMPEG API, I try to encode a x264 video to a MP4 file with 0 frame latency and also, in realtime, show the currently encoded frame on screen (with encoding artifacts). The encoding to the file works, but so far I don't get the frames decoded right after writing them to the file. What I try is to feed the packetdata that is returned from avcodec_encode_video() right into avcodec_decode_video2() but the function returns -1 and the cmd output shows:
[h264 # 00000000025F0710] non-existing PPS 0 referenced
[h264 # 00000000025F0710] decode_slice_header error
[h264 # 00000000025F0710] no frame
here is some code i use for encoding:
AVPacket FFMpegEncoder2::write_video_frame(AVFrame* pic, int &numBytes)
{
int out_size, ret;
AVPacket pkt;
/* encode the image */
out_size = avcodec_encode_video(m_cctx, m_outbuf,
m_outbufSize, pic);
/* If size is zero, it means the image was buffered. */
assert(out_size>0) //0 frame delay
av_init_packet(&pkt);
if (m_cctx->coded_frame->pts != AV_NOPTS_VALUE)
pkt.pts = av_rescale_q(m_cctx->coded_frame->pts,m_cctx->time_base, m_video_st->time_base);
if (m_cctx->coded_frame->key_frame)
pkt.flags |= AV_PKT_FLAG_KEY;
pkt.stream_index = m_video_st->index;
pkt.data = m_outbuf;
pkt.size = out_size;
/* Write the compressed frame to the media file. */
ret = av_interleaved_write_frame(m_fctx, &pkt);
if (ret != 0) {
fprintf(stderr, "Error while writing video frame\n");
exit(1);
}
numBytes = out_size;
return pkt;
}
and then I take this returned packet and feed it into the decoder:
const AVFrame* FFMpegDecoder2::decode(AVPacket* packet){
AVPacket pkt;
av_init_packet(&pkt);
pkt.size = packet->size;
pkt.data = packet->data;
int len=0;
int got_picture=0;
while (pkt.size > 0) {
len = avcodec_decode_video2(m_cctx, m_frame, &got_picture, &pkt);
if (len < 0) {
fprintf(stderr, "Error while decoding frame %d\n", m_f);
exit(1);
}
if (got_picture) {
assert(pkt.size==len);
m_f++;
}
pkt.size -= len;
pkt.data += len;
}
assert(got_picture);
return m_frame;
}
but as stated, avcodec_decode_video2() returns -1
what am I doing wrong? Do i need to feed some headerdata into the decoder first somehow?
//edit:
if i set
m_formatCtx->oformat->flags &= ~AVFMT_GLOBALHEADER;
m_codecctx->flags &= ~CODEC_FLAG_GLOBAL_HEADER;
then i can decode the returned packet without error, but the written mp4 file will be black.
//edit: this is how i setup the decoder:
FFMpegDecoder2::FFMpegDecoder2(CodecID id)
: m_codec(NULL)
, m_cctx(NULL)
{
/* Initialize libavcodec, and register all codecs and formats. */
avcodec_register_all();
m_codec = avcodec_find_decoder(id);
if (!m_codec) {
fprintf(stderr, "codec not found\n");
exit(1);
}
m_cctx = avcodec_alloc_context3(m_codec);
m_cctx->codec = m_codec;
m_cctx->pix_fmt = PIX_FMT_YUV420P;
avcodec_open2(m_cctx, m_codec, NULL);
//alloc frame
m_frame = avcodec_alloc_frame();
}
this is what the memory window shows for the first packet (didn't copy all. the size of the first packet is 7859):
0x0000000002E66670 00 00 01 06 05 ff ff 55 dc 45 e9 bd e6 d9 48 b7 96 2c d8 20 d9 23 ee ef 78 32 36 34 20 2d 20 63 6f 72 65 20 31 32 30 20 72 32 31 34 36 20 62 .....ÿÿUÜEé.æÙH·–,Ø Ù#îïx264 - core 120 r2146 b
0x0000000002E6669F 63 64 34 31 64 62 20 2d 20 48 2e 32 36 34 2f 4d 50 45 47 2d 34 20 41 56 43 20 63 6f 64 65 63 20 2d 20 43 6f 70 79 6c 65 66 74 20 32 30 30 33 cd41db - H.264/MPEG-4 AVC codec - Copyleft 2003
0x0000000002E666CE 2d 32 30 31 31 20 2d 20 68 74 74 70 3a 2f 2f 77 77 77 2e 76 69 64 65 6f 6c 61 6e 2e 6f 72 67 2f 78 32 36 34 2e 68 74 6d 6c 20 2d 20 6f 70 74 -2011 - http://www.videolan.org/x264.html - opt
0x0000000002E666FD 69 6f 6e 73 3a 20 63 61 62 61 63 3d 30 20 72 65 66 3d 33 20 64 65 62 6c 6f 63 6b 3d 31 3a 30 3a 30 20 61 6e 61 6c 79 73 65 3d 30 78 33 3a 30 ions: cabac=0 ref=3 deblock=1:0:0 analyse=0x3:0
0x0000000002E6672C 78 31 31 33 20 6d 65 3d 68 65 78 20 73 75 62 6d 65 3d 34 20 70 73 79 3d 31 20 70 73 79 5f 72 64 3d 31 2e 30 30 3a 30 2e 30 30 20 6d 69 78 65 x113 me=hex subme=4 psy=1 psy_rd=1.00:0.00 mixe
0x0000000002E6675B 64 5f 72 65 66 3d 31 20 6d 65 5f 72 61 6e 67 65 3d 31 36 20 63 68 72 6f 6d 61 5f 6d 65 3d 31 20 74 72 65 6c 6c 69 73 3d 30 20 38 78 38 64 63 d_ref=1 me_range=16 chroma_me=1 trellis=0 8x8dc
0x0000000002E6678A 74 3d 31 20 63 71 6d 3d 30 20 64 65 61 64 7a 6f 6e 65 3d 32 31 2c 31 31 20 66 61 73 74 5f 70 73 6b 69 70 3d 31 20 63 68 72 6f 6d 61 5f 71 70 t=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp
0x0000000002E667B9 5f 6f 66 66 73 65 74 3d 30 20 74 68 72 65 61 64 73 3d 31 20 73 6c 69 63 65 64 5f 74 68 72 65 61 64 73 3d 30 20 6e 72 3d 30 20 64 65 63 69 6d _offset=0 threads=1 sliced_threads=0 nr=0 decim
0x0000000002E667E8 61 74 65 3d 31 20 69 6e 74 65 72 6c 61 63 65 64 3d 30 20 62 6c 75 72 61 79 5f 63 6f 6d 70 61 74 3d 30 20 63 6f 6e 73 74 72 61 69 6e 65 64 5f ate=1 interlaced=0 bluray_compat=0 constrained_
0x0000000002E66817 69 6e 74 72 61 3d 30 20 62 66 72 61 6d 65 73 3d 30 20 77 65 69 67 68 74 70 3d 32 20 6b 65 79 69 6e 74 3d 32 35 20 6b 65 79 69 6e 74 5f 6d 69 intra=0 bframes=0 weightp=2 keyint=25 keyint_mi
0x0000000002E66846 6e 3d 32 20 73 63 65 6e 65 63 75 74 3d 34 30 20 69 6e 74 72 61 5f 72 65 66 72 65 73 68 3d 30 20 72 63 3d 61 62 72 20 6d 62 74 72 65 65 3d 30 n=2 scenecut=40 intra_refresh=0 rc=abr mbtree=0
0x0000000002E66875 20 62 69 74 72 61 74 65 3d 34 30 30 20 72 61 74 65 74 6f 6c 3d 31 2e 30 20 71 63 6f 6d 70 3d 30 2e 36 30 20 71 70 6d 69 6e 3d 30 20 71 70 6d bitrate=400 ratetol=1.0 qcomp=0.60 qpmin=0 qpm
0x0000000002E668A4 61 78 3d 36 39 20 71 70 73 74 65 70 3d 34 20 69 70 5f 72 61 74 69 6f 3d 31 2e 34 30 20 61 71 3d 31 3a 31 2e 30 30 00 80 00 00 00 01 65 88 84 ax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00.€....eˆ.
0x0000000002E668D3 11 ef ff f8 22 0f 8a 00 02 09 7e 38 00 08 45 c7 00 01 1d c9 39 3d 87 ff e0 ac 13 03 6d 05 f1 00 10 00 10 12 88 04 00 04 02 60 70 4e 2d cc 38 .ïÿø".Š...~8..EÇ...É9=.ÿà¬..m.ñ.....ˆ....`pN-Ì8
0x0000000002E66902 27 16 e6 07 21 1a e6 1c 84 6b 9f f0 f0 27 15 f2 7b 87 ff c1 58 2a 8a 00 04 b8 80 00 58 00 04 02 62 01 03 c1 c1 04 63 07 04 11 88 90 b1 89 0b '.æ.!.æ..kŸðð'.ò{.ÿÁX*Š..¸€.X...b..ÁÁ.c...ˆ.±..
0x0000000002E66931 1f 2c 11 02 b1 40 00 87 8f a4 f7 0f ff 82 b0 55 06 93 41 c4 10 51 00 00 40 14 00 04 00 a3 b7 35 b7 30 38 26 1e e6 1c 13 0f 73 f2 c1 10 2b 14 .,..±#...¤÷.ÿ.°U.“AÄ.Q..#....£·5·08&.æ...sòÁ.+.
0x0000000002E66960 1f 1f 1c 32 7f 94 11 82 a1 40 01 f1 00 00 40 14 01 22 00 01 e0 1e 22 0a e3 83 1c 19 3d f8 7f e0 b0 16 03 01 22 0f 88 00 02 00 00 16 20 01 17 ...2.”..¡#.ñ..#.."..à.".ãƒ..=ø.à°...".ˆ..... ..
0x0000000002E6698F 03 84 c2 5c 87 09 84 b9 06 4a e4 a4 ae 08 82 d8 e0 00 20 0f 1d 93 df c3 fe 0b 01 54 50 07 88 a8 80 00 64 09 88 58 88 58 83 84 1d 88 38 41 d8 ..Â\.....J䤮..Øà. ..“ßÃþ..TP.ˆ¨€.d.ˆXˆXƒ..ˆ8AØ
0x0000000002E669BE f2 c1 10 2b 14 00 08 f8 e0 00 62 38 64 ff 08 70 13 0a c1 d2 e9 b5 5d ba 10 80 09 a2 01 2e 07 04 c2 dc 87 04 c2 dc 81 c8 66 b9 0e 43 35 cb 0f òÁ.+...øà.b8dÿ.p..ÁÒéµ]º.€.¢....ÂÜ..ÂÜ.Èf..C5Ë.
0x0000000002E669ED ff c1 10 27 2c 00 7e 8e 00 05 64 e4 f6 1f ff 82 28 a0 00 21 99 e3 80 00 99 ac 70 00 11 39 93 93 d8 7f fe 0a c1 40 34 9a 0b e3 40 00 84 40 01 ÿÁ.',.~Ž..däö.ÿ.( .!™ã€.™¬p..9““Ø.þ.Á#4š.ã#..#.
0x0000000002E66A1C 00 01 02 88 fd cd 7d cc 0e 08 a4 dc c3 82 29 37 3f e0 88 14 8b f1 c3 1c 03 27 f0 c3 60 a0 50 62 86 da 36 1f 10 00 0a 80 00 80 14 40 00 20 00 ...ˆýÍ}Ì..¤ÜÃ.)7?àˆ..ñÃ..'ðÃ` Pb.Ú6....€.€.#. .
and this is the encoders output (until after encoding frame 0):
[libx264 # 00000000005ADAA0] using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShu
ffle SSE4.2
[libx264 # 00000000005ADAA0] profile High, level 3.0
[libx264 # 00000000005ADAA0] 264 - core 120 r2146 bcd41db - H.264/MPEG-4 AVC cod
ec - Copyleft 2003-2011 - http://www.videolan.org/x264.html - options: cabac=0 r
ef=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=4 psy=1 psy_rd=1.00:0.00 mixed
_ref=1 me_range=16 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pski
p=1 chroma_qp_offset=0 threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 b
luray_compat=0 constrained_intra=0 bframes=0 weightp=2 keyint=25 keyint_min=2 sc
enecut=40 intra_refresh=0 rc=abr mbtree=0 bitrate=100 ratetol=1.0 qcomp=0.60 qpm
in=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'out2.mp4':
Stream #0:0: Video: h264, yuv420p, 640x480, q=-1--1, 100 kb/s, 90k tbn, 25 t
bc
[mp4 # 0000000000467570] Encoder did not produce proper pts, making some up.
This my packet data:
00 00 00 01 67 64 00 1F AC 56 24 02 80 DA 10 00
00 03 00 10 00 00 03 03 C0 F1 83 18 98 00 00 00
01 68 E8 8E 0B CB 22 C0 00 00 00 01 65 88 82 00
The first a few bytes for a first h264 packet should somewhat look like this.
00 00 00 01 ?7 ... 00 00 00 01 ?8 ... 00 00 00 01 ?5
?7 -> sps
?8 -> pps
?5 -> idr picture
There might be something else like ?6, which is sei, etc. But with SPS, PPS and the idr picture, the decoder should be able to initialize itself properly.
Another case might be the packet contains more than one picture (00 00 00 01 ?5, or 00 00 00 01 ?1). As far as I know, the decoder cannot handle h264 packets with more than one pictures properly.