i've got a string frame looking like this in google bigquery:
S,0,2B3,8, C2 B3 00 00 00 00 03 DE
S,0,3FA,6, 00 E0 04 A5 00 0B
S,0,440,8, 80 40 4E A5 00 47 00 64
S,0,450,8, 89 50 01 12 01 19 01 B3
S,0,4B0,8, 80 B0 4E A5 00 43 00 64
my aim is to extract the 8 bytes at the end (eg 80 40 4E A5 00 47 00 64
). possible only the ones beginning with 83 and 84.
i didnt get it to work with neither split, trim, contains nor regexp_extract.
i'd be quite happy if anyone could help me.
regards
/edit
Thank you both very much for your solutions! this helped quite a lot.
#standardSQL
SELECT
*
FROM (
SELECT
timestamp,
REGEXP_EXTRACT(CAN_Frame, r', ([^,]+)$') AS bytes_string,
FROM_HEX(REPLACE(REGEXP_EXTRACT(CAN_Frame, r', ([^,]+)$'), ' ', '')) AS bytes
FROM `data.source`
)
WHERE SUBSTR(bytes, 1, 1) IN (b'\x83', b'\x84')
ORDER BY timestamp DESC
LIMIT 8000
gives me
Row timestamp bytes_string bytes
1 2017-09-29 14:31:02 UTC 84 10 00 25 00 21 00 4F hBAAJQAhAE8=
2 2017-09-29 14:30:42 UTC 83 80 00 01 00 03 00 0D g4AAAQADAA0=
3 2017-09-29 14:30:40 UTC 84 B2 00 27 00 08 00 03 hLIAJwAIAAM=
#standardSQL
SELECT
timestamp,
TRIM(SPLIT(CAN_Frame)[OFFSET(4)]) AS bytes
FROM
`data.source`
WHERE
LENGTH(CAN_Frame) > 1 and
SUBSTR(TRIM(SPLIT(CAN_Frame)[OFFSET(4)]),1,2) IN ('83', '84')
ORDER BY
timestamp DESC
LIMIT
8000
gives me
Row timestamp bytes
1 2017-09-29 14:31:02 UTC 84 10 00 25 00 21 00 4F
2 2017-09-29 14:30:42 UTC 83 80 00 01 00 03 00 0D
3 2017-09-29 14:30:40 UTC 84 B2 00 27 00 08 00 03
is there a possibility to get only the sixth and seventh byte from the bytes_string beginning with 83, to get 4th and 5th byte from the bytes_string beginning with 84 and to geht the 8th byte from string 83 and the 3rd byte from string 84 for further calculations?
best regards
Below is for BigQuery StandardSQL
#standardSQL
WITH `yourTable` AS (
SELECT 'S,0,2B3,8, C2 B3 00 00 00 00 03 DE' AS frame UNION ALL
SELECT 'S,0,3FA,6, 00 E0 04 A5 00 0B' UNION ALL
SELECT 'S,0,440,8, 80 40 4E A5 00 47 00 64' UNION ALL
SELECT 'S,0,450,8, 89 50 01 12 01 19 01 B3' UNION ALL
SELECT 'S,0,4B0,8, 80 B0 4E A5 00 43 00 64'
)
SELECT frame, TRIM(SPLIT(frame)[OFFSET(4)]) AS bytes
FROM `yourTable`
WHERE SUBSTR(TRIM(SPLIT(frame)[OFFSET(4)]), 1, 2) IN ('80', 'C2')
Here is an example that should help. It produces two columns with different interpretations of the bytes: one (bytes_string) is just the end of the strings that you showed, whereas the other (bytes) is the bytes string converted to an actual BYTES type. In the BigQuery UI, make sure to deselect "Use Legacy SQL" under "Show Options" or include the #standardSQL directive:
#standardSQL
WITH Frames AS (
SELECT 'S,0,2B3,8, C2 B3 00 00 00 00 03 DE' AS frame UNION ALL
SELECT 'S,0,3FA,6, 00 E0 04 A5 00 0B' UNION ALL
SELECT 'S,0,440,8, 80 40 4E A5 00 47 00 64' UNION ALL
SELECT 'S,0,450,8, 89 50 01 12 01 19 01 B3' UNION ALL
SELECT 'S,0,4B0,8, 80 B0 4E A5 00 43 00 64'
)
SELECT
frame,
REGEXP_EXTRACT(frame, r', ([^,]+)$') AS bytes_string,
FROM_HEX(REPLACE(REGEXP_EXTRACT(frame, r', ([^,]+)$'), ' ', '')) AS bytes
FROM Frames;
Here is another example that demonstrates filtering on the bytes column to include only values starting with \x83 or \x84 (this will return an empty result set for the sample data you provided):
#standardSQL
WITH Frames AS (
SELECT 'S,0,2B3,8, C2 B3 00 00 00 00 03 DE' AS frame UNION ALL
SELECT 'S,0,3FA,6, 00 E0 04 A5 00 0B' UNION ALL
SELECT 'S,0,440,8, 80 40 4E A5 00 47 00 64' UNION ALL
SELECT 'S,0,450,8, 89 50 01 12 01 19 01 B3' UNION ALL
SELECT 'S,0,4B0,8, 80 B0 4E A5 00 43 00 64'
)
SELECT
*
FROM (
SELECT
frame,
REGEXP_EXTRACT(frame, r', ([^,]+)$') AS bytes_string,
FROM_HEX(REPLACE(REGEXP_EXTRACT(frame, r', ([^,]+)$'), ' ', '')) AS bytes
FROM Frames
)
WHERE SUBSTR(bytes, 1, 1) IN (b'\x83', b'\x84');
I have the following contents from data.log file. I wish to extract the time value and part of the payload (after deadbeef in the payload, third row, starting second to last byte. Please refer to expected output).
data.log
print 1: file offset 0x0
ts=0x584819041ff529e0 2016-12-07 14:13:24.124834649 UTC
type: ERF Ethernet
dserror=0 rxerror=0 trunc=0 vlen=0 iface=1 rlen=96 lctr=0 wlen=68
pad=0x00 offset=0x00
dst=aa:bb:cc:dd:ee:ff src=ca:fe:ba:be:ca:fe
etype=0x0800
45 00 00 32 00 00 40 00 40 11 50 ff c0 a8 34 35 E..2..#.#.P...45
c0 a8 34 36 80 01 00 00 00 1e 00 00 08 08 08 08 ..46............
08 08 50 e6 61 c3 85 21 01 00 de ad be ef 85 d7 ..P.a..!........
91 21 6f 9a 32 94 fd 07 01 00 de ad be ef 85 d7 .!o.2...........
print 2: file offset 0x60
ts=0x584819041ff52b00 2016-12-07 14:13:24.124834716 UTC
type: ERF Ethernet
dserror=0 rxerror=0 trunc=0 vlen=0 iface=1 rlen=96 lctr=0 wlen=68
pad=0x00 offset=0x00
dst=aa:bb:cc:dd:ee:ff src=ca:fe:ba:be:ca:fe
etype=0x0800
45 00 00 32 00 00 40 00 40 11 50 ff c0 a8 34 35 E..2..#.#.P...45
c0 a8 34 36 80 01 00 00 00 1e 00 00 08 08 08 08 ..46............
08 08 68 e7 61 c3 85 21 01 00 de ad be ef 86 d7 ..h.a..!........
91 21 c5 34 77 bd fd 07 01 00 de ad be ef 86 d7 .!.4w...........
Expected output
I just want to replace the dots and colons in the time field (before UTC) and get the entire value.
141324124834649,85d79121
141324124834716,86d79121
What I have done so far
I have extracted the fields after "." but not sure how to replace the colons and get the entire time value.
awk -F '[= ]' '$NF == "UTC"{split($4,b,".");s=b[2]",";a=15} /de ad be ef/{s=s $a $(a+1);if(a==1)print s;a=1}' data.log
124834649,85d79121
124834716,86d79121
Any help is much appreciated.
awk '$NF == "UTC"{gsub("[.:]","",$3);s=$3",";a=15} /de ad be ef/{s=s $a $(a+1);if(a==1)print s;a=1}' data.log
Result:
141324124834649,85d79121
141324124834716,86d79121
PS: it can be simplified with getline :
awk '$NF == "UTC"{gsub("[.:]","",$3);s=$3","} /de ad be ef/{s=s $15 $16;getline;print(s $1 $2)}' data.log
You can extract the time part like this:
$ awk '/UTC/ {split($0,a); gsub(/[\.:]/,"",a[3]); print a[3]}' file
141324124834649
141324124834716
for the UTC part, (rest of the code is the same
awk '/UTC$/{gsub(/[\.:]/,"");print $3}' YourFile
just remove the ":" and "." and take the field value, other part of the line don't have those 2 character, so is not modified
$NF test is replaced by /UTC$/, a bit faster and simpler (OMHO)
the full code
awk -F '[= ]' '/UTC$/{gsub(/[\.:]/,"");s=$3",";a=15} /de ad be ef/{s=s $a $(a+1);if(a==1)print s;a=1}' YourFile
I have made a programmer following this:
Directly Read/Write Handshake data with Memory BIO
But now the function call SSL_do_handshake gets the error wrong version number, do you have any idea?
Here is the sample code.
client,
SSL_set_bio(pssl, rbio, wbio);
SSL_set_connect_state(pssl);
SSL_do_handshake(pssl);
length = BIO_read(wbio, buffer, length);
server,
SSL_CTX_load_verify_locations(global_sslctx , "cacerts.pem" , NULL);
SSL_CTX_use_certificate_file( global_sslctx , "cert.pem" , SSL_FILETYPE_PEM);
SSL_CTX_use_PrivateKey_file( global_sslctx , "cert.pem" , SSL_FILETYPE_PEM);
BIO_write(SSL_get_wbio(pssl) , pdata , length);
if (!SSL_is_init_finished(pssl)) {
SSL_do_handshake(pssl);
printf("%s" ,ERR_reason_error_string(ERR_get_error()));
}
And i have dumped the momery data of the client,
16 03 01 00 de 01 00 00 da 03 01 9b dc 1e ef 4d f6 74 96 9b 8a 3f c0 3f de 37 4c 1b fa d7 d8 04 12 79 f9 bf 92 38 d8 59 c0 4f b4 00 00 68 c0 14 c0 0a c0 22 c0 21 00 39 00 38 00 88 00 87 c0 0f c0 05 00
How can I generate compressed ECDSA keys in Crypto++?
AutoSeededRandomPool prng;
ECDSA<ECP, SHA1>::PrivateKey privateKey;
ECDSA<ECP, SHA1>::PublicKey publicKey;
privateKey.Initialize( prng, CryptoPP::ASN1::secp256r1());
const Integer& x1 = privateKey.GetPrivateExponent();
cout << "priv: " << std::hex << x1 << endl;
privateKey.MakePublicKey( publicKey );
const ECP::Point& q = publicKey.GetPublicElement();
const Integer& qx = q.x;
const Integer& qy = q.y;
cout << "pub x: " << std::hex << qx << endl;
cout << "pub y: " << std::hex << qy << endl;
This code generates a keypair and prints the X and Y components of the public key.
I need to know if there is a way to print the Y component of the compressed key, or if I need to generate it from the uncompressed y component. If I need to generate it, can someone link me to a good explanation of how to work with the Integer class?
How can I generate compressed ECDSA keys in CryptoPP?
You don't create a compressed key. You generate a public key, and then you compress it if that's what is required. From How to construct a public ECDSA key with point compression turned on? on the Crpyto++ user group:
ECDSA<ECP, SHA1>::Verifier verifier(...);
verifier.AccessKey().AccessGroupParameters().SetPointCompression(true);
In your case, it will be:
publicKey.AccessGroupParameters().SetPointCompression(true);
since verifier.AccessKey() returns the ECDSA public key.
I need to know if there is a way to print the Y component of the compressed key
Compression is a presentation format option or optimization. You can't print the Y component on a serialized key with compression because its not there (see the ASN.1 dumps below).
With compression on, the Y "shorthand" is going to be 1 or -1, depending on the sign of the component (the sign indicates what quadrant the point is in). The idea is someone sends you a {1,X} pair or {-1,X} pair and you can solve for Y because you know which quadrant it should be in. If Y is only allowed to be positive, you only need to serialize {X} (and not {-1,X} or {1,X}). (I did not dive into the books or standards, so there may be some errata here).
... or if I need to generate it from the uncompressed y component. If I need to generate it, can someone link me to a good explanation of how to work with the Integer class?
I have no idea what you're talking about at here. Point compression affects presentation. If you turn on point compression and fetch Y, you will still get Y out. You can't compute on a compressed point. You need the X and Y coordinates.
Add the following to your program:
publicKey.AccessGroupParameters().SetPointCompression(false);
ByteQueue q1;
publicKey.Save(q1);
cout << "Uncompressed size: " << dec << q1.MaxRetrievable() << endl;
publicKey.AccessGroupParameters().SetPointCompression(true);
ByteQueue q2;
publicKey.Save(q2);
cout << "Compressed size: " << dec << q2.MaxRetrievable() << endl;
Here's what I got for the output:
$ ./cryptopp-test.exe
priv: 4ce30d22d9593d9c7f4406eda1ce0740c7486106374d0abe7e352e1d5b1d5622h
pub x: 41a9bc936b6d1dd3a1ded997d7da08f1df990e9b50f9b58e9e4fd9319758ea34h
pub y: 4ad39ffb79c402063a99ecbc0cac8fde606db6764ace90933feee5f8d65937a2h
Uncompressed size: 311
Compressed size: 246
If you fetch Y after turning on point compression, you will still get 4ad39ffb79c402063a99ecbc0cac8fde606db6764ace90933feee5f8d65937a2h because compression affects presentation.
Now, add the following to dump the uncompressed and compressed key:
FileSink fs1("key-1.der", true);
q1.TransferTo(fs1);
FileSink fs2("key-2.der", true);
q2.TransferTo(fs2);
The keys are dumped in ASN.1's DER encoding and conform to Certicom's SEC 1: Elliptic Curve Cryptography (and to a lesser extent, ANSI 9.62 and RFC 5480, ECC SubjectPublicKeyInfo Format - see below on the Field Element's OCTET STRING vs BIT STRING).
You can run Peter Gutmann's dumpasn1 on them:
$ dumpasn1.exe key-1.der
0 307: SEQUENCE {
4 236: SEQUENCE {
7 7: OBJECT IDENTIFIER ecPublicKey (1 2 840 10045 2 1)
16 224: SEQUENCE {
19 1: INTEGER 1
22 44: SEQUENCE {
24 7: OBJECT IDENTIFIER prime-field (1 2 840 10045 1 1)
33 33: INTEGER
: 00 FF FF FF FF 00 00 00 01 00 00 00 00 00 00 00
: 00 00 00 00 00 FF FF FF FF FF FF FF FF FF FF FF
: FF
: }
68 68: SEQUENCE {
70 32: OCTET STRING
: FF FF FF FF 00 00 00 01 00 00 00 00 00 00 00 00
: 00 00 00 00 FF FF FF FF FF FF FF FF FF FF FF FC
104 32: OCTET STRING
: 5A C6 35 D8 AA 3A 93 E7 B3 EB BD 55 76 98 86 BC
: 65 1D 06 B0 CC 53 B0 F6 3B CE 3C 3E 27 D2 60 4B
: }
138 65: OCTET STRING
: 04 6B 17 D1 F2 E1 2C 42 47 F8 BC E6 E5 63 A4 40
: F2 77 03 7D 81 2D EB 33 A0 F4 A1 39 45 D8 98 C2
: 96 4F E3 42 E2 FE 1A 7F 9B 8E E7 EB 4A 7C 0F 9E
: 16 2B CE 33 57 6B 31 5E CE CB B6 40 68 37 BF 51
: F5
205 33: INTEGER
: 00 FF FF FF FF 00 00 00 00 FF FF FF FF FF FF FF
: FF BC E6 FA AD A7 17 9E 84 F3 B9 CA C2 FC 63 25
: 51
240 1: INTEGER 1
: }
: }
243 66: BIT STRING
: 04 41 A9 BC 93 6B 6D 1D D3 A1 DE D9 97 D7 DA 08
: F1 DF 99 0E 9B 50 F9 B5 8E 9E 4F D9 31 97 58 EA
: 34 4A D3 9F FB 79 C4 02 06 3A 99 EC BC 0C AC 8F
: DE 60 6D B6 76 4A CE 90 93 3F EE E5 F8 D6 59 37
: A2
: }
$ dumpasn1.exe key-2.der
0 243: SEQUENCE {
3 204: SEQUENCE {
6 7: OBJECT IDENTIFIER ecPublicKey (1 2 840 10045 2 1)
15 192: SEQUENCE {
18 1: INTEGER 1
21 44: SEQUENCE {
23 7: OBJECT IDENTIFIER prime-field (1 2 840 10045 1 1)
32 33: INTEGER
: 00 FF FF FF FF 00 00 00 01 00 00 00 00 00 00 00
: 00 00 00 00 00 FF FF FF FF FF FF FF FF FF FF FF
: FF
: }
67 68: SEQUENCE {
69 32: OCTET STRING
: FF FF FF FF 00 00 00 01 00 00 00 00 00 00 00 00
: 00 00 00 00 FF FF FF FF FF FF FF FF FF FF FF FC
103 32: OCTET STRING
: 5A C6 35 D8 AA 3A 93 E7 B3 EB BD 55 76 98 86 BC
: 65 1D 06 B0 CC 53 B0 F6 3B CE 3C 3E 27 D2 60 4B
: }
137 33: OCTET STRING
: 03 6B 17 D1 F2 E1 2C 42 47 F8 BC E6 E5 63 A4 40
: F2 77 03 7D 81 2D EB 33 A0 F4 A1 39 45 D8 98 C2
: 96
172 33: INTEGER
: 00 FF FF FF FF 00 00 00 00 FF FF FF FF FF FF FF
: FF BC E6 FA AD A7 17 9E 84 F3 B9 CA C2 FC 63 25
: 51
207 1: INTEGER 1
: }
: }
210 34: BIT STRING
: 02 41 A9 BC 93 6B 6D 1D D3 A1 DE D9 97 D7 DA 08
: F1 DF 99 0E 9B 50 F9 B5 8E 9E 4F D9 31 97 58 EA
: 34
: }
Notice the differences:
# key-1.der
243 66: BIT STRING
: 04 41 A9 BC 93 6B 6D 1D D3 A1 DE D9 97 D7 DA 08
: F1 DF 99 0E 9B 50 F9 B5 8E 9E 4F D9 31 97 58 EA
: 34 4A D3 9F FB 79 C4 02 06 3A 99 EC BC 0C AC 8F
: DE 60 6D B6 76 4A CE 90 93 3F EE E5 F8 D6 59 37
: A2
versus:
# key-2.der
210 34: BIT STRING
: 02 41 A9 BC 93 6B 6D 1D D3 A1 DE D9 97 D7 DA 08
: F1 DF 99 0E 9B 50 F9 B5 8E 9E 4F D9 31 97 58 EA
: 34
: }
The first has two Integers encoded, while the second has one Integer encoded. The common value is 41 A9 BC 93 ... 97 58 EA 34, and that's your pub x above. The missing string from key-2 is 4A D3 9F FB ... D6 59 37 A2, and that's the pub y above. For completeness, the values are encoded as either {X,Y} or {X}, and not ASN.1's encoding of integers.
The final difference is the first octet of the BIT STRING: 02 versus 04 (or 03). 04 indicates its an uncompressed point. From RFC 5480, Section 2.2:
The first octet of the OCTET STRING indicates whether the key is
compressed or uncompressed. The uncompressed form is indicated
by 0x04 and the compressed form is indicated by either 0x02 or
0x03 (see 2.3.3 in [SEC1]). The public key MUST be rejected if
any other value is included in the first octet.
And after looking at the standard and section 2.2, Crypto++ may have a bug: its writing a BIT STRING, but the standard is clearly discussing a OCTET STRING.
EDIT 1: It appears this is a Crypto++ bug under ANSI 9.62, too. Section 6.2, Syntax for Finite Field Elements and Elliptic Curve Points (p. 20)
A finite field element shall be represented by a value of type FieldElement:
FieldElement ::= OCTET STRING
The value of FieldElement shall be the octet string representation of a field
elementfollowing the conversion routine in Section 4.3.1.
An elliptic curve point shall be represented by a value of type ECPoint:
ECPoint ::= OCTET STRING
EDIT 2: Crypto++ is using the format specified by Certicom's SEC 1: Elliptic Curve Cryptography. Appendix C (page 77) states:
Finally, a specific field element is represented by the following type
FieldElement ::= OCTET STRING
whose value is the octet string obtained from the conversion routines
given in Section 2.3.5.
In the registry there is one (or more) key depending how many monitors you have HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\DISPLAY\DEL404C{Some Unique ID}\Device Parameters\EDID which is a REG_BINARY key. In my case this is :
00 ff ff ff ff ff ff 00 4c 2d 6f 03 39 31 59 4d
07 12 01 03 0e 29 1a 78 2a 80 c5 a6 57 49 9b 23
12 50 54 bf ef 80 95 00 95 0f 81 80 81 40 71 4f
01 01 01 01 01 01 9a 29 a0 d0 51 84 22 30 50 98
36 00 ac ff 10 00 00 1c 00 00 00 fd 00 38 4b 1e
51 0e 00 0a 20 20 20 20 20 20 00 00 00 fc 00 53
79 6e 63 4d 61 73 74 65 72 0a 20 20 00 00 00 ff
00 48 56 44 51 32 30 36 37 37 37 0a 20 20 00 ef
My question is how can I read only model of monitor ("SyncMaster" for example) and not all of the information using C or C++?
The format of EDID is described here: http://en.wikipedia.org/wiki/Extended_display_identification_data
What you're interested in here is the descriptor blocks of the EDID, which are found in the byte ranges 54-71, 72-89, 90-107, and 108-125. Here's those four blocks in your EDID:
#1: 9a29 a0d0 5184 2230 5098 3600 acff 1000 00
#2: 0000 00fd 0038 4b1e 510e 000a 2020 2020 20
#3: 0000 00fc 0053 796e 634d 6173 7465 720a 20
#4: 0000 00ff 0048 5644 5132 3036 3737 370a 00
You can identify the descriptor containing the monitor name because the first three bytes are all zero (so it isn't a detailed timing descriptor), and the fourth one byte FC (indicating the type). The fifth byte and beyond contain the name, which is here:
5379 6e63 4d61 7374 6572 0a20 SyncMaster..
So, in short: Check at offsets 54, 72, 90, and 108 for the sequence 00 00 00 FC; if you find a match, the monitor name is the next 12 bytes.