I'm writing a transparent intercepting HTTPS capable proxy using boost::asio + openSSL. I have a default server context where I specify that the server is a TLSv1.2 server, when a client connects, I extract the host from the hello and use SSL_set_SSL_CTX to set the context (which either already exists or I've just created it after spoofing the upstream cert) and initiate the server (downstream) read/write volley as well as the upstream.
This was working before I started storing and sharing contexts. On each new incoming connection, I was creating a new client socket and context, loading ca-bundle as verify file, then creating a new server context, getting the spoofed certificate. It was functioning, but I started developing issues where EC_KEY objects were being double freed and such. I learned from another question of mine that I was going about this the wrong way and began refactoring to recycle and share CTX objects. To be specific, I'm using a single client CTX shared across the board that loads, at program startup, the CA-Bundle for verification.
However, since this refactor, I'm getting this on both the client and the server:
decryption failed or bad record mac
..mixed with a bajillion "short read"s. If I try to force everything TLSv1.2, I get
block cipher pad is wrong
Those errors are given to me after a read/write has failed and I call async_shutdown on either upstream or downstream sockets, which in the callback, error is set (so the shutdown failed).
I've scoured the interwebs finding jira posts from places like apache httpd and nginx where this error was fixed in different ways (resizing read buffers to be larger, openSSL patches, forcing SSLv3, so on and so forth).
I thought there might be an issue with multithreading (my io-service uses a thread pool) but I can see in the code that boost do_init sets locking mechanics for openSSL and all of my IO are wrapped into a single strand.
I'm at a total loss and am wondering if anyone can shed light on what might be happening. I realize I've posted no code, that's because I've got hundreds and hundreds of lines of it and don't want to turn people off with a huge code dump. I realize however this is a rather complicated program and thus a complicated issue so please ask and I'll provide whatever I can.
Edit
I guess I should mention for completeness that I'm getting these errors on both openssl 1.0.2 and 1.0.2a, Win 8.1 x64 and I'm intercepting and routing the http/https traffic through my proxy with with WinDivert.
Edit 2
Reduced entire program to 1 thread, same effect. Created new client CTX for each client connection, same issue. Tried disabling AES-NI, issue persists. Tried different computer, same effect. Recompiled openssl from source (was using precompiled binaries), issue persists. Tried setting additional OP_ workaround flags described in current docs related to downgrade detection, padding bugs, so on and so forth, issue persist. I think I'll just start randomly mashing the keyboard and compile button soon.
I was going to just delete this question, but I decided to answer it in light of the fact that nowhere on the net (that I could find) actually pointed to a correct solution to this problem. I've read every single report about this error that one could find and every single one of those reports, the people "solved" or "reduced" this error in a different way. Every single one of them, a different solution. This is what helped make this issue so difficult to reason out, because everyone everywhere has a different underlying causal explanation.
It's complicated, ready? This error will present itself if you cancel/abort a pending async SSL operation. Mind->boom(). It'll be even more confusing if you do what the docs say and use async_shutdown to do so, because even the call back to async_shutdown will fail (error code is set) and your error message will randomly be something stupid like "decryption failed or bad record mac" or "block cipher pad is wrong" or "SSLv3 alert!" so on and so forth. When seeing errors like this, ignore the errors and analyze the control flow of your IO ops, somewhere you're either prematurely ending them or getting them out of order.
In my case, the premature end was (sort of) intentional, since during this stupid heavy refactor I decided to change things outside the scope of the problem, like my HTTPHeader parser, which I bugged out and ended up cause it to fail nearly 100% and thus aborting the connections. :) The error strings were masking the real cause by telling me encryption failed for some reason or another. Dumb mistake I know, but I take comfort in being the first one (apparently) to recognize it. :)
Open a powershell and type this
(Invoke-WebRequest -Uri status.dev.azure.com).StatusDescription
https://devblogs.microsoft.com/devops/deprecating-weak-cryptographic-standards-tls-1-0-and-1-1-in-azure-devops-services/
Related
Good morning,
I am facing a crash in my application. When the user tries to start it, he waits like a minute and then a std::exception is raised. Really I could not reproduce the bug by myself, but it seems quite a common problem.
The only thing I could track is the following line in syslog:
BUG in libdispatch client: kevent[EVFILT_MACHPORT] monitored resource vanished before the source cancel handler was invoked
Then, I start to google it and I can not find much more...I can only "suppose" that is some problem with GCD (that I do not use afaik, or at least not directly...). What I saw in Internet is that it is related with MacOSX Sierra. But the majority of forum have no answer, just a lot of tries without a unique result. Maybe the only web page that seems a bit clear about a workaround (that I have not tested, and anyway I do not want to use) is this.
So...:
someone has clear what can cause the exception in libdispatch?
someone can give me some good link, official documentation or something?
Is true that can be a bug in Sierra without updates?
Could it be related with the installer of an application?
Someone knows a way to reproduce this exception with a test program?
This libdispatch log message is not fatal, and is almost certainly not related to your crash, which sounds like an abort due to an uncaught C++ exception (without a crashreport/backtrace it is difficult to say anything more). libdispatch does not itself generate any C++ exceptions FWIW.
As to the meaning of that specific log message, it relates to the following section in the dispatch_source_create(3) manpage:
CANCELLATION:
Important: a cancellation handler is required for file descriptor and mach port based sources in order
to safely close the descriptor or destroy the port. Closing the descriptor or port before the cancellation handler has run may result in a race condition: if a new descriptor is allocated with the same
value as the recently closed descriptor while the source's event handler is still running, the event
handler may read/write data to the wrong descriptor.
If you see the EVFILT_MACHPORT "vanished" log message, somebody in your process has violated that API contract and deallocated a machport while a dispatch source was still monitoring it (causing the kernel to generate an EV_VANISHED kevent), see the source code.
This touches on some already-answered questions, so feel free to duplicate away, but chances are I've already read them and am not satisfied.
There are 2 drivers on my system (located in C:\Windows\System32\drivers) called pefndis.sys and wfpcapture.sys. I am 100% sure pefndis.sys is a kernel driver and 99.9% sure wfpcature.sys is as well. These are 3rd party drivers installed by Mircosoft's Message Analyzer. I have discovered pefndis.sys is used to capture data on the wire and wfpcapture.sys is used to capture data above the network layer (ie, this will capture loopback traffic). I have no documentation, header files, etc, for these drivers as there was no intention of Microsoft for these drivers to be used for custom solutions as I would like to do. It just so happens I've identified wfpcapture.sys as performing the exact tasks I want, and I'd love to tap into what it can do; this seems so much more reasonable than spending the time and pain of implementing my own driver. However, my efforts have failed.
This is what I've done: I have some simple c++ code here:
void Provider::InitDriver()
{
HANDLE wfpHandle = NULL;
DWORD lastError = 0;
LPCTSTR wfpName = L"\\\\.\\wfpcapture";
LPCTSTR pefName = L"\\\\.\\pefndis";
wfpHandle = CreateFile(
wfpName,
GENERIC_READ,
FILE_SHARE_READ,
NULL,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
NULL);
lastError = GetLastError();
CloseHandle(wfpHandle);
}
When I run CreateFile with wfpName, I get an invalid handle and lastError==2 meaning the file cannot be found. When I run CreateFile with pefName, I get a valid handle and lastError==0. Upon further investigation, most of my randomly-picked .sys files from the drivers folder produce invalid handles with error codes of 2. Occasionally I'd get an error code of 5 (Access Denied, which also seems odd since I'm running everything as administrator). Does anyone have an explanation why I cannot get a handle to wfpcapture.sys? I brought up the pefndis.sys driver because it was installed by the same program as wfpcapture.sys, and I can clearly get a handle to that, so all my strings are formatted correctly, and both files are in the same directory. I came across this post yesterday which told me IoCreateSymbolicLink can be used in the driver code to give the driver another alias. If I'm using the wrong alias, does that explain why so many .sys files return FILE_NOT_FOUND errors when I try to get handles to them?
I've tried to do some more research on the file using IL DASM (no luck, this is native code), DUMPBIN, WinObj, and DependencyWalker.
DUMPBIN /EXPORTS [...]wfpcapture.sys returns no exports. I find this extremely odd. These answers suggest .DLLs without exports are simply resources or the code is obfuscated. I am almost certain wfpcapture.sys does not just contain resources. Is obfuscation the most reasonable explanation.. any other ideas why it doesn't have any exports?
I could not find wfpcapture in WinObj anywhere. I located pefndis in Device\. Where is wfpcapture? It doesn't actually talk to a device, so that makes sense, but it is still a driver, correct? Do drivers need to register with Windows in some way before CreateFile can find them?
DependencyWalker verified what DUMPBIN told me, I think .. no exports. I have no idea how Message Analyzer (or anything else down its dependency stack) is actually talking to it.
Just a bit more background for a complete picture... wfpcapture.sys is an ETW Provider that taps into Microsoft's WFP architecture (used for firewall and IDS applications) to sniff packets above the network layer. I want code that "activates" wfpcapture.sys and then sits back and collects the events (packet captures) that wfpcapture publishes. It's this activation part that I can't figure out. If I setup Message Analyzer to start capturing localhost traffic, then turn on the part of my code that captures the events (using StartTrace(...) and EnableTraceEx2(...)), that works just fine. I am just dying to know how Message Analyzer is talking to wfpcapture.sys and what it's saying in order to get it to cooperate and start publishing events. Another fun fact: When I start a trace in Message Analyzer and do sc query wfpcapture, it tells me the service (here it is identified as a kernel driver) is running. When I stop the trace, the query tells me the service is stopped. If I manually sc start wfpcapture and verify the service is running,, and then run my event capturing code, I get nothing. This tells me Message Analyzer must be sending something to wfpcapture.sys to get it activated and publishing. My plan that spawned this whole thing was to get a handle to driver and start sending it control codes via DeviceIoControl to glean some knowledge on how it worked. I have also seen some very strong evidence that Message Analyzer is passing filter masks to the driver.
Am I completely wasting my time here? That driver is not meant for my consumption, and poking and prodding it to learn about it may be a long shot, but I'm certain it does exactly what I need and I've never written a driver in my life; trying to do that seems foolish when this is sitting right here. Message Analyzer is free, I'm not trying to steal software. Could there possibly be some DRM associated with the driver that's boxing me out? I'd love to hear the thoughts of anyone out there who has Windows driver experience.
Ok, lot of questions there, hope this doesn't get flagged as too broad.
I am running an ssh tunnel from an application using a QProcess:
QProcess* process = new QProcess();
process->start("ssh", QStringList()<<"-L"<<"27017:localhost:27017"<<"example.com");
So far it works great, the only problem being that there is no way for me to see when the port has actually been created.
When I run the command on a shell, it takes about 10 seconds to connect to the remote host after which the forwarded port is ready for usage. How do I detect it from my application?
EDIT:
As suggested by vahancho, I used the fact that post-connection there is some output on the terminal that can be used to detect that the connection has succeeded. However, there is a line which is run instantly after launch Pseudo-terminal will not be allocated because stdin is not a terminal, which probably would give a false alarm. The correct output is available in the second signal, emitted a bit later (which is a true indicator of the port having being opened). To get rid of the first message, I am now running ssh using ssh -t -t to force an stdin allocation.
So, the only question left is, can anyone help me without any concerns in this approach?
So, the only question left is, can anyone help me without any concerns in this approach?
This is not a stable and robust solution, unfortunately. It is similarly a broken concept to handling git outputs rather than using an actual library. The main problem is that these softwares do not have any guarantee for output compatibility, rightfully.
Just imagine that what happens if they have an unclear text, a typo, et all, unnoticed. They inherently need to fix the output respectively, and all the applications relying on the output would abruptly break.
This is also the reason behind working on dedicated libraries giving access to the functionality for reuse rather than working with the user facing output directly. In case of git, this means the libgit2 library, for instance.
Qt does not have an ssh mechanism in place by default like you can have such libraries in python, e.g. paramiko.
I would suggest to establish a way in your code by using libssh or libssh2 as you also noted yourself in the comment. I can understand the inconvenience that is not a truly Qt'ish way as of now, but at this point Qt cannot provide anything more robust without third-party.
That being said, it would be nice to see a similar add-on library in the Qt Project for the future, but this may not be happen any soon. If you write your software with proper design in mind, you will be able to switch to such a library withour major issues once someone stands up to maintain such an additional library to Qt or elsewhere.
I had the same problem, but in my case ssh do not output anything - so I couldn't just wait for output. I'm also using ssh to setupt tunnel, so I used QTcpSocket:
program = "ssh";
arguments << m_host << "-N" << "-L" << QString("3306:%1:3306").arg(m_host);
connect(tunnelProcess, &QProcess::started, this, &Database::waitForTunnel);
tunnelProcess->start(program, arguments);
waitForTunnel() slot:
QTcpSocket sock;
sock.connectToHost("127.0.0.1", 3306);
if(sock.waitForConnected(100000))
{
sock.disconnectFromHost();
openDatabaseConnection();
}
else
qDebug() << "timeout";
I hope this will help future people finding this question ;)
I am trying to figure out a "Failed to resume in time" problem. In one of our testers devices (which is an iPhone 4S with the latest OS) it happens very frequently, whereas in my own device it doesn't seem to happen at all.
Anyway, I got a few crashlogs. I am unable to trace the root of the cause though. I understand that the issue might be
1.When a process is holding up the main thread for too long.
2.When there is a memory issue.
I don't think the memory is much of an issue since it seems to happen when the user leaves the main menu and comes back. Nothing much is happening in the main menu so it probably is a task that runs too long.
Here is an excerpt from the crash log:
Can somebody help me or guide me on who I can trace the cause of the issue? Is there anyway to turn off the watchdog timer(probably not huh?) Also, what does highlighted thread refer to?
I have already checked my applicationDidBecomeActive & applicationWillEnterForeground to make sure there is nothing going on there.
To my knowledge there are no synchronous calls being made at this point. Does Reachability use synchronous calls to check for internet? How can I check for that?
I am not making any large data transfers upon resume.
I notice that GameCenter automatically logs in or check for log in upon resuming your app. Is there anyway to prevent this? Could this possibly cause a time out issue?
I tried doing a time profile, but I am not able to understand how to use it to analyze. If you can provide a good resource for that, that would be amazing.
Thanks!!!
You're currently in "trying to find the issue mode". You should switch to "try to find out how much of an issue this really is" mode.
So go find another 4S (actually as many as you can) to rule out that it's a device-specific issue. If it happens on all 4S it should be easier to pinpoint. If not, have someone else look over it, discuss possible causes. The peer programming approach often helps when you're stuck in a dead-end situation.
If the issue is only on that one device, you might want to check if it's broken (or "jailbroken") or might simply need a hard reboot (hold power and home for 10+ seconds).
If it only happens on some devices but not all, try to find what they have in common. This could be language/locale, or dictation, practically any kind of setting the user might have changed. If necessary, write a logger that logs as many settings as possible to your (web) server so you can compare settings one-by-one and quickly discard those that aren't in synch.
If only very few devices are affected, you could also ignore the issue and hope that additional crash logs from users will reveal the key to the issue.
Finally, there's always the option to disable suspend on terminate and instead terminate the app when the home button is pressed (as it was pre iOS 4). Unless of course the app has to run in background.
Well, three questions in less then 24 hours.But I suppose the lack of any helpful documentation (that i can find anyway) gives me no choice.
now i know all about the security stuff adobes got going, the policy file deal and sandbox limitations that have to be juggled around. ive done the suggested options changes, allowing network access in my AS3 project. I got my server ready to spit the policy file out instantly upon connection; but the problem is flash/AS3 whatever you wanna call it simply doesn't see my server (or the other way around)
AS3 delays for a few seconds, like the documentation say it will if its struggling to make a connection/find the policy file, but it never makes a connection period, tries for a while and then gives up and spits me a access error (because it didn't find the policy file it assumes its not allowed on the network so it get mad at me for trying... lil stubborn buggers)
Flash never gets what its looking for, and my server never detects any connections (failed or succeeded, nothing) I know my server is good because i've tested it with a test client i wrote in C++ and they talk just like best friends.
So I'm pretty much and a loss for ideas now, I thought about re-creating the winsock classes in AS3, but i don't even know how the connect() function fro winsock actually works; i got the declaration, no definition for it that i can find.
I am not sure what you would really need from my source code, but there's over a thousand lines in the client alone already so posting it all isn't really an option. ill give here what i think is relevant at least.
function hwndEnterFrame(e:Event):void
{
//trace("Frame Entered");
if (firstframe)
{
trace("try to reconcile policy file...");
//Security.loadPolicyFile("192.168.1.2:843");
TSocket = new Socket("192.168.1.2",843);//must call flush() to send data***
firstframe = false;
}
The Security.loadpolicyfile doesnt seem to do anything as far ive noticed, i get the same msgs with or without it, the code seems to be looking for the policy file at the location specified in the socket declaration. Everything compiles fine, it just ain't doing what i want it do :P everyone else seems to be writing their servers in Perl or Python i haven't found much of an abundance of C++ material, although from what i understand TCP is a universal connection across any platform that supports it (can cross-language without any funny-business)
meh, that what i got for now, if you want other parts of the code (client or sever side, whatever you want) ill stick it up here quick-like.
EDIT:
I found a swf online that i downloaded, it connected fine to the example server that came with it (but that server was written in c sparp so its not much use to me) but it wont connect to my server. So a known good flash client WONT connect to my C++ server, but a C++ client WILL connect to my C++ server. So its clearly somthing on the server side, becuase it dosnt hear any connection whatsoever from the flash apps (mine or the expamle one)
Cheers;
-Tyler
I finally figured it out. I needed to AllowDomian(IPofserver). I thought I had already tried that but apparently not.