Techniques to Identify Root Cause of winspool!EnumPorts Incorrectly Returning 0 Ports - c++

Have a Windows environment where the "Ports" tab is empty on all network printers, but only on some win7 clients. All other printing functionality from these clients works to printers on those print servers. Due to the way the environment is set up I cannot easily attach a kernel debugger to these vSphere virtual machines.
Server 2008 R2 machines - OK
Clean Win7 machine - OK
Win7 machine + much 3rd party software, including 3rd party device drivers, multiple API hooking software, etc - BROKEN
All machines are joined to same domain, and logged in as same user.
Using Rohitab API Monitor on Explorer.exe I identified the winspool EnumPorts is succeeding, but returning 0 ports on the failing machine, but under same user account/same network and same parameters it returns 600+ ports on both working machines..
I wrote a test C++ program to test EnumPorts on its own. The Visual Studio 2015 project + binary is here
The key part of this program is:
pPortInfo = (PORT_INFO_2 *)malloc(cbNeeded);
bResult = EnumPorts(argv[1],
2,
(LPBYTE)pPortInfo,
cbNeeded,
&cbNeeded,
&cbReturned);
On a working machine output from pPortInfo is :
CALL: EnumPorts("\\PRINTSERVER01",(LPBYTE)pPortInfo,4096,&cbNeeded,&cbReturned)
RETURNED: false ERROR CODE: 122 ERROR MSG: The data area passed to a system call is too small.
CALL: EnumPorts("\\PRINTSERVER01",(LPBYTE)pPortInfo,121800,&cbNeeded,&cbReturned)
RETURNED: true ERROR CODE: 0 ERROR MSG: The operation completed successfully.
cbReturned = 629
PORT #0
Port Name: TS005
Monitor Name: Dynamic Print Monitor
Description: Inactive TS Port
Port Type: WRITE READ
PORT #1
Port Name: TS004
Monitor Name: Dynamic Print Monitor
Description: Inactive TS Port
Port Type: WRITE READ
etc...
On the broken machine:
CALL: EnumPorts("\\PRINTSERVER01",(LPBYTE)pPortInfo,4096,&cbNeeded,&cbReturned)
RETURNED: true ERROR CODE: 0 ERROR MSG: The operation completed successfully.
cbReturned = 0
What I have identified/tried so far
Using SysInternals AutoRuns to compare machines there are hundreds of differences between Windows Services Configuration/Drivers/Hooking DLLs on both machines. I want to find a technique to narrow this down to potential culprit without having to remove/disable all these items
I've tried disabling all hooking DLLs by renaming AppInit_DLLs key and confirming these DLLs no longer load in the process, but it remains broken
Captured WireShark trace - On the working machine network traffic is sent/received to the print server when EnumPorts is called. On the broken machine there is no network traffic sent to the print server at all. However on the broken machine you can still run commands like net view /all \ and map to printers/drives on the remote print server.
Used Microsoft Message Analyzer RPC trace, this logs the RPC calls but not sure how to interpret this data. None of the calls appear to be failing.
Looking deeper into winspool!EnumPorts it uses NdrClientCall2 to communicate with print server but these calls are not failing.
Tried disabling local firewall
Tried netsh winsock reset and restarting
Tested with antivirus disabled
Created a filter in Windows Event viewer to look at ALL windows event log entries during the time test is run and can't find any seemingly related errors
Running my test tool via WinDbg there are no first chance exceptions or other errors
Tried setting a breakpoint at the completion of EnumPort function then created a complete memory dmp with NotMyFault but couldn't find anything, and didn't really know what I was looking for in the first place
Used Process Monitor to look for missing registry keys/files and couldn't find any

Related

UDP Communication not working outside of Visual Studio Debugger

I have two separate c++ applications (running on Windows 10) that need to communicate with each other on multiple UDP ports. Both applications are running on one machine so I am just using the localhost IP on both ends. The first application is a robust control program that generates a lot of data that is sent out on these ports. The second application is a basic server that is designed to receive and process all this message traffic and occasionally send commands back to the first application.
When running in Visual Studio everything works as expected however running from the command line or a bash script my server app receives all the data as expected, but I cannot sending data to the control app.
I have tried tweaking the Windows Defender and symantic settings with no luck. Is it possible something else is blocking communication only in one direction? And why does it work in the Visual Studio debugger but not when running the executables from the command line?

Winsock client / server tutorial fails with desktop as client, but works with laptop as client

I have successfully built the client and server modules from the Getting Started with Winsock tutorial.
I have a desktop and a laptop both connected to my wireless router – both running Windows 10.
Running the client module on the laptop, I am able to successfully transmit data back-and-forth to the desktop (running the server module) using the desktop's IP address.
Running the client module on the desktop with the laptop's IP address as the command line argument, I get an "Unable to connect to server!" message after a ten second delay.
If I try to run both modules on the desktop in separate console windows using the "localhost" command line argument, the client console displays "Bytes sent: 14" and hangs waiting for a response from the server – however this works if I use either the desktop name or the desktop IP address in place of "localhost".
I am able to run both modules on the laptop using either "localhost", the laptop name, or the laptop IP address as arguments.
I have gone through the same motions with port 27015 forwarded on the router and incoming and outgoing firewall exceptions added to both the desktop and the laptop – there is no difference either way.
Any assistance would be greatly appreciated as I cannot figure out why this works in one direction but not the other.
Thank you for the suggestions Karsten and Andriy. I first tried getting the two computers to ping each other and neither was successful. After researching online, I was able to get them to ping after turning on "echo requests" in the firewall settings, but my original problem persisted. I then tried turning off both firewalls and I was able to get my server and client programs to work both ways. That wasn't a great long-term solution, so I tried selectively disabling the firewalls and realized it was an issue on the laptop's end. I noticed that my "server.exe" program was in the allowed apps list twice – one instance granting private access and one granting public access – but only one instance was active. I deleted both and added "server.exe" again with both public and private access boxes checked, which solved my issue.

ODBC SQL Driver Locked to Single User On Windows 2008 x64

I have two different C++ ISAPI DLLs that have been setup and configured for IIS 7 to use as web applications. They are setup using different application pools that are running under different domain user accounts. The problem that I am having is that, once one of the applications connects to SQL, the other application cannot. There are no differences in code between how the applications attempt to establish the connection – they both call the same static library method with identical parameters. The only way to get the second application to connect to SQL is to set its Application Pool user to the same domain user as the first application or vice versa (which rules out one user having insufficient permissions). I have double checked that ODBC connection pooling is turned on for the SQL Driver and I've monitored the ODBC Data Source Administrator Trace logs. If I monitor the trace log while both application pools are running under different users, SQLAllocHandle shows up successfully for both enter and exit in the Trace, but SQLDriverConnect will only succeed for both enter and exit for the first application that calls it. When the second application calls SQLDriverConnect with the exact same parameters, the trace only shows enter and debug logs also indicate that this function call never returns. Packet traces indicate that the second application never even attempts to connect to the SQL server. However, the first application will connect just fine and running a profiler in SQL will show proper execution of SQL statements. These applications ran just fine using the same multi-user configuration on Windows 2003 x86 under IIS 6 and older versions of the ODBC drivers.
I am hoping that there is a simple ODBC setting that I've missed, but my searches come up empty. Any help anyone can provide would be greatly appreciated, thanks in advance.
After over a week of troubleshooting and talking with Microsoft, we've finally found a solution. From what Microsoft told us, they changed the way that processes can interact with each other between 2003 and 2008 and using our static linked library like we were was causing a locking issue when it was running under different users. When we introduced a new thread before calling the ODBC connector, everything works fine.

Multiple NetApi calls failing inconsistently

We have a multi-threaded process which makes multiple calls to multiple target machines from a source machine using NetApi’s eg. NetServerGetInfo, LSAOpenPolicy, NetShareEnum, NetWKstaGetInfo, NetWKstaUserEnum etc… We make quite significant number of calls and have observed that over a period of time these calls fail. For example NetServerGetInfo starts returning error 53 after a while. This issue persist until we restart Workstation service or the machine. Accessing the target shares directly also does not work after such error is returned by our process.
The source machine from where we are making calls is a Win 2k8 R2 and the target machines are 2k3 servers.
We are suspecting some kind of issue with NetApi calls or some kind of handle leak.
Has anyone faced similar issues while using these APIs and managed to figure out a solution?
I found few references online for similar issues:
http://social.technet.microsoft.com/Forums/en-US/windowsserver2008r2networking/thread/9f93508c-71fa-4807-b41a-8f558563afe3/
Snippet from above link:
Experiencing the exact same issue as stated about except we have 2 Windows Server 2008 R2's acting as Terminal Servers connecting to Server 2003 Shares. Rebooting the terminal servers seems to resolve the problem for about 2-4 days and then re-appears. The XP/Vista/Win7 workstations on the network has no problem accessing the shares on the 2003 Server, only the 2008 R2 servers.
Connecting the the 2003 Shares using the FQDN or IP address works, but using \servername returns network path not found. Setting up WINS on the network did not resolve this, or adding a static entry in the hosts file to the server.
There is no firewall software installed on the servers and we don't use Symantec products on the network (No Symantec Endpoint security).
Viewing of the eventlog also turned up the Event ID: 1006, could not validate DNS server, even though name resolution appears to be functioning without a problem.
http://support.microsoft.com/kb/816621
http://technet.microsoft.com/en-us/library/dd296694%28WS.10%29.aspx
https://serverfault.com/questions/205043/windows-share-the-specified-network-name-is-no-longer-available

Jrun ColdFusion service intermittently fails to start

We occasionaly have a problem where we attempt to start the Jrun service and it fails with the following two errors:
error JRun Naming Service unable to start on port 2902
java.net.BindException: Port in use by another service or process: 2902
info No JDBC data sources have been configured for this server (see jrun-resources.xml)
error java.net.BindException: Port in use by another service or process: 8300
We then have to reboot the machine and Jrun comes up with no problem. This is very intermittent - happens perhaps one out of every 10 times we restart Jrun services.
I saw another reference on StackOverflow that if Windows Services take longer than 30 seconds to restart Windows shuts down the startup proccess. Perhaps that is the issue here? The logs indeed indicate that these errors are thrown about 37+ seconds after the restart command is issued.
We are on a 64bit platform on WinServer 2008.
Thanks!
We've been experiencing a similar problem on some of our servers. Unfortunately, netstat never indicated any sort of actual port conflict for us. My suspicion is that it's related to our recent deployment of a ColdFusion "cumulative hotfix" to our servers. We use the multi-server edition of CF 8.0.1 enterprise with a large number of instances on each machine -- each with its own JVM and its own distinct set of ports. Each CF instance is attached to its own IIS website and runs as its own Windows Service.
Within the past few weeks, we started getting similar "port in use" exceptions on startup, on our 32-bit machines as well as our 64-bit machines, all of which are running Windows Server 2003. I found several possible culprits and tried the following:
In jrun-jms.xml for each CF instance, there's an entry for the RMI transport layer that reads <port>0</port> -- which, according to the JRun documentation, means "choose a random port." I made that non-random and distinct per instance (in the 2600-2650 range) and restarted each instance. Things improved temporarily, perhaps coincidentally.
In the same file, under the entry for the TCPIP transport later, every instance defaulted to <port>2522</port> -- so I changed those to distinct ports per instance in the 2500-2550 range and restarted each instance. That didn't seem to help at all.
I tried researching whether ports in the 2500-3000 range might be used for any other purpose, and I couldn't find anything obvious, and besides, netstat wasn't telling me that any of my choices were in use.
I found something online about Windows designating ports from 1024 to 5000 as the "dynamic port" range, so I added 10000 to the port numbers I had set in jrun-jms.xml and restarted each instance again. Still didn't help.
I tried changing the port in jndi.properties, also by adding 10000 to the port numbers. Unfortunately this meant wiping out all my wsconfig connections to IIS and creating them again from scratch. I had to edit wsconfig_jvm.config as well, adding -DWSConfig.PortScanStartPort=12900 to java.args, so it could detect my CF instances. (By default it only scans ports 2900-3000. See bpurcell.org for details. It's an old post but still relevant.) So far so good!
My best guess is that Adobe (or MS Windows) changed the way some of its code grabs "random" ports. But all I know for sure so far is that the steps outlined above appear to have fixed the problem.
Have you verified that the services are in fact stopping? Task manager should show no instances of jrun.exe. You can also check to see what is bound to that port by opening a command window and running
netstat -a -b
This will list all your open ports, plus what program is using them. You can also use
netstat -a -o
Which does the same thing as the above, but will list the process id instead of the program name. You can then cross-reference those with task manager. You'll need to enable showing the PIDs in task manager by going to View->Select Columns and making sure PID is checked. My guess would be that the jrun processes are not shutting down in a timely fashion.