Visual Studio Team Services build fails because CoreCompile does not reference NuGet assembly - visual-studio-2017

Question

I am running into an weird issue where my build works perfectly on local machine but almost always fails on Visual Studio Team Services build agent. I say almost always the same code sometimes builds fine on VSTS build agent.
The error is that for a project in my solution, the compiler could not find System.Web.Http.dll, which is referenced in Microsoft.AspNet.WebApi.Core NuGet package. I've checked the reference and it is there. I've removed the NuGet and re-added it to fix any potential NuGet hiccup.
The exact error is this:
Error CS0234: The type or namespace name 'Http' does not exist in the namespace 'System.Web' (are you missing an assembly reference?)
The reference is in the csproj file:
<Reference Include="System.Web.Http, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<HintPath>..\..\Microsoft.AspNet.Cors.5.2.3\lib\net45\System.Web.Cors.dll</HintPath>
</Reference>
The NuGet restore task from VSTS suggested that all NuGet packages were installed, but the build still failed.
I then did a diff between the command lines of the successful build and failed build, both from Visual Studio Team Services build agent and found that the failed build CoreCompile task was missing some assembly references.
CoreCompile gets the assemblies to reference from ResolveAssemblyReferences. But ResolveAssemblyReferences claims that it could not find System.Web.Http.dll.
For SearchPath "{HintPathFromItem}".
Considered "....\Microsoft.AspNet.WebApi.Core.5.2.2\lib\net45\System.Web.Http.dll", but it didn't exist.
Is there any known issue with NuGet restore from VSO? We are using NuGet 4.0.
UPDATE:
Visual Studio Team Services support came back to me. I had my own nuget.config which had the following 2 lines. This interfered with the global NuGet cache on the build agents. Once I removed these 2 lines everything went smoothly as expected.
<add key="globalPackagesFolder" value="..\nupkgs" />
<add key="PackageSaveMode" value="nuspec" />

Answer 1

It looks like, at some point, this reference was added manually, and the relative paths match up on your local PC.
In every valid nuget reference I've seen, those relative paths are prefixed with "packages". Try deleting the nuget package, and delete this reference, then reinstall it.

Answer 2

Here is the answer to the issue I ran into.
Visual Studio Team Services support came back to me. I had my own nuget.config which had the following 2 lines. This interfered with the global NuGet cache on build agents. Once I removed these 2 lines everything went smoothly as expected.
<add key="globalPackagesFolder" value="..\nupkgs" />
<add key="PackageSaveMode" value="nuspec" />