Sharepoint Office 365 OAuth Service authentication to list.asmx - web-services

Question

I´m trying to access Sharepoint list data in Office 365 from an external website. I registered my app in Azure Active Directory and I've done all the process of creating and trusting a certifacte and getting the access token.
Add-Type -Path ".\Microsoft.IdentityModel.Clients.ActiveDirectory.dll"
$authenticationContext = New-Object Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext -ArgumentList "https://login.microsoftonline.com/{myTenantId}/", $false
$cer = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2
$cer.Import(".\WithPrivateKey.pfx", "privateKey", [System.Security.Cryptography.X509Certificates.X509KeyStorageFlags]::MachineKeySet)
$clientAssertion = New-Object Microsoft.IdentityModel.Clients.ActiveDirectory.ClientAssertionCertificate -ArgumentList "{myClientId}", $cer
$authenticationResult = $authenticationContext.AcquireToken("https://{tenantName}.sharepoint.com", $clientAssertion)
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization", "Bearer " + $authenticationResult.AccessToken)
I can successfully call to Sharepoint REST Api by presenting the access token in request headers.
$response = Invoke-RestMethod -Uri https://{myTenantName}.sharepoint.com/sites/devSite/_vti_bin/ListData.svc/TestList -Method Get -Headers $headers
However I can't do the same to access asmx endpoints, such as Lists.asmx, whenever I try to call any method on those services I get 401 UNAUTHORIZED
$body = '<?xml version="1.0" encoding="utf-8"?>
<soap12:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap12="http://www.w3.org/2003/05/soap-envelope">
<soap12:Body>
<GetList xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<listName>TestList</listName>
</GetList>
</soap12:Body>
</soap12:Envelope>'
$response = Invoke-WebRequest -Uri https://{myTenantName}.sharepoint.com/sites/site/_vti_bin/Lists.asmx -Method Post -ContentType 'application/soap+xml' -Headers $headers -Body $body

Answer 1

After sometime digging into the .net Sharepoint client sdk, I found how SharepointOnlineCredential class does it, thus allowing access to Sharepoint SOAP Services.
So, as said already by Fei Xue, the Azure AD token is not valid to access Sharepoint SOAP services (althoug the token is valid to allows access to REST services...). To access Sharepoint Online services, you will need to use some sort of claims authentication, either by requesting user consent or by directly using a known user and password.
As we can't use the .net SDK in our php app, we have investigated how the SDK creates the requests to get authenticated when using user credentials directly:
First send your authentication credentials as a SAML-WSSecurity POST request to the authentication endpoint https://login.microsoftonline.com/rst2.srf:
POST https://login.microsoftonline.com/rst2.srf
Content-Type: application/soap+xml; charset=utf-8
Content-Length: [calculate]
Host: login.microsoftonline.com
<?xml version="1.0" encoding="UTF-8"?>
<S:Envelope xmlns:S="http://www.w3.org/2003/05/soap-envelope" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:wsa="http://www.w3.org/2005/08/addressing" xmlns:wst="http://schemas.xmlsoap.org/ws/2005/02/trust">
<S:Header>
<wsa:Action S:mustUnderstand="1">http://schemas.xmlsoap.org/ws/2005/02/trust/RST/Issue</wsa:Action>
<wsa:To S:mustUnderstand="1">https://login.microsoftonline.com/rst2.srf</wsa:To>
<ps:AuthInfo xmlns:ps="http://schemas.microsoft.com/LiveID/SoapServices/v1" Id="PPAuthInfo">
<ps:BinaryVersion>5</ps:BinaryVersion>
<ps:HostingApp>Managed IDCRL</ps:HostingApp>
</ps:AuthInfo>
<wsse:Security>
<wsse:UsernameToken wsu:Id="user">
<wsse:Username>[user]</wsse:Username>
<wsse:Password>[password]</wsse:Password>
</wsse:UsernameToken>
</wsse:Security>
</S:Header>
<S:Body>
<wst:RequestSecurityToken xmlns:wst="http://schemas.xmlsoap.org/ws/2005/02/trust" Id="RST0">
<wst:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</wst:RequestType>
<wsp:AppliesTo>
<wsa:EndpointReference>
<wsa:Address>sharepoint.com</wsa:Address>
</wsa:EndpointReference>
</wsp:AppliesTo>
<wsp:PolicyReference URI="MBI"></wsp:PolicyReference>
</wst:RequestSecurityToken>
</S:Body>
</S:Envelope>
The response, in xml format, will come with the security token in it:
....
<wst:RequestedSecurityToken>
<wsse:BinarySecurityToken Id="Compact0">t=EwA4A06hBwAUNfDkMme61kIdXqvj9tWnUbHtXWEAAREB5clgLb8J/VvxRFIKLUnd9SRyoBHmTHFk0viit2FMlGXak5NJKJhicT8MiZmgA2HoTrJM1EgXCNUpmWqrX1LQRNfs0PHEV4XncjI9lnphsSTiFSCDjmdCKtW4TmV8n18xJHvBtDUWdvCT2lBti8/gf1oiqD5lQBPtxr+d4OwNtJHADEKpC/YIoatcKqgxI480tlWOZHpEL1wifo5EDMDRRc985ObMCZ31fPdSpA7WIbDzlZYX9ou6Cq7EybrIHsAcr5cPIJ8y0FRUacma9+dMxqr/lILAIyAYz/GdTNffa2Q3zJOSWW5RcnigtCApHgf83HjW8DqC6NgTrXs6rpUDZgAACC/4JZlSLB3JCALIntkKmNtRl2JLvwUljkXrP5jg5ipK/J/fGF3oc46aP/YT3VnrrD6TCV5ZECki5ycYZ6JR5RDK6OSqI9c5FDfFS/YmSCcdcaJ1cG2Ug3Oz3w14mznYGwmvrgyGvw35aoyjnKZALw8OQ2Ddi97gbe03L4rrM7CxTGwEPgoKCK7USkwxZT+myLJASVhd29+eNsTqd7wuphhLrzbgYZ+7swlJb3oIJw/2T7YvJ4fTPByaLxGaBt7iry74aSh/RnXdH3snOQnsr63bXqqoDJGcj7A3aIpElw2LlW2/PGh84zke3corp2q/jg7PEKCnV8PYN2xiwSfqY9vNCny14xhHEPsK8FWDBOPDpgeC18qz+FpTN0rGUMXl20bxJGxGqnQ+s8k0Gu9yTxoZKWPSeVihJk6qUQo6KJb/NE/QRco94QDUjMYi+gccGN0D1ouUe+O0fb0InXeM+98qfXJLQAjoUtgS8rRJUAqFk5XVwebGbx0ICRv3Q/wiJ7T5yUryMBTtwbaGf/07QuGTv9CW2UTsV9zT1nMSRDfUpelZrgZt6huLnDRLC8yVHfXjndMwONdymxWcD9sLb8EcNmTUFHDfBrv8XFb50PNJAV4qvK8CgVmWu2C2GWXoZfYkaR6o6jpliQdT5NcNnb9wNy36OqDnIWl0ZNM1SOzVX0yQOLeaf8+1bslaafyYMAbhcAI=&p=
</wsse:BinarySecurityToken>
</wst:RequestedSecurityToken>
...
Then using the security token extracted from the previous response, you need to send a GET request to your tenant credentials endpoint: https://yourtenantname.sharepoint.com/_vti_bin/idcrl.svc/
You will need to send an Authorization header with the format: BPOSIDCRL + space + token. Like:
GET https://yourtenantname/_vti_bin/idcrl.svc/
Host: yourtenant.sharepoint.com
Authorization: BPOSIDCRL t=EwA4A06hBwAUNfDkMme61kIdXqvj9tWnUbHtXWEAAREB5clgLb8J/VvxRFIKLUnd9SRyoBHmTHFk0viit2FMlGXak5NJKJhicT8MiZmgA2HoTrJM1EgXCNUpmWqrX1LQRNfs0PHEV4XncjI9lnphsSTiFSCDjmdCKtW4TmV8n18xJHvBtDUWdvCT2lBti8
The response to this request will set a cookie that we need to capture and use in our next requests to the soap services:
Set-Cookie: SPOIDCRL=77u/PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz48U1A+VHJ1ZSwwaC5mfG1lbWJlcnNoaXB8MTAwMzdmZmU5NTc2YzdlZUBsaXZlLmNvbSwwIy5mfG1lbWJlcnNoaXB8Y2xvdWRAYXNlbWJsaWFkZXYub25taWNyb3NvZnQuY29tLDEzMTAxMjE4ODQ0Mzc0NDcxMjsxMzA5NTg5NTQ0NTAwMDAwMDAsRmFsc2UsUCtRcmlFSFRkRnZCNkJEREZFek1mK3RVaGlzZTZtdnl1R0N4aXpjaWpyRUdxZk1BN1RpdTJNdGN0VE42TVNjdi9Cbjd2OXRxS2VPaTBzWTdlTnRqNkFESmRubFM2S0ttVjdoeHRWNjdtY3FlQVQzYWJGeDFEVFd5dEJsOWZ3MDJkZ2JTakV3eUM3WTRIWXg0ek5UYUtvUTZacGFXR0NjZ0svZEtEbloya3ozdGFBblVPM1gvUkxBeUorYkZac2RGclBCRGF4aDNTMGpBTml2VTBzb0pJR0FFRmdsQzVaMWhxU28rekZFMU5UV01oMXphMjNPYUU0TjJUNHVjd1BlaEREKzR4Ry9yMWdXMC9zOWdTaGxTMlc1U29iVDhTY2NyYi80aG9Xb1Y2TWxva0t1bXBNOWc4cCtxb0xFL3dtaElDUm9MRGhQSXR1anhoSjlqb2lZY1RBPT0saHR0cHM6Ly9hc2VtYmxpYWRldi5zaGFyZXBvaW50LmNvbS9fdnRpX2Jpbi9pZGNybC5zdmMvPC9TUD4=; path=/; secure; HttpOnly
Finally by attaching this cookie to every request to Sharepoint Services we get authenticated responses:
POST https://yourtenantname.sharepoint.com/_vti_bin/Lists.asmx HTTP/1.1
Host: yourtenantname.sharepoint.com
Content-Type: application/soap+xml; charset=utf-8
Content-Length: [calculate]
Cookie: SPOIDCRL=77u/PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz48U1A+VHJ1ZSwwaC5mfG1lbWJlcnNoaXB8MTAwMzdmZmU5NTc2YzdlZUBsaXZlLmNvbSwwIy5mfG1lbWJlcnNoaXB8Y2xvdWRAYXNlbWJsaWFkZXYub25taWNyb3NvZnQuY29tLDEzMTAxMjExNjM5MjM3NTM1ODsxMzA5NTg5NTQ0NTAwMDAwMDAsRmFsc2UsdEFEQUZZSnZiMFF6cWxFSFlKOGRPN1d1cnJ5RzJvcGxTelBueWFMUzhrNitjenBGT0JVK292M1VkSWhydGU3TXFMOHRJaFFzazRrNHd5REFqMklDUDcyMWpES3hKWmZRZjdaUlZQeisrUi92c09Qak13em5ITkg4bHVEQXdKcVlLdE16NStoaU84cUtTRzNZWEJYbWF4SDk1cDZtSDlaMVRzaFVDRXZMZ3ZIbkt6aWlPclh4UDE2RDBmZHlTRWxsU0Radmt2Tkg0UHBLT2VGbjI5S25qSk9veDVha21TZVlIbTY2ZnF5S0tpOUJmMHdjRmlyelNRZzBWZTc4NW1JZ1ZaQUY2VTArVEI0QVRvVXRVVFFqTnd4ODJEZE9jbWlqQ25NUTUzUHUrWEFIT25lenFVb1dPTXovVWk5V2VSTUMvMWZiOUpsUmZMNlZaNjNaZDRVazB3PT0saHR0cHM6Ly9hc2VtYmxpYWRldi5zaGFyZXBvaW50LmNvbS9fdnRpX2Jpbi9pZGNybC5zdmMvPC9TUD4=; path=/; secure; HttpOnly
<?xml version="1.0" encoding="utf-8"?>
<soap12:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap12="http://www.w3.org/2003/05/soap-envelope">
<soap12:Body>
<GetListCollection xmlns="http://schemas.microsoft.com/sharepoint/soap/" />
</soap12:Body>
</soap12:Envelope>

Answer 2

The token get from Azure AD by ADAL only used for the Office 365 REST API(Code above using OAuth2.0 which get an JSON Wet Token for the REST API). This token doesn't work for the SharePoint web service.
To use the SharePoint web service, we need to authenticate the SharePoint with
SharePoint Claims Authentication. More detail about SharePoint authentication you can refer to the links below:
https://msdn.microsoft.com/en-us/library/hh147177.aspx#SPO_RA_OverviewSPAuthentication