Twilio Studio RegEx in "Split Based On..." Widget - regex

Question

I'm still in my trial phase with Twilio and working on putting together a "flow" in Studio to handle my use-case properly. So far, all of the basic, plain-text steps in my flow seem to be working exactly as expected, but I'm having problems figuring out how to handle incoming media/attachments the way I want. I readily admit and acknowledge that what I've put together may not be the most effective solution, but it's what I've come up with based on the research I've done so far.
First, here is the basic breakdown of the desired steps in my flow:
I have a separate application that connects to the Twilio REST API to initiate the flow's execution. It passes a couple of parameters into the flow's flow.data namespace for inclusion in various messages sent to the end-user as a part of the flow.
Private Async Sub StartFlow(ByVal Recipient As String)
Try
Dim RefNum As String = GenerateReferenceNumber(8)
Dim NoticeParameters As New Dictionary(Of String, Object) From {
{"ref_num", RefNum},
{"contact_num", "[phone]"},
{"client_name", "Caliente Client"}
}
TwilioClient.Init(SID, Token)
Dim FlowExecution = Await ExecutionResource.CreateAsync(parameters:=NoticeParameters,
[to]:=New PhoneNumber(Recipient),
from:=MyTwilioNumber,
pathFlowSid:=NoticeFlowSID)
Catch ex As Exception
End Try
End Sub
The first step in the flow's execution is to initialize a variable in the flow.variables namespace named current_response with a value of EMPTY.
The flow's next step is to send our initial SMS contact to the end-user via a Send & Wait For Reply widget called "Notice".
When a reply is received from the user, the flow.variables.current_response value is updated by a Set Variables widget:
If the value of widgets.Notice.inbound.MediaUrl0 is not NULL, the flow.variables.current_response value is set to the URL value from that property.
If the value of widgets.Notice.inbound.MediaUrl0 is NULL, the flow.variables.current_response value is set to the value of the widgets.Notice.inbound.Body property instead.
After setting the current_response variable's value, that variable is passed on to a Split Based On... widget (named "Response_Received") that tests the flow.variables.current_response with the following Regular Expression to determine whether or not the value is a URL:
[(http(s)?):\/\/(www\.)?a-zA-Z0-9#:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9#:%_\+.~#?&//=]*)
If the value is not a URL, the flow should pass execution on to the next Split Based On... widget (named "Other_Response") to handle certain plain-text replies with a pre-defined set of responses.
If the value is a URL, it should pass execution to a Make HTTP Request widget that will POST to my MVC controller to log the incoming file.
Depending on the end-user's reply, the Other_Response widget will then pass execution on to another Send & Wait For Reply widget that can update the flow.variables.current_response property with that widget's inbound.MediaUrl0 or inbound.Body via another Set Variables widget if they continue the conversation.
After updating the current_response variable, flow execution returns to the "Response_Received" widget with the new value (see #5).
The idea is that, if the user initially responds with something like HELP, I want to be able to send them a message with additional details about the reason we reached out to them in the first place, then allow them to send us the specific information (file) we're requesting while still retaining the values I populated into the flow from my in-house application - most specifically, a unique reference number I've generated for the contact. Even if it takes them a couple of tries, as long as the execution is valid, I want any reply within that execution to have the capability of processing an incoming file based on those same values.
In my tests, I've been able to get all of the simple, plain-text responses to work as I've designed:
I reply to the flow's initial message with one of the pre-defined messages and the flow returns the correct response and waits.
I reply to that message with another of the pre-defined messages and the flow again returns the appropriate response and waits.
I reply to that message with a plain-text message that isn't defined as a valid response and the flow correctly returns a message stating, "I'm sorry. I didn't recognize your response. Please try again."
However, if I reply to the flow's initial message with a file as I expect to have our end-users doing (I've used PDF and PNG files in my testing so far), I still get the message stating, "I'm sorry. I didn't recognize your response. Please try again." This message is not defined in the path from the split designated for replies with a URL.
Based on the results, I can only assume that there's something "wrong" with the RegEx logic that's not actually matching the URL as it should. I've tested the pattern on RegExr and it "works", identifying the URL as a match. For testing purposes, I've even included the value of the flow.variables.current_response in the message sent back to the user and it shows that the value is, in fact, the Twilio API URL of the media.
For reference, here is a slightly redacted version of the JSON definition for this flow. There are some "silly" nodes in here I've been using for testing my logic. As stated above, the only thing that seems to be failing is the RegEx matching of the MediaUrl0 value to send it to my MVC controller.
{
"description": "A New Flow",
"states": [
{
"name": "Trigger",
"type": "trigger",
"transitions": [
{
"event": "incomingMessage"
},
{
"next": "Incoming_Call",
"event": "incomingCall"
},
{
"event": "incomingConversationMessage"
},
{
"next": "Initialize_Reply",
"event": "incomingRequest"
},
{
"event": "incomingParent"
}
],
"properties": {
"offset": {
"x": 0,
"y": 0
}
}
},
{
"name": "Notice",
"type": "send-and-wait-for-reply",
"transitions": [
{
"next": "Notice_Response",
"event": "incomingMessage"
},
{
"event": "timeout"
},
{
"event": "deliveryFailure"
}
],
"properties": {
"offset": {
"x": 700,
"y": 270
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"body": "Information Request from {{flow.data.client_name}}: We need some additional information from you about your recent purchase. For more information about this request, reply HELP. To stop receiving these messages from {{flow.data.client_name}}, reply STOP. Reference #{{flow.data.ref_num}}",
"media_url": "",
"timeout": "2419200"
}
},
{
"name": "Response_Received",
"type": "split-based-on",
"transitions": [
{
"next": "Other_Response",
"event": "noMatch"
},
{
"next": "Log_Document",
"event": "match",
"conditions": [
{
"friendly_name": "Document Received",
"arguments": [
"{{flow.variables.current_response}}"
],
"type": "regex",
"value": "[(http(s)?):\\/\\/(www\\.)?a-zA-Z0-9#:%._\\+~#=]{2,256}\\.[a-z]{2,6}\\b([-a-zA-Z0-9#:%_\\+.~#?&//=]*)"
}
]
}
],
"properties": {
"input": "{{flow.variables.current_response}}",
"offset": {
"x": 370,
"y": 800
}
}
},
{
"name": "Other_Response",
"type": "split-based-on",
"transitions": [
{
"next": "Unrecognized_Response",
"event": "noMatch"
},
{
"event": "match",
"conditions": [
{
"friendly_name": "More Info",
"arguments": [
"{{flow.variables.current_response}}"
],
"type": "matches_any_of",
"value": "HELP,INFO,MORE"
}
]
},
{
"next": "Log_Opt_Out",
"event": "match",
"conditions": [
{
"friendly_name": "Unsubscribe",
"arguments": [
"{{flow.variables.current_response}}"
],
"type": "matches_any_of",
"value": "STOP,QUIT,REMOVE,DELETE,CEASE,CANCEL,UNSUBSCRIBE,STOPALL,END"
}
]
},
{
"next": "Jedi_Code",
"event": "match",
"conditions": [
{
"friendly_name": "Jedi Code",
"arguments": [
"{{flow.variables.current_response}}"
],
"type": "equal_to",
"value": "JEDI"
}
]
},
{
"next": "Sith_Code",
"event": "match",
"conditions": [
{
"friendly_name": "Sith Code",
"arguments": [
"{{flow.variables.current_response}}"
],
"type": "equal_to",
"value": "SITH"
}
]
}
],
"properties": {
"input": "{{flow.variables.current_response}}",
"offset": {
"x": -130,
"y": 1020
}
}
},
{
"name": "Document_Confirmation",
"type": "send-message",
"transitions": [
{
"event": "sent"
},
{
"event": "failed"
}
],
"properties": {
"offset": {
"x": 620,
"y": 1370
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"to": "{{contact.channel.address}}",
"body": "Thank you. We will update our records as soon as possible. If there are any further questions or concerns, one of our representatives will contact you to follow up.\n\nReference #{{flow.data.ref_num}}"
}
},
{
"name": "Log_Document",
"type": "make-http-request",
"transitions": [
{
"next": "Document_Confirmation",
"event": "success"
},
{
"next": "Unrecognized_Document",
"event": "failed"
}
],
"properties": {
"offset": {
"x": 880,
"y": 1020
},
"method": "POST",
"content_type": "application/json;charset=utf-8",
"url": "https://my.domain.com/sms"
}
},
{
"name": "Jedi_Code",
"type": "send-and-wait-for-reply",
"transitions": [
{
"next": "Jedi_Response",
"event": "incomingMessage"
},
{
"event": "timeout"
},
{
"event": "deliveryFailure"
}
],
"properties": {
"offset": {
"x": -260,
"y": 1340
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"body": "There is no emotion, there is peace.\nThere is no ignorance, there is knowledge.\nThere is no passion, there is serenity.\nThere is no chaos, there is harmony.\nThere is no death, there is the Force.\nReference #{{flow.data.ref_num}}\nContact #{{flow.data.contact_num}}",
"media_url": "",
"timeout": "43200"
}
},
{
"name": "Sith_Code",
"type": "send-and-wait-for-reply",
"transitions": [
{
"next": "Sith_Response",
"event": "incomingMessage"
},
{
"event": "timeout"
},
{
"event": "deliveryFailure"
}
],
"properties": {
"offset": {
"x": 90,
"y": 1370
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"body": "Peace is a lie. There is only Passion.\nThrough Passion, I gain Strength.\nThrough Strength, I gain Power.\nThrough Power, I gain Victory.\nThrough Victory my chains are Broken.\nThe Force shall free me.\nReference #{{flow.data.ref_num}}",
"media_url": "",
"timeout": "43200"
}
},
{
"name": "Log_Opt_Out",
"type": "make-http-request",
"transitions": [
{
"next": "Opt_Out_Success",
"event": "success"
},
{
"next": "Opt_Out_Fail",
"event": "failed"
}
],
"properties": {
"offset": {
"x": -660,
"y": 1310
},
"method": "POST",
"content_type": "application/x-www-form-urlencoded;charset=utf-8",
"url": "https://my.domain.com/sms"
}
},
{
"name": "Opt_Out_Success",
"type": "send-message",
"transitions": [
{
"event": "sent"
},
{
"event": "failed"
}
],
"properties": {
"offset": {
"x": -1060,
"y": 1320
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"to": "{{contact.channel.address}}",
"body": "Thank you. Your preferences have been updated and you will no longer receive insurance notifications from {{flow.data.client_name}}. \nYou can re-enable these notifications later by texting OPT-IN to this number."
}
},
{
"name": "Opt_Out_Fail",
"type": "send-and-wait-for-reply",
"transitions": [
{
"next": "Opt_Out_Response",
"event": "incomingMessage"
},
{
"event": "timeout"
},
{
"event": "deliveryFailure"
}
],
"properties": {
"offset": {
"x": -650,
"y": 1580
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"body": "We're sorry. There was a problem registering your opt-out request. You can try again or contact {{flow.data.client_name}} to update your preferences.",
"timeout": "43200"
}
},
{
"name": "Unrecognized_Response",
"type": "send-and-wait-for-reply",
"transitions": [
{
"next": "Unknown_Response",
"event": "incomingMessage"
},
{
"event": "timeout"
},
{
"event": "deliveryFailure"
}
],
"properties": {
"offset": {
"x": -490,
"y": 1030
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"body": "I'm sorry. I didn't recognize your response. Please try again.\n\nResponse: {{flow.variables.current_response}}",
"timeout": "43200"
}
},
{
"name": "Initialize_Reply",
"type": "set-variables",
"transitions": [
{
"next": "Notice",
"event": "next"
}
],
"properties": {
"variables": [
{
"value": "EMPTY",
"key": "current_response"
}
],
"offset": {
"x": 390,
"y": 270
}
}
},
{
"name": "Notice_Response",
"type": "set-variables",
"transitions": [
{
"next": "Response_Received",
"event": "next"
}
],
"properties": {
"variables": [
{
"value": "{% if widgets.Notice.inbound.MediaUrl0 != null %}\n {{widgets.Notice.inbound.MediaUrl0}} \n{% else %}\n{{widgets.Notice.inbound.Body}} \n{% endif %}",
"key": "current_response"
}
],
"offset": {
"x": 390,
"y": 550
}
}
},
{
"name": "Jedi_Response",
"type": "set-variables",
"transitions": [
{
"next": "Response_Received",
"event": "next"
}
],
"properties": {
"variables": [
{
"value": "{% if widgets.Jedi_Code.inbound.MediaUrl0 != null %} {{widgets.Jedi_Code.inbound.MediaUrl0}} {% else %} {{widgets.Jedi_Code.inbound.Body}} {% endif %}",
"key": "current_response"
}
],
"offset": {
"x": -280,
"y": 1580
}
}
},
{
"name": "Sith_Response",
"type": "set-variables",
"transitions": [
{
"next": "Response_Received",
"event": "next"
}
],
"properties": {
"variables": [
{
"value": "{% if widgets.Sith_Code.inbound.MediaUrl0 != null %} {{widgets.Sith_Code.inbound.MediaUrl0}} {% else %} {{widgets.Sith_Code.inbound.Body}} {% endif %}",
"key": "current_response"
}
],
"offset": {
"x": 120,
"y": 1600
}
}
},
{
"name": "Unknown_Response",
"type": "set-variables",
"transitions": [
{
"next": "Response_Received",
"event": "next"
}
],
"properties": {
"variables": [
{
"value": "{% if widgets.Unrecognized_Response.inbound.MediaUrl0 != null %} {{widgets.Unrecognized_Response.inbound.MediaUrl0}} {% else %} {{widgets.Unrecognized_Response.inbound.Body}} {% endif %}",
"key": "current_response"
}
],
"offset": {
"x": -840,
"y": 1040
}
}
},
{
"name": "Opt_Out_Response",
"type": "set-variables",
"transitions": [
{
"next": "Response_Received",
"event": "next"
}
],
"properties": {
"variables": [
{
"value": "{% if widgets.Opt_Out_Fail.inbound.MediaUrl0 != null %} {{widgets.Opt_Out_Fail.inbound.MediaUrl0}} {% else %} {{widgets.Opt_Out_Fail.inbound.Body}} {% endif %}",
"key": "current_response"
}
],
"offset": {
"x": -1000,
"y": 1590
}
}
},
{
"name": "Unrecognized_Document",
"type": "send-and-wait-for-reply",
"transitions": [
{
"next": "Unrecognized_Document_Response",
"event": "incomingMessage"
},
{
"event": "timeout"
},
{
"event": "deliveryFailure"
}
],
"properties": {
"offset": {
"x": 970,
"y": 1330
},
"service": "{{trigger.message.InstanceSid}}",
"channel": "{{trigger.message.ChannelSid}}",
"from": "{{flow.channel.address}}",
"body": "I'm sorry. We were unable to process the document you sent. Please try again, or contact us at {{flow.data.contact_num}}",
"timeout": "604800"
}
},
{
"name": "Unrecognized_Document_Response",
"type": "set-variables",
"transitions": [
{
"next": "Response_Received",
"event": "next"
}
],
"properties": {
"variables": [
{
"value": "{% if widgets.Unrecognized_Document.inbound.MediaUrl0 != null %} {{widgets.Unrecognized_Document.inbound.MediaUrl0}} {% else %} {{widgets.Unrecognized_Document.inbound.Body}} {% endif %}",
"key": "current_response"
}
],
"offset": {
"x": 1060,
"y": 1610
}
}
},
{
"name": "Incoming_Call",
"type": "connect-call-to",
"transitions": [
{
"event": "callCompleted"
},
{
"event": "hangup"
}
],
"properties": {
"offset": {
"x": -460,
"y": -60
},
"caller_id": "{{contact.channel.address}}",
"noun": "number",
"to": "[phone]",
"timeout": 30
}
}
],
"initial_state": "Trigger",
"flags": {
"allow_concurrent_calls": true
}
}
Also, in case it helps, here's a screenshot of the entire flow from the Studio editor:
I am continuing to work with the Twilio Support team as well to determine if there's an actual "problem" here, or if I'm simply doing it wrong (a distinct possibility). There very well may be more effective/efficient ways to achieve my goals but, in the absence of a "generic" option to validate any incoming widget (e.g., widgets.previous_widget.inbound.MediaUrl0), this is the best solution I've been able to find so far.
The main reason I'm posting my question here is not only to try and find a solution that fits my specific needs, but also to document the troubleshooting process and hopefully help anyone else who's just beginning to work with Studio and needs to implement some "complex" business rules.
UPDATE - Alternative Solution Found
So, after talking it through with Twilio Support, we still weren't able to figure out why the above method doesn't work, but we were able to determine that the current best course of action is to add a series of Split Based On... widgets into the flow at each point where I currently just have the Set Variables widgets. While it significantly increases the overall complexity of the flow itself, this method eliminates the need to use the RegEx matching I had to identify incoming URLs.
Even so, once I was finally able to properly articulate why I was trying to make this work, the representative with whom I spoke agreed that it should work and is going to continue testing and investigating the issue. In the meantime, I'm going to update my flow to add all of the extra widgets I need for my logic, but I'd still like to find out if there's some reason I'm overlooking why this is failing.

Answer 1

Based on my conversations with the Twilio Support team, it seems that, for whatever reason, the RegEx method I had implemented will not work. I wasn't able to get a specific reason why it fails, but it appears to be a technical limitation somewhere within Studio. In this case, to achieve my specific goals, there are basically two alternatives:
As I described in the UPDATE in the OP, I can use Split Based On... widgets at each Send & Wait For Reply point that test whether the MediaUrl0 value is empty, then use a pair of Set Variables widgets - one for text-only replies and one for replies with attachments - to pass the value(s) on to the appropriate next step in the flow.
Instead of using a RegEx match to test the value of my current_response variable, I can use the Starts With test to see if the current_response begins with "my" Twilio API URL (https://api.twilio.com/2010-04-01/Accounts/[ACCOUNT SID]/Messages).
Obviously, the former method requires a bit more complexity - several additional widgets to direct the flow to the appropriate next widget - but is less dependent on the actual value of the Twilio API URL. The only "problem" with the latter method would be that if the API version (2010-04-01) changes, I would need to update each point in my flow where it's used to reference the new version.
At this time, I've opted for the former solution and have tested the flow several times to ensure that it all seems to be working correctly. After a few "false starts", I've been able to get through the entire process with all of the correct responses and operations. Simple text replies send back the appropriate response while sending a file pushes the information over to my MVC controller for further processing.
One totally tangential side note: Because the MediaUrl0 value "masks" the original file name and strips the file extension, I needed a way to determine what kind of file was being sent. In a "regular" SMS interaction using Twilio's Programmable Messaging tools, there's a MediaContentType0 property. This value is not a selectable option in the Studio flow editor. I was able to retrieve the value of this property by manually typing it into the appropriate fields (widgets.[WIDGET_NAME].inbound.MediaContentType0), so it seems that any value that is available in the current JSON should be accessible through Studio. You just have to know what you're looking for because it isn't "documented". I hope this all helps someone else looking to implement their own Studio flow.