slice the middle of a dynamically generated string in Django - django

Is there a way (Using Django filters or any other language) to find and slice certain parts of a dynamically generated string? I have tried the slice method and the truncate method
{{variable|slice:"130:-60"}} or
{{variable|truncatechars:255 }}
but neither of those methods work exactly right..... I am working on weather alerts (provided by the National Weather Service) and each alert comes with a unique ID on the front and (sometimes) on the back too.
The unique ID #'s and length vary between 60 and 130 characters and the ID at the end is longitude and latitude but it's only included about 1/2 the time.
So I am looking for / working on code to "sniff out" and remove the unique ID's and to only provide the text for the user to see.
What is the proper method to do this?
Here is an example of an alert:
INC077-437-75584393-/09584738.EGY/W.0027//KT.0215401321/ 1100 AM CDT WED MAY13 2015 THE FLOODING WILL CONTINUE FOR THE MISSISSIPPI RIVER NEAR ORLANDO FLORIDA. FROM THIS EVENING TO THE END OF TIME AT 600 AM WEDNESDAY THE STAGE WAS 30.5 FEET. FLOOD STAGE IS 30.6 FEET IMPACT BY TONIGHT AT 1000 PM SOME WATER BEGINS TO FILL SOME DITCHES. && LAT...LON 4125 5845 5458 6548 8964 5124 1234 8706 $$
and with code (where I call the variable) I want it to be:
1100 AM CDT WED MAY13 2015 THE FLOODING WILL CONTINUE FOR THE MISSISSIPPI RIVER NEAR ORLANDO FLORIDA. FROM THIS EVENING TO THE END OF TIME AT 600 AM WEDNESDAY THE STAGE WAS 30.5 FEET. FLOOD STAGE IS 30.6 FEET IMPACT BY TONIGHT AT 1000 PM SOME WATER BEGINS TO FILL SOME DITCHES.
but I can't cut or truncate because the length of every weather alert is different and each unique ID is a different # and a different length.
Any help is appreciated!

You can add a custom method to your model such as :
class Weather(models.Model):
alert = models.TextField()
#property
def get_id(self):
return self.alert.split('/')[-1]
And in your template :
<p>{{ weather.get_id }}</p>
You can also create a custom template filter :
from django import template
register = template.Library()
#register.filter(name='get_id')
def get_id(value):
return value.split('/')[-1]
And use it in your template this way :
<p>{{ weather|get_id }}</p>

Related

Rocket Universe & Unidata File

This is just for clarification, know exactly what a qpointer is but today in a meeting the concept of a dpointer was raised. Anyone know what a "D" pointer refers to? Never ever heard this term before.
This is a nice question because it helped me put together a couple of pieces I had rolling around in my head, so thanks for that!
D's are dictionary items that refer to a logical location in the the data array and you have probably seen them a million times in the DICT of any given file.
A D Item in the VOC servers the same purpose and is valid with any query. Lots of shops have some generics (F1, F2, F3, F4, F5, F6..etc) set up so you don't have to remember the dictionary name if you know what filed you want. I think the precedence for dictionary items is DICT File -> VOC but I could be wrong on that.
As an example to illiterate this I went into HS.SALES and took one of the DICT items in the CUSTOMER table and wrote it to VOC after removing the conversion in field 3. I chose BUY_DATE because it had a conversion
SORT CUSTOMER BUY_DATE 06:51:04am 10 Oct 2017 PAGE 1
CUSTOMER.. Date Purchased
1 01/07/91
10 01/28/91
01/29/91
01/30/91
Remove the conversion and save into the VOC.
>ED DICT CUSTOMER BUY_DATE
10 lines long.
0001: D Date of purchase
0002: 14
0003: D2/
0004: Date Purchased
0005: 8R
0006: M
0007: ORDERS
0008: INTEGER
0009:
0010:
----: 3
0003: D2/
----: R
0003:
----: SAVE VOC F14NOCON
"F14NOCON" filed in file "VOC".
----: Q
Now sort with new D type. Values are before the Y-1995 era when pick date were still 4 digits!
SORT CUSTOMER F14NOCON 06:45:25am 10 Oct 2017 PAGE 1
CUSTOMER.. Date Purchased
1 8408
10 8429
8430
8431
Good Luck!

Python Time Series

I am working on a real estate cash-flow simulation.
What I want in the end is a time series where everyday I report if the property is vacant, leased and if I collected rent.
In my present code, I create first a profit array with values of "Leased", "Vacant" or "Today you collected rent of $1000", so I used this to create my time series:
rng=pd.date_range('6/1/2016', periods=len(profit), freq='D')
ts=pd.Series(profit, index=rng)
To simplify, I assumed I collected rent every 30 days. Now I want to be more specific and collect it every 5th day of the month (for example) and be flexible on the day the next tenant will move in.
Do you know commands or a good source where I can learn how to iterate from month to month?
Any help would be appreciated
You can build a sequence of dates using date_range and .shift() (freq='M' is for month-end frequencies) with pd.datetools.day like so:
date_sequence = pd.date_range(start, end, freq='M').shift(num_of_days, freq=pd.datetools.day)
and then use this sequence to select dates from the DateTimeIndex using
df.loc[date_sequence, 'column_name'] = value
Alternatively, you can use pd.DateOffset() like so:
ts = pd.date_range(start=date(2015, 6, 1), end=date(2015, 12, 1), freq='MS')
DatetimeIndex(['2015-06-01', '2015-07-01', '2015-08-01', '2015-09-01',
'2015-10-01', '2015-11-01', '2015-12-01'],
dtype='datetime64[ns]', freq='MS')
Now add 5 days:
ts + pd.DateOffset(days=5)
to get:
DatetimeIndex(['2015-06-06', '2015-07-06', '2015-08-06', '2015-09-06',
'2015-10-06', '2015-11-06', '2015-12-06'],
dtype='datetime64[ns]', freq=None)

Regular expression to debatch MT940 message

I got a message with below structure, where message starts from tag :20: and ends at :86:. I want to write a regular expression to extract the all messages.
I would write a C# utility to extract each message and put it in ArrayList.
:20:160212-2359
:21:600******444
:28C:00001/00001
.
.
.
:86:DAILY SETTLEMENT /ENTRY-13 MAR
:62F:D160212GBP1229387,45
:64:D160212GBP1229387,45
:65:D120314GBP1229387,45
:65:D120315GBP1229387,45
:65:D120316GBP1229387,45
:65:D120317GBP1229387,45
:65:D120318GBP1229387,45
:86:FORWARD AVAILABLE FUNDS SHOW ITEMS KNOWN BUT NOT YET POSTED
some more comments in 86_2 segment
this is line2
:20:160212-2359
:21:B***22
:25:60*****88
.
.
.
:86:/ENTRY-13 MAR TRF/REF 6*******64 /ORD/ some line here
*********************** /BNF/ JO 88
:62F:C160212EUR13868931,00
:64:C160212EUR13868931,00
:65:C120314EUR13868931,00
:65:C120315EUR13791849,00
:65:C120316EUR13791849,00
:65:C120317EUR13791849,00
:65:C120318EUR13791849,00
:86:FORWARD AVAILABLE FUNDS SHOW ITEMS KNOWN BUT NOT YET POSTED
some more comments in 86_2 segment.
:20:160212-2359
:21:B****X
:25:6*************1
:28C:00001/00001
:86:STORE1 EUROPE B.V. /ENTRY-15 MAR RTS/REF 6*****6 RTS
SWEPT FROM 9999 1**** XX***********BILLING CHARGES -
28FEB12 TRF/REF 6641XXX43799053 /ITEMCNT/004 /BNF/ /ITEMCNT/004
BILLING CHARGES
:61:1203130313DR10000000,00****288//6*****6
:86:STORE1 CNRTY SRL /ENTRY-13 MAR CLG/REF 66**********6
:61:1*****000,00NT*****9846//6******74
:86:NAME /ENTRY-13 MAR CLG/REF 6******4 LA C****R
**** CASH DEPOSIT STORE1
:61:1203150315DR48531,00NCHGBILLING CHARGES//6641XXX43799053
:86:BILLING CHARGES - 28FEB12 /ENTRY-15 MAR TRF/REF
66******53 /ITEMCNT/004
:62F:C160212EUR0,00
:64:C160212EUR0,00
:65:C120314EUR0,00
:65:C120315EUR0,00
:65:C120316EUR0,00
:65:C120317EUR0,00
:65:C120318EUR0,00
:86:FORWARD AVAILABLE FUNDS SHOW ITEMS KNOWN BUT NOT YET POSTED
{newline}
Actual values are replaced with '*' character.
Thanks
Dhiraj Bhavsar
Try this
:20:(.*?):86:
in code
/:20:(.*?):86:/gs
https://regex101.com/r/dW4zS3/1
.*? matches any character between zero and unlimited times, as few times as possible, expanding as needed

haystack order_by not in proper

I am using django1.8 and haystack 2.4 and solr 4.10. Somehow order_by is not working as expected, Please have a look at below code,
>>> sqs = SearchQuerySet()
>>> sqs = sqs.using('entry').filter(status=0)
>>> for b in sqs.filter(content="see").order_by('title'): print b.title
501 Must-See Movies
Look See, Look at Me!
Last Chance to See
1,000 Places to See Before You Die
Pretend You Don't See Her
Learning to See Creatively : Design, Color and Composition in Photography
Behavior Solutions for the Inclusive Classroom : See a Behavior? Look It Up!
See No Evil
Last Chance to See
See It and Sink It : Mastering Putting Through Peak Visual Performance
See No Evil : The True Story of a Ground Soldier in the CIA's War on Terrorism
Voice for Now : Changing the Way We See Ourselves As Women
See Jane Win : The Rimm Report on How 1,000 Girls Became Successful Women
Kaplan Medical USMLE Medical Ethics : The 100 Cases You Are Most Likely to See on the Exam
I See You
You'll See It When You Believe It : The Way to Your Personal Transformation
Body Code : Diet and Fitness Programme: Master Your Metabolism and See the Weight Fall Off
descending Order
>>> sqs = SearchQuerySet()
>>> sqs = sqs.using('entry').filter(status=0)
>>> for b in sqs.filter(content="see").order_by('-title'): print b.title
You'll See It When You Believe It : The Way to Your Personal Transformation
Body Code : Diet and Fitness Programme: Master Your Metabolism and See the Weight Fall Off
Kaplan Medical USMLE Medical Ethics : The 100 Cases You Are Most Likely to See on the Exam
I See You
Voice for Now : Changing the Way We See Ourselves As Women
See Jane Win : The Rimm Report on How 1,000 Girls Became Successful Women
See No Evil : The True Story of a Ground Soldier in the CIA's War on Terrorism
See It and Sink It : Mastering Putting Through Peak Visual Performance
Last Chance to See
See No Evil
Behavior Solutions for the Inclusive Classroom : See a Behavior? Look It Up!
Learning to See Creatively : Design, Color and Composition in Photography
Pretend You Don't See Her
1,000 Places to See Before You Die
Last Chance to See
Look See, Look at Me!
501 Must-See Movies
Why odering is not working like A --> Z and Z --> A
Recently i had same issue with haystack order_by for title. I used python lambda function to sort object list.
ascending order using title:
sqs = sqs.using('entry').filter(status=0)
sorted_list = sorted([s.object for s in sqs], key=lambda x: x.title, reverse=False)
descending order:
sqs = sqs.using('entry').filter(status=0)
rev_sorted_list = sorted([s.object for s in sqs], key=lambda x: x.title, reverse=True)
sqs.order_by works very well with Integer fields.

Help: Extracting data tuples from text... Regex or Machine learning?

I would really appreciate your thoughts on the best approach to the following problem. I am using a Car Classified listing example which is similar in nature to give an idea.
Problem: Extract a data tuple from the given text.
Here are some characteristics of the data.
The vocabulary (words) in the text is limited to a specific domain. Lets assume 100-200 words at the most.
Text that needs to be parsed is a headline like a Car Ad data shown below. So each record corresponds to one tuple (row).
In some cases some of the attributes may be missing. So for example, in raw data row #5 below the year is missing.
Some words go together (bigrams). Like "Low miles".
Historical data available = 10,000 records
Incoming New Data volume = 1000-1500 records / week
The expected output should be in the form of (Year,Make,Model, feature). So the output should look like
1 -> (2009, Ford, Fusion, SE)
2 -> (1997, Ford, Taurus, Wagon)
3 -> (2000, Mitsubishi, Mirage, DE)
4 -> (2007, Ford, Expedition, EL Limited)
5 -> ( , Honda, Accord, EX)
....
....
Raw Headline Data:
1 -> 2009 Ford Fusion SE - $7000
2 -> 1997 Ford Taurus Wagon - $800 (san jose east)
3 -> '00 Mitsubishi Mirage DE - $2499 (saratoga) pic
4 -> 2007 Ford Expedition EL Limited - $7800 (x)
5 -> Honda Accord ex low miles - $2800 (dublin / pleasanton / livermore) pic
6 -> 2004 HONDA ODASSEY LX 68K MILES - $10800 (danville / san ramon)
7 -> 93 LINCOLN MARK - $2000 (oakland east) pic
8 -> #######2006 LEXUS GS 430 BLACK ON BLACK 114KMI ####### - $19700 (san rafael) pic
9 -> 2004 Audi A4 1.8T FWD - $8900 (Sacramento) pic
10 -> #######2003 GMC C2500 HD EX-CAB 6.0 V8 EFI WHITE 4X4 ####### - $10575 (san rafael) pic
11 -> 1990 Toyota Corolla RUNS GOOD! GAS SAVER! 5SPEED CLEAN! REG 2011 O.B.O - $1600 (hayward / castro valley) pic img
12 -> HONDA ACCORD EX 2000 - $4900 (dublin / pleasanton / livermore) pic
13 -> 2009 Chevy Silverado LT Crew Cab - $23900 (dublin / pleasanton / livermore) pic
14 -> 2010 Acura TSX - V6 - TECH - $29900 (dublin / pleasanton / livermore) pic
15 -> 2003 Nissan Altima - $1830 (SF) pic
Possible choices:
A machine learning Text Classifier (Naive Bayes etc)
Regex
What I am trying to figure out is if RegEx is too complicated for the job and a Text classifier is an overkill?
If the choice is to go with a text classifier then what would you consider to be the easiest to implement.
Thanks in advance for your kind help.
This is a well studied problem called information extraction. It is not straight forward to do what you want to do, and it is not as simple as you make it sound (ie machine learning is not an overkill). There are several techniques, you should read an overview of the research area.
Check this IE library for writing extraction rule< I think it will work best for you problem.
There also example how to create fast dictionary matching.
I think that the ARX or Phoebus systems may suit your needs if you already have annotated data and a list of words associated to each field. Their approach is a mix of information extraction and information integration.
There are a few good entity recognition libraries. Have you taken a look at Apache opennlp?
As a user looking for a specific model of car the task is easier. I'm pretty sure I could classify, say, most Ford Rangers since I know what to look for with regexp.
I think your best bet is to write a function for each car model with type String -> Maybe Tuple. Then run all these on each input and throw away those inputs resulting in zero or too many tuples.
You should use a tool like Amazon Mechanical Turk for this. Human microtasking. Another alternative is to use a data entry freelancer. upWork is a great place to look. You can get excellent quality results and the cost is very reasonable for each.