I create a stream to read data from csv and write it in Postgresql it do everything just insert data on db:
my csv consist of 1,test,1
my stream :
#App:name("StockFile")
#App:description('test ...')
#source(type='file',
dir.uri='file:C:\file',
action.after.process='NONE',
#map(type='csv'))
define stream IntputStream (Amount int, Location string, ProductId int);
#store(type = 'rdbms',
jdbc.url = "jdbc:postgresql://localhost:5432/postgres",
username = "xxx",
password = "xxx",
jdbc.driver.name = "org.postgresql.Driver",
table.name = 'Test',
operation = 'insert',
#map(type = 'keyvalue' ))
define stream outputstream (Amount int, Location string, ProductId int);
#info(name = 'Save stock records')
from IntputStream
select Amount,Location,ProductId
insert into outputstream;
When you define a table, the definition should be a table definition so the correct way to define the outputstream is
#store(type = 'rdbms',jdbc.url = "jdbc:postgresql://localhost:5432/postgres",username = "xxx",password = "xxx",jdbc.driver.name = "org.postgresql.Driver",table.name = 'Test',operation = 'insert', #map(type = 'keyvalue' ))
define table outputstream (Amount int, Location string, ProductId int);
Related
I have a list of tables (in actual data) with different columns for which, after to combine, I get a table of 15 columns. In actual data, the list of tables is get from several previous steps and each step takes less than a second, but only Table.Combine() takes almost 2 minutes with an input of about 1200 rows. In order to show the example, I show below an output of 4 columns only,
Is there a faster alternative way to get the same output given by Table.Combine()? Thanks for any help.
This is the code of the query I has so far.
let
Tables = {
Table.FromRecords({[Name = "Bob", Phone = "123-4567"],
[Name = "",Phone = ""]
}),
Table.FromRecords({[Fax = "987-6543", Phone = "838-7171"],
[Fax = "", Phone = "233-687"],
[Fax = "", Phone = "544-778"]
}),
Table.FromRecords({[Cell = "543-7890"],
[Cell = ""],
[Cell = ""]
})
},
CombinedTable = Table.Combine(Tables)
in
CombinedTable
The current output is:
UPDATE
This is the entire query, with Table.Buffer() added in step group5
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("jVNdb4JAEPwvPFty3KHUR1C4Sg+0x5lqqSF+tGlqH5q0mv787hkal41FEsLNLjczu5NQlk6kpqP7yixnceU5Pad+Vr3SCQF4qI4A9FE9AiBQXcyj6qTWlBkDYKiOSd1C8wkNTyPpdK174DntHpzscUucB8R5iOqE6rU6B1ecOXEeEueOUXmE5nejBS1uCZHt6C5JnCge3mReiiMLJ/mNELidARgZreMCNTWAsQozPMgC0N2DFbHK8unUTTOr9XxgTGzzuVIn9AKtRZrDe7M5uod3xrj7uf8I2MD9Or7u1t9rxA2VmsJRGIPui//vX/CK8rOX7zHLFXBgbntM9L+T01koSUaRnvQNiciEanw1Ir37gSLZ7mx/Uw/qcbFvDKjjFD7Bijq1Eywf64uC87egWwzS/xP3hmfG6hc=", BinaryEncoding.Base64), Compression.Deflate)),
let
_t = ((type nullable text) meta [Serialized.Text = true])
in
type table [COL1 = _t, COL2 = _t, COL3 = _t, COL4 = _t]
),
fx = each not List.IsEmpty(List.RemoveItems(_,{"",null})),
group0 = Table.Group(Source, "COL2", {"n", each _}, 0, (x, y) => Byte.From(y = "" or y = null)),
group1 = Table.TransformColumns(
group0,
{
"n",
each
let
a = Table.Skip(_),
b = Table.FirstN(a, each [COL3] = "" or [COL3] = null),
c = Table.Skip(a, Table.RowCount(b))
in
[a = a, b = b, c = c]
}
),
group2 = Table.TransformColumns(
group1,
{"n", each Table.ToColumns(Table.Transpose([b])) & Table.ToColumns([c])}
),
group3 = Table.TransformColumns(group2, {"n", each List.Select(_, fx)}),
group4 = Table.TransformColumns(group3, {"n", each Table.FromColumns(_)}),
group5 = Table.Buffer( Table.TransformColumns(group4, {"n", each Table.PromoteHeaders(_)}) ) ,
combine = Table.Combine(group5[n]),
Custom1 = Table.SelectRows(combine, each fx(Record.ToList(_)))
in
Custom1
The purpose of this query is to tabulate data that appears in repeated blocks and subblock in the way I show below.
This is the output given by the query.
No, but try wrapping the initial table definitions as you go along in Table.Buffer()
let
a= Table.Buffer(Table.FromRecords({[Name = "Bob", Phone = "123-4567"],[Name = "",Phone = ""]})),
b= Table.Buffer(Table.FromRecords({[Fax = "987-6543", Phone = "838-7171"], [Fax = "", Phone = "233-687"],[Fax = "", Phone = "544-778"]})),
c= Table.Buffer(Table.FromRecords({[Cell = "543-7890"],[Cell = ""],[Cell = ""]})),
CombinedTable = Table.Combine({a,b,c})
in CombinedTable
I am attempting (and can successfully do so) to connect to an API and loop through several iterations of the API call in order to grab the next_page value, put it in a list and then call the list.
Unfortunately, when this is published to the PBI service I am unable to refresh there and indeed 'Data Source Settings' tells me I have a 'hand-authored query'.
I have attempted to follow Chris Webbs' blog post around the usage of query parameters and relative path, but if I use this I just get a constant loop of the first page that's hit.
The Start Epoch Time is a helper to ensure I only grab data less than 3 months old.
let
iterations = 10000, // Number of MAXIMUM iterations
url = "https://www.zopim.com/api/v2/" & "incremental/" & "chats?fields=chats(*)" & "&start_time=" & Number.ToText( StartEpochTime ),
FnGetOnePage =
(url) as record =>
let
Source1 = Json.Document(Web.Contents(url, [Headers=[Authorization="Bearer MY AUTHORIZATION KEY"]])),
data = try Source1[chats] otherwise null, //get the data of the first page
next = try Source1[next_page] otherwise null, // the script ask if there is another page*//*
res = [Data=data, Next=next]
in
res,
GeneratedList =
List.Generate(
()=>[i=0, res = FnGetOnePage(url)],
each [i]<iterations and [res][Data]<>null,
each [i=[i]+1, res = FnGetOnePage([res][Next])],
each [res][Data])
Lookups
If Source1 exists, but [chats] may not, you can simplify
= try Source1[chats] otherwise null
to
= Source1[chats]?
Plus it you don't lose non-lookup errors.
m-spec-operators
Chris Web Method
should be something closer to this.
let
Headers = [
Accept="application/json"
],
BaseUrl = "https://www.zopim.com", // very important
Options = [
RelativePath = "api/v2/incremental/chats",
Headers = [
Accept="application/json"
],
Query = [
fields = "chats(*)",
start_time = Number.ToText( StartEpocTime )
],
Response = Web.Contents(BaseUrl, Options),
Result = Json.Document(Response) // skip if it's not JSON
in
Result
Here's an example of a reusable Web.Contents function
helper function
let
/*
from: <https://github.com/ninmonkey/Ninmonkey.PowerQueryLib/blob/master/source/WebRequest_Simple.pq>
Wrapper for Web.Contents returns response metadata
for options, see: <https://learn.microsoft.com/en-us/powerquery-m/web-contents#__toc360793395>
Details on preventing "Refresh Errors", using 'Query' and 'RelativePath':
- Not using Query and Relative path cause refresh errors:
<https://blog.crossjoin.co.uk/2016/08/23/web-contents-m-functions-and-dataset-refresh-errors-in-power-bi/>
- You can opt-in to Skip-Test:
<https://blog.crossjoin.co.uk/2019/04/25/skip-test-connection-power-bi-refresh-failures/>
- Debugging and tracing the HTTP requests
<https://blog.crossjoin.co.uk/2019/11/17/troubleshooting-web-service-refresh-problems-in-power-bi-with-the-power-query-diagnostics-feature/>
update:
- MaybeErrResponse: Quick example of parsing an error result.
- Raw text is returned, this is useful when there's an error
- now response[json] does not throw, when the data isn't json to begin with (false errors)
*/
WebRequest_Simple
= (
base_url as text,
optional relative_path as nullable text,
optional options as nullable record
)
as record =>
let
headers = options[Headers]?, //or: ?? [ Accept = "application/json" ],
merged_options = [
Query = options[Query]?,
RelativePath = relative_path,
ManualStatusHandling = options[ManualStatusHandling]? ?? { 400, 404, 406 },
Headers = headers
],
bytes = Web.Contents(base_url, merged_options),
response = Binary.Buffer(bytes),
response_metadata = Value.Metadata( bytes ),
status_code = response_metadata[Response.Status]?,
response_text = Text.Combine( Lines.FromBinary(response,null,null, TextEncoding.Utf8), "" ),
json = Json.Document(response),
IsJsonX = not (try json)[HasError],
Final = [
request_url = metadata[Content.Uri](),
response_text = response_text,
status_code = status_code,
metadata = response_metadata,
IsJson = IsJsonX,
response = response,
json = if IsJsonX then json else null
]
in
Final,
tests = {
WebRequest_Simple("https://httpbin.org", "json"), // expect: json
WebRequest_Simple("https://www.google.com"), // expect: html
WebRequest_Simple("https://httpbin.org", "/headers"),
WebRequest_Simple("https://httpbin.org", "/status/codes/406"), // exect 404
WebRequest_Simple("https://httpbin.org", "/status/406"), // exect 406
WebRequest_Simple("https://httpbin.org", "/get", [ Text = "Hello World"])
},
FinalResults = Table.FromRecords(tests,
type table[
status_code = Int64.Type, request_url = text,
metadata = record,
response_text = text,
IsJson = logical, json = any,
response = binary
],
MissingField.Error
)
in
FinalResults
I'm using Datatables to display tables in Django on the server side. After searching for a phrase, I have a button ready to save the current table after filtering to Excel. I would like the second button to create a new table in the database and save the same table to it as for Excel.
I don't know how I can send AJAX data to Django and read it in views in such a way that I can query the database.
javascript:
$(document).ready(function () {
function newexportaction(e, dt, button, config) {
var self = this;
var oldStart = dt.settings()[0]._iDisplayStart;
dt.one('preXhr', function (e, s, data) {
data.start = 0;
data.length = 2147483647;
dt.one('preDraw', function (e, settings) {
// Call the original action function
if (button[0].className.indexOf('buttons-excel') >= 0) {
$.fn.dataTable.ext.buttons.excelHtml5.available(dt, config) ?
$.fn.dataTable.ext.buttons.excelHtml5.action.call(self, e, dt, button, config) :
$.fn.dataTable.ext.buttons.excelFlash.action.call(self, e, dt, button, config);
}
dt.one('preXhr', function (e, s, data) {
settings._iDisplayStart = oldStart;
data.start = oldStart;
});
});
});
dt.ajax.reload();
}
$("#tb").DataTable({
"lengthChange": true,
"paging": true,
"searching": true,
"oLanguage": {
"sSearch": "Szukaj:",
},
"language": {
"processing": "Przetwarzanie",
"lengthMenu": "Pokaż _MENU_ elementów",
"info": "Wyświetlono od _START_ do _END_ z _TOTAL_ elementów",
"zeroRecords": "Nie znaleziono pasujących elementów",
"paginate": {
"first": "Pierwsza",
"last": "Ostatnia",
"next": "Kolejna",
"previous": "Poprzednia"
},
"emptyTable": "Brak danych do wyświetlenia",
},
"processing": true,
"serverSide": true,
buttons: [
{
extend: 'excel',
titleAttr: 'Excel',
title: '',
action: newexportaction
},
],
dom: 'B<"clear">lfrtip',
// "destroy": true,
"pageLength": 15,
"ordering": false,
"ajax": {
"url": "paging2/",
"type": "get"
},
});
});
views.py:
def paging2(request):
draw = int (request.GET.get ('draw')) # record the number of operations
start = int (request.GET.get ('start')) # start position
length = int (request.GET.get ('length')) # length of each page
search_key = request.GET.get ('search[value]') # search keyword
order_column = request.GET.get ('order [0] [column]') # sort field index
order_column = request.GET.get ('order [0] [dir]') #Ordering rule: ase / desc
# sql query all data, then do paging, the speed is slow
# if search_key:
# result = query(search_key)
# else:
# result = select_all()
# data = result[start:start+length]
# count = len(result)
# sql for paging, fast
if search_key:
sql_search = "SELECT NAZWA,TEL,NIP,Adres,WWW, EMAIL, Branza, NAZWA_SCRAPERA FROM gus_all t1 WHERE NOT EXISTS (SELECT * FROM matka_new t2 WHERE t2.NIP = t1.NIP) AND LENGTH(TEL) = 9 AND `STATUS` = 'AKTYWNY' AND NAZWA_SCRAPERA is NOT NULL AND Branza like '%%%s%%'" % search_key
sql_search_len = "SELECT COUNT(*) FROM gus_all t1 WHERE NOT EXISTS (SELECT * FROM matka_new t2 WHERE t2.NIP = t1.NIP) AND LENGTH(TEL) = 9 AND `STATUS` = 'AKTYWNY' AND NAZWA_SCRAPERA is NOT NULL AND Branza like '%%%s%%'"% search_key
result, count = query(sql_search, sql_search_len)
data = result[start:start + length]
else:
sql = """
SELECT NAZWA,TEL,NIP,Adres,WWW, EMAIL, Branza, NAZWA_SCRAPERA FROM gus_all t1 WHERE NOT EXISTS (SELECT * FROM matka_new t2 WHERE t2.NIP = t1.NIP) AND LENGTH(TEL) = 9 AND `STATUS` = 'AKTYWNY' AND NAZWA_SCRAPERA is NOT NULL
LIMIT %s OFFSET %s
"""
data = select_by_page(length, start, sql)
# data = select_by_page(start, start + length)
sql_len = "SELECT COUNT(*) FROM gus_all t1 WHERE NOT EXISTS (SELECT * FROM matka_new t2 WHERE t2.NIP = t1.NIP) AND LENGTH(TEL) = 9 AND `STATUS` = 'AKTYWNY' AND NAZWA_SCRAPERA is NOT NULL"
count = all_count(sql_len)
dic = {
'draw': draw,
'recordsTotal': count,
'recordsFiltered': count,
'data': data,
}
return HttpResponse(json.dumps(dic), content_type='application/json')
I have a timestamp in YYYY-MM-DD XX:XX:XX format in a csv file that is stored in S3 but when I use the timestamp data type to load into a Redshift database using Glue, the timestamp column is null. It appears that format is valid but I've also tried YYYYMMDD XXXXXX and YYMMDD XX:XX:XX formats as well just incase.
My mapping in Glue goes from timestamp to timestamp and the column datatype of the table is also timestamp. Ex of data in csv format:
1,2016 Summer,2016-06-22 00:00:00
Actual Output:
Line | Term | Date
-----+-------------+------------
1 | 2016 Summer |
Expected Output:
Line | Term | Date
-----+-------------+---------------------
1 | 2016 Summer | 2016-06-22 00:00:00
It seems that this should be a straightforward task but I can't get it right so if anyone else can find my mistake(s), that would be greatly appreciated.
Code:
val datasource37 = glueContext.getCatalogSource(database = "data", tableName = "term", redshiftTmpDir = "", transformationContext = "datasource37").getDynamicFrame()
val applymapping37 = datasource37.applyMapping(mappings = Seq(("id", "bigint", "id", "bigint"), ("name", "string", "name", "varchar(256)"), ("date", "timestamp", "date_start", "timestamp")), caseSensitive = false, transformationContext = "applymapping37")
val resolvechoice37 = applymapping37.resolveChoice(choiceOption = Some(ChoiceOption("make_cols")), transformationContext = "resolvechoice37")
val dropnullfields37 = resolvechoice37.dropNulls(transformationContext = "dropnullfields37")
val datasink37 = glueContext.getJDBCSink(catalogConnection = "dataConnection", options = JsonOptions("""{"dbtable": "term", "database": "data"}"""), redshiftTmpDir = args("TempDir"), transformationContext = "datasink37").writeDynamicFrame(dropnullfields37)
I ended up mapping from string -> timestamp and it worked. Glue had it automatically mapping from timestamp -> timestamp so I assumed it was right.
Ex:
val applymapping37 = datasource37.applyMapping
(mappings = Seq(("id", "bigint", "id", "bigint"),
("name", "string", "name", "varchar(256)"),
("date", "string", "date_start", "timestamp")),
caseSensitive = false, transformationContext = "applymapping37")
I have a Many2many type field in wizard('pack_ids') and also a Many2many('pack_id') type field in sale.order.line. And i want that the value of Many2many type field of wizard('pack_ids') return in sale.order.line field('pack_id').
For this my code is here:
class SalePackWizard(models.TransientModel):
_name = "sale.pack.wizard"
_description = "Sale Pack Wizard"
#api.onchange('product_id')
def _onchange_product_pack_name(self):
print"A:", self.product_id.product_pack
res = self.product_id.product_pack
a = {}
print "res:", res
if res:
domain = {'pack_ids': [('id', 'in', [v.id for v in res])]}
a= res
print "a:", a
return {'domain': domain}
product_id = fields.Many2one('product.product', string="Product Pack", required=True, domain="[('is_pack','=',True)]")
qty = fields.Float(string='Quantity', digits=dp.get_precision('Product Unit of Measure'), required=True, default=1.0)
pack_ids = fields.Many2many('product.pack', string='Pack Products', change_default=True,
default=_onchange_product_pack_name)
#api.multi
def action_salepack_add(self):
rec = self._context.get('active_ids', [])
print "REC", rec, self.product_id.categ_id #product_uom
if rec:
line_values = {'product_id': self.product_id.id,
#'design_id':self.design_id.id,
'pack_id': self.product_id.product_pack,
'category_id':self.product_id.categ_id.id,
'order_id':rec[0],
'product_uom_qty':self.qty,
}
sale_order_line = self.env['sale.order.line'].create(line_values)
you can update your code by that:
#api.multi
def action_salepack_add(self):
order_id = self._context.get('active_id',False)
if order_id:
line_values = {'product_id': self.product_id.id,
'pack_id': [ ( 6, 0, [self.product_id.product_pack.id] ) ],
'category_id':self.product_id.categ_id.id,
'order_id':order_id,
'product_uom_qty':self.qty,
}
sale_order_line = self.env['sale.order.line'].create(line_values)
You cant create values in a many2many field just giving it the id (this is only for many2one). If the field is a one2many or many2many:
(0, 0, { values }) link to a new record that needs to be created with the given values dictionary
(1, ID, { values }) update the linked record with id = ID (write values on it)
(2, ID) remove and delete the linked record with id = ID (calls unlink on ID, that will delete the object completely, and the link to it as well)
(3, ID) cut the link to the linked record with id = ID (delete the relationship between the two objects but does not delete the target object itself)
(4, ID) link to existing record with id = ID (adds a relationship)
(5) unlink all (like using (3,ID) for all linked records)
(6, 0, [IDs]) replace the list of linked IDs (like using (5) then (4,ID) for each ID in the list of IDs)
Error Fixed. Here is the solution for error:
#api.multi
def action_salepack_add(self):
rec = self._context.get('active_ids', [])
print "REC", rec, self.product_id.categ_id #product_uom
if rec:
line_values = {'product_id': self.product_id.id,
#'design_id':self.design_id.id,
'pack_id': [(6, 0, [v.id for v in self.product_id.product_pack])],
'category_id':self.product_id.categ_id.id,
'order_id':rec[0],
'product_uom_qty':self.qty,
}
sale_order_line = self.env['sale.order.line'].create(line_values)
Thanks,