Hive left semi join with where clause - mapreduce

I'm running this query in Hive 13, but it won't compile. (It's continuing to throw ClassCastExceptions saying it can't be converted to a query.)
The query is
set hive.auto.convert.join=false;
create table J as
select c1, c2, c3, c4
from
A
left semi join B on (A.c1 = B.c1)
where ((c2 not in ('string_literal_1', 'string_literal_2', 'string_literal_3')) and
(c3 >= (cast('${hiveconf:start_time}' as bigint)-24*60*60)*1000));
Why won't this run?

Related

Combine two (tables or lists) to create output of unique combinations in Power BI

Trying to combine two queries in Power BI to have an output of unique combinations.
for instance one list(or column): A, B, C
and another list(or column): 1, 2, 3, 4, 5, 6, 7
Output should be: A1, A2, A3, A4, A5, A6, A7, B1, B2, B3, B4, B5, B6, B7, C1, C2, C3, C4, C5, C6, C7
Is there a way to accomplish this? (Yes my rows are not equal in count)
Just don't know the best or right approach for this (tried using combine with a helper column and hit a dead end as duplicates get created, unless I did that wrong)

This is essentially a Cartesian product (a.k.a. cross product) of two lists.
If you just have two text lists, you can do a one-liner like this:
List.Combine(List.Transform(List1, (L1) => List.Transform(List2, (L2) => L1 & L2)))
This says for each item X in the first list, create a list that is the second list with X prepended to each element. This gives a list of lists that is flattened out to a single list using the combine function.
It's not uncommon to want to do this with table though too. In this case, the analogous idea is to define a new column on one table where each row is the entire list/column of the other and then expand that new column.
Assuming we want the cross product of Table1[Column1] and Table2[Column2]:
let
Table1 = <Table1 Source>,
AddCustom = Table.AddColumn(Table1 , "Custom", each Table2),
Expand = Table.ExpandTableColumn(AddCustom, "Custom", {"Column2"}, {"Column2"}),
Concatenate = Table.AddColumn(Expand, "Concatenate", each [Column1] & [Column2])
in
Concatenate
Edit:
You can do the concatenation before the expand too:
let
Table1 = <Table1 Source>,
AddCustom = Table.AddColumn(Table1 , "Custom",
(T1) => List.Transform(Table2[Column2], each T1[Column1] & _)),
Expanded = Table.ExpandListColumn(AddCustom, "Custom")
in
Expanded
References with more detail:
Cartesian Product Joins
Cartesian Join of two queries...

Is it possible to use arrayformula to copy the values above if the cell on its left is empty?

I'm trying to populate a column (Cell C2) with the value of the cell in the column to its left (Cell B2) if B2 is not empty; if B2 is empty, then C2 equals to C1.
If B3 is not empty, then C3 equals to B3; if B3 is empty, C3 equals C2.
I try using an array formula, but it returns circular reference error. And I certainly can't use INDEX and INDIRECT either.
Please help.
The sample file:
https://docs.google.com/spreadsheets/d/1TXz5m5LtTF632bMrwIIDlO-4NMavgs6HA4FE8_BFhRs/edit?usp=sharing

paste in C3 cell:
=ARRAYFORMULA(IF(D3:D<>"", IF(ROW(B3:B) <= MAX(IF(D3:D<>"", ROW(B3:B))),
TEXT(VLOOKUP(ROW(B3:B), FILTER({ROW(B3:B), B3:B}, LEN(B3:B)), 2),
"dd mmm yyyy"), ), ))

You can use vlookup on the row number:
=ArrayFormula(to_date(if(D3:D="","",vlookup(row(A3:A),if(B3:B<>"",{row(A3:A),B3:B}),2,true))))

How do I clear the following Syntax Error in Amazon Redshift

I receive the following error message when attempting to run the following:
The database reported a syntax error:
Amazon Invalid operation: syntax error at or near "." Position: 3952;
select a.*,
b.*,
c.*,
a1."lte_tbl_employee_first_name" as "lte_idb_relocation.sent_by_first_name",
a1."lte_tbl_employee_last_name" as "lte_idb_relocation.sent_by_last_name",
a2."lte_tbl_employee_first_name" as "lte_idb_return.received_by_first_name",
a2."lte_tbl_employee_last_name" as "lte_idb_return.received_by_last_name",
a3."lte_tbl_employee_first_name" as "lte_idb_shipment.order_filler_first_name",
a3."lte_tbl_employee_last_name" as "lte_idb_shipment.order_filler_last_name",
a4."lte_tbl_employee_first_name" as "lte_idb_shipment.order_taker_first_name",
a4."lte_tbl_employee_last_name" as "lte_idb_shipment.order_taker_last_name",
a5."lte_tbl_employee_first_name" as "lte_idb_shipment.shipped_by_first_name",
a5."lte_tbl_employee_last_name" as "lte_idb_shipment.shipped_by_last_name",
a6."lte_tbl_employee_first_name" as "lte_idb_shipment.requestor_first_name",
a6."lte_tbl_employee_last_name" as "lte_idb_shipment.requestor_last_name",
a7."lte_tbl_employee_first_name" as "lte_idb_shipment.shipped_by_first_name",
a7."lte_tbl_employee_last_name" as "lte_idb_shipment.shipped_by_last_name",
a8."lte_tbl_employee_first_name" as "lte_idb_transaction.entered_by_first_name",
a8."lte_tbl_employee_last_name" as "lte_idb_transaction.entered_by_last_name",
b1."lte_tbl_hospital.name" as "lte_idb_invoice.customer_name",
b2."lte_tbl_hospital.name" as "lte_idb_return.returned_by_customer_name",
b3."lte_tbl_hospital.name" as "lte_idb_relocation.relocate_from_customer_name",
b4."lte_tbl_hospital.name" as "lte_idb_transaction.invoiced_customer_name",
b5."lte_tbl_hospital.name" as "lte_idb_shipment.shipped_to_customer_name",
c1."lte_tbl_return_code.description" as "lte_tbl_return_code.return_reason_description",
c2."lte_tbl_return_code.hold_code",
c3."lte_tbl_return_code.destroyed_at_hospital",
c4."lte_tbl_inventory_type.inventory_type",
c5."lte_tbl_inventory_type.description",
c6."tbl_transaction_status.status_description",
c7."lte_idb_transaction.transaction_id",
c8.CONCAT("lte_idb_transaction.transaction_number_prefix", "lte_idb_transaction.transaction_number_sequence") as "lte_idb_transaction.transaction",
c9."lte_idb_transaction.transaction_datetime",
c10."lte_idb_transaction.invoice_number",
c11."lte_idb_transaction.entered_by",
c12."lte_idb_transaction.transaction_status",
c13."lte_idb_transaction.invoiced_customer",
c14."lte_idb_invoice.total_amount",
c15."lte_idb_invoice.invoice_status",
c16."lte_idb_invoice.date_generated",
c17."lte_idb_invoice.customer_code",
c18."lte_idb_line_item.line_item_id",
c19."lte_idb_line_item.product_id",
c20."lte_idb_line_item.price",
c21."lte_idb_line_item.product_code",
c22."lte_idb_line_item.quantity",
c23."lte_idb_line_item.comments",
c24."lte_idb_line_item.expires",
c25."lte_idb_line_item.blood_type",
c26."lte_idb_line_item.biohazard",
c27."lte_idb_line_item.licensed",
c28."lte_idb_service.po_number",
c29."lte_idb_service.auto_price",
c30."lte_idb_service.debit",
c31."lte_idb_relocation.sent_by",
c32."lte_idb_relocation.auto_price",
c33."lte_idb_relocation.po_number",
c34."lte_idb_relocation.relocate_from",
c35."lte_idb_return.returned_by",
c36."lte_idb_return.received_by",
c37."lte_idb_return.return_code" as "lte_idb_return.return_reason",
c38."lte_idb_return.ra_number" as "lte_idb_return.r/a_number",
c39."lte_idb_return.auto_price",
c40."lte_idb_shipment.shipped_to",
c41."lte_idb_shipment.shipped_by",
c42."lte_idb_shipment.order_date",
c43."lte_idb_shipment.order_filler",
c44."lte_idb_shipment.order_taker",
c45."lte_idb_shipment.requestor",
c46."lte_idb_shipment.transportation_code",
c47."lte_idb_shipment.po_number",
c48."lte_idb_shipment.autologous",
c49."lte_idb_shipment.auto_price",
c50."lte_idb_shipment.emergency_shipment",
c51."lte_idb_shipment.order_number_external"
from "lte_idb_transaction_joined" c., "raw_lte_tbl_employee" a.,
"raw_lte_tbl_hospital" b.,
left JOIN "raw_lte_tbl_employee" a1
on c."lte_idb_relocation.sent_by" = a1."lte_tbl_employee_code"
left join "raw_lte_tbl_employee" a2
on c. "lte_idb_return.received_by" = a2."lte_tbl_employee_code"
left join "raw_lte_tbl_employee" a3
on c."lte_idb_shipment.order_filler" = a3."lte_tbl_employee_code"
left join "raw_lte_tbl_employee" a4
on c."lte_idb_shipment.order_taker" = a4."lte_tbl_employee_code"
left join "raw_lte_tbl_employee" a5
on c."lte_idb_shipment.shipped_by" = a5."lte_tbl_employee_code"
left join "raw_lte_tbl_employee" a6
on c."lte_idb_shipment.requestor" = a6."lte_tbl_employee_code"
left join "raw_lte_tbl_employee" a7
on c."lte_idb_shipment.shipped_by"= a7."lte_tbl_employee_code"
left join "raw_lte_tbl_employee" a8
on c."lte_idb_transaction.entered_by" = a8."lte_tbl_employee_code"
left join "raw_lte_tbl_hospital" b1
on c."lte_idb_invoice.customer_code" = b1."lte_tbl_hospital.code"
left join "raw_lte_tbl_hospital" b2
on c."lte_idb_return.returned_by" = b2."lte_tbl_hospital.code"
left join "raw_lte_tbl_hospital" b3
on c."lte_idb_relocation.relocate_from" = b3."lte_tbl_hospital.code"
left join "raw_lte_tbl_hospital" b4
on c."lte_idb_transaction.invoiced_customer" = b4."lte_tbl_hospital.code"
left join "raw_lte_tbl_hospital" b5
on c."lte_idb_shipment.shipped_to" = b5."lte_tbl_hospital.code"

This is likely to cause that type of problem:
from "lte_idb_transaction_joined" c.,
"raw_lte_tbl_employee" a.,
"raw_lte_tbl_hospital" b.,
There should not be a period after the alias name.

Creating a Measure that returns DISTINCT COUNT by Country and Brand in Power BI

I have a table like below:
Country, Advertiser, Brand, Spend
C1, A1, B1, 10
C1, A1, B2, 5
C1, A2, B3, 0
C1, A2, B4, 20
C2, A1, B1, 8
C2, A1, B5, 7
C2, A2, B6, 4
C2, A2, B3, 3
C2, A2, B7, 2
C2, A3, B8, 9
I'd like to get the DISTINCT COUNT of Brands by Country, which is simply:
C1 => 4
C2 => 6
How do I create a measure in Power BI that I can embed in a table or a tree-map? I have already tried this:
Measure = COUNTX(DISTINCT(sourceTable),sourceTable[Brand])
which returns total Brand count (not DISTINCT COUNT) per Country...
I have also tried below and it didn't work:
DistinctBrandCount =
CALCULATE(DISTINCTCOUNT(SampleDataForDashboard[Brand]),
GROUPBY(SampleDataForDashboard, SampleDataForDashboard[Country]))

StelioK is correct, you can just use a measure like this:
DistinctBrands = DISTINCTCOUNT(sourceTable[Brand])
When you put the measure into a table with Country on the rows, the filter context passed to the measure only includes the country for that row.
If you want to keep all of your other columns as well, you can write you measure like this:
DistinctBrands = CALCULATE(DISTINCTCOUNT(sourceTable[Brand]),
ALLEXCEPT(sourceTable, sourceTable[Country]))
This removes all filter context except for the Country and then calculates the distinct count.

Redshift: insert column C1 of table T1 into column C2 of Table T2

I have two tables:
T1 with columns A1, A2, A3, A4,...., A20.
T2 with columns B1, B2, B3,...., B15.
The data type of all columns is varchar.
I want to copy all values of column range A1-A10 to B1-B10. How do I do so in Redshift? I tried:
insert into T2(B1,B2,...,B10) select A1 A2 A3 ... A10 from T1
but it failed. I corrected errors like missing ), (dot) in the column name.
How can I insert selected column from one table to another? Is there any other way to do that?

You need to do insert into T2 (select A1, A2 ... A10 from T1).
I tested with following queries and things worked fine for me:
create temp table T1 (a varchar(5), b varchar(5), c varchar(5), d varchar(5), e varchar(5));
insert into T1 values ('t11', 't12', 't13', 't14', 't15');
create temp table T2 (a varchar(5), b varchar(5), c varchar(5));
insert into T2 values ('t21', 't22', 't23');
insert into T2 (select a, b, c from T1);
select * from T2;
The last line correctly printed the following:
t21 t22 t23
t11 t12 t13