I need to generate a value with a different type from my passed type. This is the first time I write on ocaml-like, and for example, in a familiar me haskell I would use Data.Generics.
How I have understood I need to use decorator and ppx. I wrote simple example
let recordHandler = (loc: Location.t, _recFlag: rec_flag, _t: type_declaration, fields: list(label_declaration)) => {
let (module Builder) = Ast_builder.make(loc);
let test = [%str
let schema: Schema = { name: "", _type: String, properties: [] }
]
let moduleExpr = Builder.pmod_structure(test);
[%str
module S = [%m moduleExpr]
]
}
let str_gen = (~loc, ~path as _, (_rec: rec_flag, t: list(type_declaration))) => {
let t = List.hd(t)
switch t.ptype_kind {
| Ptype_record(fields) => recordHandler(loc, _rec, t, fields);
| _ => Location.raise_errorf(~loc, "schema is used only for records.");
};
};
let name = "my_schema";
let () = {
let str_type_decl = Deriving.Generator.make_noarg(str_gen);
Deriving.add(name, ~str_type_decl) |> Deriving.ignore;
};
And
open Ppxlib;
let _ = Driver.run_as_ppx_rewriter()
But in using in rescript code
module User = {
#deriving(my_schema)
type my_typ = {
foo: int,
};
};
I caught:
schema is not supported
. And I made myself sure me to connect it right when I had changed #deriving(my_schema) for #deriving(abcd) and #deriving(sschema).
I got different error
Ppxlib.Deriving: 'abcd' is not a supported type deriving generator.
And my last experiment was to copy past existing library deriving accessors .
ppx_accessor
I copied-pasted it and renamed for accessors_2. And I got same error such as experiment.
accessors_2 is not supported
Also I haven't found examples "ppx rescript". Can you please help me.
What am I doing wrong (ALL , I know)
I have found answer in the article
Dropping support for custom PPXes such as ppx_deriving (the deriving
attribute is now exclusively interpreted as bs.deriving)
Related
It is clear to me that using Actors of course it is possible: for instance https://github.com/chbatey/akka-http-typed.git is using AkkaHttp and typed actors.
But it is unclear to me if just using AkkaStreams and its Alpakka connectors library (which includes databases), if is it possible to do regular CRUD / OLTP services, or just data replication from one database to another, or other OLAP / batch / stream processing scenarios.
If you know how it can be done please indicate a few details and if you can provide an example on github for instance that would be great.
The way I am thinking it may be possible is that the server is involved in two conversations / stateful stream transformation: one with the outside world over HTTP, and one with the database. I am not sure if this is possible to be modelled like that.
https://doc.akka.io/docs/alpakka/current/slick.html seems to offer both UPDATE/INSERTS as a Sink as well as pointed SELECT to a certain id as a Source. Do you know if an example app is there or can you broadly mention how the wiring would happen with Akka Http?
I put a demo here, hope it can help you.
Creating table, database is mysql.
CREATE TABLE test(id VARCHAR(32))
sbt:
"com.lightbend.akka" %% "akka-stream-alpakka-slick" % "1.1.0",
"mysql" % "mysql-connector-java" % "5.1.40"
Code:
package tech.parasol.scala.crud
import java.sql.SQLException
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.server.Directives.{complete, get, path, _}
import akka.stream.alpakka.slick.scaladsl.{Slick, SlickSession}
import akka.stream.scaladsl.Sink
import akka.stream.{ActorAttributes, ActorMaterializer, Supervision}
import com.typesafe.config.ConfigFactory
import scala.concurrent.Future
import scala.io.StdIn
import scala.util.{Failure, Success}
object CrudTest1 {
def main(args: Array[String]): Unit = {
implicit val system = ActorSystem("CrudTest1")
implicit val materializer = ActorMaterializer()
implicit val executionContext = system.dispatcher
val hostName = "120.0.0.1"
val rocketDbConfig =
s"""
|db-config {
| profile = "slick.jdbc.MySQLProfile$$"
| db {
| dataSourceClass = "slick.jdbc.DriverDataSource"
| properties = {
| driver = "com.mysql.jdbc.Driver"
| url = "jdbc:mysql://${hostName}:3306/rocket?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&useSSL=false"
| user = "root"
| password = "passw0rd"
| }
| }
|}
|
""".stripMargin
implicit val session = SlickSession.forConfig("db-config", ConfigFactory.parseString(rocketDbConfig))
import session.profile.api._
def persistence(message: String) = {
def insert(message: String): DBIO[Int] = {
sqlu"""INSERT INTO test(id) VALUES (${message})"""
}
session.db.run(insert(message)).map {
case _ => message
}.recover {
case e : SQLException => {
throw new Exception("Database error ===>")}
case e : Exception => {
throw new Exception("Database error.")}
}
}
val route = path("hello" / Segment ) { name =>
get {
val res = persistence(name)
onComplete(res) {
case Success(value) => {
complete(s"<h1>Say hello to ${name}</h1>")
}
case Failure(e) => {
complete(s"<h1>Failed to say hello to ${name}</h1>")
}
}
}
}
val bindingFuture = Http().bindAndHandle(route, "localhost", 8088)
println(s"Server online at http://localhost:8088/\nPress RETURN to stop...")
StdIn.readLine() // let it run until user presses return
bindingFuture
.flatMap(_.unbind()) // trigger unbinding from the port
.onComplete(_ => system.terminate()) // and shutdown when done
}
}
Yes, basically at every request receive in AkkaHttp, we create an AkkaStreams Graph (just a pipeline typically), basically just the Slick Alpakka Source from the database, maybe prefixed by some operators, and then returned in AkkaHttp, which of course supports Source. More details at [https://www.quora.com/Is-it-possible-to-build-an-OLTP-CRUD-HTTP-server-using-Akka-HTTP-Akka-Streams-Alpakka-and-a-database-Do-you-know-any-examples-of-code-on-GitHub-or-elsewhere/answer/Nicolae-Marasoiu]
I have two pieces of JSON that I've successfully decoded sequentially. I would like to take the new html_fragment and update my existing html_fragment. Generally this would be simple but my data structure is giving me difficulties:
type PostDataContainer
= PostDataContainer PostData
type alias PostData =
{ title : String
, comments : List Comment
}
type alias Comment =
{ comment_id : Int
, html_fragment : String
}
type alias CommentHtml =
{ id : Int
, html_fragment : String
}
I've just gotten CommentHtml and would like to update the existing html_fragment in Comment. This is what I have so far:
MergeCommentHtml commentHtmlData ->
case commentHtmlData of
Err err ->
Debug.log ("Error decoding CommentHtmlData" ++ toString err)
( mdl, Cmd.none )
Ok commentHtml ->
case mdl.maybePostDataContainer of
Just (PostDataContainer postData) ->
let
updatedCommentData = -- I dont know how to calculate this?
in
( { mdl | postData = { postData | comments = updatedCommentData } }, Cmd.none )
Note that commentHtml here is a List CommentHtml. Any thoughts on how to update my old comment.html_fragment with the new values in commentHtml?
Option 1:
just decode the data as it stands. When it's time to display it, arrange it appropriately via some function you write like rawJsonDataToNicerData.
Option 2:
Suppose you implement the following function:
-- given a new comment, and some PostData, return the new version of the PostData
updateData : CommentHtml -> PostData -> PostData
-- so now, assuming we can decode a CommentHtml with commentHtmlDeocder
-- we can do the following
dataUpdaterDecoder : Decoder (PostData -> PostData)
dataUpdaterDecoder
commentHtmlDecoder |> Decoder.andThen (\commentHtml -> updateData commentHtml)
Now wherever we were going to decode a commentHtmlDeocder we can decode a dataUpdaterDecoder instead, and use a bunch of these to update our data.
Here is an example of a relational data decoder in action using the idea above:
https://ellie-app.com/3KWmyJmMrDsa1
Given that commentHtmlData is a List according to a comment, I think the easiest approach is to convert it to a Dict keyed by id, then map over the existing comments looking for the comment_id in the dict. If it exists, replace html_fragment, if not then return the original unmodified:
let
commentHtmlDict =
commentHtmlData
|> List.map (\c -> (c.id, c))
|> Dict.fromList
updatedCommentData =
postData.comments
|> List.map (\comment ->
case Dict.get comment.comment_id commentHtmlDict of
Just commentHtml ->
{ comment | html_fragment = commentHtml.html_fragment }
Nothing ->
comment
)
I'm having a problem with a Swift 2 to 3 conversion piece of work and some of the remains syntax giving: Value of type '[Any]' has no member errors.
I was hoping someone could point me at a good solution.
Swift 2 code
Swift 2 code
func search() {
epsonPrinters = [Printer]()
starPrinters = [Printer]()
epson_startSearching()
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)) { [unowned self] in
let devices = SMPort.searchPrinter()
self.starPrinters = devices.map { portInfo -> Printer in
let p = Printer(
id: portInfo.modelName,
make: "Star Micronics",
model: portInfo.modelName,
portName: portInfo.portName)
if let name = portInfo.modelName as? String {
p.emulation = name.containsString("TSP143") ? "StarGraphics" : "StarLine"
}
return p
}
}
}
Swift 3 Code (I've added comments above areas with errors)
func search() {
epsonPrinters = [Printer]()
starPrinters = [Printer]()
epson_startSearching()
DispatchQueue.global(qos: .background).async { [unowned self] in
let devices = SMPort.searchPrinter()
self.starPrinters = [devices.map { portInfo -> Printer in
// id, model and portName in below fails with messages like:
// Value of type '[Any]' has no member 'modelName'
let p = Printer(
id: portInfo.modelName,
make: "Star Micronics",
model: portInfo.modelName,
portName: portInfo.portName)
// error on portInfo.modelName
// Value of type '[Any]' has no member 'modelName'
if let name = portInfo.modelName as? String {
p.emulation = name.containsString("TSP143") ? "StarGraphics" : "StarLine"
}
return p
}!]
}
}
I know that I can replace the 'id:...' part with the likes of:
id: ((portInfo[0] as AnyObject).modelName) ?? "",
But this isn't correct because PortInfo can have none, 1 or multiples depending on the number of printers we find.
I'd appreciate any suggestions for refactoring this in an elegant way that is good Swift 3 syntax and likely to survive into Swift 4.
I'm working in Xcode 8.3.2
When you get some errors about types, you'd better check what type each variable has. When you select devices in the line let devices = ..., Quick Help of Xcode will show you something like this:
Declaration let devices: [Any]?
First, it's an Optional and you need to unwrap it, before using the actual content.
Second, the type of the elements in devices is Any, to which you cannot apply any methods (including property accessors). You need to cast it to an appropriate type at an appropriate place.
To solve the two things above, you can write something like this:
guard let devices = SMPort.searchPrinter() as? [PortInfo] else {
fatalError("This may never happen")
}
With guard statement above, Quick Help will show you:
Declaration let devices: [PortInfo]
Non-Optional, simple Array of PortInfo, so you can use any methods of PortInfo for the elements of this devices.
I would translate your Swift 2 code into Swift 3 like this:
func search() {
epsonPrinters = []
starPrinters = []
epson_startSearching()
DispatchQueue.global(qos: .default).async {
guard let devices = SMPort.searchPrinter() as? [PortInfo] else {
fatalError("This may never happen")
}
self.starPrinters = devices.map { portInfo -> Printer in
let p = Printer(
id: portInfo.modelName,
make: "Star Micronics",
model: portInfo.modelName,
portName: portInfo.portName)
if let name = portInfo.modelName {
p.emulation = name.contains("TSP143") ? "StarGraphics" : "StarLine"
}
return p
}
}
}
You may need some fixes (not many, I believe) to use this code, as you are not showing all relevant things such as the definition of Printer.
I'm using Scalding with Spyglass to read from/write to HBase.
I'm doing a left outer join of table1 and table2 and write back to table1 after transforming a column.
Both table1 and table2 are declared as Spyglass HBaseSource.
This works fine. But, i need to access a different row in table1 using rowkey to compute transformed value.
I tried the following for HBase get:
val hTable = new HTable(conf, TABLE_NAME)
val result = hTable.get(new Get(rowKey.getBytes()))
I'm getting access to Configuration in Scalding job as mentioned in this link:
https://github.com/twitter/scalding/wiki/Frequently-asked-questions#how-do-i-access-the-jobconf
This works when i run the scalding job locally.
But, when i run it in cluster, conf is null when this code is executed in Reducer.
Is there a better way to do HBase get/scan in a Scalding/Cascading job for cases like this?
Ways to do this...
1) You can use a managed resource
class SomeJob(args: Args) extends Job(args) {
val someConfig = HBaseConfiguration.create().addResource(new Path(pathtoyourxmlfile))
lazy val hPool = new HTablePool(someConfig, 3)
def getConf = {
implicitly[Mode] match {
case Hdfs(_, conf) => conf
case _ => whateveryou are doing for a local conf...
}
}
... somePipe.someOperation.... {
val gets = key.map { key => new Get(key) }
managed(hPool.getTable("myTableName")) acquireAndGet { table =>
val results = table.get(gets)
...do something with these results
}
}
}
2) You can use some more specific cascading code, where you write a custom scheme and inside that you will override the source method and possibly some others depending on your needs. In there you can access the JobConf like this:
class MyScheme extends Scheme[JobConf, SomeRecordReader, SomeOutputCollector, ..] {
#transient var jobConf: Configuration = super.jobConfiguration
override def source(flowProcess: FlowProcess[JobConf], ...): Boolean = {
jobConf = flowProcess match {
case h: HadoopFlowProcess => h.getJobConf
case _ => jconf
}
... dosomething with the jobConf here
}
}
I use the Exchange webservices to extract attachments from exchange mailserver.
When i call the code on linux with mono a certain text attachment contain some mixed-up strings.
like so
"sam winglin vz" becomes "sainglin vz" -> so it is missing "m w".
I see this about 3 times in a 150kb file. 3 bytes are missing in the linux output vs the windows output.
When i extract it from visual studio the text attachment is perfect.
It is like this example
Save attachments from exchange inbox
Any idea in what direction i should look to fix this?
Code:
#r "Microsoft.Exchange.WebServices.dll"
open Microsoft
open Microsoft.Exchange.WebServices.Data
open System
open System.Net
type PgzExchangeService(url,user,password) =
let service = new ExchangeService(ExchangeVersion.Exchange2007_SP1,
TimeZoneInfo.CreateCustomTimeZone("Central Standard Time",new TimeSpan(-6, 0, 0),"(GMT-06:00) Central Time (US & Canada)","Central Standard Time"))
do
ServicePointManager.ServerCertificateValidationCallback <- ( fun _ _ _ _ -> true )
service.Url <- new Uri(url)
service.Credentials <- new WebCredentials(user, password, "domain")
member this.Service with get() = service
member this.InboxItems = this.Service.FindItems(WellKnownFolderName.Inbox, new ItemView(10))
member this.GetFileAttachments ( item : Item ) =
let emailMessage =
EmailMessage.Bind( this.Service,
item.Id,
new PropertySet(BasePropertySet.IdOnly, ItemSchema.Attachments))
item, emailMessage.Attachments |> Seq.choose (fun attachment -> match box attachment with
| :? FileAttachment as x -> Some(x) | _ -> None)
let mailAtdomain = new PgzExchangeService("https://xx.xx.XX.XX/EWS/Exchange.asmx", "user", "passw")
let printsave (item : Item ,att : seq<FileAttachment>) =
if (Seq.length att) > 0 then
printfn "%A - saving %i attachments" item.Subject (Seq.length att)
att |> Seq.iter ( fun attachment -> printfn "%A" attachment.Name
attachment.Load(#"/tmp/test/" + attachment.Name ) )
// filter so we only have items with attachements and ...
let itemsWithAttachments = mailAtdomain.InboxItems
|> Seq.map mailAtdomain.GetFileAttachments
|> Seq.iter printsave
The code doesn't run on Windows with mono due to a bug in TimeZoneInfo
This sample code runs on linux but not on windows. because of the TimeZoneInfo bug.
But with this the code that works on linux to extract attachments.
Try csv attachments and see if the result is the same. i loose data ! about 3 bytes every somemany lines
mail me if you need the sample csv attachment that gives the problem
Here is a c# version that i used for testing. Running from VS2010 it works perfect but on linux with mono the attachment has wrong size , some bytes are missing?!.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Exchange.WebServices.Data;
using System.Net;
namespace Exchange_SDP_Attachment_Extracter
{
public class PgzExchangeService
{
public void Extract()
{
ExchangeService service = new ExchangeService (ExchangeVersion.Exchange2007_SP1,TimeZoneInfo.Local);
service.Credentials = new NetworkCredential("user", "pass", "domain");
service.Url = new Uri("https://xx.xx.xx.xx/EWS/Exchange.asmx");
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
FindItemsResults<Item> findResults = service.FindItems(WellKnownFolderName.Inbox, new ItemView(10));
foreach (Item item in findResults.Items)
{
EmailMessage e = EmailMessage.Bind
(service,
item.Id,
new PropertySet(BasePropertySet.IdOnly, ItemSchema.Attachments));
foreach ( Attachment att in e.Attachments )
{
if (att is FileAttachment)
{
FileAttachment fileAttachment = (FileAttachment)att;
fileAttachment.Load(#"/tmp/testsdp/" + fileAttachment.Name);
}
}
}
}
}
class Program
{
static void Main(string[] args)
{
PgzExchangeService pgz = new PgzExchangeService();
pgz.Extract();
}
}
}
My suggestion would be to try examining the text attachment with a hex editor. There has be something unusual about those three occurrences. You need to find out what those three lines have in common before any of us can recommend a course of action to you.