I have the following OCaml program
open Js
let lex s = Compiler.Parse_js.lexer_from_file s
let parse s = lex s |> Compiler.Parse_js.parse
let buffer_pp program =
let buf = Buffer.create 10 in
let pp = Compiler.Pretty_print.to_buffer buf in
Compiler.Js_output.program pp program;
Buffer.contents buf |> print_endline
let () =
parse "test.js" |> buffer_pp
and the following JavaScript program
function person(first, last, age, eyecolor) {
this.firstName = first;
this.lastName = last;
this.age = age;
this.eyeColor = eyecolor;
}
person.prototype.name = function() {
return this.firstName + " " + this.lastName;
};
When running the compiled ocaml code, it prints out
function person(first,last,age,eyecolor)
{this.firstName=first;this.lastName=last;this.age=age;this.eyeColor=eyecolor}
person.prototype.name=function(){return this.firstName+" "+this.lastName};
Is there a way to do pretty-printing which displays format better?
You can disable compact mode with
Compiler.Pretty_print.set_compact pp false;
But, AFAIK, it is on by default.
There're also lots of external tools, that prettifies javascript, that you can use, if you're still not satisfied with the result.
Related
I am new to rust and I am trying to port golang code that I had written previosuly. The go code basically downloaded files from s3 and directly (without writing to disk) ungziped the files and parsed them.
Currently the only solution I found is to save the gzipped files on disk then ungzip and parse them.
Perfect pipeline would be to directly ungzip and parse them.
How can I accomplish this?
const ENV_CRED_KEY_ID: &str = "KEY_ID";
const ENV_CRED_KEY_SECRET: &str = "KEY_SECRET";
const BUCKET_NAME: &str = "bucketname";
const REGION: &str = "us-east-1";
use anyhow::{anyhow, bail, Context, Result}; // (xp) (thiserror in prod)
use aws_sdk_s3::{config, ByteStream, Client, Credentials, Region};
use std::env;
use std::io::{Write};
use tokio_stream::StreamExt;
#[tokio::main]
async fn main() -> Result<()> {
let client = get_aws_client(REGION)?;
let keys = list_keys(&client, BUCKET_NAME, "CELLDATA/year=2022/month=06/day=06/").await?;
println!("List:\n{}", keys.join("\n"));
let dir = Path::new("input/");
let key: &str = &keys[0];
download_file_bytes(&client, BUCKET_NAME, key, dir).await?;
println!("Downloaded {key} in directory {}", dir.display());
Ok(())
}
async fn download_file_bytes(client: &Client, bucket_name: &str, key: &str, dir: &Path) -> Result<()> {
// VALIDATE
if !dir.is_dir() {
bail!("Path {} is not a directory", dir.display());
}
// create file path and parent dir(s)
let mut file_path = dir.join(key);
let parent_dir = file_path
.parent()
.ok_or_else(|| anyhow!("Invalid parent dir for {:?}", file_path))?;
if !parent_dir.exists() {
create_dir_all(parent_dir)?;
}
file_path.set_extension("json");
// BUILD - aws request
let req = client.get_object().bucket(bucket_name).key(key);
// EXECUTE
let res = req.send().await?;
// STREAM result to file
let mut data: ByteStream = res.body;
let file = File::create(&file_path)?;
let Some(bytes)= data.try_next().await?;
let mut gzD = GzDecoder::new(&bytes);
let mut buf_writer = BufWriter::new( file);
while let Some(bytes) = data.try_next().await? {
buf_writer.write(&bytes)?;
}
buf_writer.flush()?;
Ok(())
}
fn get_aws_client(region: &str) -> Result<Client> {
// get the id/secret from env
let key_id = env::var(ENV_CRED_KEY_ID).context("Missing S3_KEY_ID")?;
let key_secret = env::var(ENV_CRED_KEY_SECRET).context("Missing S3_KEY_SECRET")?;
// build the aws cred
let cred = Credentials::new(key_id, key_secret, None, None, "loaded-from-custom-env");
// build the aws client
let region = Region::new(region.to_string());
let conf_builder = config::Builder::new().region(region).credentials_provider(cred);
let conf = conf_builder.build();
// build aws client
let client = Client::from_conf(conf);
Ok(client)
}
Your snippet doesn't tell where GzDecoder comes from, but I'll assume it's flate2::read::GzDecoder.
flate2::read::GzDecoder is already built in a way that it can wrap anything that implements std::io::Read:
GzDecoder::new expects an argument that implements Read => deflated data in
GzDecoder itself implements Read => inflated data out
Therefore, you can use it just like a BufReader: Wrap your reader and used the wrapped value in place:
use flate2::read::GzDecoder;
use std::fs::File;
use std::io::BufReader;
use std::io::Cursor;
fn main() {
let data = [0, 1, 2, 3];
// Something that implements `std::io::Read`
let c = Cursor::new(data);
// A dummy output
let mut out_file = File::create("/tmp/out").unwrap();
// Using the raw data would look like this:
// std::io::copy(&mut c, &mut out_file).unwrap();
// To inflate on the fly, "pipe" the data through the decoder, i.e. wrap the reader
let mut stream = GzDecoder::new(c);
// Consume the `Read`er somehow
std::io::copy(&mut stream, &mut out_file).unwrap();
}
playground
You don't mention what "and parse them" entails, but the same concept applies: If your parser can read from an impl Read (e.g. it can read from a std::fs::File), then it can also read directly from a GzDecoder.
I have this simple stream processor (not a consumer/producer) using kafka streams binder.
#Bean
fun processFoo():Function<KStream<FooName, FooAddress>, KStream<FooName, FooAddressPlus>> {
return Function { input-> input.map { key, value ->
println("\nPAYLOAD KEY: ${key.name}\n");
println("\nPAYLOAD value: ${value.address}\n");
val output = FooAddressPlus()
output.address = value.address
output.name = value.name
output.plus = "$value.name-$value.address"
KeyValue(key, output)
}}
}
I'm trying to test it using the TopologyTestDriver:
#SpringBootTest(
webEnvironment = SpringBootTest.WebEnvironment.NONE,
classes = [Application::class, FooProcessor::class]
)
class FooProcessorTests {
var testDriver: TopologyTestDriver? = null
val INPUT_TOPIC = "input"
val OUTPUT_TOPIC = "output"
val inputKeySerde: Serde<FooName> = JsonSerde<FooName>()
val inputValueSerde: Serde<FooAddress> = JsonSerde<FooAddress>()
val outputKeySerde: Serde<FooName> = JsonSerde<FooName>()
val outputValueSerde: Serde<FooAddressPlus> = JsonSerde<FooAddressPlus>()
fun getStreamsConfiguration(): Properties? {
val streamsConfiguration = Properties()
streamsConfiguration[StreamsConfig.APPLICATION_ID_CONFIG] = "TopologyTestDriver"
streamsConfiguration[StreamsConfig.BOOTSTRAP_SERVERS_CONFIG] = "dummy:1234"
streamsConfiguration[JsonDeserializer.TRUSTED_PACKAGES] = "*"
streamsConfiguration["spring.kafka.consumer.properties.spring.json.trusted.packages"] = "*"
return streamsConfiguration
}
#Before
fun setup() {
val builder = StreamsBuilder()
val input: KStream<FooName, FooAddress> = builder.stream(INPUT_TOPIC, Consumed.with(inputKeySerde, inputValueSerde))
val processor = FooProcessor()
val output: KStream<FooName, FooAddressPlus> = processor.processFoo().apply(input)
output.to(OUTPUT_TOPIC, Produced.with(outputKeySerde, outputValueSerde))
testDriver = TopologyTestDriver(builder.build(), getStreamsConfiguration())
}
#After
fun tearDown() {
try {
testDriver!!.close()
} catch (e: RuntimeException) {
// https://issues.apache.org/jira/browse/KAFKA-6647 causes exception when executed in Windows, ignoring it
// Logged stacktrace cannot be avoided
println("Ignoring exception, test failing in Windows due this exception:" + e.localizedMessage)
}
}
#org.junit.Test
fun testOne() {
val inputTopic: TestInputTopic<FooName, FooAddress> =
testDriver!!.createInputTopic(INPUT_TOPIC, inputKeySerde.serializer(), inputValueSerde.serializer())
val key = FooName()
key.name = "sherlock"
val value = FooAddress()
value.name = "sherlock"
value.address = "Baker street"
inputTopic.pipeInput(key, value)
val outputTopic: TestOutputTopic<FooName, FooAddressPlus> =
testDriver!!.createOutputTopic(OUTPUT_TOPIC, outputKeySerde.deserializer(), outputValueSerde.deserializer())
val message = outputTopic.readValue()
assertThat(message.name).isEqualTo(key.name)
assertThat(message.address).isEqualTo(value.address)
}
}
When running it, I get this error in line inputTopic.pipeInput(key, value)
The class 'package.FooAddress' is not in the trusted packages: [java.util, java.lang]. If you believe this class is safe to deserialize, please provide its name. If the serialization is only done by a trusted source, you can also enable trust all ().*
Any ideas on how to solve this? Setting those properties in getStreamsConfiguration() is not helping. Please note that this is a stream processor, not a consumer/producer.
Thanks a lot!
When Kafka creates the Serde itself, it applies the properties by calling configure().
Since you are instantiating the Serde yourself, you need to call configure() on it passing in the map of properties.
That's how the trusted packages property gets propagated to the deserializer.
Or, you can call setTrustedPackages() on the deserializer.
So, for completeness, here's how the code looks when configuring the serde as #GaryRussell suggests:
private fun getStreamsConfiguration(): Properties? {
// Don't set the trusted packages here since topology test driver does not know about Spring
val streamsConfiguration = Properties()
streamsConfiguration[StreamsConfig.APPLICATION_ID_CONFIG] = "TopologyTestDriver"
streamsConfiguration[StreamsConfig.BOOTSTRAP_SERVERS_CONFIG] = "dummy:1234"
}
#Before
fun setup() {
val builder = StreamsBuilder()
// Set the trusted packages for all serdes
val config = mapOf<String, String>(JsonDeserializer.TRUSTED_PACKAGES to "*")
inputKeySerde.configure(config, true)
inputValueSerde.configure(config, false)
outputKeySerde.configure(config, true)
outputValueSerde.configure(config, false)
}
And the rest of the code remains as described in the question. All credit to #GaryRusell.
I am reading my data through a web service that returns a percentage as a string and I need to format it properly in my app. Currently I receive an error message that reads "Argument labels '(_:)' do not match any available overloads".
Suggestions on how to resolve this issue?
if let dUnInsured = result[0]["UnInsured"] as? String, let doubleNum = Double(dUnInsured) {
let sUnInsured = dollarFormatter.string(from: (NSDecimalNumber(Decimal(doubleNum))))!
self.inUninsured.text = sUnInsured
}
Try like this way.
if let dUnInsured = result[0]["UnInsured"] as? String, let doubleNum = Double(dUnInsured) {
let sUnInsured = dollarFormatter.string(from: (NSNumber(value: doubleNum)))!
self.inUninsured.text = sUnInsured
}
I'm trying to write a string to a file, however I cant seem to get it working, I've read though all the questions like this on stack overflow but none seem to be addressing the issue. I'm from an imperative background so usually I would, write to file, then close the output stream... However this doest work in sml.
fun printToFile pathOfFile str = printToOutstream (TextIO.openOut pathOfFile) str;
//Here is where the issues start coming in
fun printToOutStream outstream str = TextIO.output (outstream, str)
TextIO.closeOut outstream
//will not work. I've also tried
fun printToOutStream outstream str = let val os = outStream
in
TextIO.output(os,str)
TextIO.closeOut os
end;
//also wont work.
I know I need to write to the file and close the output stream but I cant figure out how to do it. Using my "sml brain" I'm telling myself I need to call the function recursively stepping towards something and then when I reach it close the output stream... but again I don't have a clue how I would do this.
You're almost there. Between the in and end you need to delimit the expressions by the semicolon. In SML ; is a sequence-operator. It evaluates expressions in turn and then only returns the value of the last one.
If you already have an outstream open, use:
fun printToOutStream outstream str = let val os = outstream
in
TextIO.output(os,str);
TextIO.closeOut os
end;
Used like thus:
- val os = TextIO.openOut "C:/programs/testfile.txt";
val os = - : TextIO.outstream
- printToOutStream os "Hello SML IO";
val it = () : unit
Then when I go to "C:/programs" I see a brand new text file that looks like this:
If you always read/write complete files at once, you could create some helper functions for this, like:
fun readFile filename =
let val fd = TextIO.openIn filename
val content = TextIO.inputAll fd handle e => (TextIO.closeIn fd; raise e)
val _ = TextIO.closeIn fd
in content end
fun writeFile filename content =
let val fd = TextIO.openOut filename
val _ = TextIO.output (fd, content) handle e => (TextIO.closeOut fd; raise e)
val _ = TextIO.closeOut fd
in () end
I use the Exchange webservices to extract attachments from exchange mailserver.
When i call the code on linux with mono a certain text attachment contain some mixed-up strings.
like so
"sam winglin vz" becomes "sainglin vz" -> so it is missing "m w".
I see this about 3 times in a 150kb file. 3 bytes are missing in the linux output vs the windows output.
When i extract it from visual studio the text attachment is perfect.
It is like this example
Save attachments from exchange inbox
Any idea in what direction i should look to fix this?
Code:
#r "Microsoft.Exchange.WebServices.dll"
open Microsoft
open Microsoft.Exchange.WebServices.Data
open System
open System.Net
type PgzExchangeService(url,user,password) =
let service = new ExchangeService(ExchangeVersion.Exchange2007_SP1,
TimeZoneInfo.CreateCustomTimeZone("Central Standard Time",new TimeSpan(-6, 0, 0),"(GMT-06:00) Central Time (US & Canada)","Central Standard Time"))
do
ServicePointManager.ServerCertificateValidationCallback <- ( fun _ _ _ _ -> true )
service.Url <- new Uri(url)
service.Credentials <- new WebCredentials(user, password, "domain")
member this.Service with get() = service
member this.InboxItems = this.Service.FindItems(WellKnownFolderName.Inbox, new ItemView(10))
member this.GetFileAttachments ( item : Item ) =
let emailMessage =
EmailMessage.Bind( this.Service,
item.Id,
new PropertySet(BasePropertySet.IdOnly, ItemSchema.Attachments))
item, emailMessage.Attachments |> Seq.choose (fun attachment -> match box attachment with
| :? FileAttachment as x -> Some(x) | _ -> None)
let mailAtdomain = new PgzExchangeService("https://xx.xx.XX.XX/EWS/Exchange.asmx", "user", "passw")
let printsave (item : Item ,att : seq<FileAttachment>) =
if (Seq.length att) > 0 then
printfn "%A - saving %i attachments" item.Subject (Seq.length att)
att |> Seq.iter ( fun attachment -> printfn "%A" attachment.Name
attachment.Load(#"/tmp/test/" + attachment.Name ) )
// filter so we only have items with attachements and ...
let itemsWithAttachments = mailAtdomain.InboxItems
|> Seq.map mailAtdomain.GetFileAttachments
|> Seq.iter printsave
The code doesn't run on Windows with mono due to a bug in TimeZoneInfo
This sample code runs on linux but not on windows. because of the TimeZoneInfo bug.
But with this the code that works on linux to extract attachments.
Try csv attachments and see if the result is the same. i loose data ! about 3 bytes every somemany lines
mail me if you need the sample csv attachment that gives the problem
Here is a c# version that i used for testing. Running from VS2010 it works perfect but on linux with mono the attachment has wrong size , some bytes are missing?!.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Exchange.WebServices.Data;
using System.Net;
namespace Exchange_SDP_Attachment_Extracter
{
public class PgzExchangeService
{
public void Extract()
{
ExchangeService service = new ExchangeService (ExchangeVersion.Exchange2007_SP1,TimeZoneInfo.Local);
service.Credentials = new NetworkCredential("user", "pass", "domain");
service.Url = new Uri("https://xx.xx.xx.xx/EWS/Exchange.asmx");
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
FindItemsResults<Item> findResults = service.FindItems(WellKnownFolderName.Inbox, new ItemView(10));
foreach (Item item in findResults.Items)
{
EmailMessage e = EmailMessage.Bind
(service,
item.Id,
new PropertySet(BasePropertySet.IdOnly, ItemSchema.Attachments));
foreach ( Attachment att in e.Attachments )
{
if (att is FileAttachment)
{
FileAttachment fileAttachment = (FileAttachment)att;
fileAttachment.Load(#"/tmp/testsdp/" + fileAttachment.Name);
}
}
}
}
}
class Program
{
static void Main(string[] args)
{
PgzExchangeService pgz = new PgzExchangeService();
pgz.Extract();
}
}
}
My suggestion would be to try examining the text attachment with a hex editor. There has be something unusual about those three occurrences. You need to find out what those three lines have in common before any of us can recommend a course of action to you.