Suppose I am writing an OCaml program and my input will be a large stream of integers separated by spaces i.e.
let string = input_line stdin;;
will return a string which looks like e.g. "2 4 34 765 5 ..." Now, the program itself will take a further two values i and j which specify a small subsequence of this input on which the main procedure will take place (let's say that the main procedure is the find the maximum of this sublist). In other words, the whole stream will be inputted into the program but the program will only end up acting on a small subset of the input.
My question is: what is the best way to translate the relevant part of the input stream into something usable i.e. a string of ints? One option would be to convert the whole input string into a list of ints using
let list = List.map int_of_string(Str.split (Str.regexp_string " ") string;;
and then once the bounds i and j have been entered one easily locates the relevant sublist and its maximum. The problem is that the initial pre-processing of the large stream is immensely time-consuming.
Is there an efficient way of locating the small sublist directly from the large stream i.e. processing the input along with the main procedure?
OCaml's standard library is rather small. It provides necessary and sufficient set of orthogonal features, as should do any good standard library. But, usually, this is not enough for a casual user. That's why there exist libraries, that do the stuff, that is rather common.
I would like to mention two the most prominent libraries: Jane Street's Core library and Batteries included (aka Core and Batteries).
Both libraries provides a bunch of high-level I/O functions, but there exists a little problem. It is not possible or even reasonable to try to address any use case in a library. Otherwise the library's interface wont be terse and comprehensible. And your case is non-standard. There is a convention, a tacit agreement between data engineers, to represent a set of things with a set of lines in a file. And to represent one "thing" (or a feature) with a line. So, if you have a dataset where each element is a scalar, you should represent it as a sequence of scalars separated by a newline. Several elements on a single line is only for multidimensional features.
So, with a proper representation, your problem can be solve as simple as (with Core):
open Core.Std
let () =
let filename = "data" in
let max_number =
let open In_channel in
with_file filename
~f:(fold_lines ~init:0
~f:(fun m s -> Int.(max m ## of_string s))) in
printf "Max number is %s is %d\n" filename max_number
You can compile and run this program with corebuild test.byte -- assuming that code is in a file name test.byte and core library is installed (with opam install core if you're using opam).
Also, there exists an excellent library Lwt, that provides a monadic high-level interface to the I/O. With this library, you can parse a set of scalars in a following way:
open Lwt
let program =
let filename = "data" in
let lines = Lwt_io.lines_of_file filename in
Lwt_stream.fold (fun s m -> max m ## int_of_string s) lines 0 >>=
Lwt_io.printf "Max number is %s is %d\n" filename
let () = Lwt_main.run program
This program can be compiled and run with ocamlbuild -package lwt.unix test.byte --, if lwt library is installed on your system (opam install lwt).
So, that is not to say, that your problem cannot be solved (or is hard to be solved) in OCaml, it is just to mention, that you should start with a proper representation. But, suppose, you do not own the representation, and cannot change it. Let's look, how this can be solved efficiently with OCaml. As previous examples represent, in general your problem can be described as a channel folding, i.e. an consequential application of a function f to each value in a file. So, we can define a function fold_channel, that will read an integer value from a channel and apply a function to it and the previously read value. Of course, this function can be further abstracted, by lifting the format argument, but for the demonstration purpose, I suppose, this will be enough.
let rec fold_channel f init ic =
try Scanf.fscanf ic "%u " (fun s -> fold_channel f (f s init) ic)
with End_of_file -> init
let () =
let max_value = open_in "atad" |> fold_channel max 0 in
Printf.printf "max value is %u\n" max_value
Although, I should note that this implementation is not for a heavy duty work. It is even not tail-recursive. If you need really efficient lexer, you can use ocaml's lexer generator, for example.
Update 1
Since there is a word "efficient" in the title, and everybody likes benchmarks, I've decided to compare this three implementations. Of course, since pure OCaml implementation is not tail-recursive it is not comparable to others. You may wonder, why it is not tail-recursive, as all calls to fold_channel is in a tail position. The problem is with exception handler - on each call to the fold channel, we need to remember the init value, since we're going to return it. This is a common issue with recursion and exceptions, you may google it for more examples and explanations.
So, at first we need to fix the third implementation. We will use a common trick with option value.
let id x = x
let read_int ic =
try Some (Scanf.fscanf ic "%u " id) with End_of_file -> None
let rec fold_channel f init ic =
match read_int ic with
| Some s -> fold_channel f (f s init) ic
| None -> init
let () =
let max_value = open_in "atad" |> fold_channel max 0 in
Printf.printf "max value is %u\n" max_value
So, with a new tail-recursive implementation, let's try them all on a big-data. 100_000_000 numbers is a big data for my 7 years old laptop. I've also added a C implementations as a baseline, and an OCaml clone of the C implementation:
let () =
let m = ref 0 in
try
let ic = open_in "atad" in
while true do
let n = Scanf.fscanf ic "%d " (fun x -> x) in
m := max n !m;
done
with End_of_file ->
Printf.printf "max value is %u\n" !m;
close_in ic
Update 2
Yet another implementation, that uses ocamllex. It consists of two files, a lexer specification lex_int.mll
{}
let digit = ['0'-'9']
let space = [' ' '\t' '\n']*
rule next = parse
| eof {None}
| space {next lexbuf}
| digit+ as n {Some (int_of_string n)}
{}
And the implementation:
let rec fold_channel f init buf =
match Lex_int.next buf with
| Some s -> fold_channel f (f s init) buf
| None -> init
let () =
let max_value = open_in "atad" |>
Lexing.from_channel |>
fold_channel max 0 in
Printf.printf "max value is %u\n" max_value
And here are the results:
implementation time ratio rate (MB/s)
plain C 22 s 1.0 12.5
ocamllex 33 s 1.5 8.4
Core 62 s 2.8 4.5
C-like OCaml 83 s 3.7 3.3
fold_channel 84 s 3.8 3.3
Lwt 143 s 6.5 1.9
P.S. You can see, that in this particular case Lwt is an outlier. This doesn't mean that Lwt is slow, it is just not its granularity. And I would like to assure you, that to my experience Lwt is a well suited tool for a HPC. For example, in one of my programs it processes a 30 MB/s network stream in a real-time.
Update 3
By the way, I've tried to address the problem in an abstract way, and I didn't provide a solution for your particular example (with j and k). Since, folding is a generalization of the iteration, it can be easily solved by extending the state (parameter init) to hold a counter and check whether it is contained in a range, that was specified by a user. But, this leads to an interesting consequence: what to do, when you have outran the range? Of course, you can continue to the end, just ignoring the output. Or you can non-locally exit from a function with an exception, something like raise (Done m). Core library provides such facility with a with_return function, that allows you to break out of your computation at any point.
open Core.Std
let () =
let filename = "data" in
let b1,b2 = Int.(of_string Sys.argv.(1), of_string Sys.argv.(2)) in
let range = Interval.Int.create b1 b2 in
let _,max_number =
let open In_channel in
with_return begin fun call ->
with_file filename
~f:(fold_lines ~init:(0,0)
~f:(fun (i,m) s ->
match Interval.Int.compare_value range i with
| `Below -> i+1,m
| `Within -> i+1, Int.(max m ## of_string s)
| `Above -> call.return (i,m)
| `Interval_is_empty -> failwith "empty interval"))
end in
printf "Max number is %s is %d\n" filename max_number
You may use the Scanf module family of functions. For instance, Scanf.fscanf let you read tokens from a channel according to a string format (which is a special type in OCaml).
Your program can be decomposed in two functions:
one which skip a number i of tokens from the input channel,
one which extract the maximum integer out of a number j from a channel
Let's write these:
let rec skip_tokens c i =
match i with
| i when i > 0 -> Scanf.fscanf c "%s " (fun _ -> skip_tokens c ## pred i)
| _ -> ()
let rec get_max c j m =
match j with
| j when j > 0 -> Scanf.fscanf c "%d " (fun x -> max m x |> get_max c (pred j))
| _ -> m
Note the space after the token format indicator in the string which tells the scanner to also swallow all the spaces and carriage returns in between tokens.
All you need to do now is to combine them. Here's a small program you can run from the CLI which takes the i and j parameters, expects a stream of tokens, and print out the maximum value as wanted:
let _ =
let i = int_of_string Sys.argv.(1)
and j = int_of_string Sys.argv.(2) in
skip_tokens stdin (pred i);
get_max stdin j min_int |> print_int;
print_newline ()
You could probably write more flexible combinators by extracting the recursive part out. I'll leave this as an exercise for the reader.
Related
Some time ago, I decided to solve a simple task on HackerRank but using OCaml and Core, in order to learn them. In one of the tasks, I'm supposed to read data from standard input:
The first line contains an integer, denoting the number of entries
in the phone book. Each of the subsequent lines describes an entry in
the form of space-separated values on a single line. The first value
is a friend's name, and the second value is an -digit phone number.
After the lines of phone book entries, there are an unknown number of
lines of queries. Each line (query) contains a to look up, and you
must continue reading lines until there is no more input.
The main issues:
I don't know how many lines there will be
Last line don't ends by newline, so I can't just read scanf "%s\n" until End_of_file
And my code became messy:
open Core.Std
open Printf
open Scanf
let read_numbers n =
let phone_book = String.Table.create () ~size:n in
for i = 0 to (n - 1) do
match In_channel.input_line stdin with
| Some line -> (
match (String.split line ~on:' ') with
| key :: data :: _ -> Hashtbl.set phone_book ~key ~data
| _ -> failwith "This shouldn't happen"
)
| None -> failwith "This shouldn't happen"
done;
phone_book
let () =
let rec loop phone_book =
match In_channel.input_line stdin with
| Some line -> (
let s = match Hashtbl.find phone_book line with
| Some number -> sprintf "%s=%s" line number
| None -> "Not found"
in
printf "%s\n%!" s;
loop phone_book
)
| None -> ()
in
match In_channel.input_line stdin with
| Some n -> (
let phone_book = read_numbers (int_of_string n) in
loop phone_book
)
| None -> failwith "This shouldn't happen"
If I solve this task in Python, then code looks like this:
n = int(input())
book = dict([tuple(input().split(' ')) for _ in range(n)])
while True:
try:
name = input()
except EOFError:
break
else:
if name in book:
print('{}={}'.format(name, book[name]))
else:
print('Not found')
This is shorter and clearer than the OCaml code. Any advice on how to improve my OCaml code? And there two important things: I don't want to abandon OCaml, I just want to learn it; second - I want to use Core because of the same reason.
The direct implementation of the Python code in OCaml would look like this:
let exec name =
In_channel.(with_file name ~f:input_lines) |> function
| [] -> invalid_arg "Got empty file"
| x :: xs ->
let es,qs = List.split_n xs (Int.of_string x) in
let es = List.map es ~f:(fun entry -> match String.split ~on:' ' entry with
| [name; phone] -> name,phone
| _ -> invalid_arg "bad entry format") in
List.iter qs ~f:(fun name ->
match List.Assoc.find es name with
| None -> printf "Not found\n"
| Some phone -> printf "%s=%s\n" name phone)
However, OCaml is not a script-language for writing small scripts and one shot prototypes. It is the language for writing real software, that must be readable, supportable, testable, and maintainable. That's why we have types, modules, and all the stuff. So, if I were writing a production quality program, that is responsible for working with such input, then it will look very differently.
The general style that I personally employ, when I'm writing a program in a functional language is to follow these two simple rules:
When in doubt use more types.
Have fun (lots of fun).
I.e., allocate a type for each concept in the program domain, and use lots of small function.
The following code is twice as big, but is more readable, maintainable, and robust.
So, first of all, let's type: the entry is simply a record. I used a string type to represent a phone for simplicity.
type entry = {
name : string;
phone : string;
}
The query is not specified in the task, so let's just stub it with a string:
type query = Q of string
Now our parser state. We have three possible states: the Start state, a state Entry n, where we're parsing entries with n entries left so far, and Query state, when we're parsing queries.
type state =
| Start
| Entry of int
| Query
Now we need to write a function for each state, but first of all, let's define an error handling policy. For a simple program, I would suggest just to fail on a parser error. We will call a function named expect when our expectations fail:
let expect what got =
failwithf "Parser error: expected %s got %s\n" what got ()
Now the three parsing functions:
let parse_query s = Q s
let parse_entry s line = match String.split ~on:' ' line with
| [name;phone] -> {name;phone}
| _ -> expect "<name> <phone>" line
let parse_expected s =
try int_of_string s with exn ->
expect "<number-of-entries>" s
Now let's write the parser:
let parse (es,qs,state) input = match state with
| Start -> es,qs,Entry (parse_expected input)
| Entry 0 -> es,qs,Query
| Entry n -> parse_entry input :: es,qs,Entry (n-1)
| Query -> es, parse_query input :: qs,Query
And finally, let's read data from file:
let of_file name =
let es,qs,state =
In_channel.with_file name ~f:(fun ch ->
In_channel.fold_lines ch ~init:([],[],Start) ~f:parse) in
match state with
| Entry 0 | Query -> ()
| Start -> expect "<number-of-entries><br>..." "<empty>"
| Entry n -> expect (sprintf "%d entries" n) "fewer"
We also check that our state machine reached a proper finish state, that is it is either in Query or Entry 0 state.
As in Python, the key to a concise implementation is to let the standard library do most of the work; the following code uses Sequence.fold in lieu of Python's list comprehension. Also, using Pervasives.input_line rather than In_channel.input_line allows you to cut down on extraneous pattern matching (it will report an end of file condition as an exception rather than a None result).
open Core.Std
module Dict = Map.Make(String)
let n = int_of_string (input_line stdin)
let d = Sequence.fold
(Sequence.range 0 n)
~init:Dict.empty
~f:(fun d _ -> let line = input_line stdin in
Scanf.sscanf line "%s %s" (fun k v -> Dict.add d ~key:k ~data:v))
let () =
try while true do
let name = input_line stdin in
match Dict.find d name with
| Some number -> Printf.printf "%s=%s\n" name number
| None -> Printf.printf "Not found.\n"
done with End_of_file -> ()
I am basically trying to read a large file (around 10G) into a list of lines. The file contains a sequence of integer, something like this:
0x123456
0x123123
0x123123
.....
I used the method below to read files by default for my codebase, but it turns out to be quit slow (~12 minutes) at this scenario
let lines_from_file (filename : string) : string list =
let lines = ref [] in
let chan = open_in filename in
try
while true; do
lines := input_line chan :: !lines
done; []
with End_of_file ->
close_in chan;
List.rev !lines;;
I guess I need to read the file into memory, and then split them into lines (I am using a 128G server, so it should be fine for the memory space). But I still didn't understand whether OCaml provides such facility after searching the documents here.
So here is my question:
Given my situation, how to read files into string list in a fast way?
How about using stream? But I need to adjust related application code, then that could cause some time.
First of all you should consider whether you really need to have all the information at once in your memory. Maybe it is better to process file line-by-line?
If you really want to have it all at once in memory, then you can use Bigarray's map_file function to map a file as an array of characters. And then do something with it.
Also, as I see, this file contains numbers. Maybe it is better to allocate the array (or even better a bigarray) and the process each line in order and store integers in the (big)array.
I often use the two following function to read the lines of a file. Note that the function lines_from_files is tail-recursive.
let read_line i = try Some (input_line i) with End_of_file -> None
let lines_from_files filename =
let rec lines_from_files_aux i acc = match (read_line i) with
| None -> List.rev acc
| Some s -> lines_from_files_aux i (s :: acc) in
lines_from_files_aux (open_in filename) []
let () =
lines_from_files "foo"
|> List.iter (Printf.printf "lines = %s\n")
This should work:
let rec ints_from_file fdesc =
try
let l = input_line fdesc in
let l' = int_of_string l in
l' :: ints_from_file fdesc
with | _ -> []
This solution converts the strings to integers as they're read in (which should be a bit more memory efficient, and I assume this was going to be done to them eventually.
Also, because it is recursive, the file must be opened outside of the function call.
I'm a beginner with OCaml and I want to read lines from a file and then examine all characters in each line.
As a dummy example, let's say we want to count the occurrences of the character 'A' in a file.
I tried the following
open Core.Std
let count_a acc string =
let rec count_help res stream =
match Stream.peek stream with
| None -> res
| Some char -> Stream.junk stream; if char = 'A' then count_help (res+1) stream else count_help res stream
in acc + count_help 0 (Stream.of_string string)
let count_a = In_channel.fold_lines stdin ~init:0 ~f:count_a
let () = print_string ((string_of_int count_a)^"\n"
I compile it with
ocamlfind ocamlc -linkpkg -thread -package core -o solution solution.ml
run it with
$./solution < huge_file.txt
on a a file with one million lines which gives me the following times
real 0m16.337s
user 0m16.302s
sys 0m0.027s
which is 4 times more than my python implementation. I'm fairly sure that it should be possible to make this go faster, but I how should I go about doing this?
To count the number of A chars in a string you can just use String.count function. Indeed, the simpliest solution will be:
open Core.Std
let () =
In_channel.input_all stdin |>
String.count ~f:(fun c -> c = 'A') |>
printf "we have %d A's\n"
update
A slightly more complicated (and less memory hungry solution), with [fold_lines] will look like this:
let () =
In_channel.fold_lines stdin ~init:0 ~f:(fun n s ->
n + String.count ~f:(fun c -> c = 'A') s) |>
printf "we have %d A's\n"
Indeed, it is slower, than the previous one. It takes 7.3 seconds on my 8-year old laptop, to count 'A' in 20-megabyte text file. And 3 seconds on a former solution.
Also, you can find this post interesting, I hope.
I am trying to do something fairly simple. I want to take a string such as "1,000" and return the string "1000".
Here was my attempt:
String.map (function x -> if x = ',' then '' else x) "1,000";;
however I get a compiler error saying there is a syntax error wrt ''
Thanks for the insight!
Unfortunately, there's no character like the one you're looking for. There is a string that's 0 characters long (""), but there's no character that's not there at all. All characters (so to speak) are 1 character.
To solve your problem you need a more general operation than String.map. The essence of a map is that its input and output have the same shape but different contents. For strings this means that the input and output are strings of the same length.
Unless you really want to avoid imperative coding (which is actually a great thing to avoid, especially when starting out with OCaml), you would probably do best using String.iter and a buffer (from the Buffer module).
Update
The string_map_partial function given by Andreas Rossberg is pretty nice. Here's another implementation that uses String.iter and a buffer:
let string_map_partial f s =
let b = Buffer.create (String.length s) in
let addperhaps c =
match f c with
| None -> ()
| Some c' -> Buffer.add_char b c'
in
String.iter addperhaps s;
Buffer.contents b
Just an alternate implementation with different stylistic tradeoffs. Not faster, probably not slower either. It's still written imperatively (for the same reason).
What you'd need here is a function like the following, which unfortunately is not in the standard library:
(* string_map_partial : (char -> char option) -> string -> string *)
let string_map_partial f s =
let buf = String.create (String.length s) in
let j = ref 0 in
for i = 0 to String.length s - 1 do
match f s.[i] with
| None -> ()
| Some c -> buf.[!j] <- c; incr j
done;
String.sub buf 0 !j
You can then write:
string_map_partial (fun c -> if c = ',' then None else Some c) "1,000"
(Note: I chose an imperative implementation for string_map_partial, because a purely functional one would require repeated string concatenation, which is fairly expensive in OCaml.)
A purely functional version could be this one:
let string_map_partial f s =
let n = String.length s in
let rec map_str i acc =
if i < n then
map_str (i + 1) (acc ^ (f (String.make 1 s.[i])))
else acc
in map_str 0 ""
Which is terminal recursive, but less performant than the imperative version.
getChar :: Int -> IO Char
getChar n = do
c <- getLine
return (c !! n)
The program must needs a number and a line and it will return char, but how do I catch exception, if the number is too big?
I tried like this but it doesnt seem to work
getChar n
= do
c <-getLine
| n>=0 && n < b
= return c !! n
| otherwise
= error "Too big number"
where
b = length c
This is not a homework, im trying to involve myself. Google didint give me useful answers
Couldn't implement catch in there. Examples?
You probably want to restructure things a bit as you've got IO mixed up in something it doesn't have to be. What about changing the signature to something like this?
getChar :: Int -> String -> Maybe Char
getChar n x | n < length x = Just (x !! n)
| otherwise = Nothing
Data.Maybe allows you to indicate that you are either going to return something (e.g. the length is within range) or Nothing (the length isn't within range). The function that calls getChar can then decide what to do with things. Data.Either provides a way of returning an error message with an error instead. From what I've seen (and I'm by no means an expert) exceptions are rarely used in Haskell, and choice types such as Either or Maybe are much more commonly used.
Now in the code that calls this, you can use pattern matching to see what happened e.g.
main :: IO ()
main = do
x <- getLine
let z = getChar' 5 x
case z of
(Just z) -> print $ "The 5th character is " ++ show z
Nothing -> print $ "The 5th character is out of range"
You can use the drop function to drop the first n characters of the line (drop will just give an empty result if there are fewer than n chars), and the listToMaybe function to turn a list to a Maybe (either Just c where c is the first element of the list, or Nothing if the list is empty):
import Data.Maybe (listToMaybe)
getchar :: Int -> IO (Maybe Char)
getchar n = do
line <- getLine
return . listToMaybe . drop n $ line
getChar' :: Int -> IO Char
getChar' n =
do
c <- getLine
if (n < length c)
then
return (c !! n)
else
getChar' n
You can do something like above. This is just an example though. But, since you are a beginner, it is strongly recommended not to play with IO and Monads. You can come to it after you get familiarized with pure functional concepts.