As I understand it Clojure does not expose the reader macro table or allow user defined reader macros.
From http://clojure.org/reader:
The read table is currently not accessible to user programs.
I'm just wondering if there is a definitive or explicit statement (presumably from Rich Hickey) stating the rationale for leaving them out of Clojure.
Note I'm not asking if it is a good or bad thing that Clojure lacks user defined reader macros. Just wondering why.
From the link in matt's comments, to quote the answer by Rich Hickey, the author of Clojure:
I am unconvinced that reader macros are needed in Clojure at this
time. They greatly reduce the readability of code that uses them (by
people who otherwise know Clojure), encourage incompatible custom mini-
languages and dialects (vs namespace-partitioned macros), and
complicate loading and evaluation.
To the extent I'm willing to accommodate common needs different from
my own (e.g. regexes), I think many things that would otherwise have
forced people to reader macros may end up in Clojure, where everyone
can benefit from a common approach.
Clojure is arguably a very simple language, and in that simplicity
lies a different kind of power.
I'm going to pass on pursuing this for now,
Rich
Speaking straight there are Tagged Literals that allow you to specify what to do with next form. For example, you can add
{to/u clojure.string/upper-case}
to data_readers.clj (see docs) and write something like this:
testapp.core> #to/u "asd"
"ASD"
but it's not so powerful as full support of reader macros, at least because of
The data reader function is invoked on the form AFTER it has been read as a normal Clojure data structure by the reader.
I found this old log (don't ask me how)
http://clojure-log.n01se.net/date/2008-11-06.html
where there is a discussion with Rich Hickey's thoughts about reader macros.
Related
I'm reading some Clojure code at the moment that has a bunch of uninitialised values as nil for a numeric value in a record that gets passed around.
Now lots of the Clojure libraries treat this as idiomatic. Which means that it is an accepted convention.
But it also leads to NullPointerException, because not all the Clojure core functions can handle a nil as input. (Nor should they).
Other languages have the concept of Maybe or Option to proxy the value in the event that it is null, as a way of mitigating the NullPointerException risk. This is possible in Clojure - but not very common.
You can do some tricks with fnil but it doesn't solve every problem.
Another alternative is simply to set the uninitialised value to a symbol like :empty-value to force the user to handle this scenario explicitly in all the handling code. But this isn't really a big step-up from nil - because you don't really discover all the scenarios (in other people's code) until run-time.
My question is: Is there an idiomatic alternative to nil-punning in Clojure?
Not sure if you've read this lispcast post on nil-punning, but I do think it makes a pretty good case for why it's idiomatic and covers various important considerations that I didn't see mentioned in those other SO questions.
Basically, nil is a first-class thing in clojure. Despite its inherent conventional meaning, it is a proper value, and can be treated as such in many contexts, and in a context-dependent way. This makes it more flexible and powerful than null in the host language.
For example, something like this won't even compile in java:
if(null) {
....
}
Where as in clojure, (if nil ...) will work just fine. So there are many situations where you can use nil safely. I'm yet to see a java codebase that isn't littered with code like if(foo != null) { ... everywhere. Perhaps java 8's Optional will change this.
I think where you can run into issues quite easily is in java interop scenarios where you are dealing with actual nulls. A good clojure wrapper library can also help shield you from this in many cases, and its one good reason to prefer one over direct java interop where possible.
In light of this, you may want to re-consider fighting this current. But since you are asking about alternatives, here's one I think is great: prismatic's schema. Schema has a Maybe schema (and many other useful ones as well), and it works quite nicely in many scenarios. The library is quite popular and I have used it with success. FWIW, it is recommended in the recent clojure applied book.
Is there an idiomatic alternative to nil-punning in Clojure?
No. As leeor explains, nil-punning is idiomatic. But it's not as prevalent as in Common Lisp, where (I'm told) an empty list equates to nil.
Clojure used to work this way, but the CL functions that deal with lists correspond to Clojure functions that deal with sequences in general. And these sequences may be lazy, so there is a premium on unifying lazy sequences with others, so that any laziness can be preserved. I think this evolution happened about Clojure 1.2. Rich described it in detail here.
If you want option/maybe types, take a look at the core.typed library. In contrast to Prismatic Schema, this operates at compile time.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
Backstory: I've made a lot of large and relatively complex projects in Java, have a lot of experience in embedded C programming. I've got acquainted with scheme and CL syntax and wrote some simple programms with racket.
Question: I've planned a rather big project and want to do it in racket. I've heard a lot of "if you "get" lisp, you will become a better programmer", etc. But every time I try to plan or write a program I still "decompose" the task with familiar stateful objects with interfaces.
Are there "design patterns" for lisp? How to "get" lisp-family "mojo"? How to escape object-oriented constraint on your thinking? How to apply functional programming ideas boosted by powerful macro-facitilties? I tried studying source code of big projects on github (Light Table, for instance) and got more confused, rather than enlightened.
EDIT1 (less ambigious questions): is there a good literatue on the topic, that you can recommend or are there good open source projects written in cl/scheme/clojure that are of high quality and can serve as a good example?
A number of "paradigms" have come into fashion over the years:
structured programming, object oriented, functional, etc. More will come.
Even after a paradigm falls out of fashion, it can still be good at solving the particular problems that first made it popular.
So for example using OOP for a GUI is still natural. (Most GUI frameworks have a bunch of states modified by messages/events.)
Racket is multi-paradigm. It has a class system. I rarely use it,
but it's available when an OO approach makes sense for the problem.
Common Lisp has multimethods and CLOS. Clojure has multimethods and Java class interop.
And anyway, basic stateful OOP ~= mutating a variable in a closure:
#lang racket
;; My First Little Object
(define obj
(let ([val #f])
(match-lambda*
[(list) val]
[(list 'double) (set! val (* 2 val))]
[(list v) (set! val v)])))
obj ;#<procedure:obj>
(obj) ;#f
(obj 42)
(obj) ;42
(obj 'double)
(obj) ;84
Is this a great object system? No. But it helps you see that the essence of OOP is encapsulating state with functions that modify it. And you can do this in Lisp, easily.
What I'm getting at: I don't think using Lisp is about being "anti-OOP" or "pro-functional". Instead, it's a great way to play with (and use in production) the basic building blocks of programming. You can explore different paradigms. You can experiment with ideas like "code is data and vice versa".
I don't see Lisp as some sort of spiritual experience. At most, it's like Zen, and satori is the realization that all of these paradigms are just different sides of the same coin. They're all wonderful, and they all suck. The paradigm pointing at the solution, is not the solution. Blah blah blah. :)
My practical advice is, it sounds like you want to round out your experience with functional programming. If you must do this the first time on a big project, that's challenging. But in that case, try to break your program into pieces that "maintain state" vs. "calculate things". The latter are where you can try to focus on "being more functional". Look for opportunities to write pure functions. Chain them together. Learn how to use higher-order functions. And finally, connect them to the rest of your application -- which can continue to be stateful and OOP and imperative. That's OK, for now, and maybe forever.
A way to compare programming in OO vs Lisp (and "functional" programming in general) is to look at what each "paradigm" enables for the programmer.
One viewpoint in this line of reasoning, which looks at representations of data, is that the OO style makes it easier to extend data representations, but makes it more difficult to add operations on data. In contrast, the functional style makes it easier to add operations but harder to add new data representations.
Concretely, if there is a Printer interface, with OO, it's very easy to add a new HPPrinter class that implements the interface, but if you want to add a new method to an existing interface, you must edit every existing class that implements the interface, which is more difficult and may be impossible if the class definitions are hidden in a library.
In contrast, with the functional style, functions (instead of classes) are the unit of code, so one can easily add a new operation (just write a function). However, each function is responsible for dispatching according to the kind of input, so adding a new data representation requires editing all existing functions that operate on that kind of data.
Determining which style is more appropriate for your domain depends on whether you are more likely to add representations or operations.
This is a high-level generalization of course, and each style has developed solutions to cope with the tradeoffs mentioned (eg mixins for OO), but I think it still holds to a large degree.
Here is a well-known academic paper that captured the idea 25 years ago.
Here are some notes from a recent course (I taught) describing the same philosophy.
(Note that the course follows the How to Design Programs curriculum, which initially emphasizes the functional approach, but later transitions to the OO style.)
edit: Of course this only answers part of your question and does not address the (more or less orthogonal) topic of macros. For that I refer to Greg Hendershott's excellent tutorial.
A personal view:
If you parameterise an object design in the names of the classes and their methods - as you might do with C++ templates - then you end up with something that looks quite like a functional design. In other words, functional programming does not make useless distinctions between similar structures because their parts go by different names.
My exposure has been to Clojure, which tries to steal the good bit from object programming
working to interfaces
while discarding the dodgy and useless bits
concrete inheritance
traditional data hiding.
Opinions vary about how successful this programme has been.
Since Clojure is expressed in Java (or some equivalent), not only can objects do what functions can do, there is a regular mapping from one to the other.
So where can any functional advantage lie? I'd say expressiveness. There are lots of repetitive things you do in programs that are not worth capturing in Java - who used lambdas before Java provided compact syntax for them? Yet the mechanism was always there.
And Lisps have macros, which have the effect of making all structures first class. And there's a synergy between these aspects that you will enjoy.
The "Gang of 4" design patterns apply to the Lisp family just as much as they do to other languages. I use CL, so this is more of a CL perspective/commentary.
Here's the difference: Think in terms of methods that operate on families of types. That's what defgeneric and defmethod are all about. You should use defstruct and defclass as containers for your data, keeping in mind that all you really get are accessors to the data. defmethod is basically your usual class method (more or less) from the perspective of an operator on a group of classes or types (multiple inheritance.)
You'll find that you'll use defun and define a lot. That's normal. When you do see commonality in parameter lists and associated types, then you'll optimize using defgeneric/defmethod. (Look for CL quadtree code on github, for an example.)
Macros: Useful when you need to glue code around a set of forms. Like when you need to ensure that resources are reclaimed (closing files) or the C++ "protocol" style using protected virtual methods to ensure specific pre- and post-processing.
And, finally, don't hesitate to return a lambda to encapsulate internal machinery. That's probably the best way to implement an iterator ("let over lambda" style.)
Hope this gets you started.
I have been trying to think more about what abstraction actually means in functional programming. The very best post I have found that talks the type of language I can understand is the following 4 abstractions. However since I'm a wannabe Clojure programmer I'm wondering what sort of abstractions macros provide. It seems that they fit in stage 2 together with HOF but at the same time they are more then a HOF. I find stage 3 to be related to the Expression problem and would be protocol and multi methods in Clojure. So my question is:
When implementing a macro in a Lisp language what would you say you are abstracting over?
What would stage 3 and 4 be in a Lisp language?
I don't really view macros as an abstraction, but more as a compiler hook.
Most languages implement what is known as an Abstract Syntax Tree (or AST). This is a representation of the code of a program in a sort of data structure. Lisp macros expose parts of this AST as data that can be transformed via a macro function. But since lisp programs are themselves data structures, macros tend to be a bit cleaner in lisp programs then they would be in Rust or Scala.
So one could say that macros are simply abstractions of language semantics...but I don't know that I agree with that. One could say that macros are extensions of the lisp compiler, but that's not exactly true either.
As it turns out, macros are quite limited. They can only see a small subsection of the code being compiled. In other words, a macro can't see up the tree, only down. In addition while macros that perform deep inspection of children in the AST are possible (known as deep walking macros) these macros tend to be complex and error prone (just look at the guts of core.async's go or the contents of midje to see how complex these can get). So I hesitate to call them abstractions, perhaps they are, perhaps they are just very limited abstractions.
So I see macros as a weird mix between the more powerful Fexprs (http://en.wikipedia.org/wiki/Fexpr) and the more complete compiler code transforms found in projects like LLVM. They provide a very limited controlled way to transform code at compile time, that's about it.
And in the end it all comes down to the lisp mantra that "code is data is code". If your code is data it makes sense to provide ways to transform it at compile time.
For example pattern matching is a programming language feature that can be added to the clojure language through macros: http://www.brool.com/index.php/pattern-matching-in-clojure
What other language features can be added to the language?
Off the top of my hat I have two examples, but I'm sure there are more.
Contracts programming: https://github.com/fogus/trammel
Declarative logic: https://github.com/jduey/mini-kanren
I think its a stupid question to ask what can be added, what you should ask is what you cant add. Macros allow you to hook into the compiler that mean you can do almost anything.
At the moment you cant add your own syntax to the language. Clojure does not have a user extenseble reader, this means you don't have any reader-macros (http://dorophone.blogspot.com/2008/03/common-lisp-reader-macros-simple.html). This is not because of a technical problem but more a decition by Rich Hickey (the Clojure creator).
What you can not do is implement features that need virtual maschine support like add tail call semantics or goto.
If you want to see some stuff that has been done: Are there any Clojure DSLs?
Note that this list is not 100% up to date.
Edit:
Since you seem you took pattern matching as an example (it is a really good example for the power of macros) you should really look at the match library. Its probebly the best fastest pattern matching library in Clojure. http://vimeo.com/27860102
You can effectively add any language features you like.
This follows from the ability of macros to construct arbitrary code at compile time: as long as you can figure out what code you need to generate in order to implement your language features, it can be achieved with macros.
Some examples I've seen:
Query languages (Korma)
Logic programming (core.logic)
Image synthesis DSL (clisk)
Infix notation for arithmetic
Algebraic manipulation
Declarative definition of realtime data flows (Storm, Aleph)
Music programming (Overtone, Music As Data)
There are a few caveats:
If the feature isn't supported directly by the JVM (e.g. tail call optimisation in the mutually recursive case) then you'll have to emulate it. Not a big deal, but may have some performance impact.
If the feature requires a syntax not supported by the Clojure reader, you'll need to provide your own reader (since Clojure lacks an extensible reader at present). As a result, it's much easier if you stick to Clojure syntax/forms.
If you do anything too unusual / unidiomatic, it probably won't get picked up by others. There is a lot of value in sticking to standard Clojure conventions.
Beware of using macros where they are not needed. Often, just using normal functions (perhaps higher order functions) is sufficient to implement many new language features. The general rule is: "don't use macros unless you absolutely need to".
I find myself attached to a project to integerate an interpreter into an existing application. The language to be interpreted is a derivative of Lisp, with application-specific builtins. Individual 'programs' will be run batch-style in the application.
I'm surprised that over the years I've written a couple of compilers, and several data-language translators/parsers, but I've never actually written an interpreter before. The prototype is pretty far along, implemented as a syntax tree walker, in C++. I can probably influence the architecture beyond the prototype, but not the implementation language (C++). So, constraints:
implementation will be in C++
parsing will probably be handled with a yacc/bison grammar (it is now)
suggestions of full VM/Interpreter ecologies like NekoVM and LLVM are probably not practical for this project. Self-contained is better, even if this sounds like NIH.
What I'm really looking for is reading material on the fundamentals of implementing interpreters. I did some browsing of SO, and another site known as Lambda the Ultimate, though they are more oriented toward programming language theory.
Some of the tidbits I've gathered so far:
Lisp in Small Pieces, by Christian Queinnec. The person recommending it said it "goes from the trivial interpreter to more advanced techniques and finishes presenting bytecode and 'Scheme to C' compilers."
NekoVM. As I've mentioned above, I doubt that we'd be allowed to incorporate an entire VM framework to support this project.
Structure and Interpretation of Computer Programs. Originally I suggested that this might be overkill, but having worked through a healthy chunk, I agree with #JBF. Very informative, and mind-expanding.
On Lisp by Paul Graham. I've read this, and while it is an informative introduction to Lisp principles, is not enough to jump-start constructing an interpreter.
Parrot Implementation. This seems like a fun read. Not sure it will provide me with the fundamentals.
Scheme from Scratch. Peter Michaux is attacking various implementations of Scheme, from a quick-and-dirty Scheme interpreter written in C (for use as a bootstrap in later projects) to compiled Scheme code. Very interesting so far.
Language Implementation Patterns: Create Your Own Domain-Specific and General Programming Languages, recommended in the comment thread for Books On Creating Interpreted Languages. The book contains two chapters devoted to the practice of building interpreters, so I'm adding it to my reading queue.
New (and yet Old, i.e. 1979): Writing Interactive Compilers and Interpreters by P. J. Brown. This is long out of print, but is interesting in providing an outline of the various tasks associated with the implementation of a Basic interpreter. I've seen mixed reviews for this one but as it is cheap (I have it on order used for around $3.50) I'll give it a spin.
So how about it? Is there a good book that takes the neophyte by the hand and shows how to build an interpreter in C/C++ for a Lisp-like language? Do you have a preference for syntax-tree walkers or bytecode interpreters?
To answer #JBF:
the current prototype is an interpreter, and it makes sense to me as we're accepting a path to an arbitrary code file and executing it in our application environment. The builtins are used to affect our in-memory data representation.
it should not be hideously slow. The current tree walker seems acceptable.
The language is based on Lisp, but is not Lisp, so no standards compliance required.
As mentioned above, it's unlikely that we'll be allowed to add a full external VM/interpreter project to solve this problem.
To the other posters, I'll be checking out your citations as well. Thanks, all!
Short answer:
The fundamental reading list for a lisp interpreter is SICP. I would not at all call it overkill, if you feel you are overqualified for the first parts of the book jump to chapter 4 and start interpreting away (although I feel this would be a loss since chapters 1-3 really are that good!).
Add LISP in Small Pieces (LISP from now on), chapters 1-3. Especially chapter 3 if you need to implement any non-trivial control forms.
See this post by Jens Axel Søgaard on a minimal self-hosting Scheme: http://www.scheme.dk/blog/2006/12/self-evaluating-evaluator.html .
A slightly longer answer:
It is hard to give advice without knowing what you require from your interpreter.
does it really really need to be an interpreter, or do you actually need to be able to execute lisp code?
does it need to be fast?
does it need standards compliance? Common Lisp? R5RS? R6RS? Any SFRIs you need?
If you need anything more fancy than a simple syntax tree walker I would strongly recommend embedding a fast scheme subsystem. Gambit scheme comes to mind: http://dynamo.iro.umontreal.ca/~gambit/wiki/index.php/Main_Page .
If that is not an option chapter 5 in SICP and chapters 5-- in LISP target compilation for faster execution.
For faster interpretation I would take a look at the most recent JavaScript interpreters/compilers. There seem to be a lot of thought going into fast JavaScript execution, and you can probably learn from them. V8 cites two important papers: http://code.google.com/apis/v8/design.html and squirrelfish cites a couple: http://webkit.org/blog/189/announcing-squirrelfish/ .
There is also the canonical scheme papers: http://library.readscheme.org/page1.html for the RABBIT compiler.
If I engage in a bit of premature speculation, memory management might be the tough nut to crack. Nils M Holm has published a book "Scheme 9 from empty space" http://www.t3x.org/s9fes/ which includes a simple stop-the-world mark and sweep garbage collector. Source included.
John Rose (of newer JVM fame) has written a paper on integrating Scheme to C: http://library.readscheme.org/servlets/cite.ss?pattern=AcmDL-Ros-92 .
Yes on SICP.
I've done this task several times and here's what I'd do if I were you:
Design your memory model first. You'll want a GC system of some kind. It's WAAAAY easier to do this first than to bolt it on later.
Design your data structures. In my implementations, I've had a basic cons box with a number of base types: atom, string, number, list, bool, primitive-function.
Design your VM and be sure to keep the API clean. My last implementation had this as a top-level API (forgive the formatting - SO is pooching my preview)
ConsBoxFactory &GetConsBoxFactory() { return mConsFactory; }
AtomFactory &GetAtomFactory() { return mAtomFactory; }
Environment &GetEnvironment() { return mEnvironment; }
t_ConsBox *Read(iostream &stm);
t_ConsBox *Eval(t_ConsBox *box);
void Print(basic_ostream<char> &stm, t_ConsBox *box);
void RunProgram(char *program);
void RunProgram(iostream &stm);
RunProgram isn't needed - it's implemented in terms of Read, Eval, and Print. REPL is a common pattern for interpreters, especially LISP.
A ConsBoxFactory is available to make new cons boxes and to operate on them. An AtomFactory is used so that equivalent symbolic atoms map to exactly one object. An Environment is used to maintain the binding of symbols to cons boxes.
Most of your work should go into these three steps. Then you will find that your client code and support code starts to look very much like LISP too:
t_ConsBox *ConsBoxFactory::Cadr(t_ConsBox *list)
{
return Car(Cdr(list));
}
You can write the parser in yacc/lex, but why bother? Lisp is an incredibly simple grammar and scanner/recursive-descent parser pair for it is about two hours of work. The worst part is writing predicates to identify the tokens (ie, IsString, IsNumber, IsQuotedExpr, etc) and then writing routines to convert the tokens into cons boxes.
Make it easy to write glue into and out of C code and make it easy to debug issues when things go wrong.
The Kamin Interpreters from Samuel Kamin's book Programming Languages, An Interpreter-Based Approach, translated to C++ by Timothy Budd. I'm not sure how useful the bare source code will be, as it was meant to go with the book, but it's a fine book that covers the basics of implementing Lisp in a lower-level language, including garbage collection, etc. (That's not the focus of the book, which is programming languages in general, but it is covered.)
Lisp in Small Pieces goes into more depth, but that's both good and bad for your case. There's a lot of material on compiling and such that won't be relevant to you, and its simpler interpreters are in Scheme, not C++.
SICP is good, definitely. Not overkill, but of course writing interpreters is only a small fraction of the book.
The JScheme suggestion is a good one, too (and it incorporates some code by me), but won't help you with things like GC.
I might flesh this out with more suggestions later.
Edit: A few people have said they learned from my awklisp. This is admittedly kind of a weird suggestion, but it's very small, readable, actually usable, and unlike other tiny-yet-readable toy Lisps it implements its own garbage collector and data representation instead of relying on an underlying high-level implementation language to provide them.
Check out JScheme from Peter Norvig. I found this amazingly simple to understand and port to C++. Uh, dunno about using scheme as a scripting language though - teaching it to jnrs is cumbersome and feels dated (helloooo 1980's).
I would like to extend my recommendation for Programming Languages: Application and Interpretation. If you want to write an interpreter, that book takes you there in a very short path. If you read through writing the code you read and doing the exercise you end up with a bunch of similar interpreters but different (one is eager, the other is lazy, one is dynamic, the other has some typing, one has dynamic scope, the other has lexical scope, etc).