I have been trying to think more about what abstraction actually means in functional programming. The very best post I have found that talks the type of language I can understand is the following 4 abstractions. However since I'm a wannabe Clojure programmer I'm wondering what sort of abstractions macros provide. It seems that they fit in stage 2 together with HOF but at the same time they are more then a HOF. I find stage 3 to be related to the Expression problem and would be protocol and multi methods in Clojure. So my question is:
When implementing a macro in a Lisp language what would you say you are abstracting over?
What would stage 3 and 4 be in a Lisp language?
I don't really view macros as an abstraction, but more as a compiler hook.
Most languages implement what is known as an Abstract Syntax Tree (or AST). This is a representation of the code of a program in a sort of data structure. Lisp macros expose parts of this AST as data that can be transformed via a macro function. But since lisp programs are themselves data structures, macros tend to be a bit cleaner in lisp programs then they would be in Rust or Scala.
So one could say that macros are simply abstractions of language semantics...but I don't know that I agree with that. One could say that macros are extensions of the lisp compiler, but that's not exactly true either.
As it turns out, macros are quite limited. They can only see a small subsection of the code being compiled. In other words, a macro can't see up the tree, only down. In addition while macros that perform deep inspection of children in the AST are possible (known as deep walking macros) these macros tend to be complex and error prone (just look at the guts of core.async's go or the contents of midje to see how complex these can get). So I hesitate to call them abstractions, perhaps they are, perhaps they are just very limited abstractions.
So I see macros as a weird mix between the more powerful Fexprs (http://en.wikipedia.org/wiki/Fexpr) and the more complete compiler code transforms found in projects like LLVM. They provide a very limited controlled way to transform code at compile time, that's about it.
And in the end it all comes down to the lisp mantra that "code is data is code". If your code is data it makes sense to provide ways to transform it at compile time.
Related
I know they are dialects of the same family of language called lisp, but what exactly are the differences? Could you give an overview, if possible, covering topics such as syntax, characteristics, features and resources.
They all have a lot in common:
Dynamic languages
Strongly typed
Compiled
Lisp-style syntax, i.e. code is written as a Lisp data structures (forms) with the most common pattern being function calls like: (function-name arg1 arg2)
Powerful macro systems that allow you to treat code as data and generate arbitrary code at runtime (often used to either "extend the language" with new syntax or create DSLs)
Often used in functional programming style, although have the ability to accommodate other paradigms
Emphasis in interactive development with a REPL (i.e. you interactively develop in a running instance of the code)
Common Lisp distinctive features:
A powerful OOP subsystem (Common Lisp Object System)
Probably the best compiler (Common Lisp is the fastest Lisp according to http://benchmarksgame.alioth.debian.org/u64q/which-programs-are-fastest.html although there isn't much in it.....)
Clojure distinctive features:
Largest library ecosystem, since you can directly use any Java libraries
Vectors [] and maps {} used as standard in addition to the standard lists () - in addition to the general usefullness of vectors and maps some believe this is a innovation which makes generally more readable
Greater emphasis on immutability and lazy functional programming, somewhat inspired by Haskell
Strong concurrency capabilities supported by software transactional memory at the language level (worth watching: http://www.infoq.com/presentations/Value-Identity-State-Rich-Hickey)
Scheme distinctive features:
Arguably the simplest and easiest to learn Lisp
Hygienic macros (see http://en.wikipedia.org/wiki/Hygienic_macro) - elegantly avoids the problems with accidental symbol capture in macro expansions
The people above missed a few things
Common Lisp has vectors and hash tables as well. The difference is that Common Lisp uses #() for vectors and no syntax for hash tables. Scheme has vectors, I believe
Common Lisp has reader macros, which allow you to use new brackets (as does Racket, a descendant of Scheme).
Scheme and Clojure have hygienic macros, as opposed to Common Lisp's unhygienic ones
All of the languages are either modern or have extensive renovation projects. Common Lisp has gotten extensive libraries in the past five years (thanks mostly to Quicklisp), Scheme has some modern implementations (Racket, Chicken, Chez Scheme, etc.), and Clojure was created relatively recently
Common Lisp has a built-in OO system, though it's quite different from other OO systems you might have used. Notably, it is not enforced--you don't have to write OO code.
The languages have somewhat different design philosophies. Scheme was designed as a minimal dialect for understanding the Actor Model; it later became used for pedagogy. Common Lisp was designed to unify the myriad Lisp dialects that had sprung up. Clojure was designed for concurrency. As a result, Scheme has a reputation of being minimal and elegant, Common Lisp of being powerful and paradigm-agnostic (functional, OO, whatever), and Clojure of favoring functional programming.
Don't forget about Lisp-1 and Lisp-2 differences.
Scheme and Clojure are Lisp-1:
That means both variables and functions names resides in same namespace.
Common Lisp is Lisp-2:
Function and variables has different namespaces (in fact, CL has many namespaces).
Gimp is written in Scheme :)
In fact allot of software some folks think might be written in C++ was probably done under the Lisp umbrella, its hard to pick out the golden apples out of the bunch. The fact is C++ was not always popular, it only seems to be popular today because of a history of updates. For the lesser half of the century C++ didn't even utilize multithreading, it was where Python is today a cesspool of useless untested buggy glue code. Fasterforward a little and now we are seeing a rise in functional programming, its more like adapt or die. I think Java has it right as far as the adapt part is concerned.
Scheme was designed to simplify the Lisp language, that was its only intent except it never really caught on. I think Clojure does something similar its meant to simplify Scheme for the JVM nothing more. Its just like every other JVM language just there to inflate the user experience, only to simplify writting boilerplate in Java land.
Now I'm generally in Java/C# (love both of them, can't really say I'm dedicated to one).
And I've recently been discussing the differences between F# and C# with a friend, when he surprised me saying: "So.. F# sounds a lot like lisp, but with way less 'Swiss-army knife' feel to it."
Now, I was partly ashamed of saying this but I have no idea what lisp was.
After some searching, I saw that lisp is very interesting, but got stumped by the multiple dialects and running environments.
Here is what I know:
I know of 3 dialects:
Common Lisp (I have the Practical Common Lisp book in my bookmarks.
Scheme (a more "theoretical" version of CL)
Clojure. Seems to be a version of CL that runs on JVM.
The basic idea of lisp seems to be about using code as data.
What I want to know:
What is the running environment for different dialects? How do they work/get installed (by this I mean is it a runtime like Java Virtual Machine, or if it requires something else, or if it's supported generally by the OS (as in compiled)). And how to get them (if something is to be gotten)
What is the better dialect to learn (I want the dialect not to be a "learning language" but one you can fully use afterwards without regret of not learning some other one, for example one should first learn C++ before trying out Visual C++, if you know what I mean)
What are the main advantages of lisp in general (I've seen many pages about that saying it's faster in development and execution, but they were all pretty vague about the details)
Can it be generally used for general purpose, or is it concentrated on AI? (By this I mean if, for example, one could make a full console app with it, and then implement OpenGL just as easily and make a game. Learning a language specialized on something precise is worthwhile, but not at the moment for me)
I would also be very happy about any additional details you guys can give me! (Links are appreciated too! E-Books and whatnot.)
Edit: all of the answers here were very useful. As such, I gave them all a +1 to rep, but chose the more concrete one as best. Thank you all.
I also learnt Java and C# intensively before coming to Lisp so hopefully can share some useful perspectives.
Firstly, all Lisps are great and you should definitely consider learning one. There's a famous quote by Eric Raymond:
"Lisp is worth learning for the profound enlightenment experience you
will have when you finally get it; that experience will make you a
better programmer for the rest of your days, even if you never
actually use Lisp itself a lot."
Reasons that Lisps are particularly interesting and powerful are:
Homoiconicity - in Lisp "code is data" - the language itself is written in Lisp data structures. In itself this is interesting, but where it gets really powerful is when you start using this for code generation and advanced macros. Some believe that this features is a key reason why Lisp can help you be more productive than anyone else (short Paul Graham essay)
Interactice development at the REPL - a few other languages also have this, but it is particularly idiomatic and deep-rooted in Lisp culture. It's remarkably productive and liberating to develop while altering a live running program. Recent examples that caught my eye include music hacking with overtone and editing a live game simulation.
Dynamic typing - opinion is more divide on whether this is an advantage or not (I'm personally neutral) but many people thing that dynamically typed langauges give you a productivity advantage, at least in terms of building things quickly. YMMV.
My personal recommendation for a Lisp to learn nowadays would be Clojure. Clojure has a few distinct advantages that make it stand out:
Modern language design - Clojure "refines" Lisp in a number of ways. For example, Clojure adds some new syntax for vectors [] and hashmaps {} in addition to lists (). Purists may disapprove, but I personally believe these find of innovations make the language much nicer to use and read.
Functional first and foremost - all the Lisps are good as functional languages, however Clojure takes it much further. All the standard library is written in terms of pure functions. All data structures are immutable. Mutable state is strictly limited. Lazy sequences (including infinite sequences) are supported. In some senses it feels a bit more like Haskell than the other Lisps.
Concurrency - Clojure has a unique approach to managing concurrency, supported by a very good STM implementation. Worth watching this excellent video for a much deeper explanation.
Runs on the JVM - whatever you think of Java, the JVM is a great platform with extremely good GC, JIT compilation, cross platform portability etc. This can be a barrier to entry for some, but anyone used to Java or C# should quickly feel at home.
Library ecosystem - since Clojure runs on the JVM, it can use Java libraries extremely easily. Calling a Java API from Clojure is trivial - it's just like any other function call with a syntax of (.methodName someObject arg1 arg2). With the availability of the huge Java library ecosystem (mostly open source) Clojure basically leapfrogs all the "niche" languages in terms of practical usefulness
In terms of applications, Clojure is designed to be a fully general purpose langauge so can be used in any field - certainly not limited to AI. I know of people using it in startups, using it for big data processing, even writing games.
Finally on the performance point: you are basically always going to pay a slight performance penalty for using higher level language constructs. However Clojure in my experience is "close enough" to Java or C# that you won't notice the difference for general purpose development. It helps that Clojure is always compiled and that you can use optional type hints to get the performance benefits of static typing.
The flawed benchmarks (as of early 2012) put Clojure within a factor of 2-3 of the speed of statically typed languages like Java, Scala and C#, a little bit behind Common Lisp and a little bit ahead of Scheme (Racket).
Lisp, as you've discovered, is not one language; it's a family of languages that have certain features in common.
There are two primary dialects of Lisp: Common Lisp and Scheme. Each of those two dialects has many implementations, each with their own features. However, both Common Lisp and Scheme are standardized, and the standards define a certain baseline of features which you can expect any implementation to have.
Scheme is a minimalistic language with a very small standard library. It is used primarily by students and theoreticians. Common Lisp has many more language features and a much larger standard library, including a powerful object system, and has been used in large production systems.
Clojure is another minor, more recent dialect. If you want to understand Lisp, you're better off first learning either Common Lisp or Scheme.
My recommendation is to learn Scheme first; it's a purer expression of the ideas that Lisp is made of, and will help you understand the essence of the language. In many ways, Lisp is completely different from Java and other imperative languages; however, what you learn from it will make you a better programmer in those languages. You can easily learn Common Lisp after you know Scheme.
The advantage of Lisp is, simply put, that it's more powerful than other languages. All Lisp code is Lisp data and can be manipulated as such; this allows you to do really cool things with metaprogramming that simply can't be done in other languages, because they don't give you direct access to the data structures that comprise your code. (The reason Lisp can do this and they can't is intimately related to its strange-looking syntax. Every compiler or interpreter, after reading the source code, must translate it into abstract syntax trees. Unlike other languages, Lisp's syntax is a direct representation of the ASTs that Lisp code is translated into, so you know what those trees look like and can manipulate them directly.) The most commonly used metaprogramming feature is macros; Lisp macros can literally translate a bit of source code into anything you can program. You can't do that with, say, C macros.
The "faster in development and execution" thing may have been a reference to one specific feature which most Lisp implementations provide: the read-eval-print loop. You can type an expression into a prompt and the interpreter will evaluate it and print the result. This is wonderful both for learning the language and for debugging or otherwise investigating code.
Lisp is dynamically typed (though statically typed flavors do exist). Most implementations of Lisp run on their own virtual machine; however, many can also be compiled to machine code. Clojure was written specifically to target the JVM; it can also target .NET and JavaScript.
Though originally created for AI research, Lisp is by no means exclusively for AI. The main reason why it's not more popular in mainstream production environments (apart from the self-perpetuating dominance of Java and C#) is library support. Common Lisp has many good libraries out there (Scheme less so), but it pales in comparison to the vast amount of library support available for Java or Python.
If you want to get started, I recommend downloading Racket, a highly popular implementation of Scheme. It has everything you need, including a simple-but-very-powerful IDE with a read-eval-print loop, right out of the box. Though originally developed as a teaching language, it comes with a very large standard library more characteristic of Common Lisp than of Scheme. As a result, it's seeing use in real production environments.
Runtime Environments
Common Lisp and Scheme generally have their own unique runtime environments. There are some variants of Scheme (Chicken and Gambit) which can be translated to C and then linked with their environments so as to be able to be deployed as stand alone executable programs. Clojure runs in the JVM, and there is also a CLR port, but its not clear to me that the CLR port is current with the JVM. Clojure also has Clojurescript, which targets a Javascript runtime.
Which is Better to Learn First
I don't think that question has a good answer. Its up to you. Although if you have experience with the JVM, Clojure might be a bit smoother to start with.
What is Better about Lisp
That's a question liable to start a flame war. I don't have much lisp experience. I started learning Clojure a few months ago in earnest, have looked at Common Lisp and Scheme on and off over the years.
What I like is their dynamic natures. You need to change a function at runtime while your program is running? No problem! Like any power tool, you have to be careful not to chop your bits off when using this.
The power and expressiveness is addicting too. I am able to do some things with little effort that I know I could not achieve in Java, or I know would require a lot more work. Specifically, I was able to put together a description of a data structure - and though the use of macros, delay evaluation of parts of the data until the right time. If I had done that in Java, I would not have been able to nest the declarations like I did because they would have evaluated in the wrong order. Pain would have ensued.
I also like Clojure's view of functional programming, although I have to say it requires work to adjust.
Is Lisp General Purpose
Yes.
--
Mark Volkman has a really good article on Clojure. Many basics are there. One thing that I did in the beginning was to just fire up a repl and experiment when I needed to figure something out programmatically. e.g. explore an API or do some calculations. After a short period of time with that I started working on more building up levels of effort, and I have a project that I'm working on right now that involves Clojure.
There isn't a bad book about Clojure that has been written. The Stuart Sierra book is being updated; and the Oreilly book is about to come out soon, so you might want to wait. The Joy of Clojure is good, but I don't think its a good starter book.
For Common Lisp, I highly recommend the Land of Lisp.
For Scheme, there are several classics including The Little Schemer and SICP.
Oh, and this: http://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hickey (maybe one of the most important talks you'll ever watch), and this http://www.infoq.com/presentations/hickey-clojure (IIRC, really good intro to Clojure).
common lisp
Common Lisp is both compiled and interpreted. Deployments (in Windows) can be done by an exe with DLLs. Or by a precompiled bytecode. Or by installing a Lisp system on the target device and executing the source against it.
Common Lisp is a fully usable industrial language with an active community and libraries for many different tasks.
Lisps are generally faster for development and due to the abstraction capabilities, better at developing higher level concepts. It's hard to explain. Ruby vs. C is an example of this sort of thing. All Lisps carry this capacity IMO.
Common Lisp is a general purpose language. I don't know offhand if modern Common Lisp implementations directly support executing assembly, so it may be difficult to write drivers or use compiler-unsupported CPU instructions.
I like Common Lisp, but Clojure and Racket are not to be sneezed at either. Clojure in particular represents a very interesting track, in my opinion.
For e-books, you can get On Lisp by Graham and Gentle Introduction to Symbolic Computation. Possibly others but those are the ones I can recall.
My understanding is Clojure's homoiconicity exists so as to make writing macros easier.
Based on this stackoverflow thread, it looks like Macros are used sparingly, except for DSLs in which higher-order functions are not to be used.
Could someone share some examples of how macros are used in real-life?
It's correct that homoiconicity makes writing Clojure macros very easy. Basically they enable you to write code that builds whatever code you want, exploiting the "code is data" philosophy of Lisp.
Macros in a homoiconic language are also extremely powerful. There's a fun example presentation I found where they implement a LINQ-like query syntax in just three lines of Clojure.
In general, Clojure macros are potentially useful for many reasons:
Control structures - it's possible to create certain control structures using macros that can never be represented as functions. For example, you can't write if as a function, because if it was a function then it would have to evaluate all three arguments, whereas with a macro you can make it only evaluate two (the condition value and either the true or false expression)
Compile time optimisation - sometimes you want to optimise your code based on a constant that is known or can be computed at compile time. For example, you could create a "logging" function that logs only if the code was compiled in debug mode, but creates zero overhead in the production application.
Code generation / boilerplate elimination - if you need to produce a lot of very similar code with similar structure, then you can use macros to automatically generate these from a few parameters. If you hate boilerplate, then macros are your friends.
Creating new syntax - if you see the need for a particular piece of syntax that would be useful (perhaps encapsulating a common pattern) then you can create a macro to implement this. Some DSLs for example can be simplified with additional syntax.
Creating a new language with entirely new semantics (Credits to SK-Logic!) theoretically you could even go so far to create a new language using macros, which would effectively compile your new language down into Clojure. The new langauge would not even have to be Lisp-like: it could parse and compile arbitrary strings for example.
One important piece of advice is only use macros if you need them and functions won't work. Most problems can be solved with functions. Apart for the fact that macros are overkill for simple cases, functions have some intrinsic advantages: they are more flexible, can be stored in data structures, can be passed as parameters to higher order functions, are a bit easier for people to understand etc.
For example pattern matching is a programming language feature that can be added to the clojure language through macros: http://www.brool.com/index.php/pattern-matching-in-clojure
What other language features can be added to the language?
Off the top of my hat I have two examples, but I'm sure there are more.
Contracts programming: https://github.com/fogus/trammel
Declarative logic: https://github.com/jduey/mini-kanren
I think its a stupid question to ask what can be added, what you should ask is what you cant add. Macros allow you to hook into the compiler that mean you can do almost anything.
At the moment you cant add your own syntax to the language. Clojure does not have a user extenseble reader, this means you don't have any reader-macros (http://dorophone.blogspot.com/2008/03/common-lisp-reader-macros-simple.html). This is not because of a technical problem but more a decition by Rich Hickey (the Clojure creator).
What you can not do is implement features that need virtual maschine support like add tail call semantics or goto.
If you want to see some stuff that has been done: Are there any Clojure DSLs?
Note that this list is not 100% up to date.
Edit:
Since you seem you took pattern matching as an example (it is a really good example for the power of macros) you should really look at the match library. Its probebly the best fastest pattern matching library in Clojure. http://vimeo.com/27860102
You can effectively add any language features you like.
This follows from the ability of macros to construct arbitrary code at compile time: as long as you can figure out what code you need to generate in order to implement your language features, it can be achieved with macros.
Some examples I've seen:
Query languages (Korma)
Logic programming (core.logic)
Image synthesis DSL (clisk)
Infix notation for arithmetic
Algebraic manipulation
Declarative definition of realtime data flows (Storm, Aleph)
Music programming (Overtone, Music As Data)
There are a few caveats:
If the feature isn't supported directly by the JVM (e.g. tail call optimisation in the mutually recursive case) then you'll have to emulate it. Not a big deal, but may have some performance impact.
If the feature requires a syntax not supported by the Clojure reader, you'll need to provide your own reader (since Clojure lacks an extensible reader at present). As a result, it's much easier if you stick to Clojure syntax/forms.
If you do anything too unusual / unidiomatic, it probably won't get picked up by others. There is a lot of value in sticking to standard Clojure conventions.
Beware of using macros where they are not needed. Often, just using normal functions (perhaps higher order functions) is sufficient to implement many new language features. The general rule is: "don't use macros unless you absolutely need to".
Both seem to be good for building extensible API's and code generation.
What are the main differences between them?
What do you see as their strengths, weaknesses, ...
Disclaimer: I'm more familiar with Clojure than Tcl so apoligies to any Tclers if I misrepresent anything. However here are some points I'm generally aware of:
Both are extremely flexible - you can use meta-programming to generate and execute pretty much any code you like at runtime.
Clojure is a JVM language whereas Tcl is relatively standalone.
Advantages of being on the JVM include being able to generate interoperable code for and act as a DSL for libraries in other JVM languages like Java and Scala. Also you get access to the huge range of Java libraries.
Disadvantage of being on the JVM is relatively substantial start-up time. So you'd probably prefer Clojure for longer-running server applications rather than command line tools that need to execute quickly.
Clojure metaprogramming will get compiled to native code (via JIT in the JVM). Will depend on your application, but I expect this will perform faster than Tcl in most circumstances
Both languages are dynamic (a good thing for metaprogramming on average!) and support functional programming
Clojure contains some interesting abstractions that are very useful for metaprogramming - in particular the considerable in-language support for sequences
I personally find the simlicity of Lisp syntax to be a great advantage for metaprogramming - it's much easier to generate code when there is only one syntactical construct (the s-expression) to worry about...
On average both languages are great for metaprogramming, but if I was choosing between the two:
I'd choose Clojure if I was building a server-side application or if I had a particular need for JVM interoperability
I'd choose Tcl if I wanted a DSL for managing scripts / tools at the command line
I think mikera's answer is excellent.
I would add only that where in clojure metaprogramming tends to focus on macros, a grand unified metaprogramming solution without peer, tcl has several small sharp tools available for metaprogramming, among them the "unknown" command, which can be used to do all sorts of nifty tricks. Also interesting is that even though clojure has only a small number of keywords or "special forms", tcl actually has none, which sort of makes tcl it's own dsl, and changing (or extending) the behavior of any command is possible.
One thing i have to disagree with from mikera's answer is the part about clojure's syntax being more amenable to metaprogramming. One of the unpleasant surprises for me when coming to clojure is actually how much syntactic variation there is in clojure, with the various uses of () [] {} "" ^{} #' :key ... and on and on. I totally grok the justification of this type of syntactic sugar, but i actually find tcl's syntax easier to deal with for "ham-handed" metaprogramming and code generation. And tcl's "everything is a string" nature adds to the simplicity.
As for mulit-processing capabilities, immutable data structures, purely functional nature and many other instances of clojure's distinctives, there really is nothing comparable in tcl.
I couldn't agree more with mikera's conclusion.