Designing Crux – Record Traits

I’ve intentionally tried not to break much new ground with Crux’s type system. I mostly want the language to be a pleasant, well-executed, modern language and type system for the web.

That said, when porting some web APIs to Crux, I ran into an bothersome issue with an interesting solution. I’ll motivate the problem a bit and then share the solution below.

Records and Row Polymorphism

Crux has row-polymorphic record types. Think of records as anonymous little structs that happen to be represented by JavaScript objects. Row polymorphism means that functions can be written to accept many different record types, as long as they have some subset of properties.

fun length(vec) {
    math.sqrt(vec.x * vec.x + vec.y * vec.y)
}

length({x: 10, y: 20})
length({x: 3, y: 4, name: "p1"})
length({x: 15})               // type error, missing property y
length({x: "one", y: "two"})  // type error, x and y must be numbers

I lived in a TypeScript codebase for nine months or so and the convenience of being able to quickly define anonymous record types is wonderful. Sometimes you simply want to return a value that contains three named fields, and defining a new type and giving it a name is busywork that contributes to the common belief that statically typed languages feel “heavy”.

Crux supports both nominal types (String, CustomerID, Array, …) and anonymous records, sometimes called structural types.

The problem here is how structural types interact with traits.

Imagine coming to Crux for the first time and trying to encode some values as JSON.

json.encode(10)              // returns "10"
json.encode("hello")         // returns "\"hello\""
json.encode(True)            // "true"
json.encode([15, 30, 12])    // "[15,30,12]"
json.encode(js.Null)         // "null"

json.encode({code: 200, text: "OK"})
// Type error: records don't implement traits

Wait, what? Why does everything work but records?

Records and Traits

json.encode‘s type is fun encode<V: ToJSON>(value: V) — that is, it accepts any value which implements the ToJSON trait. Why don’t records implement traits? Well, record types are anonymous, as described before. They don’t necessarily have unique definitions or names, so how would we define a ToJSON instance for this record?

Haskell, and other statically-typed languages, simply have the programmer manually construct a HashMap and JSON-encode that. In Crux, that approach might look something like:

let map = hashmap.new()
map["x"] = 10
map["y"] = 20
json.encode(map)

Not too bad, but not as nice as using record syntax here.

To solve this problem, which is “only” a human factors problem (but remember that Crux is intended to be delightful), I came up with something that I believe to be novel. I haven’t seen this technique before in the row polymorphism literature or any languages I’ve studied.

There are two parts to this problem. First we want to use record literals to construct key-value maps, and then we want to define trait instances that can use said maps.

Dictionaries

As I mentioned, Crux records are implemented as JavaScript objects. This is pretty convenient when interfacing with JavaScript APIs.

data jsffi ReadyState {
    UNSENT = 0,
    OPENED = 1,
    HEADERS_RECEIVED = 2,
    LOADING = 3,
    DONE = 4,
}

type XMLHttpRequest = {
    mutable onreadystatechange: () => (),
    readyState: ReadyState,
    responseText: JSOption<String>,
    // ...
}

Sometimes, however, JavaScript objects are used as key-value maps instead of records. Crux provides a Dict type for that case. Dict is like Map, except keys are restricted to strings to allow implementation on a plain JavaScript object.

let d = dict.new()
d["happy"] = True
d["sad"] = False
json.encode(d)  // "{\"happy\":true,\"sad\":false}"

There is a ToJSON instance for any Dict<V> where V: ToJSON. But can we provide better syntax for creating one? It would be nicer to write:

let d = dict.from({
    happy: True,
    sad: False,
})

To implement dict.from, we need a new type of record constraint: one that says, I accept any set of fields, as long as every value has a consistent type.

fun from<V>(record: {...: V}): Dict<V> {
    // some unsafe JS code to copy the record into a mutable JS object
}

Read ... as “arbitrary set of fields” and : V as “has type V”. Thus, {...: V} is the new type of record constraint, and it’s read as “record with arbitrary set of fields where each field’s value has type V”.

So now we can write:

json.encode(dict.from({
    happy: True,
    sad: False,
}))

Better, but we’re still not done. For one, dict.from requires the values to have the same type — not necessary for JSON encoding. Two, it’s annoying to have to call two functions to quickly create a JSON object.

Defining Record Instances

Here’s where record trait instances come in. First we need to convert all the record’s field values into a common type T. Then, once the record has been converted into a record of type {...: T}, it can be passed to dict.from and then json.encode. The resulting syntax is:

“`crux
// Read as: implement ToJSON for records where…
impl json.ToJSON {…} {
// …this code runs across each property of the record,
// producing a value of type {…: json.Value}…
for fieldValue {
json.toJSON(fieldValue)
}

// which is then encoded into a JSON object.
toJSON(record) {
    json.toJSON(dict.from(record))
}

}
“`

And there you have it, a ToJSON instance for any record, at least as long as the record’s field types implement ToJSON themselves.

json.encode({
    code: 200,
    status: "OK",
})

I’ve never seen a feature like this in the literature or languages I’ve studied, but if you have, please let me know!

Why isn’t Crux a pure functional language?

Nicolas Hery asked on Twitter: “[Why isn’t Crux] pure like Haskell/Elm/etc.?” Good question, and one worth writing about.

First of all, the term “pure functional language” isn’t terribly enlightening. It’s certainly not useful for communication, because everyone has or invents their own definition on the fly.

There are a few independent ideas here: mutable vs. immutable data structures, mutable vs. immutable names/variables, and whether effects are tracked.

Regarding mutable vs. immutable data structures: in reality, mutable data structures are simply unavoidable. At some point you need to update your in-memory representation. There are ways to encode that with immutable data structures, but they get awkward fast. In addition, constantly producing new immutable values places a great deal of pressure on the garbage collector – often it’s fastest just to update your data in-place. Immutable data is a fantastic tool, used appropriately, but it sucks when it’s a straitjacket.

Regarding mutable vs. immutable local variables: imagine you want to sum up the elements in an array. If Crux were pure-functional, that
could be written something like:

fun sum_(list, index) {
  if index >= len(list) {
    0
  } else {
    list[index] + sum_(list, index + 1)
  }
}
fun sum(list) {
  sum_(list, 0)
}

Oh wait, that’s not tail-recursive, so it’ll blow up the stack with a long enough list. Let’s rewrite it:

fun sum_(list, index, total) {
  if index >= len(list) {
    total
  } else {
    sum_(list, index + 1, total + list[index])
  }
}
fun sum(list) {
  sum_(list, 0, 0)
}

OK, now we’re tail recursive. For efficiency, however, the compiler needs to turn that into a tight loop of instructions, storing the accumulators in registers. Wait, so you’re saying the language forced me to translate a simple imperative function (sum the elements in a list) into a tail-recursive loop, which the compiler then has to convert back into imperative code for efficiency? Why can’t the compiler just turn my natural imperative loop into a tail-recursive implementation, if that’s what it really wants? (It definitely could.) And it gets even worse the larger your functions become, especially if they have early exits and multiple “mutable variables”. In short, tail calls are great, when appropriate, but always programming with tail recursion is like someone rejecting all your commits with “Nope, you must contort your code for my pleasure.” And I shudder to think of what a tail-recursive implementation of a huge interpreter loop with lots of mutable variables would look like.

Tail calls are great, but being forced to write tail-recursive loops is dumb and one of my biggest annoyances with Haskell. Instead, why not just write:

fun sum(list) {
  let mutable total = 0
  for i in list {
      total += i
  }
  return total
}

Crux makes that natural.

However! Everything I said above is independent of what the defaults are. Crux variables and values are immutable by default, since that’s the common case. Sum types are always immutable. The only things that can be mutable are record fields and locals, but, again, you must opt into that functionality. It’s not annoying in practice; since everything’s an expression in Crux, there’s less need to mutate locals.

Okay, so instead let’s presume that most people consider pure functional to mean “has restricted side effects”, like Haskell.

Personally, I’m a huge fan of monads in Haskell, and the amazing
things they enable.

However, they come at a large cost: since nearly everything is a Monad, the error messages get really complicated, especially when the compiler thinks you meant the Maybe monad or a function monad. (Isn’t it crazy how, in Haskell, Integers aren’t a Monoid by default, but all functions are Monads?)

Eventually I’d love Crux to have an effect tracking system. Different syntax between pure code and effectful code is unacceptable, so the Haskell approach is a no-go. I am excited, however, about row-typed effect systems like Koka. (Notably, it should be okay for pure functions to use mutable locals, like the sum function above.) Future work!

To summarize, I think we can achieve most of the benefits of “pure functional programming” while retaining accessible syntax and allowing mutability as desired and convenient.

Designing Crux – Import Statements

Manipulating a module’s import list is a high-frequency activity in modern programming languages. Sadly, few popular languages perfect the usability of the import list.

To do a good job in Crux, let’s start from the use cases, roughly sorted by importance:

  • import a module and refer to it by a qualified import name
  • import a list of names unqualified into this module’s scope
  • import everything unqualified into this module’s scope
  • import a module solely for its side effects, without importing any symbols
  • visually scan the import list
  • sort the import list (with Emacs’s sort-lines or the like)
  • and, of course, tools must easily parse the import list

Designing for usability means optimizing for the most common use cases, while making sure the less common use cases are still possible.

The most common use case is to import a module and refer to it qualified. Consider this hypothetical syntax:

import fs.path
path.combine("/usr", "local")

The qualified import reference (here, path) defaults to the last segment of the module name. Now, for this to work well, the basename has to be short, distinct, and meaningful.

Haskell gets this wrong.

import qualified Data.ByteString as BS

From left to right: import qualified is long to start with. Then you have most packages in some kind of arbitary namespace like Data. Then, since ByteString is too long to be convenient (especially with the mixed case), people typically name the import BS. Typically being the key word. Sometimes people name it Bytes. Sometimes ByteString. Sometimes B. Sometimes Data.ByteString.Char8 is imported as BS. The fact that the programmer has to make a choice here naturally leads to inconsistency.

Go does a great job here. In Go, you import like this:

import "fs/path"

Afterwards, path functions can be accessed as path.Function. Short and meaningful.

Go goes even further and provides guidelines for naming functions in the context of a module. Conventions tend to be copied, so by setting a good standard early on, Go positively affected every project built then on.

ES6 modules are also pretty interesting. They are statically analyzable and syntactically lightweight, but they have a fairly significant problem at scale: sorting imports.

import Dispatcher from 'flux/dispatcher';
import {foo, bazzle} from 'app/utils';

The problem with having the list of imported symbols first is that they can’t easily be machine-sorted. Every ES6 project I’ve worked on has encouraged its engineers to sort the imports by hand. Also, the import list coming first gets annoying when import lists are long.

import {
  foo, bar, baz,
  qux, harf, barf,
  hurp, floop
} from 'my_module';

With Crux, we learned from all of the above import systems and came up with something that meets every use case. Here is the proposed syntax:

import {
  bytes,
  flux.dispatcher,
  my_encoding_system as mes,
  imported_only_for_side_effects as _,
  utils(foo, bar, baz),
  more_utils(
    foo2,
    bar2,
    baz2,
  ),
  all_the_utils(...), // import everything into scope
}

The easiest import syntax is most common one, but other types of imports, like renaming the import or importing some symbols unqualified, are straightforward too.

This directly satisfies all of our use cases!

We may change our minds and make the commas after each import optional in the future.

Side Note: Implicit vs. Explicit Imports

OCaml and Java allow qualifying names at their use sites, making import statements unnecessary. Import statements are syntax sugar at that point. I don’t have a strong justification here, but Crux went with explicit import statements (like Go, Python, Haskell, ES6) because it’s easier for humans and machines to see a module’s dependencies at a glance.

Designing Crux – Methods

Crux’s foundations are rooted in the ML language family. Everything is a function. Haskell has this property too, but there is a really sucky aspect of an everything-is-a-function world.

Imports.

Here’s a common experience I have with Haskell:

let url = getCustomerUrl myCustomerId

OK, I have a string.

I need to check if it starts with “https://”

Scroll to top of file… That means I need to call the startsWith function. What module does it live in again?

Think… is this a Text or a ByteString? Or maybe a String? That determines which module I import…

This is a very common bit of friction, and it comes up all the time in Haskell, both for the standard library and your own code.

Now let’s hypothesize what reading from a data store might feel like in a world with explicit imports:

OK, my React component has an env prop.

I know the UserStore is on the env, but I need to import the Environment module to call getUserStore:

let userStore = Environment.getUserStore(env).

Now I want the list of users, so import UserStore and:

let users = UserStore.getUserList(userStore)

I can see it now: “Why can’t I just write let users = env.getUserStore().getUserList()? Crux sucks!”

In addition, assuming a roughly-one-type-per-module structure, each type gets two “names” that must be used: the module name and the type name. If types are refactored to live in different modules, all the callers have to be updated with new imports. Finally, if those names diverge, it could get confusing.

Dynamic languages like JavaScript and Python solve this usability issue by implementing all method calls with dynamic lookup. Besides the obvious performance hit, this can make analyzing the call graph nontrivial — both by humans and tools.

Languages like C++ and Java and Go take a different approach: they make methods a first-class concept. This adds some complication to the language when you want to use a method as a normal function. C++ has adapters that let you use members as functions sometimes but it’s nicer if methods are just functions.

Crux is not an object-oriented language, but we want the usability benefits of “method syntax”, sometimes known as type-directed name resolution:

  1. with no dynamic dispatch
  2. without any specialize method concept — methods should just be normal functions
  3. and, importantly, while still supporting bidirectional type inference for record types

The reason we have to worry about a conflict with type inference is because records already use dot syntax. Consider the following function:

fun distance(start, end) {
  return sqrt(sqr(end.x - start.x) + sqr(end.y - start.y))
}

Notice no type annotations! The distance function is inferred to take two records, each of which has floating point x and y properties. It returns a float.

Since we think it’s important that . syntax be used for records, Crux uses an arrow -> for method calls. Method calls only work when the type is known. Fortunately, in most programs, the type inference engine already knows the concrete types of most things. If you try to write a function like:

fun isHTTPSURL(url) {
  return url->startsWith("https://")
}

you will get a type error, since it doesn’t know what url is. The fix is easy: provide a type annotation on the argument:

fun isHTTPSURL(url: String) {
  return url->startsWith("https://")
}

In practice, most of the time, the type is known from context. Let’s go back to the Flux UserStore example. With method syntax, it would be spelled:

let users = env->getUserStore()->getUserList()

-> is mainly a way to avoid imports. a->b() can be read as “assuming the type of a is known, look up the symbol b in the module that defined a, and call b(a)”

The env example works because the type of env is known. Therefore, the location of the getUserStore() function is known, and it’s known to return a UserStore, so getUserList is looked up there. No type annotations required, there’s nothing special about member functions, and we achieve all the performance and tooling benefits of static dispatch!

One note: for simplicity, we have been applying the rule that the type must have been inferred before the method call is resolved. Technically, this is more restrictive than necessary. Consider the following example:

fun addSuffix(p) {
  if p->endsWith(".json") {
    return p
  } else {
    return p + ".json"
  }
}

The p + ".json" expression proves that p’s type is String, and therefore p->endsWith(".json") could be resolved. However, supporting this kind of backwards knowledge transfer would complicate the compiler (I think). Perhaps we’ll look into fixing it later. :)

Another note: sometimes you simply want to reference a “member function” without having a particular instance in mind. You might have the type name in scope, but don’t want to bother importing. We’ve considered support syntax like &String::startsWith to reference the startsWith function in the module that defines the String type. It could be useful in situations like ["foo", "bar", "baz"]->map(&String::length).

Designing Crux – Function Calls

Tupled vs. Curried Functions

Programming languages specify function argument lists in one of two ways: tuples or currying. Most mainstream languages use the former:

function add(x, y) {
    return x + y;
}

The function add takes two arguments. You can imagine the caller building up a tuple of arguments on the stack before calling the function.

Haskell and ML effect multi-argument functions with currying. For example:

add :: Int -> Int -> Int
add x y = x + y

This is equivalent to:

add :: Int -> (Int -> Int)
add x = \y -> x + y

That is, add is a function that takes an Int and returns another function which also takes an Int and returns an Int.

Currying is quite elegant. All functions take one argument, which makes certain types of metaprogramming easy. Curried function syntax is also extremely convenient: map (add 1) [1, 2, 3] returns [2, 3, 4].

However, there are some downsides to curried functions:

The first has to do with knowing when effects occur. In Haskell, where function application is always pure, this isn’t an issue. But since Crux allows unrestricted effects (more on this later), we want it to be obvious when side effects can happen. Consider this snippet of a hypothetical curried language:

add x y =
  print "adding " + toString x + " and " + toString y
  return x + y

Now consider this other definition:

add x =
  print "adding " + toString x
  return \y ->
    print " and " + toString y
    return x + y

Both functions have the same behavior when called like this:

add 1 2
add 1 3

But they would have different behavior with a partial application:

let addOne = add 1
addOne 2
addOne 3

That’s one argument against curried functions, though you could argue it’s a rare issue not worth worrying much about.

However, a more serious issue has to do with consistent performance. Curried languages don’t actually produce new functions for every argument of every call. That would be way too slow. In practice, functions take multiple arguments at the machine-code level (where the number of arguments is their arity), and when the function’s definition is not known at compile-time, some dynamic arity checks are added to decide whether to create a new partially-applied function, call straight through to the implementation, or perform oversaturation.

These dynamic checks add a bit of runtime overhead, and since we want Crux to have obvious execution semantics, we decided not to use curried functions.

Finally, I want to mention a third downside of curried functions: the error messages. At least in Haskell, if you pass too few or too many arguments, you don’t get a very obvious error message.

Prelude> let f a b = a + b
Prelude> f 1 2
3
Prelude> f 1
<interactive>:31:1:
    No instance for (Show (a0 -> a0))
      (maybe you haven't applied enough arguments to a function?)
      arising from a use of ‘print’
    In the first argument of ‘print’, namely ‘it’
    In a stmt of an interactive GHCi command: print it
Prelude> f 1 2 3
<interactive>:32:1:
    Non type-variable argument in the constraint: Num (a -> t)
    (Use FlexibleContexts to permit this)
    When checking that ‘it’ has the inferred type
      it :: forall a t. (Num a, Num (a -> t)) => t
```

Syntax

Syntax is somewhat tangential to the issue at hand, and it’s certainly possible to use the f x y z syntax with tupled functions, but my experience teaching Haskell is that people get unnecessarily hung up on the "f x y z" call syntax and we decided against spending our strangeness budget here.

Conclusion

That was a long justification for saying that Crux’s function syntax and semantics are just like every other mainstream language. :)

fun add(x, y) {
    x + y
}

let four = add(2, 2)

Designing DFPL – The Name (Update)

Previously, I’d talked about the import of a programming language’s name.

I have a quick update on that front. Andy had originally named the project Crux, but we questioned if we wanted that name in the long term. So… when in doubt, run a survey!

I put together a simple survey with some front-runners. Two questions: “Which is your favorite?” followed by “Check all that you’d be okay with.” Crux came in 2nd place for the favorite but it was the one most people were okay with, and since it was the name we already had, we stuck with it. :)

Designing DFPL – The Name

Without hard evidence, it’s hard to say how much a language’s name matters, at least beyond basic Googleability.

Marketing has an effect on any kind of product, and programming languages are no exception.  Some names convey their different from other well-known languages, like C++ (an extension of C) or TypeScript (JavaScript but with a type system).  Java, Go, Python, and Ruby, on the other hand, are short, memorable words with neutral or positive connotations.  Bonus points for names that carry a family of associated words, e.g. Ruby and Gem.

Since our language is more of a synthesis of good ideas than a modification to an existing language, a short positive word seems to be the right strategy.

Crux is the first name we tried, though neither of us is thrilled with it.  We briefly tried the name Sneak, but everyone hated it.  Fig is short, neutral-to-positive, but nobody seemed excited by that name either.

Do you have suggestions?

The name affects the file extension, which ought to be short.  (Or am I the only one perpetually annoyed by the horizontal real estate consumed by .java and .coffee?)

Designing DFPL – The Broad Strokes

Why not Python?

I love Python. It’s easy to learn, predictable, and works well for a wide variety of programs. Python’s the first language where I regularly found myself writing programs and having them work on the first try. The problem with Python is that it breaks down at scale.

Performance

In my experience, whenever a Python codebase reaches a couple hundred thousand lines of code, performance becomes an issue. It’s well-known that Python’s straight-line perf is not even close to C or even Java. In addition, Python programs don’t make efficient use of memory: objects are large and sparse — programs consist of a lot of pointer-chasing. The garbage collector’s not great, and startup and module loading times become an issue at scale, especially for short-lived processes like unit tests, build scripts, or desktop applications.

There are Python JIT compilers, but JITs don’t really help with startup time or memory usage.

Parallelism

Python has a GIL, meaning it’s hard to write programs that make efficient use of the hardware. It’s possible to move CPU-bound work into C, but any kind of coordination between Python and C has to ping-pong across the GIL, which becomes a central bottleneck.

Launching multiple processes can help, but each process pays Python’s base process memory usage overhead, and they can’t easily share in-memory caches.

As we’ve seen from Haskell and Go, the ability to have convenient, lightweight, efficient parallelism is a big deal.

Maintainability

I talked about this in The Long-Term Problem With Dynamically Typed Languages. As codebases get huge, you really want to be able to lean on the computer in order to safely make changes. Type systems are mini proof assistants for expressing guarantees like "I never mix up binary data with textual data" or "I always check the case that optional values are empty". Unit tests help, but are "proof by example". Type systems add a meaningful layer of confidence, allowing software to scale to larger teams and faster changes.

Why not Haskell/Go/Rust/Scala/F#/Whatever?

I agree 100% with Patrick Walton here:

There is room for some statically-typed language to occupy the network-services niche where performance needs to be good—maybe 2x slower than optimal (note that this is not acceptable for all network services)—and speed of development trumps most other considerations. Some sort of hybrid of Swift, Rust, and Go, perhaps, as none of them are 100% ideal for this space.

The dynamic language space is relatively saturated at this point. Rust is doing a great job in the native machine performance space. F# and Swift and, to some degree, Scala are excellent choices for the .NET, Apple, and JVM ecosystems, respectively.

But as of yet, there’s nothing great in either the web frontend space or the application server space.

Haskell’s laziness, curried functions, and nonobvious operational semantics are not suitable for frontend code. While it’s an otherwise perfect language for writing application servers, its syntax is tragically divisive, and I have no hope that Haskell as it stands today will become mainstream.

Go’s type system somehow manages to be both verbose and not expressive enough. Go is sufficient for single-purpose servers, but, from first-hand experience, the lack of generics really hurts in application servers. (I probably owe an essay on the difference between the two types of servers.) The Go community tends to rely on idioms, but it would be better if the language was general and expressive enough to allow direction expression of things like timeouts and cancellation. Also, Go is not actually memory-safe. Go is especially frustrating because, if it had, say, generics and tuples instead of the more limited multiple-return-values, it would be a truly excellent programming language.

Rust’s concurrency is not lightweight enough, and the syntax is extremely verbose.

Scala has a significant platform dependency on the JVM. Check out the size of some Scala.js programs. Also Scala’s type system is insanely complex.

F# brings along a large .NET dependency. It also suffers from the ML syntax. (Actually, it has two syntaxes.)

Maybe Swift is great, but it was only just open sourced, and I’d be worried about the Obj-C platform dependency. I’ve yet to write any real Swift programs, but what I’ve seen looks great.

All of these languages are lovely and have their uses, but none are particularly satisfactory replacements for Go or TypeScript/JavaScript. That is, lightweight code that can run at high performance in a browser and on a concurrent, scalable server.

But, hold on, what about js_of_ocaml? This is where things get interesting. Andy used js_of_ocaml for a while and found that, on paper, Ocaml has all the right properties: it’s safe, high-performance, compiles to very concise, efficient JavaScript, expressive, concurrent, established… In fact, it’s nearly perfect… if you can stomach the syntax, that is. As I mentioned previously, at this point I’m convinced that ML-family languages haven’t caught on largely for reasons of taste (or lack thereof — just look at the js_of_ocaml website’s font selection).

The Goals

So what are we trying to do with this thing?

The language doesn’t need anything new or innovative. ML gives us how bidirectional type inference, sum types with pattern matching, and soundness. Haskell gives us overloading (i.e. type classes, i.e. type-indexed values). Go shows us how important a solid experience is, both for the first time and in daily use. Python shows us how it’s possible to connect predictable syntax and solid, composable semantics.

Thus, this is largely a human factors project. Our goal is to see if we can marry all of those things into one language.

Our goals are to have

  • Memory safety.
  • Static typing without heavyweight type annotations everywhere.
  • Language features that are composable.
  • A minimal runtime, with no large platform dependency.
  • Obvious execution semantics. Like imperative languages, you should be able to look at a program and understand its runtime behavior.
  • Tight, clean compilation to JavaScript. Should be a suitable alternative to TypeScript or CoffeeScript.
  • A reasonable replacement for Go or Node on the backend.
  • Cost-free abstractions.

Non-goals

  • C- or Rust-like performance. Rust is already doing a great job there.
  • Support for restricted effects. It’s not clear the usability cost is worth it.

Next time, I’ll either dive into one of the specific design decisions we’ve made, or talk about the language’s overall gestalt or texture.

Designing a Delightful Functional Programming Language

ML and Haskell have amazingly powerful type systems. Their type systems, and to some degree syntax, have widely influenced the modern crop of languages. Both can achieve machine performance. Haskell has fantastic concurrency. Both have concise, boilerplate-free syntax — some would say too concise. Both have the ability to build abstractions without impacting performance.

So why aren’t there more Haskell or ML programmers? Like, it’s fantastic that Rust and Swift borrowed some of their ideas, but why did it take two decades? Why do things like CoffeeScript and Go spread like wildfire instead? Go is especially bizarre because it’s not really very different from Java in many ways, and a lot worse in others.

I’m sure there are many reasons. Tooling is important. Marketing is important. Haskell is (or perhaps just was) largely a research language. The first-time user experience is important. I don’t claim to know every reason why some languages spread and others don’t, but I know at least one thing: syntax matters a whole lot more than we’d like to admit.

Consider all of the bracing and spacing and indentation arguments people have about C.

Remember that brilliant people have whined about the fact that Python uses whitespace to indicate nesting, as if that materially affects how much work they can get done.

Hell, CoffeeScript gained significant popularity by adding nothing but syntax to JavaScript. There were many opportunities to add actual features to CoffeeScript, like lightweight coroutine syntax, but CoffeeScript’s motto was “It’s just JavaScript.” Its popularity is based on the strength of the syntax alone.

There are two ways to interpret this phenomenon. One is “stupid humans and their quirks and biases, if only the uneducated masses would see the light and focus on what really matters”. Another is the realization that programming is a deeply human activity. A programming language lets humans take their human thoughts and map them into something a computer can execute. The aesthetics of that activity are important.

If a language is going to get adopted, at least without significant corporate backing, it needs to appeal to peoples’ tastes.

Back to Haskell and ML. They have their proponents, but it’s only until now that they’re starting to dance with the mainstream. It’s a shame, because software would be in a much better place if all the material benefits of ML and Haskell (bidirectional type inference, cheap and safe concurrency, sum types) had spread faster.

I’ll share some relevant experiences.

My personal OCaml experience occurred in university. One day I decided to play with this OCaml thing. My rationale is perhaps embarrassing, but I quit for two reasons: the addition operator for integers is + but for floats it’s +.. The other thing was the pervasive use of ;;, at least in the repl. Honestly, it felt tasteless, so I dropped it and moved on, never getting to see the real benefits (polymorphism, type inference, modules, sum types).

Now, Haskell. I was part of the effort to get Haskell adopted at IMVU. The benefits of using Haskell were simply too great to ignore. But when we tried to get everyone to learn it, a nontrivial fraction of engineers developed a visceral aversion. “I understand all the benefits but… I just hate it.” “Why do you hate it?” “The whitespace, $, the two let forms, the weird function names… I dunno, it’s just ugly and I hate it.”

It made me deeply sad that these seemingly superficial complaints prevented what should have been a slam dunk, at least when it came to safety and performance and maintainability and development speed.

When I joined Dropbox, I had some conversations with teammates about the benefits of something like Haskell (most people either don’t know about it or think it’s some academic obscurity), and I got a serious negative reaction to the idea of static types. “They’re heavyweight.” “They’re hard to learn.” “They hinder refactoring.” “They hurt development speed.”

None of these are necessarily factual. After I poked and prodded, I figured out that the resistance to the idea of static typing comes from prior exposure to things like C++ or Java (or Go), which have especially verbose syntaxes and type systems. This is similar to what happens with functional programming, where a common reaction to the idea is “Ugh, I hated my lisp class in college.”

These realizations led Andy and I to take on a project: design a programming language with all the expressive power and semantic correctness of an ML, yet with a syntax and aesthetic that would be instantly recognizable to your typical Go or JavaScript programmer. Surely it’s achievable. This is largely a human factors project – while the PL theory is important, we do not intend to significantly innovate there. Instead, the main innovation is applying familiarity and usability to a domain with the opposite reputation.

In subsequent posts, I will describe some of DFPL’s design decisions.