Assume Good Intentions

Email and text messaging are cold. It’s easy to assume the person on the other side of the screen is aggressive or upset or thinks poorly of you, and respond in kind.

Many years ago, I worked on the SCons build system. Much of the development occurred on mailing lists. Every so often someone would come in, guns blazing, asking why X was so bad or why Y didn’t work. The project lead, Steven Knight, always managed to either get useful information from the person or, if they were truly trolling, deflect and ignore the onslaught.

I once asked him how he did such a great job keeping the community calm. He said “I assume everyone has a good heart.” If a person comes in with a problem or does something you don’t like, gently figure what the real issue is. It’s probably something you can solve.

Quasi-Succinct Data Structures

I thought this paper was really cool: Semi-Indexing
Semi-Structured Data in Tiny Space
. The idea is pretty simple but it uses two ideas I wasn’t familiar with: Elias-Fano coding and balanced parentheses coding.

Imagine you have a bunch of JSON documents that you want to keep around without reencoding in some way, and you want to occasionally query the document for a selection of fields. Parsing JSON is not terribly efficient, and you have to parse most of the file just to read, say, a single key.

The key observation is that you can store a bitstream of information alongside the original JSON document that describes enough of the parse tree to be able to find a key without parsing the entire document. JSON is LL(1) so the type of each node is determined by its first byte in the file. Thus, the locations of each JSON node are a monotonically-increasing set of integer indices into the source file. Elias-Fano codes are a very efficient (quasi-succinct) encoding for such a set. In addition, the nesting hierarchy is encoding with a balanced parentheses bit pattern code.

The result is that it’s possible to preparse the document into a data structure about 25% of the size of the original JSON document that allows direct field lookups.

The Quasi-Succinct Indices paper gives some background information on the encoding and its efficient implementation. An implementation of a set of succinct data structures is available on GitHub.

The author also provides an implementation of this preparsing algorithm for JSON on GitHub.

All of that said, I do slightly question this approach’s utility: it’s probably a bigger win to come up with an isomorphic encoding of JSON that allows direct access and still allows reconstruction of the original document if necessary.

s/

Companies are built of many projects and repositories. Each tends to develop its own culture and tools for interacting with it.

The client is compiled with MSVC, and tests are run with ./runner.py --test foo/bar/baz_test.py.

The website builds with SCons and tests are run with bin/run-tests.sh.

The utility library has ./build and ./test.

Over time, some projects put scripts in the root, some in bin, maybe some in a directory named scripts. Some projects might use all three.

This makes it hard for people coming to the project to know 1) how to do common project tasks and 2) what the set of available commands even is.

When code is collectively owned and engineers contribute across the entire stack, it’s important that anyone can easily check out a repository and start developing in it, no matter the language or tooling.

At IMVU we solved this problem by introducing a directory to every project named s/.

  • How do you build the project? s/build
  • How do you run the tests? s/test
  • How do you deploy? s/deploy
  • How do you lint? s/lint
  • How do you launch the program? s/run
  • What if there’s a server component? s/server
  • How do you see what commands are available? ls s

At first there was some resistance, primarily to the name. “s is too short. Nobody will know what it means.” But after living with it, it was perfect:

  • It’s short.
  • It’s on home row for both qwerty and dvorak.
  • Unlike bin and scripts, s doesn’t already have a semantic meaning. bin, for example, is often used for programs generated by the build script. Conflating that with project tooling is confusing.

s is a home only for project commands – the interface that developers use to interact with the project. These commands should use the same nomenclature as your company (e.g. if people say build, call it s/build; if people say compile, call it s/compile).

The scripts shouldn’t have extensions, because, importantly, the programming language is an implementation detail. That is, s/build.sh or s/build.py are wrong. s/build lets you be consistent across projects and have the option to migrate from Python to Bash or whatever.

s/ is a simple trick, but it goes a long way towards helping people migrate between projects!

Thinking About Performance – Notes

I gave an internal presentation at Dropbox (sorry, video is not sharable publicly) about engineering software for performance. Here are a bunch of resources that went into the presentation.

Similar Presentations

The Economics of Performance

Humans

Vision

Touch

Reaction Times

Response Time

Perception of Time

Computers

Latency

Examples of High-Performance Code

CPU architecture

In the talk I intentionally left out some detail – technically the branch predictor and branch target predictor are different things.

Agner Fog has amazing resources for CPU optimization, including his famous x86 instruction tables. It’s helpful to scan the latencies and reciprocal throughputs of common instructions.

Haswell

Apple A9

Caches, Memory, and Atomics

Memory Bandwidth

Branch Prediction

Miscellaneous

Designing Crux – Record Traits

I’ve intentionally tried not to break much new ground with Crux’s type system. I mostly want the language to be a pleasant, well-executed, modern language and type system for the web.

That said, when porting some web APIs to Crux, I ran into an bothersome issue with an interesting solution. I’ll motivate the problem a bit and then share the solution below.

Records and Row Polymorphism

Crux has row-polymorphic record types. Think of records as anonymous little structs that happen to be represented by JavaScript objects. Row polymorphism means that functions can be written to accept many different record types, as long as they have some subset of properties.

fun length(vec) {
    math.sqrt(vec.x * vec.x + vec.y * vec.y)
}

length({x: 10, y: 20})
length({x: 3, y: 4, name: "p1"})
length({x: 15})               // type error, missing property y
length({x: "one", y: "two"})  // type error, x and y must be numbers

I lived in a TypeScript codebase for nine months or so and the convenience of being able to quickly define anonymous record types is wonderful. Sometimes you simply want to return a value that contains three named fields, and defining a new type and giving it a name is busywork that contributes to the common belief that statically typed languages feel “heavy”.

Crux supports both nominal types (String, CustomerID, Array, …) and anonymous records, sometimes called structural types.

The problem here is how structural types interact with traits.

Imagine coming to Crux for the first time and trying to encode some values as JSON.

json.encode(10)              // returns "10"
json.encode("hello")         // returns "\"hello\""
json.encode(True)            // "true"
json.encode([15, 30, 12])    // "[15,30,12]"
json.encode(js.Null)         // "null"

json.encode({code: 200, text: "OK"})
// Type error: records don't implement traits

Wait, what? Why does everything work but records?

Records and Traits

json.encode‘s type is fun encode<V: ToJSON>(value: V) — that is, it accepts any value which implements the ToJSON trait. Why don’t records implement traits? Well, record types are anonymous, as described before. They don’t necessarily have unique definitions or names, so how would we define a ToJSON instance for this record?

Haskell, and other statically-typed languages, simply have the programmer manually construct a HashMap and JSON-encode that. In Crux, that approach might look something like:

let map = hashmap.new()
map["x"] = 10
map["y"] = 20
json.encode(map)

Not too bad, but not as nice as using record syntax here.

To solve this problem, which is “only” a human factors problem (but remember that Crux is intended to be delightful), I came up with something that I believe to be novel. I haven’t seen this technique before in the row polymorphism literature or any languages I’ve studied.

There are two parts to this problem. First we want to use record literals to construct key-value maps, and then we want to define trait instances that can use said maps.

Dictionaries

As I mentioned, Crux records are implemented as JavaScript objects. This is pretty convenient when interfacing with JavaScript APIs.

data jsffi ReadyState {
    UNSENT = 0,
    OPENED = 1,
    HEADERS_RECEIVED = 2,
    LOADING = 3,
    DONE = 4,
}

type XMLHttpRequest = {
    mutable onreadystatechange: () => (),
    readyState: ReadyState,
    responseText: JSOption<String>,
    // ...
}

Sometimes, however, JavaScript objects are used as key-value maps instead of records. Crux provides a Dict type for that case. Dict is like Map, except keys are restricted to strings to allow implementation on a plain JavaScript object.

let d = dict.new()
d["happy"] = True
d["sad"] = False
json.encode(d)  // "{\"happy\":true,\"sad\":false}"

There is a ToJSON instance for any Dict<V> where V: ToJSON. But can we provide better syntax for creating one? It would be nicer to write:

let d = dict.from({
    happy: True,
    sad: False,
})

To implement dict.from, we need a new type of record constraint: one that says, I accept any set of fields, as long as every value has a consistent type.

fun from<V>(record: {...: V}): Dict<V> {
    // some unsafe JS code to copy the record into a mutable JS object
}

Read ... as “arbitrary set of fields” and : V as “has type V”. Thus, {...: V} is the new type of record constraint, and it’s read as “record with arbitrary set of fields where each field’s value has type V”.

So now we can write:

json.encode(dict.from({
    happy: True,
    sad: False,
}))

Better, but we’re still not done. For one, dict.from requires the values to have the same type — not necessary for JSON encoding. Two, it’s annoying to have to call two functions to quickly create a JSON object.

Defining Record Instances

Here’s where record trait instances come in. First we need to convert all the record’s field values into a common type T. Then, once the record has been converted into a record of type {...: T}, it can be passed to dict.from and then json.encode. The resulting syntax is:

“`crux
// Read as: implement ToJSON for records where…
impl json.ToJSON {…} {
// …this code runs across each property of the record,
// producing a value of type {…: json.Value}…
for fieldValue {
json.toJSON(fieldValue)
}

// which is then encoded into a JSON object.
toJSON(record) {
    json.toJSON(dict.from(record))
}

}
“`

And there you have it, a ToJSON instance for any record, at least as long as the record’s field types implement ToJSON themselves.

json.encode({
    code: 200,
    status: "OK",
})

I’ve never seen a feature like this in the literature or languages I’ve studied, but if you have, please let me know!

Why isn’t Crux a pure functional language?

Nicolas Hery asked on Twitter: “[Why isn’t Crux] pure like Haskell/Elm/etc.?” Good question, and one worth writing about.

First of all, the term “pure functional language” isn’t terribly enlightening. It’s certainly not useful for communication, because everyone has or invents their own definition on the fly.

There are a few independent ideas here: mutable vs. immutable data structures, mutable vs. immutable names/variables, and whether effects are tracked.

Regarding mutable vs. immutable data structures: in reality, mutable data structures are simply unavoidable. At some point you need to update your in-memory representation. There are ways to encode that with immutable data structures, but they get awkward fast. In addition, constantly producing new immutable values places a great deal of pressure on the garbage collector – often it’s fastest just to update your data in-place. Immutable data is a fantastic tool, used appropriately, but it sucks when it’s a straitjacket.

Regarding mutable vs. immutable local variables: imagine you want to sum up the elements in an array. If Crux were pure-functional, that
could be written something like:

fun sum_(list, index) {
  if index >= len(list) {
    0
  } else {
    list[index] + sum_(list, index + 1)
  }
}
fun sum(list) {
  sum_(list, 0)
}

Oh wait, that’s not tail-recursive, so it’ll blow up the stack with a long enough list. Let’s rewrite it:

fun sum_(list, index, total) {
  if index >= len(list) {
    total
  } else {
    sum_(list, index + 1, total + list[index])
  }
}
fun sum(list) {
  sum_(list, 0, 0)
}

OK, now we’re tail recursive. For efficiency, however, the compiler needs to turn that into a tight loop of instructions, storing the accumulators in registers. Wait, so you’re saying the language forced me to translate a simple imperative function (sum the elements in a list) into a tail-recursive loop, which the compiler then has to convert back into imperative code for efficiency? Why can’t the compiler just turn my natural imperative loop into a tail-recursive implementation, if that’s what it really wants? (It definitely could.) And it gets even worse the larger your functions become, especially if they have early exits and multiple “mutable variables”. In short, tail calls are great, when appropriate, but always programming with tail recursion is like someone rejecting all your commits with “Nope, you must contort your code for my pleasure.” And I shudder to think of what a tail-recursive implementation of a huge interpreter loop with lots of mutable variables would look like.

Tail calls are great, but being forced to write tail-recursive loops is dumb and one of my biggest annoyances with Haskell. Instead, why not just write:

fun sum(list) {
  let mutable total = 0
  for i in list {
      total += i
  }
  return total
}

Crux makes that natural.

However! Everything I said above is independent of what the defaults are. Crux variables and values are immutable by default, since that’s the common case. Sum types are always immutable. The only things that can be mutable are record fields and locals, but, again, you must opt into that functionality. It’s not annoying in practice; since everything’s an expression in Crux, there’s less need to mutate locals.

Okay, so instead let’s presume that most people consider pure functional to mean “has restricted side effects”, like Haskell.

Personally, I’m a huge fan of monads in Haskell, and the amazing
things they enable.

However, they come at a large cost: since nearly everything is a Monad, the error messages get really complicated, especially when the compiler thinks you meant the Maybe monad or a function monad. (Isn’t it crazy how, in Haskell, Integers aren’t a Monoid by default, but all functions are Monads?)

Eventually I’d love Crux to have an effect tracking system. Different syntax between pure code and effectful code is unacceptable, so the Haskell approach is a no-go. I am excited, however, about row-typed effect systems like Koka. (Notably, it should be okay for pure functions to use mutable locals, like the sum function above.) Future work!

To summarize, I think we can achieve most of the benefits of “pure functional programming” while retaining accessible syntax and allowing mutability as desired and convenient.

The Story of Haskell at IMVU

An interesting blog post and its corresponding comment thread on Hacker News made me realize that, while I’d told the story of Haskell at IMVU to many people, I’d never written it down. :)

IMVU’s backend had been written in PHP from the beginning. Startup code is always gross, but after years of learning PHP’s ins and outs, we bent it to our will. Part of me looks back and says “It wasn’t that bad”, and indeed, it has a lot of great properties. But then I remember the horror.

The terrible straight-line performance, the lack of concurrent IO, the insane memory usage… Once the codebase reached about a million lines, the latency on our REST service response times was over half a second and in the worst case several seconds. For some types of services, that’s okay, but IMVU strives to be a real-time, delightful experience, and service latency on interactive things like dressing up, browsing the catalog, sending messages to your friends, and shopping is really important.

Some of the performance-minded of us tried to get PHP to be fast, but it was basically intractable, and we couldn’t afford to fork PHP with a high-performance JIT like Facebook ended up doing years later.

The combination of PHP’s awful performance, terrifying pitfalls, and the fact that millions of lines of code in a dynamic language (even with copious test coverage) makes it really hard to refactor, led us to evaluate alternatives. Someone suggested Java (which would have been totally fine in hindsight) but the infrastructure team decided to adopt this newfangled Node.js thing. Node sucked. This predated ES6, promises, and the rich ecosystem of npm, so we spent far too long rebuilding basic, basic, infrastructure (like propagating stack traces across async operations) in Node that we already had in PHP, and it still wasn’t as good. In the end we decided to drop Node and stick with PHP.

Fast forward a year or two…

There were some people at IMVU who had dabbled with Haskell, but it always seemed too idealistic and not very practical. In fact, one weekend, Andy went home with the explicit goal of proving that Haskell was stupid and not good for writing production software.

He came back on Monday glowing. “Haskell is perfect for web services! We should use it! It’s fast and safe and concurrent!” We smiled and nodded (sometimes Andy gets enthusiastic about new things) and went on with our daily work. But in the evenings and weekends he plugged away, and built a prototype web service that was 1) dramatically faster than the corresponding PHP services 2) actually quite short in implementation and 3) surprisingly clear. Huh, maybe this Haskell thing did have legs.

OK, so let’s consider the possibility that Haskell could be something IMVU could adopt. How we would vet it?

We started by, during the next hack week, forming a small team of everyday engineers to learn Haskell and work on a feature. We learned two things: teaching Haskell was a nontrivial problem. BUT, Haskell newcomers hacking on the codebase only caused one regression, and that regression actually exposed a bug in some of our infrastructure. Well that’s pretty cool. For years, IMVU had a tradition of throwing new engineers into the deep end, after which they’d invariably take down the site, and then we’d demonstrate that we’re egoless and all failures are an opportunity to improve our process with a postmortem. What if Haskell could help us avoid some of these problems?

Over time we slowly ramped up Haskell within the company. At first we had growing pains. The infrastructure in our PHP stack was extremely mature and well-baked. The corresponding infrastructure in our Haskell (error reporting, DB access, unit testing) was not. It took us a while to sort out exactly what our Haskell best practices would be, and by that point a few people started to get a negative vibe about Haskell. But we kept pushing forward.

(By the way, I keep saying “we”, but at this point I hadn’t written any Haskell yet, and was not particularly for or against it, even though I did think PHP was terrible.)

By this point we’d shipped at least two sets of mobile services on Haskell, and many people considered it a success. However… Tensions about Haskell had really started to heat up, with some people saying things like “I never want to even look at Haskell again” or “at this stage in my career I don’t want another programming language” or “I think we’re making a horrible mistake by building another stack, especially in Haskell.” People were complaining to their managers, and the managers were coming to me. Some people loved Haskell, some people hated it, but very few were in the middle.

As a neutral third party, and as someone with a lot of respect in the organization, the engineering managers sent me out to gather ground truth. I took my little black notebook and a pen and interviewed something like half the engineering team on their opinions on Haskell, whether they were fans or not. I spent an hour with each person, asking what they liked, what they didn’t like, what they thought we should do, etc.

I then collated all the feedback and presented back to the engineering managers and leadership team (of which I was a part too, but since I had no direct reports and had no experience with Haskell at IMVU, I was fairly unbiased). I don’t have my report handy, but it went approximately like this:

75% of the people I interviewed were positive about or fine with Haskell.

25% of the people had major concerns. The concerns, in roughly decreasing priority, were:

  • Haskell strings suck
    • Specifically there’s a lot of cognitive overhead in switching between the five common string representations as needed
  • Syntactic concerns
    • Haskell’s syntax is iconoclastic: parens, $, /= instead of !=
    • some people in the Haskell community love point-free style
    • cognitive overhead in switching between pure and monadic style
    • let vs. where
  • concerns about a second backend stack
    • duplicate infrastructure across PHP and Haskell
  • concerns about it not being a mainstream language
    • teaching the whole organization
    • interaction between Haskell experts and novices sometimes leaves the novices feeling dumb or inadequate
    • bus factor to organization on key infrastructure
    • Haskell is less debuggable by ops engineers

The wins were:

  • GHC’s support for safe concurrent programming is world-class
  • Sound type safety led to refactoring being a breeze
  • Fun, stretches the mind (for some people)

Ultimately, we settled on a middle ground. We would not deprecate PHP, but we would continue to support Haskell as a technology stack. We had specific services in mind that needed high throughput and low latency (<100 ms) and we knew that PHP could not achieve those. That said, we respected the fact that several engineers, even those in leadership roles, put their foot down and said they didn’t want to touch Haskell. (From my perspective it was kind of a weird experience. Highly-skilled people that I respect just got really angry and irrational and basically refused to even try to learn Haskell, which resulted in a culture divide in the company.)

In the end, I never did manage to convince some of my peers to try it. That said, there were many engineers who loved Haskell and we built a great deal of critical infrastructure in it. The Haskell infrastructure basically hums like a well-oiled machine. Perfectly reliable unit tests. Concurrent batched IO. Abstractions for things like timeouts and running tasks on other threads while correctly bubbling failures. Fantastic support for safely encoding and decoding JSON. One of the best benchmarking tools I’ve ever used.

Since one of the biggest concerns about Haskell was education, I taught a sequence of four or five classes for the engineering team with my take on how it should be taught. Traditional Haskell materials tend to talk about laziness and currying and all this FP junk that is certainly interesting and different, but mostly irrelevant to people trying to come in hot and get the job done. Instead, I focused on specifics first, and then generalizations.

“You know how you can add two integers and get a new integer back? Well, you can also add two floating point numbers. Int and Float are instances of Num.”

“You know how you can concatenate strings? And concatenating with the empty string gives you the same value? It turns out many things satisfy these properties. They’re called Monoids.”

“You know how you can map across a list? We do it all the time in PHP or Python. Well you can also map across Maybe. Think of Maybe as a list that can have either zero or one element. Things that can be mapped are called Functors.”

etc. etc.

We started with specific examples of how to do stuff with Haskell, then talked about type classes, and the final optional class was about how Haskell is compiled down to the machine. You’d have to ask the attendees how much they remember, but the classes were well-reviewed at the time. :)

In summary, I definitely wouldn’t say that Haskell was a failure. GHC is an amazing piece of technology, and IMVU built infrastructure in it that wouldn’t have been possible in PHP. I also can’t say Haskell was a resounding success. I was truly disturbed by the culture drama that it brought. And, frankly, saddened and confused by how some people closed their minds. Personally, I would have happily traded a lot of Haskell’s safety for avoiding drama. I sometimes wonder what would have happened had we chosen Java instead of Node.js back in 2010. We could have hit our performance goals in Java too.

Oh well. Many of us still love Haskell. It’s not going away. I left IMVU last year (needed some more experience elsewhere) but I personally miss Haskell a great deal. If I were starting a new project with people that I trusted, I’d definitely reach for Haskell first. The expressiveness, brevity, and safety let you move extremely quickly, and Haskell is definitely mature enough for production use.

Designing Crux – Import Statements

Manipulating a module’s import list is a high-frequency activity in modern programming languages. Sadly, few popular languages perfect the usability of the import list.

To do a good job in Crux, let’s start from the use cases, roughly sorted by importance:

  • import a module and refer to it by a qualified import name
  • import a list of names unqualified into this module’s scope
  • import everything unqualified into this module’s scope
  • import a module solely for its side effects, without importing any symbols
  • visually scan the import list
  • sort the import list (with Emacs’s sort-lines or the like)
  • and, of course, tools must easily parse the import list

Designing for usability means optimizing for the most common use cases, while making sure the less common use cases are still possible.

The most common use case is to import a module and refer to it qualified. Consider this hypothetical syntax:

import fs.path
path.combine("/usr", "local")

The qualified import reference (here, path) defaults to the last segment of the module name. Now, for this to work well, the basename has to be short, distinct, and meaningful.

Haskell gets this wrong.

import qualified Data.ByteString as BS

From left to right: import qualified is long to start with. Then you have most packages in some kind of arbitary namespace like Data. Then, since ByteString is too long to be convenient (especially with the mixed case), people typically name the import BS. Typically being the key word. Sometimes people name it Bytes. Sometimes ByteString. Sometimes B. Sometimes Data.ByteString.Char8 is imported as BS. The fact that the programmer has to make a choice here naturally leads to inconsistency.

Go does a great job here. In Go, you import like this:

import "fs/path"

Afterwards, path functions can be accessed as path.Function. Short and meaningful.

Go goes even further and provides guidelines for naming functions in the context of a module. Conventions tend to be copied, so by setting a good standard early on, Go positively affected every project built then on.

ES6 modules are also pretty interesting. They are statically analyzable and syntactically lightweight, but they have a fairly significant problem at scale: sorting imports.

import Dispatcher from 'flux/dispatcher';
import {foo, bazzle} from 'app/utils';

The problem with having the list of imported symbols first is that they can’t easily be machine-sorted. Every ES6 project I’ve worked on has encouraged its engineers to sort the imports by hand. Also, the import list coming first gets annoying when import lists are long.

import {
  foo, bar, baz,
  qux, harf, barf,
  hurp, floop
} from 'my_module';

With Crux, we learned from all of the above import systems and came up with something that meets every use case. Here is the proposed syntax:

import {
  bytes,
  flux.dispatcher,
  my_encoding_system as mes,
  imported_only_for_side_effects as _,
  utils(foo, bar, baz),
  more_utils(
    foo2,
    bar2,
    baz2,
  ),
  all_the_utils(...), // import everything into scope
}

The easiest import syntax is most common one, but other types of imports, like renaming the import or importing some symbols unqualified, are straightforward too.

This directly satisfies all of our use cases!

We may change our minds and make the commas after each import optional in the future.

Side Note: Implicit vs. Explicit Imports

OCaml and Java allow qualifying names at their use sites, making import statements unnecessary. Import statements are syntax sugar at that point. I don’t have a strong justification here, but Crux went with explicit import statements (like Go, Python, Haskell, ES6) because it’s easier for humans and machines to see a module’s dependencies at a glance.

Designing Crux – Methods

Crux’s foundations are rooted in the ML language family. Everything is a function. Haskell has this property too, but there is a really sucky aspect of an everything-is-a-function world.

Imports.

Here’s a common experience I have with Haskell:

let url = getCustomerUrl myCustomerId

OK, I have a string.

I need to check if it starts with “https://”

Scroll to top of file… That means I need to call the startsWith function. What module does it live in again?

Think… is this a Text or a ByteString? Or maybe a String? That determines which module I import…

This is a very common bit of friction, and it comes up all the time in Haskell, both for the standard library and your own code.

Now let’s hypothesize what reading from a data store might feel like in a world with explicit imports:

OK, my React component has an env prop.

I know the UserStore is on the env, but I need to import the Environment module to call getUserStore:

let userStore = Environment.getUserStore(env).

Now I want the list of users, so import UserStore and:

let users = UserStore.getUserList(userStore)

I can see it now: “Why can’t I just write let users = env.getUserStore().getUserList()? Crux sucks!”

In addition, assuming a roughly-one-type-per-module structure, each type gets two “names” that must be used: the module name and the type name. If types are refactored to live in different modules, all the callers have to be updated with new imports. Finally, if those names diverge, it could get confusing.

Dynamic languages like JavaScript and Python solve this usability issue by implementing all method calls with dynamic lookup. Besides the obvious performance hit, this can make analyzing the call graph nontrivial — both by humans and tools.

Languages like C++ and Java and Go take a different approach: they make methods a first-class concept. This adds some complication to the language when you want to use a method as a normal function. C++ has adapters that let you use members as functions sometimes but it’s nicer if methods are just functions.

Crux is not an object-oriented language, but we want the usability benefits of “method syntax”, sometimes known as type-directed name resolution:

  1. with no dynamic dispatch
  2. without any specialize method concept — methods should just be normal functions
  3. and, importantly, while still supporting bidirectional type inference for record types

The reason we have to worry about a conflict with type inference is because records already use dot syntax. Consider the following function:

fun distance(start, end) {
  return sqrt(sqr(end.x - start.x) + sqr(end.y - start.y))
}

Notice no type annotations! The distance function is inferred to take two records, each of which has floating point x and y properties. It returns a float.

Since we think it’s important that . syntax be used for records, Crux uses an arrow -> for method calls. Method calls only work when the type is known. Fortunately, in most programs, the type inference engine already knows the concrete types of most things. If you try to write a function like:

fun isHTTPSURL(url) {
  return url->startsWith("https://")
}

you will get a type error, since it doesn’t know what url is. The fix is easy: provide a type annotation on the argument:

fun isHTTPSURL(url: String) {
  return url->startsWith("https://")
}

In practice, most of the time, the type is known from context. Let’s go back to the Flux UserStore example. With method syntax, it would be spelled:

let users = env->getUserStore()->getUserList()

-> is mainly a way to avoid imports. a->b() can be read as “assuming the type of a is known, look up the symbol b in the module that defined a, and call b(a)”

The env example works because the type of env is known. Therefore, the location of the getUserStore() function is known, and it’s known to return a UserStore, so getUserList is looked up there. No type annotations required, there’s nothing special about member functions, and we achieve all the performance and tooling benefits of static dispatch!

One note: for simplicity, we have been applying the rule that the type must have been inferred before the method call is resolved. Technically, this is more restrictive than necessary. Consider the following example:

fun addSuffix(p) {
  if p->endsWith(".json") {
    return p
  } else {
    return p + ".json"
  }
}

The p + ".json" expression proves that p’s type is String, and therefore p->endsWith(".json") could be resolved. However, supporting this kind of backwards knowledge transfer would complicate the compiler (I think). Perhaps we’ll look into fixing it later. :)

Another note: sometimes you simply want to reference a “member function” without having a particular instance in mind. You might have the type name in scope, but don’t want to bother importing. We’ve considered support syntax like &String::startsWith to reference the startsWith function in the module that defines the String type. It could be useful in situations like ["foo", "bar", "baz"]->map(&String::length).