# A Critique of The Programming Language J

I’ve spent around a year now fiddling with and eventually doing real
data analytic work in the The Programming Language J. J is one of
those languages which produces a special enthusiasm from its users and
in this way it is similar to other unusual programming languages like
Forth or Lisp. My peculiar interest in the language was due to no
language to do analysis in, and an attraction to brevity and the point
free programming style, two aspects of programming which J emphasizes.

Sorry, Ken.

I’ve been moderately happy with it, but after about a year of light
work in the language and then a month of work-in-earnest (writing
interfaces to gnuplot and hive and doing Bayesian inference and
spectral clustering) I now feel I am in a good position to offer a
friendly critique of the language.

## First, The Good

J is terse to nearly the point of obscurity. While terseness is not a
particularly valuable property in a general purpose programming
language (that is, one meant for Software Engineering), there is a
case to be made for it in a data analytical language. Much of my work
involves interactive exploration of the structure of data and for that sort
of workflow, being able to quickly try a few different ways of
chopping, slicing or reducing some big pile of data is pretty
handy. That you can also just copy and paste these snippets into some
analysis pipeline in a file somewhere is also nice. In other words,
terseness allows an agile sort of development style.

Much of this terseness is enabled by built in support for tacit
programming. What this means is that certain expressions in J are
interpreted at function level. That is, they denote, given a set of
verbs in a particular arrangement, a new verb, without ever explicitly
mentioning values.

For example, we might want a function which adds up all the maximum
values selected from the rows of an array. In J:

+/@:(>./"1)


J takes considerable experience to read, particularly in Tacit
style. The above denotes, from RIGHT to LEFT: for each row ("1)
reduce (/) that row using the maximum operation >. and then (@:)
reduce (/) the result using addition (+). In english, this means:
find the max of each row and sum the results.

Note that the meaning of this expression is itself a verb, that is
something which operates on data. We may capture that meaning:

sumMax =: +/@:(>./"1)


Or use it directly:

+/@:(>./"1) ? (10 10 $10)  Tacit programming is enabled by a few syntactic rules (the so-called hooks and forks) and by a bunch of function level operators called adverbs and conjuctions. (For instance, @: is a conjunction rougly denoting function composition while the expression +/ % # is a fork, denoting the average operation. The forkness is that it is three expressions denoting verbs separated by spaces. The details obscure the value: its nice to program at function level and it is nice to have a terse denotation of common operations. J has one other really nice trick up its sleeve called verb rank . Rank itself is not an unusual idea in data analytic languages: it just refers to the length of the shape of the matrix; that is, its dimensionality. We might want to say a bit about J’s basic evaluation strategy before explaining rank, since it makes the origin of the idea more clear. All verbs in J take one or two arguments on the left and the right. Single argument verbs are called monads, two argument verbs are called dyads. Verbs can be either monadic or dyadic in which case we call the invocation itself monadic or dyadic. Most of J’s built-in operators are both monadic and dyadic, and often the two meanings are unrelated. NB. monadic and dyadic invocations of < 4 < 3 NB. evaluates to 0 <3 NB. evalutes to 3, but in a box. Give that the arguments (usually called x and y respectively) are often matrices it is natural to think of a verb as some sort of matrix operator, in which case it has, like any matrix operation, an expected dimensionality on its two sides. This is sort of what verb rank is like in J: the verb itself carries along some information about how its logic operates on its operands. For instance, the built-in verb -: (called match) compares two things structurally. Naturally, it applies to its operands as a whole. But we might want to compare two lists of objects via match, resulting in a list of results. We can do that by modifying the rank of -: x -:”(1 1) y The expression -:”(1 1) denotes a version of match which applies to the elements of x and y, each treated as a list. Rank in J is roughly analogous the the use of repmat, permute and reshape in Matlab: we can use rank annotations to quickly describe how verbs operate on their operands in hopes of pushing looping down into the C engine, where it can be executed quickly. To recap: array orientation, terseness, tacit programming and rank are the really nice parts of the language. ## The Bad and the Ugly As a programming environment J can be productive and efficient, but it is not without flaws. Most of these have to do with irregularities in the syntax and semantics which make the language confusing without offering additional power. These unusual design choices are particularly apparent when J is compared to more modern programming languages. ### Fixed Verb Arities As indicated above, J verbs, the nearest cousin to functions or procedures from other programming languages, have arity 1 or arity 2. A single symbol may denote expressions of both arity, in which case context determines which function body is executed. There are two issues here, at least. The first is that we often want functions of more than two arguments. In J the approach is to pass boxed arrays to the verb. There is some syntactic sugar to support this strategy: multiArgVerb =: monad define ‘arg1 arg2 arg3’ =. y NB. do stuff ) If a string appears as the left operand of the =. operator, then simple destructuring occurs. Boxed items are unboxed by this operation, so we typically see invocations like: multiArgVerb('a string';10;'another string')  But note that the expression on the right (starting with the open parentheses) just denotes a boxed array. This solution is fine, but it does short-circuit J’s notion of verb rank: we may specify the the rank with which the function operates on its left or right operand as a whole, but not on the individual “arguments” of a boxed array. But nothing about the concept of rank demands that it be restricted to one or two argument functions: rank entirely relates to how arguments are extracted from array valued primitive arguments and dealt to the verb body. This idea can be generalized to functions of arbitrary argument count. Apart from this, there is the minor gripe that denoting such single use boxed arrays with ; feels clumsy. Call that the Lisper’s bias: the best separator is the space character.1 A second, related problem is that you can’t have a zero argument function either. This isn’t the only language where this happens (Standard ML and OCaml also have this tradition, though I think it is weird there too). The problem in J is that it would feel natural to have such functions and to be able to mention them. Consider the following definitions: o1 =: 1&- o2 =: -&1 (o1 (0 1 2 3 4)); (o2 (0 1 2 3 4)) ┌────────────┬──────────┐ │1 0 _1 _2 _3│_1 0 1 2 3│ └────────────┴──────────┘  So far so good. Apparently using the & conjunction (called “bond”) we can partially apply a two-argument verb on either the left or the right. It is natural to ask what would happen if we bonded twice. (o1&1) o1&1 Ok, so it produces a verb.  3 3$ ''
;'o1'
;'o2'
;'right'
;((o1&1 (0 1 2 3 4))
; (o2&1 (0 1 2 3 4))
;'left'
; (1&o1 (0 1 2 3 4))
; (1&o2 (0 1 2 3 4)))

┌─────┬────────────┬────────────┐
│     │o1          │o2          │
├─────┼────────────┼────────────┤
│right│1 0 1 0 1   │1 0 _1 _2 _3│
├─────┼────────────┼────────────┤
│left │1 0 _1 _2 _3│_1 0 1 2 3  │
└─────┴────────────┴────────────┘


I would describe these results as goofy, if not entirely impossible to
understand (though I challenge the reader to do so). However, none of
them really seem right, in my opinion.

I would argue that one of two possibilities would make some sense.

1. (1&-)&1 -> 0 (eg, 1-1)
2. (1&-)&1 -> 0″_ (that is, the constant function returning 0)

That many of these combinations evaluate to o1 or o2 is doubly
confusing because it ignores a value AND because we can denote
constant functions (via the rank conjunction), as in the expression
0"_.

## Generalizations

What this is all about is that J doesn’t handle the idea of a
function very well. Instead of having a single, unified abstraction
representing operations on things, it has a variety of different ideas
that are function-like (verbs, conjuctions, adverbs, hooks, forks,
gerunds) which in a way puts it ahead of a lot of old-timey languages
like Java 7 without first order functions, but ultimately this
handful of disparate techniques fails to acheive the conceptual unity
of first order functions with lexical scope.

Furthermore, I suggest that nothing whatsoever would be lost (except
J‘s interesting historical development) by collapsing these ideas
into the more typical idea of closure capturing functions.

## Other Warts

### Weird Block Syntax

Getting top-level2 semantics right is hard in any
language. Scheme is famously ambiguous on the subject, but at
least for most practical purposes it is comprehensible. Top-level has
the same syntax and semantics as any other body of code in scheme
(with some restrictions about where define can be evaluated) but in
J neither is the same.

We may write block strings in J like so:

blockString =: 0 : 0
Everything in here is a block string.
)


When the evaluator reads 0:0 it switches to sucking up characters
into a string until it encounters a line with a ) as its first
character. The verb 0:3 does the same except the resulting string is
turned into a verb.

plus =: 3 : 0
x+y
)


However, we can’t nest this syntax, so we can’t define non-tacit
functions inside non-tacit functions. That is, this is illegal:

plus =: 3 : 0
plusHelper =. 3 : 0
x+y
)
x plusHelper y
)


This forces to the programmer to do a lot of lambda lifting
manually, which also forces them to bump into the restrictions on
function arity and their poor interaction with rank behavior, for if
we wish to capture parts of the private environment, we are forced to
pass those parts of the environment in as an argument, forcing us to
give up rank behavior or forcing us to jump up a level to verb
modifiers.

### Scope

Of course, you can define local functions if you do it tacitly:

plus =: 3 : 0
plusHelper =. +
x plusHelper y
)


But, even if you are defining a conjunction or an adverb, whence you
are able to “return” a verb, you can’t capture any local functions –
they disappear as soon as execution leaves the conjunction or adverb
scope.

That is because J is dynamically scoped, so any capture has to be
handled manually, using things like adverbs, conjunctions, or the good
old fashioned fix f., which inserts values from the current scope
directly into the representation of a function. Essentially all modern
languages use lexical scope, which is basically a rule which says: the
value of a variable is exactly what it looks like from reading the
program. Dynamic scope says: the valuable of the variable is whatever
its most recent binding is.

# Recapitulation!

The straight dope, so to speak, is that J is great for a lot of
reasons (terseness, rank) but also a lot of irregular language
features (adverbs, conjunctions, hooks, forks, etc) which could be
folded all down into regular old functions without harming the
benefits of the language, and simplifying it enormously.

If you don’t believe that regular old first order functions with
lexical scope can get us where we need to go, check out my
tacit-programming libraries in R and Javascript. I
even wrote a complete, if ridiculously slow implementation of J‘s
rank feature, literate-style, here.

## Footnotes

1 It bears noting that ; in an expression like (a;b;c)
is not a syntactic element, but a semantic one. That is, it is the
verb called “link” which has the effect of linking its arguments into
a boxed list. It is evaluated like this:

(a;(b;c))


(a;b;c) is nice looking but a little strange: In an expression
(x;y) the effect depends on y is boxed already or not: x is always boxed regardless, but y is boxed only if it wasn’t boxed before.

2 Top level? Top-level is the context where everything
“happens,” if anything happens at all. Tricky things about top-level
are like: can functions refer to functions which are not yet defined,
if you read a program from top to bottom? What about values? Can you
redefine functions, and if so, how do the semantics work? Do functions
which call the redefined function change their behavior, or do they
continue to refer to the old version? What if the calling interface
changes? Can you check types if you imagine that functions might be
instances created before a change in the class definition. Believe or
not, Common Lisp tries to let you do this – and its confusing!

On the opposite end of the spectrum are really static languages like
Haskell, wherein type enforcement and purity ensure that the top-level
is only meaningful as a monolith, for the most part.

# Compose is Better than Dot (or: Function Level Programming In Javascript)

That great minds think alike is a reflection of the fact that certain ideas have an appeal that is, if not innate, then compelling in context. Hence the use of dot notation in the creation of domain specific languages in Javascript: dot is a handy way of succinctly representing a computation with a context. This is closely related to monads, and many javascript libraries relying heavily on dot for syntactic sugar are very closely related to one monad or another (eg _ to the sequence monad, promises to a sort of continuation monad, etc).

(NB. Much of the code here is inspired by a pointfree/function level programming library I am building for Javascript called puff, available here);

What I aim to prove today is that function composition subsumes the functionality of dot in this regard and that we can embed in Javascript a powerful, succinct function-level programming language in the vein of APL, J, or K/Q based primarily on function composition.

First of all, the traditional definition of function composition:

/** compose functions
*  return a new function which applies the last argument to compose
*  to the values passed in and then applies each previous function
*  to the previous result.  The final result is returned.
*/
function compose(){
var fs = Array.prototype.slice.call(arguments, 0, arguments.length);
return function(){
var inArgs = Array.prototype.slice.call(arguments, 0, arguments.length);
var i = fs.length-1;
var res = fs[i].apply(null, inArgs);
i = i - 1;
for(; i>=0; i = i - 1){
res = fs[i](res);
}
return res;
}
}


This version of compose allows the programmer to pass in an unlimited number of functions and returns a new function which takes any number of arguments and then threads the result through all the previous functions, transforming it each time, starting with the last function and ending with the first.

This order of composition is inspired by the fact that the function in a traditional application expression precedes the argument (on the left), transforming:

f(h(g(o)))


“naturally” to

compose(f,h,g)(o)


or, if succinctness is of interest:

var c = compose;
c(f,h,g)(o)


In this way we drop a few parentheses and hopefully express more clearly our intent.

Of course we might not wish to apply our composition immediately: we can produce useful functions via composition, after all.

var plusOne(x){ return x + 1 };
var timesTen(x){ return x*10 };

var plusOneTimesTen = c(timesTen, plusOne);


We can now apply plusOneTimesTen to as many values as we wish. Handy. However, now our intuition about naming and the order of the arguments to c are at odds. Hence, we introduce:

function rCompose(){
return compose.apply(null,
Array.prototype.slice.call(arguments, 0, arguments.length).reverse());
}
var r = rCompose;


So that the above looks a bit nicer:

var plusOne(x){ return x + 1 };
var timesTen(x){ return x*10 };

var plusOneTimesTen = r(plusOne, timesTen);


This reverse composition operator is similar in many respects to dot in javascript except we have abstracted away this, to which each method invocation is implicitly addressed in a dot chain. In addition, instead of simple method names, each element in our r list can be any Javascript expression which evaluates to a function. This means that we can denote any sequence of operations this way without worrying whether or not they have been associated with any particular Javascript object.

With the right set of primitives and combinators, r forms the basis of a powerful, succinct function level programming language in which we can build, among other things, expressive domain specific languages. Or with which we can rapidly denote complex operations in a very small number of characters.

Well, first of all we want the bevy of basic functions:

plus, minus, div, times, split, join, toString, index, call,
apply, etc, array


These behave as you might expect, more or less (eg, some plausible implementations to give you the spirit of the idea):

function plus(){
var out = arguments[0];
Array.prototype.slice.call(arguments, 1, arguments.length)
.forEach(function(x){
out = out + x;
});
return out;
}

function index(o, i){
return o[i];
}

function call(f){
return f.apply(null, Array.prototype.slice.call(arguments, 0, arguments.length));
}


You get the idea.

Astute readers may realize that our composition function seems to only work with functions of a single argument. Remedying this will be a matter of some interest. The simplest approach is to provide for partial application:

/** partially fix arguments to f (on the right)
*
*/
function partialRight(f /*... fixedArgs */){
var fixedArgs = Array.prototype.slice.call(arguments, 1, arguments.length);
var out = function(/*... unfixedArgs */){
var unfixedArgs = Array.prototype.slice.call(arguments, 0, arguments.length);
return f.apply(null,unfixedArgs.concat(fixedArgs));
}
out.toString = function(){
return "partialRight("+f.toString()+","+fixedArgs.join(",")+")";
}
return out;
}

/** partially fix arguments to f (on the left)
*
*/
function partialLeft(f /*... fixedArgs */){
var fixedArgs = Array.prototype.slice.call(arguments, 1, arguments.length);
var out = function(/*... unfixedArgs */){
var unfixedArgs = Array.prototype.slice.call(arguments, 0, arguments.length);
return f.apply(null,fixedArgs.concat(unfixedArgs));
}
out.toString = function(){
return "partialLeft("+f.toString()+","+fixedArgs.join(",")+")";
}
return out;
}


These functions (they might be adverbs in J) take a function and a set of values and return a new function, “fixing” the arguments to the left or right of the argument list, depending on whether we’ve usedpartialLeft or partialRight.

Its handy to introduce the following bindings:

var p_ = partialRight;
var _p = partialLeft;


I hope these are relatively mnemonic (Javascript unfortunately isn’t an ideal environment for very short, expressive names).

We can get a surprising amount of mileage out of these ingredients already. For instance, a function to remove break tags from a string and replace them with newlines (sort of contrived):

var remBreaks = r(p_(split,'<br>'),p_(join,'\n'));


compared to

function remBreaks(s){
return s.split('<br>').join('\n');
}


(NB. If split and join are written as curried functions, as they are in puff, the above is a little shorter:

var remBreaks = r(split('<br>'),join('\n'));


Providing a meaningful default currying (which args should be applied first) is a little tricky, though.)

## Going Further

The above example demonstrates that we can do some handy work with r as long as it involves simply transforming a single value through a set of functions. What may not be obvious here is that a determined programmer can denote any computation whatsoever this way, even if multiple values might need to be kept around and transformed.

Consider the function which I will call augment or au:

/** given a function f and a additional functions gs
*  return a new function h which applies each g
*  to its single argument and then applies f to the
*  resulting list of values
*/
function augment(f /*... gs*/){
var gs = Array.prototype.slice.call(arguments, 1, arguments.length);
var out = function(a){
return f.apply(null, gs.map(function(g){
return g(a);
}));
}
out.toString = function(){
return "augment("+f.toString()+","+gs.map(toString).join(", ")+")";
}
return out;
}


And a nice default currying of index:

function index(o, ix){
if(typeof ix === "undefined"){
var realIx = o;
return function(o){
return o[realIx];
}
} else {
return o[ix];
}
}
var ix = index;


Now:

var fullName = augment(join(' '), ix('first'), ix('last'));


Such that:

fullName({first:'Ronald',last:'Reagan'}) -> "Ronald Reagan"


What have we just accomplished? We’ve demonstrated that we can take an arbitrary function of any arity and automatically transform it into a function which reads its input arguments from a single object. The sort of single object which we might be passing from function to function via r.

## Putting it Together

To go further we need to add just one more utility function: cleave

function cleave(v /*... fs*/){
var fs = Array.prototype.slice.call(arguments, 1, arguments.length);
return fs.map(function(f){
return f(v);
});
}


and the shortcut:

function cl_(){
return function(f){
return cleave.apply(null, [f].concat(Array.prototype.slice.call(arguments,0,arguments.length)));
}
}


(This is just cleave curried on the right, eg in puff: c_(cleave).)

// replaceMiddleName name newMiddleName -> transformedName
var replaceMiddleName = r(args(2),
cl_(r(first, split(' ')), second),
cl_(r(first,first),second,r(first, third)),
au(join(' '), first, second, third));


Let’s go nuts with some of the extra utility functions in puff:

var replaceMiddleName = f(2,
cl_(r(n0,split(' ')),n1),
cl_(n00, n1, n02),
join(' '));


Puff lets you do all of this insanity quite easily. There are browser and nodejs builds. You use it by saying, in node:

require('puff').pollute(global);


Which pollutes the global namespace with the puff functions. Since terseness is the rule I can’t imagine why you’d do otherwise, but you can also say:

var puff = require('puff');
puff.r(...)


<script src="./build/puff-browser.js"></script>


To your head. This will define a global puff object, which you can then use directly or say:

puff.pollute(window);

# Optional Keyword Arguments in J

J is great! It is a wonderful little language for data analysis tasks. However, to programmers used to working in modern dynamic languages, it sometimes feels a little restrictive. In particular, a ubiquitous feature in the post-Python (let’s call it) era is hash-maps. Even in older languages, like Common Lisp, the association list – allowing abritrary mappings between keys and values, is a very common idiom.

J exposes no native functionality that exactly meets this use case, one common application of which is named, optional arguments to functions.

In developing an interactive gnuplot interface for J, I wanted to pass optional keyword arguments to functions so that plots can be easily customized. So I developed a simple representation of key/val pairs which is efficient enough for small collections of named values.

Consider:

nil =: (0$0) rankOne =: 1: -: (#@:$)
rankTwo =: 2: -: (#@:$) talliesEven =: 0: -: (2: | #) twoColumns =: 2: -: 1&{@:$

keysandvals =. y
assert. rankOne keysandvals
assert. talliesEven keysandvals
ii =. 2 | (i. #y)
keys =. (-. ii) # y
vals =. ii # y
keys (>@:[ ; ])"(0 0) vals
)


Opts is a monadic verb which takes a flat boxed array of even length and returns a rank two boxed array whose first column is keys and whose second column is values:

opts 'key1';10;'key2';11
+----+--+
|key1|10|
+----+--+
|key2|11|
+----+--+


Look up is simple: find the key’s index in the first column and index the second column with it. Return nil if the key isn’t found.

getopt =: dyad define
options =. x
key =. y
assert. rankTwo options
assert. twoColumns options
if. 0 -: #options do.
nil
else.
ii =. ((i.&1)@:(((key&-:@:>)"0)@:(((0&{)@:|:)))) options
if. ii < #options do.
(>@:(ii&{)@:(1&{)@:|:) options
else.
nil
end.
end.
)


Eg:

(opts 'key1';10;'key2';11) getopt 'key1'
-> 10


We can now define a handy conjunction to allow the specification of a default value:

dft =: conjunction define
:
r =. x u y
if. r -: nil do.
n
else.
r
end.
)


Which we use this way:

(opts 'key1';10;'key2';11) getopt dft '___' 'key1'
-> 10
(opts 'key1';10;'key2';11) getopt dft '___' 'key3'
'---'


This allows us to pass optional arguments to verbs and specify default values relatively easily, as in this example from my gnuplot library:

histogram =: verb define
(ensureGnuPlot'') histogram y
:
data =. y
if. boxedP x do.
options =. x
gph =. options getopt dft (ensureGnuPlot'') 'gnuplot'
else.
gph =. x
options =. (opt '')
end.
mn =. <./ data
mx =. >./ data
bw =. options getopt dft (0.1&* mx-mn) 'binwidth'
pttl =. options getopt dft '' 'plot-title'
'binwidth=%f\n' gph gpfmt <bw
gph send 'set boxwidth binwidth'
gph send 'bin(x,width)=width*floor(x/width) + binwidth/2.0'
s =. 'plot "%s" using (bin(\$1,binwidth)):(1.0) smooth freq title "%s" with boxes'
s gph gpfmt ((asFile (asVector data));pttl)
)


Where, in the dyadic case, we detect whether x is boxed, and if so, treat it as a list of options. We extract each option by name, providing a reasonable default.

This seems like a common problem, so I am wondering if anyone else in the J community has solved it before in a different way?