rpypkgs

Fork

regiux: Implement Booleans and if-expressions.

Add Pixie.

Also do a bunch of required fixups. Pixie doesn't yet build, but we are
starting to get some patterns about broken packages, since they are
broken in similar ways due to bitrot.

Topaz is still broken, and now marked as such. There is now a flag to
allow building with PyPy's full source tree, for Topaz or others which
need it.

Uniformly build all three PyPys from a single file.

We no longer pass through any `nixpkgs` derivation directly. Instead,
everything is built via bootstrapped RPython.

reguix: Wire up basic inherits.

reguix: Implement builtin attrset and application.

I had to figure out a fair amount of structure. We now compile on top of
a live scope, directly embedding our objects into the heap and
incorporating live references.

regiux: Sketch a loop-detecting heap.

The basic idea is borrowed from GHC, which does something similar.

regiux: Sketch out the construction of attrsets.

An attrset's names are statically known! So, we should probably compile
it to some sort of key-thunk structure.

regiux: Set up a basic division of responsibility.

The idea is that we need two workflows:

* REPL: given a top-level expr, repeatedly stir the heap
* Recursive Nix: From within a heap action, invoke the parser and
possibly the builder

We're punting on building for as long as possible, but even without
builders we may still have a situation where we import code from the Nix
store while mid-evaluation. To handle this, we want to slowly pivot
towards a single unified workflow:

* Given a heap object, reduce it to WHNF or NF (two methods)
* REPL: Compile to a heap object, reduce it to WHNF, print it
* Recursive Nix: While reducing a heap object, invoke the parser

Now, the *real* complexity will be in invoking the *builder* while
reducing the heap; that would require some sort of scheduler and
reactor, possibly using CPS techniques to add continuations as
callbacks. Mother of monads, event loops, etc. This is already
well-known to be a monadic structure, so we'll have to tackle it
eventually, but not today.

regiux: Split parser.

reguix: More of the parser.

The remaining parts have to do with lexing and parsing of quasiliteral
strings. Parsing these is always...involved. It seems rply doesn't
support the CppNix approach of stateful lexing, so instead we'll have
to carefully lex quasiliterals into pieces and parse the pieces.

regiux: Disentangle bad refs.

Now lambdas pretty-print. Let's figure out attrsets next!

regiux: More parsing.

regiux: Hack out about half of Nix's grammar.

Okay, maybe more like a third. There's a bunch of gnarly regexes that
probably can be ported directly for the lexer, a bunch of implicit
characters that have to be explicitly catalogued and named, and a lot of
boxes to be written for the parser.

I've used as much metaprogramming as I can here, but historically
RPython doesn't like autogeneration of box classes, so they are
handwritten. Yuck.

Set up scaffolding for a Nix parser.

The name is based on a suggestion from
https://fosstodon.org/@IslandUsurper/112322426945766066 and is a pun on
*Python regius*, or "royal python", the Latin name for ball pythons.

Remove non-reproducible builds from the bootstrap.

This is eventually going to go into a tarball; reproducibility will only
grow in importance and we have already been paying the build cost
without a problem. (PyPy's build is RAM-limited first and CPU-limited
second, and doesn't spend much time in stdlib code compared to RPython
code, so optimizing CPython's stdlib bytecode doesn't help much.)

Remove Tcl/Tk support from the bootstrap.

This is functionally dead code even at runtime; everything already
builds without it.

bf: Better JIT debugging; one more peephole.

bf: Hash-cons all ops.

This is so many lines of code, but it pays off. In exchange for around
30 lines of code, we get about a 50% improvement on Mandelbrot time
spent in JIT, and maybe a 10-15% speedup.

What's happening here? RPython JIT greens are compared by pointer when
they are structs, not by contents. There's no way to override this,
because the comparison is on the JIT's hot path for interp. Instead, we
have to hash-cons the entire structure of ops so that two ops will
compare equal by pointer when they are equal by contents. This could be
generated, somewhat, but there's no good answer for the linear scan on
loops or other variadic components.

So, the hash-consing means fewer code objects, which improves the JIT's
performance because it doesn't recompile equivalent code multiple times.

bf: Recognize add-and-scale idiom.

bf: Add optimizing ops, peephole, delooper.

Simpler than it sounds! I get to exploit the Abelian group actions on
the two pairs of ops Inc/Dec and Right/Left to track them with integers,
and they are free other than the type tag; this works because integers
are the initial Abelian group.

I've seen a couple folks go further down this particular direction, but
I'm going to say that this is sufficient for now, because we're now 20%
faster on Mandelbrot than Brown's original interpreter. Our main source
of speedups is fundamentally from reducing the tape from a
dynamically-allocated compound structure into a statically-allocated
arena. We also use op fusion to overcome otherwise-difficult sequences
for the JIT, like repeated subtraction and copying/moving memory.

bf: Rewrite.

This mostly trades speed for lines, in the sense that it's about twice
as slow to perform an indirect dispatch as to have the direct dispatch
inlined into an opcode loop. However, I have about 40 spare lines...

Factor RPython libs, set up bootstrap and checks.

Well, a single check, at least.

Use re-bootstrapped PyPy2 to translate DIVSPL.

Surprisingly, I didn't need any of the nixpkgs framing for this.

Yoink CPython 2.7 over from nixpkgs to form a bootstrap.

Demonstrate that we can build PyPy for Python 2.7 as an intermediate
step to building other interpreters.

Don't re-export Typhon.

Instead, let Monte users (all 2 of them!) import it from the downstream
flake. This lets us focus on providing generic RPython builders here.

Add a known-working version of macropy.

Export library functions.

Add JIT to DIVSPL interpreter.

I only saw a minor speedup in my testing, but the assembly code is
well-structured and should do a decent job for small numbers of rules.
The number of failed guards is exponential in the number of rules, so
I'm not sure if there's any good solution.

Add a DIVSPL interpreter.

I wrote the original. This version is licensed along with the rest of
the repository.

Incorporate some feedback.

Permalink: https://lobste.rs/s/ffhqcr/rpypkgs_nix_flake_for_rpython#c_ebpji9