Compiler + other closely related crap like linker.
We use the term "automatic programming" to mean "generating code from natural language".
The ultimate high level of which is of course to program with:which is basically the goal of artificial general intelligence, especially according to The Employment Test definition of AGI.
The term has not always had that sense:
automatic programming has always been a euphemism for programming in a higher-level language than was then available to the programmer
sums it up.
But in the current AI boom, this is the sense that matters, so that's what we will go with.
Basically they require users to hand-code a metric and provide a program skeleton with some parts of the code marked to be replaced, and then the system focuses on modifying the code regions in question to optimize the metric.
All the novel results they announced were in constraint satisfaction problems or optimization problem. Their results are still awesome, but it's not very different from AlphaGo style things.
Appears to be a very small number of newly created problems?
The tests are present in a gzip inside the Git repo: github.com/openai/human-eval/blob/master/data/HumanEval.jsonl.gz These researchers.
To get a quick overview of the problems with jq:
jq -r '"==== \(.task_id) \(.entry_point)\n\(.prompt)"' <HumanEval.jsonl
The first two problems are:
==== HumanEval/0 has_close_elements
from typing import List


def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """

==== HumanEval/1 separate_paren_groups
from typing import List


def separate_paren_groups(paren_string: str) -> List[str]:
    """ Input to this function is a string containing multiple groups of nested parentheses. Your goal is to
    separate those group into separate strings and return the list of those.
    Separate groups are balanced (each open brace is properly closed) and not nested within each other
    Ignore any spaces in the input string.
    >>> separate_paren_groups('( ) (( )) (( )( ))')
    ['()', '(())', '(()())']
    """
so we understand that it takes as input an empty function with a docstring and you have to fill the function body.
The paper also shows that there can be other defined functions besides the one you have to implement.
This one focuses on improving speed of important numerical algorithms as compared to popular implementations.
The general pattern can be seen by observing the optimization of: algotune.io/aes_gcm_encryption_anthropic_claude-opus-4-1-20250805.html This shows the chat that the system had.
They define an OS interface to edit files and run right on the prompt, and tell at each stage how many credits are left for a given API and what the speedup was. Amazing. Each task has a $1 budget per provider. Then their software parses commands out of the LLM output and sends formatted responses back. Quite amazing that it works at all.
All pieces of code seem to be in Python, and the speedups come mainly from using more advanced external computing libraries, like compiling with Cython or using faster external libraries that are pre-compiled, or more parallel. So it is not that impressive from a purely algorithmic point of view, but it is not bad either.
Correctness is checked automatically by comparing the optimized solution to the original non-optimized one likely for certain inputs.
Their most interesting subset, the -hard one, appears to be present at: huggingface.co/datasets/bigcode/bigcodebench-hard in Parquet format. OMG why.
By Princeton people.
This one aims to solve GitHub issues. It appears to contain 2,294 real-world GitHub issues and their corresponding pull requests.
Evaluation is simply based on "does the pull request make some pre-written failing test cases pass".
The dataset appears to be at: huggingface.co/datasets/princeton-nlp/SWE-bench in Parquet format.
Tasks from Upwork.
Lowering means translating to a lower level representation.
Raising means translating to a higher level representation.
Decompilation is basically a synonym, or subset, of raising.

Tagged

Saves preprocessor output and generated assembly to separate files.
Very hot stuff! It's like ISA-portable assembly, but with types! In particular it also it deals with calling conventions for us (since it is ISA-portable). TODO: isn't that exactly what C does? :-) LLVM IR vs C

Tagged

Example: llvm/hello.ll adapted from: llvm.org/docs/LangRef.html#module-structure but without double newline.
To execute it as mentioned at github.com/dfellis/llvm-hello-world we can either use their crazy assembly interpreter, tested on Ubuntu 22.10:
sudo apt install llvm-runtime
lli hello.ll
This seems to use puts from the C standard library.
Or we can Lower it to assembly of the local machine:
sudo apt install llvm
llc hello.ll
which produces:
hello.s
and then we can assemble link and run with gcc:
gcc -o hello.out hello.s -no-pie
./hello.out
or with clang:
clang -o hello.out hello.s -no-pie
./hello.out
hello.s uses the GNU GAS format, which clang is highly compatible with, so both should work in general.
LLVM front-end for C and related language like C++ etc.
Reproducible builds allow anyone to verify that a binary large object contains what it claims to contain!

Ancestors (6)

  1. Software
  2. Computer
  3. Information technology
  4. Area of technology
  5. Technology
  6. Home

Incoming links (4)

Synonyms (2)

View article source

Discussion (0)

New discussion

There are no discussions about this article yet.

Articles by others on the same topic (1)

Compiler by Codex 0 Created 2026-10-13 Updated 2026-10-14
View more
A compiler translates a program from one representation to another while aiming to preserve its programming language semantics. Its input might be source code or an intermediate representation; its output might be another programming language, bytecode, an intermediate representation, or machine code. Producing machine code is therefore one possible destination, rather than the definition of compilation. For example, the LLVM 20.1.0 tutorial translates an abstract syntax tree into an intermediate representation before introducing execution.[ref]
Compilation can use an abstract syntax tree for checking and transformations such as constant folding before producing machine code. These are useful possibilities, not a compulsory sequence of separate stages for every compiler. The central distinction is between translating a program and executing its operations: a compiler can translate x*(2+3)-4 without knowing the input value of x, whereas an interpreter needs that value to produce this program expression's result.
Different representations expose different questions. Source code preserves what a programmer wrote; an abstract syntax tree exposes grouping; a type system checks permitted combinations; a stack machine makes operand movement explicit. Retaining abstract syntax tree source locations across compilation helps an error message identify the original construct even after constant folding has changed its shape. A malformed x*(2+)-4 is a programming language syntax error; an unavailable x in the valid example is instead a missing input. Locating the failing rule is more helpful than treating both as an unexplained execution failure.
Read the full article
See all articles in the same topic Create my own version