Fix an effective enumeration of unary partial computable functions. The diagonal halting set is
It is a computably enumerable set: on input , simulate and accept if that computation halts.
Suppose were a computable set. A program could then halt on input exactly when the decider says . Let be its own index. If , the program does not halt, while if , it halts. Both alternatives contradict the definition of . Thus is computably enumerable but not computable.
Let be the diagonal halting set and put
If were a computably enumerable set, then would many-one reduce to , making the complement of computably enumerable. Together with the computable enumerability of , that would make a computable set, a contradiction.
Moreover
If this complement were computably enumerable, would again make computably enumerable. Thus neither nor its complement is computably enumerable.
Rice theorem says that every nontrivial extensional property of programs computing partial functions is undecidable. Equivalently, a nontrivial index set cannot be a computable set.
Let be such an index set. Replacing by its complement if necessary, assume that the nowhere-defined function has no index in . Since the property is nontrivial, choose an index . If membership in were decidable, we could decide the diagonal halting problem as follows. For each , define
By the S-m-n theorem, a total computable function produces an index for . If diverges, then is nowhere defined and . If it halts, then and extensionality gives . A decider for would therefore decide whether halts, a contradiction. Hence is undecidable.
No. The property
is an index set: it depends only on the domain of the partial function computed, not on the syntax of its program. It is nontrivial. For example, an index whose domain is a singleton belongs to , whereas an index whose domain has two elements does not. The Rice theorem therefore shows that is not a computable set, so no algorithm can decide the requested property for arbitrary .
Invoking the Church–Turing thesis, this also rules out any effective procedure in the informal sense, because such a procedure would be represented by a computable function.