Incoming links: Hogwarts
A good explanation of how this insane system came up is given at Video "History of Oxford University by Chris Day (2018)".
As if it weren't enough, there are also the 6 Halls: permanent private hall.
The colleges are controlled by its fellows, a small self-electing body of highly successful scholars, usually in the dozens per college number it seems. Each college also usually has different types of fellows, e.g. see he university college page: www.univ.ox.ac.uk/about/college-fellowships/ (archive)
The college system does has its merits though, as it instates a certain sense of Hogwarts "belonging" to a certain group, so it might help students get better support for their learning projects from older students, or through the tutoring system. Of course, all such "belonging" feelings are bad, the correct thing would be to make great online tutorials for all, and answer questions in the open. But oh well, humans are dumb.
The college you are in impacts the quality of your courses, because tutorials are per-college. As of 2023, Ciro Santilli spoke to some students of the Computer science course of the University of Oxford, and was told that in some cases where you don't have anyone who can give the tutorial, you instead get a "class", i.e. a P.h.D. student going through question sheets with no interaction in the C.S. department, rather than a deep interactive discussion over the college fire. How can this system be so broken, it is beyond belief
This functionality is somewhat related to fraternities and sororities in 2000's United States.