Incoming links: Next.js
Failed attempt at minimizing nodejs/next/ref-twice outside of Next.js.
Founder: Peter Armstrong
The general idea is publishing entire books with usual copyright, but with gradual updates.
ruboss.com/ documents their stack, a somewhat similar choice to OurBigBook.com as of 2021, notably Next.js. But backend in Ruby on Rails. They actually managed Apollo/GraphQL, which Ciro Santilli would have liked, but din't have the patience for.
The founder/CEO Peter Armstrong www.linkedin.com/in/peterburtonarmstrong/ He looks like a nice guy.
Mission: to live in a world where you can learn university-level mathematics, physics, chemistry, biology and engineering from perfect free books that anyone can write to get famous.
Live website: ourbigbook.com
Further information and rationale: Section "OurBigBook.com"
The project's mission is one of, or perhaps the most important, life objective of Ciro Santilli. Reproductive goals aside. These two types of goal are incommensurable. This is one of the great challenges of life.
This is ongoing project.
Ciro's goals in advertising this half done project are is partly to obtain some feedback, and partly to give the idea to someone else who might help push it further, be it in this stack or not.
Better editing support is a must, likely WYSIWYG.
But besides that, it is already in broad strokes the best approach Ciro Santilli can come up with to try and reach the mission statement only with technical advances, i.e. without large amounts of money or political influence which Ciro Santilli does not have.
Maybe that website isn't enough of a technical advance to reach its mission. Maybe there is some further not yet imagined technical insight that would push it into viability. Maybe not. But one must try. Only God can know the answer to these questions.
As of 2022, Ciro has spent about 2.5 years full time working on this project. First he spent about 1 year in 2014 on the first iteration: github.com/booktree/booktree, a GitLab fork, but then decided it was not the way to go.
Then around 2021 he put in some more 1.5 year of full time work, now with a possibly overly complicated (or perhaps just insane/immature) Next.js/Sequelize from scratch website stack.
It makes Ciro a bit ashamed to see that "so little user visible stuff was achieved in so much time". It is partly because he and many people underestimate the difficulty of web development. Perhaps there were some bad stack/useless feature choices issues. And a good dose of indulging in studying the natural sciences to bootstrap content and have fun. But really trying is the only way to learn.
Website: reactjs.org
React officially recommends that you use Next.js[ref], so just do it. It just sets up obvious missing functionality from raw React.
React feels like a good. But it also feels impossible to use/learn sometimes.
Its main design goal is to reduce DOM changes to improve rendering times.
And an important side effect of that is that it becomes easier to do stuff of the type:and then the new comment easily gets the callback attached to it.
- user creates a new comment that appears on screen without page reload
- comment has a delete button, which is JavaScript callback activated
And it also ends up naturally doubling as a template engine.
But React can also be extremely hard to use. It can be very hard to know what you can and cannot do sometimes, then you have to stop and try to understand how react works things better:The biggest problem is that it is hard to automatically detect such errors, but perhaps this is the same for other frontend stuff. Though when doing server-side rendering, the setup should really tell you about such errors, so you don't just discover them in production later on.
- cannot update a component while rendering a different component warning in React
- Rendered more hooks than during the previous render.
- cannot use hooks from helpers:
Is is also very difficult to understand precisely why hooks run a certain number of times.
Examples under: react.
- react/hello.html
- react/hello-func.html: Hello World with a React function component instead of classes. At page load console shows:and then after each click:
Mainso we understand thatonClick MainMaininsanely functions both as the constructor and as the render function in React function components. - react/hello-func-use-callback.html: same as react/hello-func.html but with useCallback. TODO no advantages in this case? When does it help?
- react/hello-without-jsx.html: Hello World in pure JavaScript, without JSX. Exactly equivalent to react/hello.html. Documented at: reactjs.org/docs/react-without-jsx.html Understanding this is fundamental to understanding React.
- react/prop-change.html: shows what gets called as parameters flow down through the tree.By looking at the console, we see all
renderget called every time, even ifpropsdidn't change, but not the constructors.After page load the console contains:Main.constructor Main.render NotMain.constructor NotMain.render NotMain2.constructor NotMain2.renderThen, every time we click the button it adds:handleClick Main.render NotMain.render NotMain2.renderNote how thepropsofNotMainonly change every other click, butrenderstill gets called every time.In order to makeReactnot re-render when there are not changes, you have to either:- define the
shouldComponentUpdatemethod of class components - wrap functional components in
React.memo
- define the
- react/prop-change-hook.html: same as react/prop-change.html, but using hooks. The notable difference is that functional components don't have a clear constructor/render separation, the function just gets called every time. Then React does some magic to ensure that
useStatereturns the current state, except for the first render where they return the initial value. - react/prop-change-hook-use-memo.html: TODO forgot if this example is useful, was tring to use
useMemo - react/prop-change-child.html: shows what child prop changes do not call render on parent,
Maindoes not show up on console when you click underNotMain - react/hook-from-function-fail.html: TODO got some errors that seemed linked to this on a larger program, but failed to minimize them here
- react/hook-different-number-of-times.html: this illustrates one of the cardinal points of using hooks: you must always call them the same number of times, otherwise it fails with:In the case of
React has detected a change in the order of Hooks called by Main. This will lead to bugs and errors if not fixed.
useState, we can kind of understand why this happens: React must use the order of calls to determine which state variable to return at each point in time. - react/hello-hook-use-effect.html: just checking when it gets called. Happens after every render
handleClick Main useEffect useEffect2 - TODO create a test
\a[react/img-broken.html]
How React works bibliography:
- www.netlify.com/blog/2019/03/11/deep-dive-how-do-react-hooks-really-work/ shows how
uesStateworks under the hood with crazy closures - medium.com/@gethylgeorge/how-virtual-dom-and-diffing-works-in-react-6fc805f9f84e
github.com/cirosantilli/node-express-sequelize-nextjs-realworld-example-app contains the same baseline tech as OurBigBook, and I have been use to quickly test/benchmark new concepts for the website base.
I'm almost proud about that project, as a reasonable template for a Next.js project. It is not perfect, notably see issues on the issue tracker, but it it quite reasonable.
The side effects of ambitious goals are often the most valuable thing achieved once again? I to actually make the project be more important thatn the side effects this time, but we'll see.
Since the last update, I've made some major improvements to the baseline tech of the website, which I'll move little by little into OurBigBook. Some of the improvements actually started in OurBigBook.com. The improvements were:
- got a satisfactorily comprehensive linting working at: this commit. Nothing is easy, not even that! Part of the wisdom extracted to: stackoverflow.com/questions/58233482/next-js-setting-up-eslint-for-nextjs/70519682#70519682.
- fully rationalized directory structure to avoid nasty errors that show up in Next.js when accidentally requiring backend stuff on the client. Commit. A detailed explanation of this was extracted to: stackoverflow.com/questions/64926174/module-not-found-cant-resolve-fs-in-next-js-application/70363153#70363153.
- create an extremely clean and rationalized way to do API tests from a simple
npm test. These now actually start a clean server, and make full HTTP requests to that server. Commit. Wisdom extracted to: stackoverflow.com/questions/63762896/whats-the-best-way-to-test-express-js-api/70479940#70479940. - greatly reduce the number of SQL queries, fully understood every problem
- more intelligently using JOINs where I have managed to get Sequelize to do what I fucking want. This also led to several sequelize Stack Overflow answers as usual: stackoverflow.com/search?tab=newest&q=user%3a895245%20%5bsequelize%5dEverything that I didn't manage to do because of crappy sequelize is documented at: github.com/cirosantilli/node-express-sequelize-nextjs-realworld-example-app/issues/5
- better understanding Next.js/React/useSWR do avoid doing double API queries
In any case, the outcome of that is that the tech has improved. And I have done a relatively good job of clearly publishing any "more user visible" improvements to docs.ourbigbook.com/news and social media such as though it is important to note that there have been more than one "fix a hard bug" weeks that were not published because they would just bore readers.
During this period the main focus has been on improving OurBigBook Web, i.e. the dynamic website that powers OurBigBook.com. There are two reasons for that:As a result, Web is now way less buggy and much more usable.
- Web is what has the OurBigBook topics feature for mind-melding, which is the killer feature of OurBigBook compared to other note taking apps and therefore deserves the highest levels of priorityStatic website generation is an indispensable escape valve that ensures that your content can be published forever even if OurBigBook.com goes down one day, which it won't as long as I live. But the innovation is Web.
- static website generation was closer to good enough, but web was much further and is fundamentally harder.I'm extremely satisfied with OurBigBook static website generation and haven't touched it as much. It wasn't easy to reach this state, but I'm there.But Web is a different and much more complex beast.Making CLI software that will run on a person's local computer under full trust and building a bunch of HTML from lightweight markup in bulk is one thing.But making a public dynamic website that has to continuously maintain a coherent database state on granular updates, while giving users some trust but not enough for them to blow everything up is on a totally different level. See e.g. the recent SPAM attack we've had to fend off.
Figure 1. Screenshot showing voting manipulated SPAM as the most highly upvoted article on OurBigBook.com. Source.And then there's also the issue of front-end being mega-hard to get right.
If you look through the list of Web updates, there is nothing specifically mind blowing. The core ideas have largely crystallized, and we are just trying to making them click. I have a few more punches up my sleeve, but the core is decided.
OurBigBook Web search
. Source. This is one of the many basic quality of life improvements that have been done on OurBigBook Web.
OurBigBook Web article announcement
. Source. Another cute new feature, you can send an email to your followers about a new amazing article you created.Web process has been somewhat slower than what I'd like. Of course, it is the case of any project that things are easily said than done. But there are two other main structural factors that have played into it:
- I have my first baby now, and we're learning how to deal with that on the fly.For example, we could have put him on childcare a bit earlier, but due to inexperience we've kept him a bit longer than we maybe should have.Things are well sorted out now, but not matter how good your support system is, at the end of the day, and more often night, it is you the parents that have to deal with a lot of inevitable baby issues. Unless you want them to turn into psychopaths and drug addicts that is, which I don't. I've reached the point of semi failure middle age that the baby feels like my best moonshot.All of this sets a fundamental limit on how many hours you can work per week.But at least with the donations I was able to work on OurBigBook at all. Because if it weren't for that, I would have to focus entirely on the generic job instead and OurBigBook would have been put on hold.
- the choice of Web stack. I was allured by Next.js. I can see the beauty and usefulness of a Node.js render front-end that also runs on backend and hydration. That is awesome.But:
- React is insanely hard to learn and understand. Furthermore, it is also hard to understand the performance problem that it solves, and actually have a benchmark where this problem is solved faster than just delivering some HTML files with ad-hoc Js on top.
- the lack (or perhaps excess of shitty) actual web framework like Ruby on Rails and Django means that I have to rediscover the wheel many times over for all the essential support activities like testing, login and so one
At this point a rewrite is out of the question. I've managed to master things well enough to get a decent result, and given up on the few things that I couldn't for the life of me achieve, after documenting them very well for posterity of course.
Aside from Web, there was only one thing that received a significant improvement, and that was the OurBigBook VS Code extension. The extension is not perfect, and it is not the "final UI", which has to be some WYSIWYG implementation, and there are some fundamental limitations that cannot be overcome without patching VS Code itself. However, the extension is already extremely usable, and I'm writing this on it right now. Basics like syntax highlighting, jump to definition and autocomplete are very useful and usable.
Tree navigation in the OurBigBook Visual Studio Code extension
.