I had meant to make an update earlier, but I wanted to try and add some more "visible end-user changes" to OurBigBook.com.
I still haven't the user visible ones I wanted, but I've hit major milestones, and it feels like time for an update.
I have now finished all the OurBigBook CLI features that I wanted for 1.0, all of which will be automatically reused in ourbigbook.com.
The two big things since last email were the following:
A secondary but also important advance was: further improvements to the website's base technology.
I knew I was going to do them for several months now, and I knew they were going to hurt, and they did, but I did them.
These change caused two big bugs that I will solve next, one them infinite recursion in the database recursive query, but they shouldn't be too hard.
And do 5 big queries instead of hundreds of smaller ones.
For example, a README.ciro document that references another document saying:needs to fetch "speed-of-light" from the ID database (previously populated e.g. by preparsing light.ciro:to decide that it should display as "Speed of light" (the title rather than the ID).
The \x[speed-of-light] is fast.
= Light == Speed of light
Previously, I was doing a separate fetch for each
\xas they were needed, leading to hundreds of them at different times.
Now I refactored things so that I do very few database queries, but large ones that fetch everything during parsing. And then at render time they are all ready in cache.
This will be fundamental for the live preview on the browser, where the roundtrip to server would make it impossible
README.cironot-readme.cirothe table of contents for
= My website == h2 \Include[not-readme]
= Not readme == Not readme h2
index.htmlalso contains the headers for
- My website
- Not readme
- Not readme h2
- Not readme
This feature means that you can split large input files if rendering starts to slow you down, and things will still render exactly the same, with the larger table of contents.
This will be especially important for the website because initially I want users to be able to edit one header at a time, and join all headers with
\Include. But I still want the ToC to show those children.
This was a bit hard because it required doing RECURSIVE SQL queries, something I hadn't done before: stackoverflow.com/questions/192220/what-is-the-most-efficient-elegant-way-to-parse-a-flat-table-into-a-tree/192462#192462 + of course the usual refactor a bunch of stuff and fix tests until you go mad.
github.com/cirosantilli/node-express-sequelize-nextjs-realworld-example-app contains the same baseline tech as OurBigBook, and I have been use to quickly test/benchmark new concepts for the website base.
I'm almost proud about that project, as a reasonable template for a Next.js project. It is not perfect, notably see issues on the issue tracker, but it it quite reasonable.
The side effects of ambitious goals are often the most valuable thing achieved once again? I to actually make the project be more important thatn the side effects this time, but we'll see.
Since the last update, I've made some major improvements to the baseline tech of the website, which I'll move little by little into OurBigBook. Some of the improvements actually started in OurBigBook.com. The improvements were:
- got a satisfactorily comprehensive linting working at: this commit. Nothing is easy, not even that! Part of the wisdom extracted to: stackoverflow.com/questions/58233482/next-js-setting-up-eslint-for-nextjs/70519682#70519682.
- fully rationalized directory structure to avoid nasty errors that show up in Next.js when accidentally requiring backend stuff on the client. Commit. A detailed explanation of this was extracted to: stackoverflow.com/questions/64926174/module-not-found-cant-resolve-fs-in-next-js-application/70363153#70363153.
- create an extremely clean and rationalized way to do API tests from a simple
npm test. These now actually start a clean server, and make full HTTP requests to that server. Commit. Wisdom extracted to: stackoverflow.com/questions/63762896/whats-the-best-way-to-test-express-js-api/70479940#70479940.
- greatly reduce the number of SQL queries, fully understood every problem
- more intelligently using JOINs where I have managed to get Sequelize to do what I fucking want. This also led to several sequelize Stack Overflow answers as usual: stackoverflow.com/search?tab=newest&q=user%3a895245%20%5bsequelize%5dEverything that I didn't manage to do because of crappy sequelize is documented at: github.com/cirosantilli/node-express-sequelize-nextjs-realworld-example-app/issues/5
- better understanding Next.js/React/useSWR do avoid doing double API queries
Currently, none of the crucial cross file features like
\Includeand table of contents are working. I was waiting until the above mentioned features were done, and now I'm going to get to that.