Sent to the Findka Essays mailing list on 17 May 2021
As you may have noticed, Findka Essays has been strugglin' this week. It fails to send emails some days, and it looks like my attempts to trigger the emails manually resulted in some people getting the same recommendations on multiple days.
The root cause is that I wrote the code quick & dirty, and enough data has accumulated now that I'll need to do some refactoring before it'll scale further. The Sample (https://sample.findka.com) is also barely starting to have some flakiness but still works most of the time.
For those interested in technical details: I think the best solution will be to move the algorithm code to its own API server (currently it runs on the web server), and then implement things more scalably as I do that. I'm planning to design the data model so it can handle different content types. Then we can use the same API server for both Findka Essays and The Sample. That'll also make it easy to launch more recommendation services for other content types, which could be interesting. Plus, that will make it easy to share data between services. For example, if you subscribe to both Findka Essays and The Sample, then we can use your essay preferences to recommend newsletters (and vice-versa). (If you were around for Findka pre-October, that last bit might ring a bell!)
Also, my brother has recently graduated and joined as a partner. He specializes in data science, and he'll take over the algorithm work (!). Consolidating it all to a single server means that his improvements will benefit all our recommendation services at once.
Given all that, I would rather move immediately to implementing the API server than spend time putting a band-aid on the current setup. So Findka Essays won't send any more emails until after we get the new API server running. It'll probably take at least a couple weeks.
Other news: for those of you who write Clojure, I just barely finished a huge update for Biff (https://biff.findka.com). It's been almost a complete re-write. It's simpler, easier, and production-ready now. I'm doing the official announcement tomorrow.
I've decided to start freelancing so that I can continue improving my various projects without feeling pressure to spend all my time on growth. I'm looking for opportunities to build things for clients with Biff (or train others how to). Let me know if you have any leads.
Jacob