The development repository for LessWrong2 and the EA Forum, based on Vulcan JS
Forum Magnum is the codebase powering LessWrong and the Effective Altruism Forum.
The team behind LessWrong created this codebase in 2017 as a rewrite of the original version of LessWrong, which was a difficult-to-maintain fork of reddit.
Forum Magnum is built on top of a number major open-source libraries.
Vulcan is a framework for designing social applications like forums and news aggregators. We started out using it as a library in the usual way, then forked its codebase and diverged considerably. Read their docs to understand where we've come from, but be wary of outdated information. This page is still particularly useful. CEA: see notion for more.
Typescript is the programming language we're using. It's like Javascript, but with type annotations. We're gradually moving from un-annotated Javascript towards having annotations on everything, and any new code should have type annotations when it's added.
React is a user interface programming library developed by Facebook that lets us define interface elements declaratively in the form of components. We use it to define how to render and manage state for all parts of the site.
GraphQL is a query language that the browser uses to get data from our servers. Most usage of GraphQL is hidden behind utility functions, but occasionally we use it directly to define APIs for accessing and mutating our data.
Apollo is a client-side ORM which we use for managing data on the client. We interact with it primarily via the React hooks API.
CkEditor5 is the default text editor for posts, comments, and some other form fields. Draft is an alternative rich text editor, which is no longer the default but which we still support.
.nvmrc
for the required node versionbrew install nvm
to install nvm, then nvm install
in the ForumMagnum directoryClone our repo:
git clone https://github.com/ForumMagnum/ForumMagnum.git
Install dependencies:
cd ForumMagnum
yarn install
CEA Devs, see the ForumCredentials repository for access to a remote dev database. Otherwise, do the following:
Run a local postgres instance, version 15. For example, if you're on macos:
brew install postgresql@15
brew services start postgresql@15
createdb forummagnum
(DB name is an arbitrary choice.)
You must also have the pgvector Postgres extension installed.
Configure the schema:
psql forummagnum -f ./schema/accepted_schema.sql
TODOs:
yarn ea-load-dev-db <settings file>
to load the database settings from the file. (TODO: example of the file)When developing features that require changing database schemas, it can be desirable to do this work without changing schemas in shared database instances that other developers are working on. To solve this problem, we have a script that can be used to create temporary clones of the dev database. The following commands are supported:
yarn branchdb create
clones a new dev database for the current git branchyarn branchdb drop
drops the cloned database for the current git branchyarn branchdb clean
drops all cloned dev databases created by this git cloneyarn branchdb list
lists all cloned dev databases created by this git cloneYour postgres URL for the locally-running database will be postgres://localhost/forummagnum.
yarn start-local-db --postgresUrl $YOUR_LOCAL_POSTGRES_URL
Next, select which website you are working on. If you're making a new site, open packages/lesswrong/lib/instanceSettings.ts, find the line that says
export const allForumTypes = ...
and add your forum type to the end. Then regardless of whether you're making a
new site or editing an existing one, open settings-dev.json and change the line
that says "forumType": "LessWrong"
to the correct forum.
Or, if (and only if!) you have access to CEA's ForumCredentials repository, use
yarn ea-start
You should now have a local version running at http://localhost:3000.
It will start out with an empty database. (This means that some of the hardcoded links on the frontpage will not work). You can create users via the normal sign up process (entering a fake email is fine). The first user youâll create will be an admin, so youâll probably want to create at least two users to check how the site looks for non-admins. [Note for CEA: this doesn't apply to you, your database is shared with other developers.]
Some relevant pieces of documentation that will help you understand aspects of the design:
You can also see auto-generated documentation of our GraphQL API endpoints and try out queries using GraphiQL on our server or on a development server.
Eventually, itâll be helpful to have a good understanding of each of those technologies (both to develop new features and fix many kinds of bugs). But for now, the most useful things to know are:
Collections â Mongo databases are organized around collections of documents. For example, the Users collection is where the user objects live. Mongo databases do not technically have a rigid schema, but VulcanJS has a pattern for files that determine the intended schema (which is used by the API, forms and permissions systems to determine what database modifications are allowed). Each collection is a subdirectory in packages/lesswrong/lib/collections
.
Components â Our React components are organized in a folder structure based loosely on our collections. (i.e. components related to the User
collection go in the packages/lesswrong/components/users
folder). Each component is (usually) defined in a separate .tsx
file in packages/lesswrong/components
and imported from packages/lesswrong/lib/components.ts
.
Some edge cases just go in a randomly picked folder (such as the RecentDiscussion components, which involve both comments and posts, but live in the comments folder)
There are multiple ways of creating a ReactJS component. New components should be functional components, using hooks and ideally minimizing usage of higher-order components. Ideally, each component does one (relatively) simple thing and does it well, with smart components and dumb components separated out. In practice, we havenât done a great job with this. (Scope creep turns what were once simple components into increasingly complex monstrosities that we should really refactor but havenât gotten around to it).
We use Vulcanâs registerComponent
function to add them as children to a central âComponentsâ table.
Smart Forms - Vulcan also allows us to automatically generate simple forms to create and edit Documents (in the Mongo sense of the word Document, any instance of a Collection). This functionality is called Smart Forms.
You can create an EditFoo
page, which renders WrappedSmartForm
, which then automagically creates a form for you. We use this to edit just about every Document in the codebase. How does it know what type of input you want though? This is the interesting part. You define the way you want to edit fields in the collection schema. So in Posts you have (selected examples):
'checkbox'
, which makes it editable by a simple checkbox.'EditTitle'
, which means the Smart Form will look in Components for an EditTitle component, and then use that as the UI for modifying the Title.useFoo (React Hooks) - We make heavy use of React hooks for querying data, managing state, and accessing shared data like the current user.
withFoo (Higher Order Components) â Higher-order components exist as alternatives for most hooks, and are sometimes used because class-components cannot use hooks. However, these are deprecated and we are migrating towards only using hooks.
Fragments â GraphQL queries are made using fragments, which describe the fields from a given database object you want to fetch information on. Thereâs a common failure mode where someone forgets to update a fragment with new fields, and then the site breaks the next time a component attempts to use information from the new field.
makeEditable - To add a long text field to a schema, use makeEditable
. It add the correct control component, and creates the necessary callbacks to sync it with the Revisions table.
Configuration and Secrets We store most configuration and secrets in the
database, not in environment variables like you might expect. See
packages/lesswrong/server/databaseSettings.ts
for more.
Logging - If there's a part of the codebase you often want to see debug
logging for, you can create a specific debug logger for that section by using
loggerConstructor(scope)
. You can then enable or disable that logger by
setting the public database setting debuggers
to include your scope, or by
setting the instance setting instanceDebuggers
. See
packages/lesswrong/lib/utils/logging.ts
for more.
Collection callbacks - One important thing to know when diving into the
codebase is how much logic is done by callbacks. Collections have hooks on
them where we add callbacks that react to CRUD operations. They can fire
before or after the operation. The running of these callbacks is found in
packages/lesswrong/server/vulcan-lib/mutators.ts
.
When developing, we make good use of project-wide search. One benefit of a
monorepo codebase at a non-megacorp is that we can get good results just by
searching for hiddenRelatedQuestion
to find exactly how that database field is
used.
console.warn(variable)
when you want to see the stacktrace of variable
$r
. For example, you can check the props or state using $r.props
or $r.state
.forcePendingEvents
.All migrations should be designed to be idempotent and should represent a one-off operation (such as updating a table schema). Operations that need to be run multiple times should instead be implemented as a server script.
yarn migrate up [dev|staging|prod]
yarn migrate down [dev|staging|prod]
, but note that we treat down migrations as optionally so this may or may not work as expectedyarn migrate create --name=my-new-migration
, although usually you will want to do yarn makemigrations
instead (see below)yarn migrate pending [dev|staging|prod]
yarn migrate executed [dev|staging|prod]
Instead of using [dev|staging|prod] above, you can also manually pass in a postgres connection string through a PG_URL
environment variable. Use that option if you are not using the [EA] ForumCredentials repo.
Many (most) migrations will just be to update the database schema to be in line with what the code expects. For these we have scripts to autogenerate a migration template, and assert that the new schema has been "accepted" (i.e. you have remembered to write a migration). For these the development process will be like this:
schema.ts
filesyarn makemigrations
, to check the schema and generate a new migration file if there are changesacceptsSchemaHash = ...
line when you are doneyarn acceptmigrations
to accept the changes (this updates two files which should be committed, accepted_schema.sql
and schema_changelog.json
)accepted_schema.sql
and schema_changelog.json
)yarn makemigrations
again, to generate new hashes for the schemaacceptedSchemaHash
value and timestamp in the file nameschema_changelog.json
if it's still there)yarn acceptmigrations
We use Jest for unit and integration testing, and Cypress for end-to-end testing.
yarn unit
yarn integration
(you may need to run yarn create-integration-db
first so that the db is up-to-date)Both commands support a -watch
suffix to watch the file system, and a -coverage
suffix to generate a code coverage report. After generating both code coverage reports they can be combined into a single report with yarn combine-coverage
.
yarn ea-start-testing-db
, then in a separate terminal run either yarn ea-cypress-run
for a CLI version, or yarn ea-cypress-open
for a GUI version. To run specific tests in the CLI, you can use the -s <glob-file-pattern>
option../settings-test.json
.cy.get()
to find elements via CSS selectors, cy.contains()
to find elements via text contents, cy.click()
and cy.type()
for input, and cy.should()
for assertions. Feel free to steal from existing tests in ./cypress/integration/
../cypress/support/commands.js
, and access them via cy.commandName()
../cypress/fixtures
, and can be accessed using cy.fixture('<filepath>')
. See here for more../cypress/plugins/index.js
. Tasks are executed using cy.task('<task-name>', args)
.Branch off of master
and submit to master
. Deploys occur when master
is
merged into ea-deploy
and lw-deploy
.
The local development database is actually hosted in the cloud like staging and production. There's no reason to host your own database. It's also shared with other developers, which means if someone adds a feature which requires manual database work, there's no need for you to also do that manual work.
The test user admin credentials are in 1password. You're also welcome to create your own admin user.