The Invisible Framework

• Published at August 26, 2019
• 8 mins reading timei

Over the past few weeks, I've been writing code for the new Traveloka web framework. Even though I was the one writing most of the code, this is a team effort because there's a lot of discussions happen beforehand. We learned a lot from previous framework and tried our best not to repeat the same mistakes.

It's weird to share something that's not proven in production yet, but I like the direction that we're going. That's why I'm writing this in my blog instead of our engineering blog. Let's see in a few months, we might write a follow-up post there.

It's not a framework

There's no easy way to call this project. Internally, we call it TVLK5 because this is the fifth iteration. We used to have a semi-official framework that we used, but due to unclear goal, we ended up writing a couple of micro frameworks on top of that, each with more specific functionality and their defining features.

There's a limit on how many refactor you can introduce to your codebase. Incremental changes are good and that's what we've been doing, but sometimes new paradigm warrants a rewrite. We don't like rewrite for the sake of rewrite, and we need to be careful when we're proposing big changes. Our web engineering team has grown a lot since I worked here, and we need to make sure the framework is still relevant for the next few years, at least the guiding principle. The internal implementation can be rewritten every month for all I care.

What we ended up is not your typical framework. I call it a framework because it's easier for people to associate it with. Inside, they are a collection of modules that work well together.

Consider this example of how we format price based on user preferred locale and currency:

import { useLocale, useCurrency } from '@traveloka/core';
export default function Page(props) {
  const locale = useLocale();
  const currency = useCurrency();
  const priceText = new Intl.NumberFormat(locale.languageTag, {
    style: 'currency',
    currency: currency.code,
  }).format(props.price);
  return <div>{priceText}</div>;
}
Page.getInitialProps = async (ctx) => {
  const price = await getPrice(ctx.query.id);
  return { price };
};

Wait, isn't that just Next.js with some react hooks? Where's the framework?

Exactly!

If your page/component doesn't need any data from our backend, you just write React component inside pages/ directory, like you would on normal Next.js page. You don't need to wrap your page component with anything, you don't need to initialize reducer to do anything.

Here's the hello world version in comparison:

export default function Page() {
  return <div>{'Hello, world!'}</div>;
}

As you can see, it's as basic as you can get.

Instead of focusing on what kind of framework that we like, we focus on what kind of API that we want to have. These APIs should have 2 things in common: they have to be composable, and they're low level enough that we can build anything on top of it, inspired from the extensible web manifesto. From there, we proposed the change one by one, starting from the most common source of frustration.

We wrote a bunch of RFCs

We already have rules in place when there's a major change affecting multiple product teams, the one proposing changes must write an RFC. Our RFC template has fields you need to fill that forces you to think about edge cases and alternative methods to achieve the same goal. Overall, it's similar to what you see in the open-source world.

We started by creating one RFC to handle module organization and directory structure. We call these micro packages. Instead of having one package containing product code, each product will have its own self-contained modules.

We used to be able to import from the root because of how our directory is structured. Having platform-specific code and product code means you can end up with an import like

import FlightBookingHeader from 'desktop/flight/booking/components/FlightBookingHeader/FlightBookingHeader';

Imagine writing relative import from FlightBookingHeader.

To fix that issue, we're leveraging Yarn workspace. We've been using it to manage dependency of our reusable UI components and end-to-end tests, but now we extend the usage to almost everything.

This means import is now scoped to NPM package name, without us having to publish the package to the NPM registry. Instead of deep hierarchy, we advocate for flatter structure. Rather than importing by path, we use named import. Any reusable logic can now be imported using @traveloka/ prefix, like this:

import { sortFlightSearch } from '@traveloka/flight';

We're also moving from monolith Node.js app to microservices where we can deploy different services unrelated to one other. For this, we'll have another 2 packages for desktop service and mobile service to handle www and m traffic respectively. To reduce friction when developing locally, we're creating dev proxy inspired from Zeit's Now route definition.

Engineer can simply run yarn dev and proxy starts. The way our proxy work is it forward requests and starts the services only if path matching the service is visited by the user. Then relying on Next.js mechanism for building pages, we only build a page that user visit, so less resource is used compared to before where we run webpack --watch for entire entry points before running our service locally.

There's 2 way to define a route, using name or package. If name is used, it forward requests traffic to package depending request hostname. Request to m.traveloka.com will be forwarded to @traveloka/{name}-mobile package. Using package is reserved for services that don't care about hostname, for example, our API proxy.

{
  "routes": [
    {
      "name": "flight",
      "paths": ["/flight"]
    },
    {
      "package": "@traveloka/api-proxy",
      "paths": ["/api"]
    }
  ]
}

This routing rule is also what we're going to use for production load balancers routing logic. This is to make sure that dev environment is as close as possible with production.

This is one of the few changes that we introduce using RFC. There's one RFC for each of the major pain points that we have with our codebase and each of them are only related to each other in one thing: how they work together. After writing one RFC, usually, another one pops up that reuse part of that RFC or at least the pattern. I wouldn't say this is the best planning for big changes like this, but this works for us. One important thing to note is we defer from writing any code before all major issues are addressed in the RFC(s).

Not all changes are written in the form of RFC though. Reaching consensus is hard and sometimes we just need to pick one implementation and gather feedback after the implementation, especially if these are less used patterns.

Interesting thing under the hood

While the RFCs are useful to uncover various cases of how engineers write code, the implementation is mostly "boring" work. React context here, another hooks there, sprinkled in some conventions. But there are times when we need to build things using something interesting that we haven't had a chance to learn before, and that's what we did.

We want to create API to fetch translation data using hooks so any query requirements are described at render time. The benefit of doing this is when we want to code split components, translation data is automatically split without having to do anything.

function Parent() {
  const cr = useContentResource({
    Some: {
      query: '',
    },
  });
  return (
    <div>
      {cr.Some.query}
      <Child />
    </div>
  );
}
function Child() {
  const cr = useContentResource({
    Another: {
      query: '',
    },
  });
  return <div>{cr.Another.query}</div>;
}

To know all query, we need to render entire tree at first, to do that we render the component using renderToString and replace how our hooks resolve data by using JavaScript Proxy. This is why we can render the entire tree without error because we know the required shape in advance, we can mock it for initial render, while we're collecting queries.

We rely on Next.js getInitialProps to batch query and fetch data on server-render and client-side navigation, but on client-side rerender, we use Suspense for data fetching that allow us to place loading boundary anywhere in the tree. In my opinion, this is the best way to handle loading states compared to typical async component where you can only define loading state in the same tree that has async dependencies.

This server-client bundle difference is made possible because of one RFC we proposed called client-side extension where we can have two files module.js and module.client.js and webpack will create a different bundle for server and client using different file based on the environment.

Originally that RFC was created to reduce changes to webpack config whenever there's an incompatible NPM package that isn't created with universal JavaScript in mind (like the one that relies on fs and friends). Another goal was to reduce client bundle size where we can optimize by serving different dependencies depending on the environment. We can also optimize by only calling API call on the server and only using hydrated data in the client.

This is the overall theme of most RFCs that were created, one thing enables the other thing to be built, some are coincidental, few are planned and discussed.

This will not be open-sourced

When you hear someone writing framework for their company, one of the things in your mind may be "when this will be open sourced?". I'm sorry to say that this one will probably never make it to the open-source. It's tightly coupled with the rest of our infra and I don't mean that in a bad way.

You don't want a flexible framework that can do a lot of things but need boilerplate that looks like a micro-framework. What you need is a solid pattern implemented on top of good low-level abstraction. Opinionated framework integrates better with the rest of your app, and you can still have an extension point through that low-level abstraction.

Even though we call it TVLK5 internally, there's no official name that people can import from because all you see is something like this:

import { TravelokaPageComponent } from '@traveloka/core';

We made it look like it's just modules that maintained by the web infrastructure team because that's what it is. No fancy name, no hidden meaning. It's just another dependency in your package that helps you write your code for your features, and stay hidden when you don't need it.

It's the invisible framework.