Web Scraping with Next.js 13 using a Generated Fetch Typescript client with Python, Playwright and a FastAPI Backend

Web Scraping with Next.js 13 using a Generated Fetch Typescript client with Python, Playwright and a FastAPI Backend
Web Scraping with Next.js 13 using a Generated Fetch Typescript client with Python, Playwright and a FastAPI Backend.

For my SaaS backlinkgpt.com (AI powered backlink building), I need to scrape website content. Sounds, easy right. It took my quite some time to figure out on how to piece together Playwright, FastAPI, OpenAPI, TypeScript, and Next.js. Here's the story.

Problem 1: Scrape Website Content

While it's tempting to think scraping is as easy as sending a request using Javascript's fetch or Python's requests, the reality is different. Modern websites often use Javascript to render content dynamically, meaning a straightforward fetch often won't capture everything. The real challenge is to scrape sites that depend heavily on dynamic content loading techniques.

Enter the world of headless browsers.

Headless Browsers

A headless browser, as the name suggests, operates without a graphical interface. It can interact with web pages, execute JavaScript, and perform tasks similar to a standard browser - all without user visibility. Popular choices include Puppeteer (JavaScript) and Selenium and Playwright, which caters to multiple languages, including Python.

Problem 2: Running a Headless Browser in a Vercel Serverless Function

My hope was that running a headless browser on Vercel was straight forward. Turns out it was not. I soon ran into a few issues regarding bundle size and memory limits.

To run them, you must connect to a browser instance via websockets. Browserless.io is one option and has a good tutorial on it:

Vercel + Puppeteer
Vercel is the platform for frontend developers, providing the speed and reliability innovators need…

Browserless.io Next.js Tutorial

So I just went ahead and created a route, which would allow me to use the Browserless API and scrape the content:

Overall, it worked out pretty fine and the scraping was also fairly quick as I shared in this tweet:

First Implementation Showcase

While the pricing for Browserless.io was reasonable and I quickly surpassed the free tier's limit of 1,000 scrapes, I remained unsatisfied due to additional challenges.

Browserless.io Pricing Page

Problem 3: Cookies & Data Manipulation

I encountered two primary challenges with Browserless.io.

  1. When a website presented a cookie consent banner, only this banner was scraped, omitting the actual site content.
  2. I aimed to extract more than just opengraph tags; I wanted to capture the website's text in a more structured format. By converting HTML into markdown, I could reduce token usage for OpenAI's GPT and maintain the crucial structure that plain text would sacrifice.

While addressing these issues with Browserless.io seemed feasible, my Python background made it enticing to use Python Playwright. This approach granted me greater ease in debugging and crafting custom logic, ensuring adaptability for future enhancements.

Solution: FastAPI App with Playwright on Modal, Complemented by a Fetch TypeScript Client

I've long admired Modal for its unparalleled developer experience in deploying Python apps. While I intend to share a detailed review on its merits soon, feel free to check out my tech stack in the meantime:

Tech Stack 2023
Dive straight into the heart of my development process in this quick tour of my personal tech stack as of 2023. Discover the languages, frameworks, and tools I use to bring ideas to life. Ready for a sneak peek into my digital toolkit? Let’s get started!

Felix Vemmer Tech Stack 2023

FastAPI App with simple Post Route

First I created a simple FastAPI app on Modal with the POST route scrape-website. This in turn calls the get_website_content function which takes care of parsing the HTML with Beautiful Soup and converting the HTML content to Markdown with html2text:

For those keen on delving deeper into web scraping or deploying a FastAPI app on Modal, here is a curated a list of invaluable resources:

Web endpoints
Modal gives you a few ways to expose functions as web endpoints. You can either turn any Modal function into a web endpoint with a single line of code, or you can serve an entire ASGI-compatible app (including existing apps written with frameworks such as FastAPI or Flask).

FastAPI on Modal.com

A simple web scraper
In this guide we’ll introduce you to Modal by writing a simple web scraper. We’ll explain the foundations of a Modal application step by step.

Building a Python Playwright Webscraper on Modal.com

Generating a Typescript Client

One very known feature of FastAPI is its ability to generate OpenAPI (formerly known as Swagger) documentation for your API out of the box. This documentation not only serves as a great tool for understanding and testing your API endpoints but also provides a JSON schema that can be utilized to generate client libraries in various languages, including TypeScript.

FastAPI OpenAPI Documentation

I thought doing so was quite easy, especially since Sebastián Ramírez even wrote some amazing docs on how to do it:

There are many tools to generate clients from OpenAPI. A common tool is OpenAPI Generator. If you are building a frontend, a very interesting alternative is openapi-typescript-codegen.

Turns out I tried too many tools and code generators and was quite amazed on how many new ones are built, but there's not one single super well working one. Here's what I found.

  1. openapi-typescript-codegen: Currently does not work with Next13 as outlined here:
CancelablePromise fails to resolve with Next.js 13 · Issue #1626 · ferdikoomen/openapi-typescript-codegen
Describe the bug I’ve tried to upgrade an application from Next.js 12 (v12.3.4) to 13 (v13.4.8) and suddenly experienced issues when fetching data from the API through the generated typescript clie…
  1. stOpenAPI Generator: Looks way too verbose, require some Java installation and also not sure if it fully works based on this discussion:
OpenAPI Client Generators with support 3.1.0 specs · tiangolo/fastapi · Discussion #9810
First Check I added a very descriptive title here. I used the GitHub search to find a similar question and didn’t find it. I searched the FastAPI documentation, with the integrated search. I alread…
  1. openapi-zod-client: Sadly uses axios and I did not want an additonal dependencies. Also all the functions are in snake case and customizing them was a bit confusing to me since I never used handlebars.
  2. Fern: Looks like a cool startup but a bit of an overkill. Also creating more yaml and custom things was too much work, since I wanted to keep it simple.

Solution: openapi-fetch

Finally after almost giving up I came across this project, which finally worked with PydanticV2 and OpenApi 3.1.

GitHub - drwpow/openapi-typescript: Generate TypeScript types from OpenAPI 3 specs
Generate TypeScript types from OpenAPI 3 specs. Contribute to drwpow/openapi-typescript development by creating an account on GitHub.

The command to generate the types was quite straight forward:

npx openapi-typescript [Link to your openapi spec] -o ./src/lib/fast-api/v1.d.ts

Then from the generated v1.d.ts I could create my client.ts with type completion and use in the sample page.tsx

Final Step: Next.js Rewrites

With everything set up, I wanted to ensure that our frontend could seamlessly interface with our backend without having to juggle different URLs or face CORS issues. To do this, I turned to the rewrites feature in Next.js, which provides a mechanism to map an incoming request path to a different destination path.

Here's how I configured the rewrites in the next.config.js:

  async rewrites() {
    const fastApiBaseUrl = process.env.NEXT_PUBLIC_FAST_API_BASE_URL;

    if (!fastApiBaseUrl) {
      throw new Error('Please set the NEXT_PUBLIC_FAST_API_BASE_URL environment variable');
    return [
        source: "/ingest/:path*",
        destination: "https://eu.posthog.com/:path*",
        source: "/fast-api/:path*",
        destination: `${process.env.NEXT_PUBLIC_FAST_API_BASE_URL}/:path*`,

Next.js rewrites

The above configuration tells Next.js to forward any request starting with /fast-api to our backend server. This way, on our frontend, we can simply call /fast-api/scrape-website and it will be proxied to our backend on Modal.com.

With these rewrites in place, the integration of frontend and backend was smooth, and my development experience was greatly enhanced. I no longer had to remember or handle different URLs for different environments, and everything just worked.

And that's how I bridged PydanticV2, OpenAPI, TypeScript, and Next.js together. Hope this helps anyone looking to do something similar!