~/leohuynh.dev/log/crawling-goodreads-books-data

Crawling Goodreads books data in Node.js

·–– views#goodreads#goodreads-api#crawling#rss#rss-parser#csv#csv-parser

Since Goodreads no longer supports fetching user’s books data via their API, I’ve decided to crawl / scrape user’s book data. There are two primary ways to do this: using the RSS feed or by exporting your library as a CSV file.

Goodreads API

No matter which method you choose, we’ll parse everything into a clean, consistent format. Here’s what our final GoodreadsBook type looks like:

export type GoodreadsBook = {
  guid: string
  pubDate: string
  title: string
  link: string
  id: string
  bookImageUrl: string
  bookSmallImageUrl: string
  bookMediumImageUrl: string
  bookLargeImageUrl: string
  bookDescription: string
  authorName: string
  isbn: string
  userName: string
  userRating: string
  userReadAt: string
  userDateAdded: string
  userDateCreated: string
  userShelves: string
  userReview: string
  averageRating: string
  bookPublished: string
  content: string
}

Using the RSS Feed

The first method is to use the rss-parser package to parse the RSS feed and extract the book data.

import Parser from 'rss-parser'
import type { GoodreadsBook } from '~/types'


let parser = new Parser<{[key: string]: any}, GoodreadsBook>({
  customFields: {
    // Define all the custom fields you want to extract from the RSS feed
    // Here I'm listing all the available fields from the Goodreads RSS feed
    item: [
      'guid',
      'title',
      'link',
      'pubDate',
      ['book_id', 'id'],
      ['book_image_url', 'bookImageUrl'],
      ['book_small_image_url', 'bookSmallImageUrl'],
      ['book_medium_image_url', 'bookMediumImageUrl'],
      ['book_large_image_url', 'bookLargeImageUrl'],
      ['book_description', 'bookDescription'],
      ['author_name', 'authorName'],
      ['isbn', 'isbn'],
      ['user_name', 'userName'],
      ['user_rating', 'userRating'],
      ['user_read_at', 'userReadAt'],
      ['user_date_added', 'userDateAdded'],
      ['user_date_created', 'userDateCreated'],
      ['user_shelves', 'userShelves'],
      ['user_review', 'userReview'],
      ['average_rating', 'averageRating'],
      ['book_published', 'bookPublished'],
    ],
  },
})

Then you can fetch the data from the RSS feed using the parser object, and process it as needed.

const GOODREADS_RSS_FEED_URL = '<YOUR_GOODREADS_RSS_FEED_URL>'


export async function fetchGoodreadsBooks() {
  if (GOODREADS_RSS_FEED_URL) {
    try {
      let data = await parser.parseURL(GOODREADS_RSS_FEED_URL)
      // All the books data will be stored in the `data.items` array
      // Use the parsed data as needed, for example, you can write it to a JSON file:
      writeFileSync(`./json/books.json`, JSON.stringify(data.items))
    } catch (error) {
      console.error(`Error fetching the Goodreads RSS feed: ${error.message}`)
    }
  } else {
    console.log('📚 No Goodreads RSS feed found.')
  }
}

[!NOTE] You can get a Goodreads user’s RSS feed URL by going to their profile and navigating to the bookshelf page and copy the RSS feed URL. This is my bookshelf page for example: https://www.goodreads.com/review/list/179720035

Now that you have the data you might need to prettify them before storing or using in your application since the data is stored in a raw format.

let data = await parser.parseURL(/* GOODREADS_RSS_FEED_URL */)
// Loop through the `data.items` array to prettify the data
for (let book of data.items) {
  book.content = book.content.replace(/\n/g, '').replace(/\s\s+/g, ' ') // Remove line breaks
  book.book_description = book.book_description
    .replace(/<[^>]*(>|$)/g, '') // Remove HTML tags
    .replace(/\s\s+/g, ' ') // Replace multiple spaces with a single space
    .replace(/^[\"|“]|[\"|“]$/g, '') // Remove leading and trailing quotation marks
    .replace(/\.([a-zA-Z0-9])/g, '. $1') // Add a space after a period
}
// Use the parsed and prettified data as needed...

Using a CSV Export

The second method involves exporting your Goodreads library as a CSV file and parsing it. This method gives you more data fields than the RSS feed.

First, you need to export your data from the Goodreads import/export page.

Once you have the goodreads_library_export.csv file, you can use the csv-parser package to parse it.

import fs from 'node:fs'
import path from 'node:path'
import csv from 'csv-parser'
import type { GoodreadsCsvBook } from '~/types'


const GOODREADS_CSV_FILE_PATH = path.join(
  process.cwd(),
  'your/path/to',
  'goodreads_library_export.csv',
)


export async function parseGoodreadsCsv() {
  if (!fs.existsSync(GOODREADS_CSV_FILE_PATH)) {
    console.log('📚 Goodreads CSV file not found.')
    return
  }


  let csvBooks: GoodreadsCsvBook[] = []
  await new Promise<void>((resolve, reject) => {
    fs.createReadStream(GOODREADS_CSV_FILE_PATH)
      .pipe(
        csv({
          mapHeaders: ({ header }) => {
            // Map CSV headers to a more usable format
            let headerMap: Record<string, string> = {
              'Book Id': 'id',
              'Title': 'title',
              'Author': 'authorName',
              'ISBN': 'isbn',
              'My Rating': 'userRating',
              'Average Rating': 'averageRating',
              'Publisher': 'publisher',
              'Number of Pages': 'numberOfPages',
              'Year Published': 'bookPublished',
              'Original Publication Year': 'bookPublished',
              'Date Read': 'userReadAt',
              'Date Added': 'userDateAdded',
              'Bookshelves': 'userShelves',
              'Exclusive Shelf': 'exclusiveShelves',
              'My Review': 'userReview',
              'Binding': 'binding',
            }
            return headerMap[header] || header.toLowerCase().replace(/\s+/g, '')
          },
        }),
      )
      .on('data', (book: GoodreadsCsvBook) => {
        csvBooks.push(book)
      })
      .on('error', reject)
      .on('end', resolve)
    })


    // Now csvBooks contains all the data from the CSV
    // You can then transform it to your desired format
    console.log(`Found ${csvBooks.length} books in the CSV.`)
}

The data from the CSV needs to be transformed to a consistent format, similar to the one used for the RSS feed data. Notice that some fields like bookImageUrl are not available in the CSV export.

// Transform CSV data to match a common GoodreadsBook format
let books: GoodreadsBook[] = csvBooks.map(book => {
  let transformedBook: GoodreadsBook = {
    id: book.id || '',
    guid: `goodreads-${book.id}` || '',
    pubDate: book.userDateAdded || new Date().toISOString(),
    title: book.title || '',
    link: `https://www.goodreads.com/book/show/${book.id}`,
    bookDescription: book.userReview || book.title || '',
    authorName: book.authorName || '',
    isbn: book.isbn?.replace(/[=""]/g, '') || '',
    userRating: book.userRating || '0',
    userReadAt: book.userReadAt || '',
    userDateAdded: book.userDateAdded || new Date().toISOString(),
    userDateCreated: book.userDateAdded || new Date().toISOString(),
    userShelves: book.userShelves || book.exclusiveShelves || '',
    userReview: book.userReview || '',
    averageRating: book.averageRating || '0',
    bookPublished: book.bookPublished || book.yearPublished || '',
    content: book.userReview || book.title || '',
  }
  // ... any further processing like cleaning up descriptions
  return transformedBook
})
// Use the transformed data as needed, for example, write it to a JSON file or store in a database

RSS vs CSV

RSS Feed

  • Pros: Can be automated to fetch data periodically.
  • Cons: Data refresh is not instant (can take hours). Provides fewer data fields compared to the CSV export.

CSV Export

  • Pros: Contains more detailed information about the books (e.g., publisher, number of pages, binding). The data is available immediately after export.
  • Cons: No image fields. You’ll need to fetch book covers separately. Exporting CSV is manual.

Choose your preferred method and happy crawling! clinking beer mugs

react to this post