Migrating From Wagtail To Astro (5): Data Sync And Scripts

This is the fifth article in the Wagtail-to-Astro series. It explains how to use scripts to sync Wagtail data and media files from the online site to local storage, then import them as Astro Markdown files through a repeatable workflow.

What The Sync Script Should Do

1. Pull from the server: Copy the SQLite database used by Wagtail and media directories such as media/original_images/.
2. Write locally: Store them under legacy-wagtail/ or another agreed directory, without polluting the current src/content/blog/ structure.
3. Be repeatable and configurable: Keep host, paths, and authentication in a configuration file so the script can be run multiple times for full or incremental updates.

Sync Script Design

Configuration And Safety

• Put server address, username, paths, password, or key path in a configuration file, such as config/deploy.local.json, and add it to .gitignore so secrets do not enter the repository.
• Use SSH + SCP/rsync, or a library such as Paramiko, for remote copying. For large directories, rsync incremental sync avoids repeated transfers.

Sync Scope

• Database: Copy the entire database to local storage, for example legacy-wagtail/zicodedb.
• Media: Sync only the directories that are actually used, such as media/original_images/. You can first collect image paths from the database, or perform one full sync and then incremental syncs later.

Example Directory Structure

project/
  config/
    deploy.local.json    # Local config, not committed
  legacy-wagtail/
    zicodedb             # Copied SQLite database
    media/
      original_images/   # Copied images
  scripts/
    sync_legacy.py       # Sync script
  astro-site/
    src/content/blog/imported/  # Import script output
    public/uploads/              # Copied images for the static site

Example configuration:

{
  "wagtail_host": "user@example.com",
  "wagtail_db_path": "/path/on/server/zicodedb",
  "wagtail_media_path": "/path/on/server/media",
  "local_legacy_dir": "legacy-wagtail"
}

The script can use rsync to pull the database and media directory into local_legacy_dir. Remote commands can be executed through Paramiko/SSH, or everything can be handled through rsync over SSH.

Import Script Design

Reading Wagtail Data

• Use sqlite3 or SQLAlchemy to connect to the local copy of zicodedb.
• Query post title, slug, publication time, body content, StreamField JSON, category, tags, column, author, hero image, and other fields.
• Parse StreamField by block type, such as paragraph, markdown, image, and heading, then assemble the final Markdown body.

Generating Astro Content Files

• Generate one .md file for each post under src/content/blog/imported/.
• Fill frontmatter according to the site’s schema: title, description, pubDate, updatedDate, slug, draft, category, column, authors, tags, and heroImage.
• Rewrite body image paths to site paths such as /uploads/original_images/xxx.png, and copy the corresponding files into astro-site/public/uploads/original_images/.

Idempotence And Overwrite Policy

• Decide whether to overwrite or skip based on slug or legacyId, so hand-written new posts are not accidentally deleted.
• The import script should only overwrite files that it owns, such as files in imported/ generated by the script. Hand-written posts should live elsewhere or follow a different naming rule.

One-Command Workflow

From the project root, the workflow can be wrapped as:

python scripts/sync_legacy.py && python scripts/import_wagtail.py

Or through a Makefile:

sync:
	python scripts/sync_legacy.py
import: sync
	python scripts/import_wagtail.py
build: import
	cd astro-site && pnpm build

This turns “pull data -> import -> build” into one repeatable flow, useful for periodic old-site syncs or pre-release updates.

Common issues:

• Inconsistent StreamField structure: Different Wagtail versions or custom blocks may have different JSON shapes. The import script should degrade gracefully for unknown block types, for example by rendering them as plain text or skipping with logs for later handling.
• Duplicate images: Full imports can accumulate files under public/uploads/. Use a stable overwrite rule based on legacy ID or slug, or clean the generated import directory before re-importing.
• Time zones: Wagtail fields such as first_published_at may be UTC. When writing frontmatter, either convert to local time deliberately or write consistent ISO strings that Astro parses correctly.

Summary

With separate sync and import scripts, Wagtail data and media can be pulled locally and converted into Astro content files. This preserves historical content while making the process repeatable and maintainable.

Series: Part 1: Why migrate and how I chose · Part 2: The main phases · Part 3: Adding new posts · Part 4: Deployment · Part 5: Data sync and scripting · Part 6: Theme and feature polish

WeChat

Follow ZiCode on WeChat

If this post was useful, you can follow later updates on WeChat as well.

X / Twitter

Follow @ax2_zicode

Faster technical notes, short thoughts, and new-post alerts are posted on X.

X Open X Profile @ax2_zicode