Historical migrations overview

Prior to starting a historical data migration, ensure you do the following:

1. Create a project on our US or EU Cloud.
2. Sign up to a paid product analytics plan on the billing page (historic imports are free but this unlocks the necessary features).
3. Raise an in-app support request with the Data pipelines topic detailing where you are sending events from, how, the total volume, and the speed. For example, "we are migrating 30M events from a self-hosted instance to EU Cloud using the migration scripts at 10k events per minute."
4. Wait for the OK from our team before starting the migration process to ensure that it completes successfully and is not rate limited.
5. Set the historical_migration option to true when capturing events in the migration.

Historical migrations refer to ingesting and importing past data into PostHog for analysis. This includes:

• Migrating from a different tool or platform like Mixpanel, Amplitude, Heap, or LaunchDarkly.

• Migrating from a self-hosted PostHog instance to PostHog Cloud.

• Migrating from one PostHog Cloud instance to another, for example US -> EU.

• Adding past data from a third-party source into PostHog.

What about exporting data from PostHog? Use our batch export feature to export data from PostHog to external services like S3 or BigQuery.

The basics of migrating data into PostHog

Start your migration by formatting your data correctly. There is no way to selectively delete event data in PostHog, so getting this right is critical. This means:

• Using the correct event names. For example, to capture a pageview event in PostHog, you capture a $pageview event. This might be different than the "name" other services use.

• Including the timestamp field. This ensures your events are ingested with the correct time in PostHog. It needs to be in the ISO 8601 format.

• Use the correct distinct_id. This is the unique identifier for your user in PostHog. Every event needs one. For example, posthog-js automatically generates a uuidv7 value for anonymous users.

To capture events, you must use the PostHog Python SDK or the PostHog API batch endpoint with the historical_migration set to true. This ensures we handle this data correctly and you aren't charged standard ingestion fees for it.

An example Python implementation looks like this:

from posthog import Posthog
from datetime import datetime

posthog = Posthog(
    '<ph_project_api_key>', 
    host='https://us.i.posthog.com',
    debug=True,
    historical_migration=True
)

events = [
  {
    "event": "batched_event_name",
    "properties": {
      "distinct_id": "user_id",
      "timestamp": datetime.fromisoformat("2024-04-02T12:00:00")
    }
  },
  {
    "event": "batched_event_name",
    "properties": {
      "distinct_id": "used_id",
      "timestamp": datetime.fromisoformat("2024-04-02T12:00:00")
    }
  }
]

for event in events:
  posthog.capture(
    distinct_id=event["properties"]["distinct_id"],
    event=event["event"],
    properties=event["properties"],
    timestamp=event["properties"]["timestamp"],
  )

An example cURL implementation using the batch API endpoint looks like this:

curl -v -L --header "Content-Type: application/json" -d '{
  "api_key": "<ph_project_api_key>",
  "historical_migration": true,
  "batch": [
    {
      "event": "batched_event_name",
      "properties": {
        "distinct_id": "user_id"
      },
      "timestamp": "2024-04-03T12:00:00Z"
    },
    {
      "event": "batched_event_name",
      "properties": {
        "distinct_id": "user_id"
      },
      "timestamp": "2024-04-03T12:00:00Z"
    }
  ]
}' https://us.i.posthog.com/batch/

Best practices for migrations

• We highly recommend testing at least a part of your migration on a test project before running it on your production project.

• Separate exporting your data from your service from importing it into PostHog. Store it in a storage service like S3 or GCS in between to ensure exported data is complete.

• Build resumability into your exports and imports, so you can just resume the process from the last successful point if any problems occur. For example, we use a cursor-based approach in our self-hosted migration tool.

• To batch user updates, use the same request but with the $identify event. Same for groups and the $group_identify event.