1.10.7 • Published 2 years ago

@posthog/plugin-server v1.10.7

Weekly downloads
3,514
License
MIT
Repository
github
Last release
2 years ago

PostHog Plugin Server

npm package MIT License

This service takes care of processing events with plugins and more.

Get started

Let's get you developing the plugin server in no time:

  1. Have virtual environment from the main PostHog repo active.

  2. Install dependencies and prepare for takeoff by running command yarn.

  3. Start a development instance of PostHog - instructions here. After all, this is the PostHog Plugin Server, and it works in conjuction with the main server. To avoid interference, disable the plugin server there with setting the PLUGIN_SERVER_IDLE env variable before running. PLUGIN_SERVER_IDLE=true ./bin/start

  4. Make sure that the plugin server is configured correctly (see Configuration). Two settings that you MUST get right are DATABASE_URL and REDIS_URL - they need to be identical between the plugin server and the main server.

  5. If developing the enterprise Kafka + ClickHouse pipeline, set KAFKA_ENABLED to true and provide KAFKA_HOSTS plus CLICKHOUSE_HOST, CLICKHOUSE_DATABASE, CLICKHOUSE_USER, andCLICKHOUSE_PASSWORD.

    Otherwise if developing the basic Redis + Postgres pipeline, skip ahead.

  6. Start the plugin server in autoreload mode with yarn start, or in compiled mode with yarn build && yarn start:dist, and develop away!

  7. To run migrations for the test, run yarn setup:test:postgres or setup:test:clickhouse. Run Postgres pipeline tests with yarn test:postgres:{1,2}. Run ClickHouse pipeline tests with yarn test:clickhouse:{1,2}. Run benchmarks with yarn benchmark.

Alternative modes

This program's main mode of operation is processing PostHog events, but there are also a few alternative utility ones. Each one does a single thing. They are listed in the table below, in order of precedence.

NameDescriptionCLI flags
HelpShow plugin server configuration options-h, --help
VersionOnly show currently running plugin server version-v, --version
HealthcheckCheck plugin server health and exit with 0 or 1--healthcheck
MigrateMigrate Graphile job queue--migrate
IdleStart server in a completely idle, non-processing mode--idle

Configuration

There's a multitude of settings you can use to control the plugin server. Use them as environment variables.

NameDescriptionDefault value
DATABASE_URLPostgres database URL'postgres://localhost:5432/posthog'
REDIS_URLRedis store URL'redis://localhost'
BASE_DIRbase path for resolving local plugins'.'
WORKER_CONCURRENCYnumber of concurrent worker threads0 – all cores
TASKS_PER_WORKERnumber of parallel tasks per worker thread10
REDIS_POOL_MIN_SIZEminimum number of Redis connections to use per thread1
REDIS_POOL_MAX_SIZEmaximum number of Redis connections to use per thread3
SCHEDULE_LOCK_TTLhow many seconds to hold the lock for the schedule60
CELERY_DEFAULT_QUEUECelery outgoing queue'celery'
PLUGINS_CELERY_QUEUECelery incoming queue'posthog-plugins'
PLUGINS_RELOAD_PUBSUB_CHANNELRedis channel for reload events'reload-plugins'
CLICKHOUSE_HOSTClickHouse host'localhost'
CLICKHOUSE_DATABASEClickHouse database'default'
CLICKHOUSE_USERClickHouse username'default'
CLICKHOUSE_PASSWORDClickHouse passwordnull
CLICKHOUSE_CAClickHouse CA certsnull
CLICKHOUSE_SECUREwhether to secure ClickHouse connectionfalse
KAFKA_ENABLEDuse Kafka instead of Celery to ingest eventsfalse
KAFKA_HOSTScomma-delimited Kafka hostsnull
KAFKA_CONSUMPTION_TOPICKafka incoming events topic'events_plugin_ingestion'
KAFKA_CLIENT_CERT_B64Kafka certificate in Base64null
KAFKA_CLIENT_CERT_KEY_B64Kafka certificate key in Base64null
KAFKA_TRUSTED_CERT_B64Kafka trusted CA in Base64null
KAFKA_PRODUCER_MAX_QUEUE_SIZEKafka producer batch max size before flushing20
KAFKA_FLUSH_FREQUENCY_MSKafka producer batch max duration before flushing500
KAFKA_MAX_MESSAGE_BATCH_SIZEKafka producer batch max size in bytes before flushing900000
LOG_LEVELminimum log level'info'
SENTRY_DSNSentry ingestion URLnull
STATSD_HOSTStatsD host - integration disabled if this is not providednull
STATSD_PORTStatsD port8125
STATSD_PREFIXStatsD prefix'plugin-server.'
DISABLE_MMDBwhether to disable MMDB IP location capabilitiesfalse
INTERNAL_MMDB_SERVER_PORTport of the internal server used for IP location (0 means random)0
DISTINCT_ID_LRU_SIZEsize of persons distinct ID LRU cache10000
PLUGIN_SERVER_IDLEwhether to disengage the plugin server, e.g. for developmentfalse
CAPTURE_INTERNAL_METRICSwhether to capture internal metrics for posthog in posthogfalse
PISCINA_USE_ATOMICScorresponds to the piscina useAtomics config option (https://github.com/piscinajs/piscina#constructor-new-piscinaoptions)true
PISCINA_ATOMICS_TIMEOUT(advanced) corresponds to the length of time (in ms) a piscina worker should block for when looking for tasks - instances with high volumes (100+ events/sec) might benefit from setting this to a lower value5000

Releasing a new version

Just bump up version in package.json on the main branch and the new version will be published automatically, with a matching PR in the main PostHog repo created.

It's advised to use bump patch/minor/major label on PRs - that way the above will be done automatically when the PR is merged.

Courtesy of GitHub Actions.

Walkthrough

The story begins with pluginServer.ts -> startPluginServer, which is the main thread of the plugin server.

This main thread spawns WORKER_CONCURRENCY worker threads, managed using Piscina. Each worker thread runs TASKS_PER_WORKER tasks (concurrentTasksPerWorker).

Main thread

Let's talk about the main thread first. This has:

  1. pubSub – Redis powered pub-sub mechanism for reloading plugins whenever a message is published by the main PostHog app.

  2. hub – Handler of connections to required DBs and queues (ClickHouse, Kafka, Postgres, Redis), holds loaded plugins. Created via hub.ts -> createHub. Every thread has its own instance.

  3. piscina – Manager of tasks delegated to threads. makePiscina creates the manager, while createWorker creates the worker threads.

  4. scheduleControl – Controller of scheduled jobs. Responsible for adding Piscina tasks for scheduled jobs, when the time comes. The schedule information makes it into the controller when plugin VMs are created.

    Scheduled tasks are controlled with Redlock (redis-based distributed lock), and run on only one plugin server instance in the entire cluster.

  5. jobQueueConsumer – The internal job queue consumer. This enables retries, scheduling jobs in the future (once) (Note: this is the difference between scheduleControl and this internal jobQueue). While scheduleControl is triggered via runEveryMinute, runEveryHour tasks, the jobQueueConsumer deals with meta.jobs.doX(event).runAt(new Date()).

    Jobs are enqueued by job-queue-manager.ts, which is backed by Postgres-based Graphile-worker (graphile-queue.ts).

  6. queue – Event ingestion queue. This is a Celery (backed by Redis) or Kafka queue, depending on the setup (EE/Cloud is Kafka due to high volume). These are consumed by the queue above, and sent off to the Piscina workers (src/main/ingestion-queues/queue.ts -> ingestEvent). Since all of the actual ingestion happens inside worker threads, you'll find the specific ingestion code there (src/worker/ingestion/ingest-event.ts). There the data is saved into Postgres (and ClickHouse via Kafka on EE/Cloud).

    It's also a good idea to see the producer side of this ingestion queue, which comes from Posthog/posthog/api/capture.py. The plugin server gets the process_event_with_plugins Celery task from there, in the Postgres pipeline. The ClickHouse via Kafka pipeline gets the data by way of Kafka topic events_plugin_ingestion.

  7. mmdbServer – TCP server, which works as an interface between the GeoIP MMDB data reader located in main thread memory and plugins ran in worker threads of the same plugin server instance. This way the GeoIP reader is only loaded in one thread and can be used in all. Additionally this mechanism ensures that mmdbServer is ready before ingestion is started (database downloaded from http-mmdb and read), and keeps the database up to date in the background.

Worker threads

This begins with worker.ts and createWorker().

hub is the same setup as in the main thread.

New functions called here are:

  1. setupPlugins – Loads plugins and prepares them for lazy VM initialization.

  2. createTaskRunner – Creates a Piscina task runner that allows to operate on plugin VMs.

Note: An organization_id is tied to a company and its installed plugins, a team_id is tied to a project and its plugin configs (enabled/disabled+extra config).

Questions?

Join our Slack community. 🦔

1.10.7

2 years ago

1.10.6

2 years ago

1.10.5

2 years ago

1.10.4

2 years ago

1.10.3

2 years ago

1.10.2

2 years ago

1.10.1

2 years ago

1.10.0

2 years ago

1.9.5

2 years ago

1.9.4

2 years ago

1.9.3

3 years ago

1.9.2

3 years ago

1.9.1

3 years ago

1.9.0

3 years ago

1.8.3

3 years ago

1.8.2

3 years ago

1.8.1

3 years ago

1.8.0

3 years ago

1.7.0

3 years ago

1.6.5

3 years ago

1.6.4

3 years ago

1.6.3

3 years ago

1.6.2

3 years ago

1.6.1

3 years ago

1.6.0

3 years ago

1.5.1

3 years ago

1.5.0

3 years ago

1.4.2

3 years ago

1.4.1

3 years ago

1.2.3

3 years ago

1.4.0

3 years ago

1.3.0

3 years ago

1.2.2

3 years ago

1.2.0

3 years ago

1.2.1

3 years ago

1.1.15

3 years ago

1.1.14

3 years ago

1.1.13

3 years ago

1.1.12

3 years ago

1.1.11

3 years ago

1.1.10

3 years ago

1.1.9

3 years ago

1.1.8

3 years ago

1.1.7

3 years ago

1.1.6

3 years ago

1.1.5

3 years ago

1.1.4

3 years ago

1.1.3

3 years ago

1.1.1

3 years ago

1.1.2

3 years ago

1.1.0

3 years ago

1.0.2

3 years ago

1.0.3

3 years ago

1.0.1

3 years ago

0.21.27

3 years ago

1.0.0

3 years ago

0.21.26

3 years ago

0.21.25

3 years ago

0.21.23

3 years ago

0.21.24

3 years ago

0.21.22

3 years ago

0.21.21

3 years ago

0.21.20

3 years ago

0.21.18

3 years ago

0.21.19

3 years ago

0.21.16

3 years ago

0.21.17

3 years ago

0.21.14

3 years ago

0.21.15

3 years ago

0.21.12

3 years ago

0.21.13

3 years ago

0.21.10

3 years ago

0.21.11

3 years ago

0.21.9

3 years ago

0.21.8

3 years ago

0.21.7

3 years ago

0.20.1

3 years ago

0.20.0

3 years ago

0.17.2

3 years ago

0.17.0

3 years ago

0.17.1

3 years ago

0.21.6

3 years ago

0.21.5

3 years ago

0.21.4

3 years ago

0.21.3

3 years ago

0.21.2

3 years ago

0.21.1

3 years ago

0.21.0

3 years ago

0.18.1

3 years ago

0.18.0

3 years ago

0.19.0

3 years ago

0.16.15

3 years ago

0.16.18

3 years ago

0.16.19

3 years ago

0.16.16

3 years ago

0.16.17

3 years ago

0.16.21

3 years ago

0.16.22

3 years ago

0.16.20

3 years ago

0.16.23

3 years ago

0.16.24

3 years ago

0.20.5

3 years ago

0.20.4

3 years ago

0.20.3

3 years ago

0.20.2

3 years ago

0.16.14

3 years ago

0.16.11

3 years ago

0.16.12

3 years ago

0.16.13

3 years ago

0.16.10

3 years ago

0.16.9

3 years ago

0.16.3

3 years ago

0.16.4

3 years ago

0.16.5

3 years ago

0.16.6

3 years ago

0.16.7

3 years ago

0.16.8

3 years ago

0.16.2

3 years ago

0.16.0

3 years ago

0.16.1

3 years ago

0.15.5

3 years ago

0.15.4

3 years ago

0.15.3

3 years ago

0.15.2

3 years ago

0.15.1

3 years ago

0.15.0

3 years ago

0.14.8

3 years ago

0.14.6

3 years ago

0.14.7

3 years ago

0.14.5

3 years ago

0.14.2

3 years ago

0.14.3

3 years ago

0.14.4

3 years ago

0.14.1

3 years ago

0.14.0

3 years ago

0.13.0

3 years ago

0.13.1

3 years ago

0.12.0

3 years ago

0.11.8

3 years ago

0.11.9

3 years ago

0.11.7

3 years ago

0.11.5

3 years ago

0.11.6

3 years ago

0.11.2

3 years ago

0.11.3

3 years ago

0.11.4

3 years ago

0.11.0

3 years ago

0.11.1

3 years ago

0.10.2

3 years ago

0.10.3

3 years ago

0.10.1

3 years ago

0.10.0

3 years ago

0.9.27

3 years ago

0.9.28

3 years ago

0.9.29

3 years ago

0.9.25

3 years ago

0.9.26

3 years ago

0.9.23

3 years ago

0.9.24

3 years ago

0.9.22

3 years ago

0.9.21

3 years ago

0.9.20

3 years ago

0.9.18

3 years ago

0.9.19

3 years ago

0.9.16

3 years ago

0.9.17

3 years ago

0.9.15

3 years ago

0.9.12

3 years ago

0.9.8

3 years ago

0.9.13

3 years ago

0.9.7

3 years ago

0.9.14

3 years ago

0.9.9

3 years ago

0.9.4

3 years ago

0.9.3

3 years ago

0.9.10

3 years ago

0.9.6

3 years ago

0.9.11

3 years ago

0.9.5

3 years ago

0.9.0

3 years ago

0.9.2

3 years ago

0.9.1

3 years ago

0.8.5

3 years ago

0.8.4

3 years ago

0.8.6

3 years ago

0.8.1

3 years ago

0.8.3

3 years ago

0.8.2

3 years ago

0.8.0

3 years ago

0.7.5

3 years ago

0.7.4

3 years ago

0.7.3

3 years ago

0.7.2

3 years ago

0.7.1

3 years ago

0.7.0

3 years ago

0.6.11

3 years ago

0.6.10

3 years ago

0.6.9

3 years ago

0.6.7

3 years ago

0.6.6

3 years ago

0.6.8

3 years ago

0.6.5

3 years ago

1.0.0-alpha.3

3 years ago

0.6.4

3 years ago