Skip to content

Instantly share code, notes, and snippets.

@rud
Last active April 3, 2016 13:57
Show Gist options
  • Save rud/c54d9d769b21f724d10989da71bccbe8 to your computer and use it in GitHub Desktop.
Save rud/c54d9d769b21f724d10989da71bccbe8 to your computer and use it in GitHub Desktop.

Revisions

  1. rud revised this gist Apr 3, 2016. 1 changed file with 6 additions and 3 deletions.
    9 changes: 6 additions & 3 deletions chunked-event-publishing.md
    Original file line number Diff line number Diff line change
    @@ -1,20 +1,23 @@
    # Chunked event publishing - design overview

    Class: `EventCollector`
    ## Class: `EventCollector`

    Roles:
    - `#enqueue_event(payload, target queue name)`: stores messages to be sent in a simple array
    - `#publish()` send everything currently enqueued to external queue system (here: RabbitMQ); if the `publish` fails for any reason, messages are kept in memory for later retry (along with grumbling in logs, etc).
    - `#clear_events_to_publish!` is useful in rollback scenarios
    - `.thread_singleton_instance` accessor with lazy initialization, like `ActiveRecord::Base.connection`. This allows for easy setup, and persistent state across requests (should flushing fail we can retry on next request on the same thread). The `thread_singleton_instance` also easy replacement with a simple test implementation, think `ActionMailer`. Sending actual messages in test is best restricted to a few integration test scenarios.

    Class: `AutoPublishingEventCollector`
    ## Class: `AutoPublishingEventCollector`

    Roles:
    - ensure a `PublishingEventCollector` is setup for the current block of code
    - call `#publish` after a block of code has completed successfully
    - useful in a trivial Rails middleware, for instance, or in the console


    Main patterns of use:
    ## Main patterns of use

    - in the Rails console it is usually convenient to autoflush after each enqueued event.
    - when handling requests, it makes sense to automatically call `#publish` after each successful request.
    - message consumers (worker processes) are usually explicitly calling `#publish` at the appropriate times. Most other code paths ignore the existence of queues and events.
  2. rud created this gist Apr 3, 2016.
    23 changes: 23 additions & 0 deletions chunked-event-publishing.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,23 @@
    # Chunked event publishing - design overview

    Class: `EventCollector`
    Roles:
    - `#enqueue_event(payload, target queue name)`: stores messages to be sent in a simple array
    - `#publish()` send everything currently enqueued to external queue system (here: RabbitMQ); if the `publish` fails for any reason, messages are kept in memory for later retry (along with grumbling in logs, etc).
    - `#clear_events_to_publish!` is useful in rollback scenarios
    - `.thread_singleton_instance` accessor with lazy initialization, like `ActiveRecord::Base.connection`. This allows for easy setup, and persistent state across requests (should flushing fail we can retry on next request on the same thread). The `thread_singleton_instance` also easy replacement with a simple test implementation, think `ActionMailer`. Sending actual messages in test is best restricted to a few integration test scenarios.

    Class: `AutoPublishingEventCollector`
    Roles:
    - ensure a `PublishingEventCollector` is setup for the current block of code
    - call `#publish` after a block of code has completed successfully
    - useful in a trivial Rails middleware, for instance, or in the console


    Main patterns of use:
    - in the Rails console it is usually convenient to autoflush after each enqueued event.
    - when handling requests, it makes sense to automatically call `#publish` after each successful request.
    - message consumers (worker processes) are usually explicitly calling `#publish` at the appropriate times. Most other code paths ignore the existence of queues and events.
    - in tests it makes sense to assert on the existence or absence of messages matching specific patterns after some code has run. Having collected the messages with topics in memory these matchers are usually easy to create.

    The above was implemented on top of [sneakers](http://sneakers.io) / RabbitMQ, but I think the pattern is generally useful.