Goldrush is a small Erlang app that provides fast event stream processing

Event processing compiled to a query module

• per module private event processing statistics
• query module logic can be combined for any/all filters
• query module logic can be reduced to efficiently match event processing

Complex event processing logic

• match input events with greater than (gt) logic
• match input events with less than (lt) logic
• match input events with equal to (eq) logic
• match input events with wildcard (wc) logic
• match input events with notfound (nf) logic
• match no input events (null blackhole) logic
• match all input events (null passthrough) logic

Handle output events

• Once a query has been composed the output action can be overriden with one or more erlang functions. The functions will be applied to each output event from the query.

Handle low latency retrieval of compile-time stored values.

• Values stored are also provided to functions called on event output.
• Handle job execution and timing which can also get values stored
• create input events that include runtime on successful function executions.

Handle fastest lookups of stored values.

• provide state storage option to compile, caching the values in query module.

Usage

To use goldrush in your application, you need to define it as a rebar dep or include it in erlang’s path.

Before composing modules, you’ll need to define a query. The query syntax matches any number of `{erlang, terms}’ and is composed as follows:

Simple Logic

• Simple logic is defined as any logic matching a single event filter

Select all events where ‘a’ exists and is greater than 0.

glc:gt(a, 0).

Select all events where ‘a’ exists and is greater than or equal to 0.

glc:gte(a, 0).

Select all events where ‘a’ exists and is equal to 0.

glc:eq(a, 0).

Select all events where ‘a’ exists and is not equal to 0.

glc:neq(a, 0).

Select all events where ‘a’ exists and is less than 0.

glc:lt(a, 0).

Select all events where ‘a’ exists and is less than or equal to 0.

glc:lte(a, 0).

Select all events where ‘a’ exists.

glc:wc(a).

Select all events where ‘a’ does not exist.

glc:nf(a).

Select no input events. User as a black hole query.

glc:null(false).

Select all input events. Used as a passthrough query.

glc:null(true).

Combined Logic

• Combined logic is defined as logic matching multiple event filters

Select all events where both ‘a’ AND ‘b’ exists and are greater than 0.

glc:all([glc:gt(a, 0), glc:gt(b, 0)]).

Select all events where ‘a’ OR ‘b’ exists and are greater than 0.

glc:any([glc:gt(a, 0), glc:gt(b, 0)]).

Select all events where ‘a’ AND ‘b’ exists where ‘a’ is greater than 1 and ‘b’ is less than 2.

glc:all([glc:gt(a, 1), glc:lt(b, 2)]).

Select all events where ‘a’ OR ‘b’ exists where ‘a’ is greater than 1 and ‘b’ is less than 2.

glc:any([glc:gt(a, 1), glc:lt(b, 2)]).

Reduced Logic

• Reduced logic is defined as logic which can be simplified to improve efficiency.

Select all events where ‘a’ is equal to 1, ‘b’ is equal to 2 and ‘c’ is equal to 3 and collapse any duplicate logic.

glc_lib:reduce(
    glc:all([
        glc:any([glc:eq(a, 1), glc:eq(b, 2)]),
        glc:any([glc:eq(a, 1), glc:eq(c, 3)])])).

The previous example will produce and is equivalent to:

glc:all([glc:eq(a, 1), glc:eq(b, 2), glc:eq(c, 3)]).

Composing Modules

• All query modules must be compiled before use

To compose a module you will take your Query defined above and compile it.

glc:compile(Module, Query).
glc:compile(Module, Query, State).
glc:compile(Module, Query, State, ResetStatistics).

• At this point you will be able to handle an event using a compiled query.

Begin by constructing an event list.

Event = gre:make([{'a', 2}], [list]).

Now pass it to your query module to be handled.

glc:handle(Module, Event).

Handling output events

• You can override the output action with an erlang function.

Write all input events as info reports to the error logger.

glc:with(glc:null(true), fun(E) ->
     error_logger:info_report(gre:pairs(E)) end).

Write all input events where `error_level’ exists and is less than 5 as info reports to the error logger.

glc:with(glc:lt(error_level, 5), fun(E) ->
     error_logger:info_report(gre:pairs(E)) end).

Write all input events where `error_level’ exists and is 3 or 5 as info reports to the error logger.

glc:any([
  glc:with(glc:lt(error_level, 3), fun(E) ->
      error_logger:info_report(gre:pairs(E)) end),
  glc:with(glc:lt(error_level, 5), fun(E) ->
      error_logger:info_report(gre:pairs(E)) end)]).

To compose a module with state data you will add a third argument (orddict).

glc:compile(Module, Query, [{stored, value}]).

Return the stored value in this query module.

{ok, value} = glc:get(stored).

Return all stored values in this query module.

[...] = Module:get().

Composing Modules with stored data

• You can create query modules with local state to compare to event data in `with’ and `run’

To compose a module with state data you will add a third argument (orddict).

glc:compile(Module, Query, [{stored, value}]).

Accessing stored data in constant time

• You can use query modules in a way similar to mochiglobal

Return the stored value in this query module.

{ok, value} = glc:get(stored).

Job processing with composed modules

• You can use query modules to execute jobs, if the job errors or not, process an event.
• `with’ is similar to `run’, the main difference is additional statistics and execution order
• when a job completes in error, the event data will contain an additional {error, _} item

To execute a job through the query module, inputting an event on success.

Event = gre:make([{'a', 2}], [list]).
{ExecutionTime, Result}= glc:run(Module, fun(Event, State) ->
    %% do not end with {error, _} or throw an exception 
end, Event).

Event Processing Statistics

Return the number of input events for this query module.

glc:input(Module).

Return the number of output events for this query module.

glc:output(Module).

Return the number of filtered events for this query module.

glc:filter(Module).

Job Processing Statistics

Return the number of job runs for this query module.

glc:job_run(Module).

Return the number of job errors for this query module.

glc:job_error(Module).

Return the number of job inputs for this query module.

glc:job_input(Module).

Return the amount of time jobs took for this query module.

glc:job_time(Module).

Some Tips & Tricks

• This is really just a drop in the bucket.

Return the average time jobs took for this query module.

glc:job_time(Module) / glc:job_input(Module) / 1000000.

Return the query combining the conditional logic of multiple modules

glc_lib:reduce(glc:all([Module1:info('query'), Module2:info('query')]).

Return all statistics from this query module.

glc:info(Module).

Build

$ ./rebar compile

or

$ make

CHANGELOG

0.1.9

• Add support for running jobs

0.1.8

• Add support for not equal

0.1.7

• Support multiple functions specified using `with/2`
• Add support for greater than or less than operators
• Add state storage option for output events or lookup

0.1.7

• Add job execution and timings
• Add state storage option

0.1.7

• Add job execution and timings
• Add state storage option

0.1.6

• Add notfound event matching

0.1.5

• Rewrite to make highly crash resilient
  • per module supervision
  • statistics data recovery
• Add wildcard event matching
• Add reset counters