Baratine on GitHub

Baratine Service Architecture

Core aims of Baratine’s service architecture:

  • Strong encapsulation boundary for code, objects, and thread context.
  • Allow single-threaded programming for high-performance services
  • Application use their own POJO APIs.
  • Continuation-style asynchronous programming.
  • High performance.

Service Components

../../_images/service-proxy-stub.png

Components of the service model include:

  • Strict encapsulation boundary
  • An application service implementation
  • An optional application API
  • Thread safe concurrent proxy
  • Service thread for synchronization, cpu affinity, and isolation
  • Inbox queue to sequence and synchronize calls
  • Proxy and stub to marshal calls to messages and messages to calls
  • Outbox for outgoing messages.

Hello Service

Lets take a quick look at how services are used in practice.

1
2
3
4
5
6
7
 public class Hello
 {
   public void hello(Result<String> result)
   {
     result.ok("hello");
   }
 }

Service Creation and Use

1
2
3
4
5
6
7
8
9
public void test()
{
  Services services = Services.newManager().get();

  Hello hello = services.newService(Hello.class)
                        .as(Hello.class);

  hello.hello((value,exn)-> System.out.println("Hello: " + value));
}

Synchronous APIs

A traditional blocking API for a service can be useful when calling Baratine services from traditional threaded Java code. This blocking API is discouraged inside Baratine services themselves, but is useful for gateway code.

1
2
3
4
 public interface HelloSync
 {
   String hello();
 }

Sync Service Creation and Use

1
2
3
4
5
6
7
8
9
public void test()
{
  Services services = Services.newManager().get();

  HelloSync hello = services.newService(new Hello())
                            .as(HelloSync.class);

  System.out.println("Hello: " + hello.hello());
}

hello.hello() is a proxy call that waits for the message response using a future.

Because the future waits for a message, it ties up the requesting thread. While this thread parking is useful in gateway or testing code, it’s discourage in services because it blocks the service and because of extra overhead.

Send Methods

Baratine proxy methods can also send unidirectional fire-and-forget messages by using a void return type. When no response is required, Baratine can reduce the messaging cost by skipping the return message.

Send methods have advantages and disadvantages. Performance is better, particularly with large batches, but lose the ability to receive exceptions, which makes debugging more difficult.

1
2
3
4
 public interface HelloSend
 {
   void hello();
 }

Method Call Walkthrough

  • Client calls proxy method hello
  • Proxy creates a query message for the method with the target stub for Hello.
  • Proxy puts the query message in the Hello inbox and wakes Hello if it’s asleep.
  • Hello service wakes if not already awake.
  • Hello service receives messages from its inbox in order.
  • The Hello stub invokes the hello() method.
  • hello() executes and calls result.ok()
  • result.ok() puts the reply into the caller’s inbox and wakes the caller if it’s asleep.
  • The caller wakes if it’s not already awake.
  • The caller processes its messages from the inbox in order.
  • The caller processes the reply message and calls the Result continuation.

Pipes

Pipes are a secondary messaging system when flow control is required. A websocket clients for a chat system might freeze its network connection, but the system itself must not freeze. Flow control lets the producer avoid blocking when the consumer is blocked.

Pipes are unidirectional messaging between two services. Unlike service calls, they use application messages and require a subscription call to setup.

Flow control is managed by a credit sequence. The consumer adds credits, which the producer can use to send messages. For convenience, a prefetch can issue credits automatically for simpler applications.

Pipes are designed to resemble the JDK 9 Flow API and Reactive Streams.

Performance

Continuation/async performance is consistently better than async/future, especially under load. Under light load, async performance is around 9M method calls/sec while future performance is 4M calls/sec, which is reasonably close. As the load increases future performance drops dramatically to 0.5M calls/sec. Async performance drops as well but stays over 2.2M calls/sec.

../../_images/async-perf.png

The extra cost of the blocking/future calls is primarily wake/unpark cost, which is very expensive. The blocking/future calls have a fixed blocking thread per client, while the continuation/async calls release their thread to the thread pool. For the future call, “clients” is the number of threads, while for the continuation call, “clients” is the number of in-flight messages with fewer active, pooled threads.

Send vs Query Performance

Under light load, send performance is approximately double call performance because a call is two sends.

  solo 2 clients 3 clients 4 clients
Pipe 39M n/a n/a n/a
Send 18M 17M 15M 14M
Query Batch=1 9.2M 3.0M 2.2M 2.3M
Query Batch=4 7.4M 3.4M 4.1M 3.8M
Sync Query 4.1M 1.9M 0.65M 0.48M

Pipe has the best solo performance because it avoids blocking with its credit flow control, and because it’s optimized for a single producer and consumer.

While send has lower performance than pipe, it supports multiple producers. When its inbox queue is filled, it will force the producers to block, which incurs some overhead, but more importantly freezes the producers, preventing them from processing more messages while the queue is full.

Continuation/async calls have about half the performance of a solo send, because a call is two send messages: call and reply. With multiple clients, continuation performance drops, because a reply must wake the caller’s service if it’s asleep.

Batched continuation calls improve performance under load, nearly double the unbatched performance. Continuation calls can batch requests because the calling thread sends its next request while the previous call is in flight.

While sync/future calls are nearly as fast under light load, they quickly drop in performance under heavy load. Unlike continuation calls, they can’t batch requests because the calling thread is blocked until the call completes.