Insight Engineering Platform

Set of base libraries used primarily by the Insight Engineering team at Netflix to support applications that need to run internally and externally.

Overview

The internal platform libraries used at Netflix, provide many useful capabilities that help make the applications more consistent, easier to debug, and generally integrate well into the internal environment. For example it is straightforward to:

• Examine the properties and jars being used on running instance.
• Get consistent logging and ability to adjust log levels dynamically for debugging.
• Configuring JMX so common JVM tooling can be used through the firewall.
• Register with Eureka and use it to communicate with other services.

Over time some of this functionality was extracted into standalone libraries, many of which have been open sourced as NetflixOSS Common Runtime Services and Libraries. Examples are Archaius (configuration), Eureka (discovery service), Karyon (base server), Ribbon (Eureka aware HTTP client), Governator (dependency injection), and blitz4j (logging).

However, there are still some gaps and for many of the libraries mentioned above an internal wrapper library needs to be used to work well internally. Originally the intent was for the internal wrappers to get deprecated and phased out, but that turned out to be harder than expected and never actually happened. Further some of these libraries like Ribbon and blitz4j are no longer receiving much investment. The IEP libraries were created as part of the work to open source Atlas to allow us to:

• Have our open source applications be able to work the same way internally and externally.
• Ensure that the core debugging capabilities and key internal integrations work.
• Opt-in instead of opt-out. The internal platform provides a lot by default and you typically have to explicitly opt-out to turn off stuff you do not need. To make our apps lighter weight we wanted to explicitly opt-in instead. This has improved with some of the newer internal libraries, but is not available as part of NetflixOSS and thus doesn't satisfy our goal of consistency.

Modules

Libraries named with a prefix of iep-module- are Guice modules that can be pulled in. All Insight libraries should work using plain Guice, we do not rely on Governator extensions. However, some libraries do require some lifecycle management with methods that are annotated with PostConstruct and PreDestroy. Guice does not support that out of the box, for minimal support on top of guice you can use the iep-guice library.

You can pick and choose just the set of modules you need. If one module requires another, then it will install that module explicitly so you do not need to worry about the transitive dependencies.

Libraries

These are standalone libraries used in various Insight products.

Compatibility

The IEP libraries follow a semantic versioning scheme. Backwards incompatible changes should be marked with an incremented major version number. Forwards compatibility may work, but is in not required or guaranteed. It is highly recommended that all iep-* versions in the classpath are the same.

Prior to 1.0, it was mostly backwards compatible with major changes resulting in the minor version being incremented.

Common Runtime Libraries Comparison

Where possible we will use the other NetflixOSS Common Runtime Libraries, but as the support for some of those libraries has waned the usage of several have been dropped. This section will provide a quick summary of which parts we use and the differences.

Archaius

Archaius is the primary configuration library used at Netflix. The primary feature over other alternatives is that it can communicate with a property service to allow for properties that can be changed at runtime. This can be used for things like feature flags to enable or quickly disable functionality. Insight uses Archaius 2 configured via iep-module-archaius2.

It should be noted that as deployment automation and velocity has increased our (Insight team) interest in runtime properties has waned. Changing runtime properties can be just as risky as a deployment and adding proper checks such as canary analysis, staggered rollout, etc to the property path means there is little advantage to using properties over just doing a new deployment. The base layer for our configuration is using the Typesafe Config library and many new uses inject that directly rather than injecting the Archaius2 Config object.

Blitz4j

Logging library providing performance improvements on top of log4j 1.x and making the logging levels configurable via properties. Most code should be using the slf4j interfaces so the logging framework is pluggable, so this decision should only matter when selecting a binding for running the application. For Insight apps we bind slf4j to vanilla log4j2 when running internally. The logging configuration is mapped to a file that is monitored by log4j2 so we can tune log levels on an instance.

Eureka

Eureka is the Netflix service discovery system. All of the Insight apps should register with Eureka to integrate well with internal systems that check this as part of ensuring the service is healthy. If using iep-module-eureka, then registration should happen automatically and the Eureka status will be mapped to the service state so it accurately reflects if the application is fully up and ready to receive traffic. Insight apps will also map the healthcheck endpoint to the service state so that healthcheck polling and the state from Eureka heartbeats should match.

For client side uses, see section discussing Ribbon.

Governator

IEP is compatible with, but does not directly use or require Governator. All Insight libraries should work using plain Guice, we do not rely on Governator extensions. However, some libraries do require some lifecycle management with methods that are annotated with PostConstruct and PreDestroy. Guice does not support that out of the box, for minimal support on top of guice you can use the iep-guice library.

JMX

The internal platform-management library restricts JMX to use a single port so that it is easier to make tools like VisualVM and JMC work via SSH tunnels or firewalls where there is no desire to open up a full range of ports. This functionality is not available in other NetflixOSS, but is supported if the iep-module-jmxport module is used.

Karyon

Karyon provides the server framework and admin for internal applications. Internally it is wrapped by the base-server library, though newer applications can use Karyon 2 directly. For a long time we have only used the admin aspect of Karyon. The IEP admin is much lighter weight and was originally developed as part of Karyon 3. The Runtime team decided to go a different direction so we simplified and inlined the aspects we care about as iep-admin.

Ribbon

Ribbon is the Eureka aware HTTP client layer used heavily at Netflix. Internally it is often wrapped by the platform-ipc library and often referred to as NIWS (Netflix Integrated Web Services). Similar to the story with Karyon, Insight was an early adopter of RxNetty that was intended to become the underlying library powering Ribbon. Insight developed the iep-rxhttp library to guinea pig RxNetty internally. It is still heavily used by the Insight team for interacting with services that require middle tier load balancing. However, the Runtime team has since de-prioritized RxNetty and is focusing on gRPC for new use-cases. Ribbon is not used by Insight and we'll likely phase out iep-rxhttp over time.

Another trend is that since all Netflix services are now in the VPC, the AWS ELBs can have proper security groups. That was not true in classic and was a big reason for initially building out Eureka. Using ELBs and standard DNS means the selection of client library is less important. The main add-on we want is consistent metrics and access logs which we achieve by using the Spectator HTTP log util.

Related Projects

Open source projects maintained by the Netflix Insight team are:

• Apps
  • Atlas: in-memory dimensional time series database.
  • Edda: read-only cache of AWS resources.
  • IEP Apps: small example apps using Insight libraries.
• Libraries
  • AWSObjectMapper: provides mappings for Jackson 2 to allow the AWS Java SDK model objects to easily be converted to/from JSON.
  • Spectator: client library for instrumenting an application to report data into Atlas or similar data stores.
• Legacy Libraries
  • Edda Client: client library to access Edda via the AWS Java SDK interfaces.
  • Rx AWS Java SDK: experimental implementation of an AWS Java SDK that uses RxNetty internally and automatically handles pagination via RxJava Observables.
  • Servo: legacy library for instrumenting code. Use Spectator for new projects.