Sending OpenTelemetry Data To LightStep Using Cribl LogStream

Will Hallam · ‎01-10-2022

Cribl LogStream is a handy tool for managing logs. It lets you set up routes and pipelines for log data with support for many-to-many source/destination relationships. This can facilitate a few handy things when used in tandem with OpenTelemetry, such as the following:

allows alteration to the observability pipeline without application-level changes
provides a way to perform transformation and/or filtering of the raw OTel data before it is ingested by an observability tool like LightStep
facilitates a transparent means to perform "observability bakeoffs" between different tools
provides a way to store raw data indefinitely, e.g. saving to cloud storage

Recently I set up a demo app with OpenTelemetry instrumentation and was able to send the OTel data to Cribl as well as pass it on to LightStep. Here are some highlights.

Configure Cribl OTel Source

I added my Cribl OpenTelemetry sources by doing the following in the Cribl LogStream UI:

Navigate to Data/Sources
Select "OpenTelemetry"
Create a meaningful Input ID
Take the default Address and Port, which is port 4317 on all available interfaces
Click "Save"

NOTE: enabling TLS for real implementations is heartily recommended

Send OTel Data to Cribl

My example application is a Python script using Flask; all that was required to send telemetry to Cribl LogStream was to populate the endpoint with the IP/FQDN of my Cribl endpoint:

# flask_example.py
import flask
import requests

from opentelemetry import trace
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
    ConsoleSpanExporter,
)
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter,
)

span_exporter = OTLPSpanExporter(
    # optional
    endpoint="http://<cribl box>:4317",
    # credentials=ChannelCredentials(credentials),
    # headers=(("metadata", "metadata")),
)

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(span_exporter)
)
#trace.get_tracer_provider().add_span_processor(
#    BatchSpanProcessor(ConsoleSpanExporter())
#)

app = flask.Flask(__name__)
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

tracer = trace.get_tracer(__name__)


@app.route("/")
def hello():
    with tracer.start_as_current_span("example-request"):
        requests.get("http://www.example.com")
    return "hello"


app.run(debug=True, port=5000)

Verify Cribl OTel Source

Once I'd configured my source, I verified it was receiving data by starting my example app and sending requests to it using curl:

Once I had requests going to my test app, I opened my OpenTelemetry source in Cribl LogStream and selected the "Live Data" tab. Cribl pulled a sample of the data stream and populated the window.

Configure Cribl LightStep Destination

I created a destination in Cribl for my LightStep project via the Cribl UI as follows:

Navigate to Data/Destinations
Select "OpenTelemetry"
Populate Output ID with a meaningful name
Set Endpoint to "https://ingest.lightstep.com:443"
Set TLS Settings (Client Side)->Enabled to "Yes"
Create a new Metadata key under Advanced Settings:
Key set to "lightstep-access-token"
Value set to my LightStep access token, _single-quoted_
Click Save

Create Cribl Pipeline

For this example, I went with a straight pipeline without any transformation.

In the Cribl UI, navigate to Processing/Pipelines, click "+ Pipeline->Create Pipeline"
Enter a meaningful "ID" value and an optional Description, then click "Save"
Click "+ Function->Standard->Eval" to add a passthrough function
Click "Save"

Create Cribl Route

To glue all my Cribly bits together, I created a Route as follows:

In the Cribl UI, navigate to "Routing->Data Routes"
Click "+ Route"
Populate "Route Name" with something meaningful
Add a filter to handle only the data from my OpenTelemetry data source
Set Pipeline to my newly-created passthrough pipeline from the previous step
Set the Output to my newly-created LightStep data output
Click "Save"

Validate Cribl Destination

To validate that my OTel data is making the trip within Cribl, I do the following:
In the Cribl UI, navigate to "Data/Destinations"
Select my newly-created LightStep destination
Select the "Live Data" tab
Validate a few recent datagrams show up

Validate LightStep Ingestion

To make sure my OTel data is getting to LightStep, I login to my LightStep account, view the Reporting Status for my project and ensure I see counts for the applicable service (in my case, Python-based "unknown_service").

Conclusion

I'm finding that Cribl LogStream is super-useful and complementary to multiple ServiceNow solutions which consume log and/or instrumentation data. Once I have the data flowing through the Cribl ecosystem, I have many options with routing, retention and transformation. It's making my life a lot easier right now.

RJ11 · ‎03-27-2022