opentelemetry（OpenTelemetry 命令行）

OpenTelemetry with Grafana

Overview

OpenTelemetry (OTel) is a vendor-neutral framework for collecting observability data (metrics, logs, traces, profiles). Grafana Labs integrates it as a core strategy, offering a full stack to collect, ingest, store, analyze, and visualize telemetry data.

Four-Step Implementation Model

1. Instrument - Add telemetry using Grafana SDKs, Beyla (eBPF), or upstream OTel SDKs
2. Pipeline - Build processing infrastructure with Grafana Alloy or OTel Collector
3. Ingest - Route data to Grafana Cloud OTLP endpoint or self-managed backends
4. Analyze - Dashboards, alerts, Application Observability, Drilldown apps

Grafana Backends

OTLP Endpoint and Authentication

Grafana Cloud OTLP Endpoint

Grafana Cloud exposes a managed OTLP gateway endpoint:

https://otlp-gateway-<region>.grafana.net/otlp

Example regions: prod-us-east-0, prod-eu-west-0, prod-ap-southeast-0

Full example:

https://otlp-gateway-prod-us-east-0.grafana.net/otlp

Authentication - Basic Auth

Grafana Cloud OTLP uses HTTP Basic Auth:

• Username: Grafana Cloud Instance ID (numeric, e.g. 123456)
• Password: Grafana Cloud API token (with MetricsPublisher, LogsPublisher, TracesPublisher roles)

Via environment variable (recommended)

# Base64-encode "instanceID:apiToken"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n '123456:glc_eyJ...' | base64)"

Via Alloy environment variables

export GRAFANA_CLOUD_INSTANCE_ID=123456
export GRAFANA_CLOUD_API_KEY=glc_eyJ...
export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp

Direct Send (no collector) - Environment Variables

export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64(instanceID:apiToken)>"
export OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=production"

Instrumentation by Language

Go

Requirements: Go 1.22+

Install packages:

go get "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" \
  "go.opentelemetry.io/contrib/instrumentation/runtime" \
  "go.opentelemetry.io/otel" \
  "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp" \
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace" \
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" \
  "go.opentelemetry.io/otel/sdk" \
  "go.opentelemetry.io/otel/sdk/metric"

Run with environment variables:

OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
go run .

See references/instrumentation.md for full Go code example.

Java (Grafana Distribution - JVM Agent)

Requirements: JDK 8+

Download: grafana-opentelemetry-java.jar from https://github.com/grafana/grafana-opentelemetry-java/releases

Run:

OTEL_RESOURCE_ATTRIBUTES="service.name=shoppingcart,service.namespace=ecommerce,deployment.environment=production" \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
java -javaagent:/path/to/grafana-opentelemetry-java.jar -jar myapp.jar

Optional: Data saver mode (reduces metric cardinality):

export GRAFANA_OTEL_APPLICATION_OBSERVABILITY_METRICS=true

Debug:

export OTEL_JAVAAGENT_DEBUG=true
# Enable console output alongside OTLP
export OTEL_TRACES_EXPORTER=otlp,console
export OTEL_METRICS_EXPORTER=otlp,console
export OTEL_LOGS_EXPORTER=otlp,console

Node.js

Install:

npm install --save @opentelemetry/api
npm install --save @opentelemetry/auto-instrumentations-node

Run:

OTEL_TRACES_EXPORTER="otlp" \
OTEL_METRICS_EXPORTER="otlp" \
OTEL_LOGS_EXPORTER="otlp" \
OTEL_NODE_RESOURCE_DETECTORS="env,host,os" \
OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register" \
node app.js

Warning: Bundlers like @vercel/ncc can break auto-instrumentation hooks.

See references/instrumentation.md for manual SDK setup example.

Python

Install:

pip install "opentelemetry-distro[otlp]"
opentelemetry-bootstrap -a install

Run:

OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
opentelemetry-instrument python app.py

Multi-process servers (Gunicorn, uWSGI): implement post-fork hooks to reinitialize OTel providers per worker.

.NET (Grafana Distribution)

Install NuGet:

dotnet add package Grafana.OpenTelemetry

ASP.NET Core setup:

using Grafana.OpenTelemetry;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetry()
    .WithTracing(configure => configure.UseGrafana())
    .WithMetrics(configure => configure.UseGrafana());
builder.Logging.AddOpenTelemetry(options => options.UseGrafana());

Run:

OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
dotnet run

Requirements:.NET 6+ or.NET Framework 4.6.2+

See references/instrumentation.md for full.NET examples.

Beyla (eBPF - Language Agnostic)

Grafana Beyla instruments at the network layer - no code changes required, works with any language.

# Docker
docker run --rm -it \
  --privileged \
  -e BEYLA_SERVICE_NAME=myapp \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
  -v /sys/kernel/security:/sys/kernel/security \
  grafana/beyla

Verify with: curl http://localhost:9090/metrics

Full docs: https://grafana.com/docs/beyla/

Grafana Alloy Collector

Grafana Alloy is the recommended OTel Collector distribution. It combines upstream OTel Collector components with Prometheus exporters for infrastructure + application observability correlation.

Why Use a Collector?

• Cost control: Aggregate, sample, and drop data before sending
• Reliability: Buffer and retry on connection failures
• Enrichment: Add resource attributes, transform, redact, and route data

Alloy Ports

Application -> Alloy -> Grafana Cloud

Application env vars (point to local Alloy):

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Alloy config env vars (Alloy -> Grafana Cloud):

export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export GRAFANA_CLOUD_INSTANCE_ID=123456
export GRAFANA_CLOUD_API_KEY=glc_eyJ...

See references/collector-config.md for full Alloy configuration.

Kubernetes Setup

Option 1: Grafana Kubernetes Monitoring Helm Chart (recommended)

The Grafana Kubernetes Monitoring Helm chart deploys Alloy with OTLP receivers pre-configured.

1. Enable "OTLP Receivers" in the Cluster Configuration tab
2. Get gRPC/HTTP endpoints from "Configure Application Instrumentation" section
3. Point apps to the in-cluster Alloy endpoint:

export OTEL_EXPORTER_OTLP_ENDPOINT=<GRPC_ENDPOINT_FROM_HELM>
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Option 2: OpenTelemetry Operator

Install via official docs, then use Instrumentation CR for auto-injection:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://otelcol:4317
  propagators:
    - tracecontext
    - baggage
  java:
    # Use Grafana distribution image
    image: us-docker.pkg.dev/grafanalabs-global/docker-grafana-opentelemetry-java-prod/grafana-opentelemetry-java:2.3.0-beta.1
  nodejs: {}
  python: {}

Inject into pods with annotation:

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
    # or: inject-nodejs, inject-python, inject-dotnet

See references/collector-config.md for Kubernetes Alloy Helm values and OTel Collector YAML.

Sampling Strategies

Head-Based Sampling

Decision made at trace start - low overhead, may miss rare errors.

Environment variable (probability sampler):

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1   # 10% of traces

Alloy head sampling config:

otelcol.processor.probabilistic_sampler "default" {
  sampling_percentage = 10
  output {
    traces = [otelcol.exporter.otlphttp.grafana_cloud.input]
  }
}

Tail-Based Sampling

Decision made after all spans collected - can sample based on outcome (e.g. keep all errors).

Alloy tail sampling config:

otelcol.processor.tail_sampling "default" {
  decision_wait            = "10s"
  num_traces               = 100000
  expected_new_traces_per_sec = 10

  policy {
    name = "keep-errors"
    type = "status_code"
    status_code {
      status_codes = ["ERROR"]
    }
  }

  policy {
    name = "probabilistic-sample"
    type = "probabilistic"
    probabilistic {
      sampling_percentage = 10
    }
  }

  output {
    traces = [otelcol.exporter.otlphttp.grafana_cloud.input]
  }
}

Key Environment Variables Reference

Key Resource Attributes

Useful Links

• Grafana OTel docs: https://grafana.com/docs/opentelemetry/
• Grafana Cloud OTLP: https://grafana.com/docs/grafana-cloud/send-data/otlp/
• Grafana Java Agent: https://github.com/grafana/grafana-opentelemetry-java
• Grafana.NET SDK: https://github.com/grafana/grafana-opentelemetry-dotnet
• Grafana Alloy: https://grafana.com/docs/alloy/
• Grafana Beyla: https://grafana.com/docs/beyla/
• OTel Collector: https://opentelemetry.io/docs/collector/
• OTel Operator: https://opentelemetry.io/docs/kubernetes/operator/