Monitoring Apache with OpenTelemetry and Cloud Observability

Heather_Waters
ServiceNow Employee
ServiceNow Employee
‎09-26-2023 09:18 AM

The majority of today’s websites and web applications are served up with the Apache web server. Web servers act as a traffic cop to a host machine, routing incoming requests to their corresponding websites. For this reason, monitoring Apache performance metrics—such as the connection count, the rate of incoming requests, and the rate of traffic flow—is essential. Your uptime depends on the availability and proper functioning of your Apache web server.

A simple way to track Apache server metrics is to use the OpenTelemetry Collector. In this guide, I’ll walk through how
to set up the OpenTelemetry Collector to ingest metrics from Apache and then send those metrics to Cloud Observability. Then, you'll create charts and alerts in Cloud Observability to help monitor your Apache server.

First, let’s look briefly at the metrics available from Apache.

Apache Metrics

The mod_status module, when turned on, exposes server metrics through a specific endpoint. By default, that endpoint is /server-status. The metrics available include, as described by Apache’s documentation:

  • The number of workers serving requests

  • The number of idle workers

  • The status of each worker, the number of requests that worker has performed, and the total number of bytes served by the worker

  • A total number of accesses and byte count served

  • The time the server was started/restarted and the time it has been running for

  • Averages of the number of requests per second, bytes served per second, and the bytes per request

  • The current CPU percentage used by each worker and in total by all workers combined

  • The current hosts and requests being processed

Assuming you have Apache started on your local host, here's what you’ll see when you visit  http://localhost:80/server-status.

 

 
00-browser-apache-server-status.png
 

The metrics are also available in a simple list format as plain text. This structured format is what the OpenTelemetry Collector uses to collect metrics. To see the structured format, visit http://localhost:80/server-status?auto. The result is:

localhost
ServerVersion: Apache/2.4.41 (Ubuntu)
ServerMPM: event
Server Built: 2023-03-08T17:32:54
CurrentTime: Monday, 27-Mar-2023 21:19:43 MST
RestartTime: Monday, 27-Mar-2023 21:14:22 MST
ParentServerConfigGeneration: 1
ParentServerMPMGeneration: 0
ServerUptimeSeconds: 320
ServerUptime: 5 minutes 20 seconds
Load1: 1.13
Load5: 2.25
Load15: 1.57
Total Accesses: 3
Total kBytes: 10
Total Duration: 44
CPUUser: 0
CPUSystem: .27
CPUChildrenUser: 0
CPUChildrenSystem: 0
CPULoad: .084375
Uptime: 320
ReqPerSec: .009375
BytesPerSec: 32
BytesPerReq: 3413.33
DurationPerReq: 14.6667
BusyWorkers: 1
IdleWorkers: 49
Processes: 2
Stopping: 0
BusyWorkers: 1
IdleWorkers: 49
ConnsTotal: 0
ConnsAsyncWriting: 0
ConnsAsyncKeepAlive: 0
ConnsAsyncClosing: 0

These metrics can give us an indication of Apache’s current load and performance. By monitoring these metrics, we can keep a careful watch to ensure Apache is serving up web applications as expected.

 

Let’s look at how to use the OpenTelemetry Collector and Cloud Observabilitycv to monitor your Apache instance.

 

Configure OpenTelemetry Collector

The Apache receiver for OpenTelemetry Collector gathers metrics by periodically scraping the data from server-status?auto. The specific metrics captured can be found here. 

 

Install the Collector

Because the Apache receiver is part of the contributor distribution of the OpenTelemetry Collector, we need to install this distribution. We can follow the installation instructions on GitHub. For our demo environment, we run this command:

$ wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.72.0/otelcol-contrib_0.72.0_linux_amd64.deb
$ sudo dpkg -i otelcol-contrib_0.72.0_linux_amd64.deb

 

Then, we can verify that the Collector is running:

$ sudo systemctl status otelcol-contrib
● otelcol-contrib.service - OpenTelemetry Collector Contrib
     Loaded: loaded (/lib/systemd/system/otelcol-contrib.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-03-30 18:43:36 MST; 17s ago
   Main PID: 403935 (otelcol-contrib)
      Tasks: 14 (limit: 18864)
     Memory: 139.2M
     CGroup: /system.slice/otelcol-contrib.service
             ├─403935 /usr/bin/otelcol-contrib --config=/etc/otelcol-contrib/config.yaml
             └─403951 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session

 

Configure the Collector

The first step is to set up the Apache receiver. To do this, you need to configure the OpenTelemetry Collector. The configuration file for the Collector can be found here: /etc/otelcol-contrib/config.yaml.

Modify the configuration to add the apache receiver and then specify the endpoint that exposes Apache metrics (http://localhost:80/server-status?auto). Set your collection interval to 10 seconds.

receivers:
  apache:
    endpoint: "http://localhost:80/server-status?auto"
    collection_interval: 10s

processors:
  batch:

exporters:
  logging:
    verbosity: detailed

service:
  pipelines:
    metrics:
      receivers: [apache]
      processors: [batch]


Use the batch processor and the logging exporter, as those are the basic options that help you get up and running quickly. The logging exporter helps verify that the Collector is working, but you’ll soon change the exporter to use Cloud Observability instead.


Now, save the configuration file and restart the Collector.

$ sudo systemctl restart otelcol-contrib

 

Then, run journalctl to verify that the Collector is properly gathering your Apache metrics. Here’s a snippet of what we see.

 
$ journalctl -u otelcol-contrib -f
…
Mar 30 18:43:37 info otelcol-contrib: 2023-03-30T18:43:37.801-0700        info        service/service.go:140        Everything is ready. Begin running and processing data.
Mar 30 18:43:48 info otelcol-contrib: 2023-03-30T18:43:48.021-0700        info        MetricsExporter        {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 11}
Mar 30 18:43:48 info otelcol-contrib: 2023-03-30T18:43:48.022-0700        info        ResourceMetrics #0
Mar 30 18:43:48 info otelcol-contrib: Resource SchemaURL:
Mar 30 18:43:48 info otelcol-contrib: Resource attributes:
Mar 30 18:43:48 info otelcol-contrib:      -> apache.server.name: Str(localhost)
Mar 30 18:43:48 info otelcol-contrib:      -> apache.server.port: Str(80)
Mar 30 18:43:48 info otelcol-contrib: ScopeMetrics #0
Mar 30 18:43:48 info otelcol-contrib: ScopeMetrics SchemaURL:
Mar 30 18:43:48 info otelcol-contrib: InstrumentationScope otelcol/apachereceiver 0.68.0
Mar 30 18:43:48 info otelcol-contrib: Metric #0
Mar 30 18:43:48 info otelcol-contrib: Descriptor:
Mar 30 18:43:48 info otelcol-contrib:      -> Name: apache.cpu.time
Mar 30 18:43:48 info otelcol-contrib:      -> Description: Jiffs used by processes of given category.
Mar 30 18:43:48 info otelcol-contrib:      -> Unit: {jiff}
Mar 30 18:43:48 info otelcol-contrib:      -> DataType: Sum
Mar 30 18:43:48 info otelcol-contrib:      -> IsMonotonic: true
Mar 30 18:43:48 info otelcol-contrib:      -> AggregationTemporality: Cumulative
Mar 30 18:43:48 info otelcol-contrib: NumberDataPoints #0
Mar 30 18:43:48 info otelcol-contrib: Data point attributes:
Mar 30 18:43:48 info otelcol-contrib:      -> level: Str(children)
Mar 30 18:43:48 info otelcol-contrib:      -> mode: Str(user)
Mar 30 18:43:48 info otelcol-contrib: StartTimestamp: 2023-03-31 01:43:37.779914959 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Timestamp: 2023-03-31 01:43:47.845740707 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Value: 0.000000
Mar 30 18:43:48 info otelcol-contrib: NumberDataPoints #1
Mar 30 18:43:48 info otelcol-contrib: Data point attributes:
Mar 30 18:43:48 info otelcol-contrib:      -> level: Str(self)
Mar 30 18:43:48 info otelcol-contrib:      -> mode: Str(user)
Mar 30 18:43:48 info otelcol-contrib: StartTimestamp: 2023-03-31 01:43:37.779914959 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Timestamp: 2023-03-31 01:43:47.845740707 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Value: 0.000000
Mar 30 18:43:48 info otelcol-contrib: NumberDataPoints #2
Mar 30 18:43:48 info otelcol-contrib: Data point attributes:
Mar 30 18:43:48 info otelcol-contrib:      -> level: Str(self)
Mar 30 18:43:48 info otelcol-contrib:      -> mode: Str(system)
Mar 30 18:43:48 info otelcol-contrib: StartTimestamp: 2023-03-31 01:43:37.779914959 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Timestamp: 2023-03-31 01:43:47.845740707 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Value: 0.000000
Mar 30 18:43:48 info otelcol-contrib: NumberDataPoints #3
Mar 30 18:43:48 info otelcol-contrib: Data point attributes:
Mar 30 18:43:48 info otelcol-contrib:      -> level: Str(children)
Mar 30 18:43:48 info otelcol-contrib:      -> mode: Str(system)
Mar 30 18:43:48 info otelcol-contrib: StartTimestamp: 2023-03-31 01:43:37.779914959 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Timestamp: 2023-03-31 01:43:47.845740707 +0000 UTC
Mar 30 18:43:48 info otelcol-contrib: Value: 0.000000
…

 

Now that you've verified proper metrics collection, you're ready to start exporting those metrics to Cloud Observability.

 

Send metrics from the OpenTelemetry Collector to Cloud Observability

After you sign into your Cloud Observability account, you’ll need to create an access token. You'll use this token in your Collector configuration, as the Collector needs to authenticate its requests when sending metrics data to Cloud Observability. 

 

Navigate to Project Settings > Access Tokens. Click on Create New to create a new access token. Copy the token value.

 

02-access-token.png
 
 

Configure the Collector to export to Cloud Observability

Please note: Cloud Observability continues to use lightstep (the former product name) in code for ongoing compatibility.

Now, you’ll replace the logging exporter in your configuration. Next, you'll use an exporter to ship your data to Cloud Observability with OTLP. Modify /etc/otelcol-contrib/config.yaml to look like this:

 

receivers:
  apache:
    endpoint: "http://localhost:80/server-status?auto"
    collection_interval: 10s

processors:
  batch:

exporters:
  otlp/lightstep:
    endpoint: ingest.lightstep.com:443
    headers: {"lightstep-access-token": "INSERT YOUR TOKEN HERE"}

service:
  pipelines:
    metrics:
      receivers: [apache]
      processors: [batch]
      exporters: [otlp/lightstep]

 

Make sure to insert your own access token, and then restart the Collector:

$ sudo systemctl restart otelcol-contrib


With your Collector gathering metrics from Apache and sending them to Lightstep, you can begin putting together some charts.


Work with metrics in Cloud Observability

In Cloud Observability, you’ll create a dashboard for your Apache metrics. Then, you'll add a few charts to the dashboard in order to visualize the metrics. You'll create some basic charts, but additional examples of dashboards and charts can be found here.

 

Create a dashboard

On the Dashboard pages, click on Create Dashboard. Provide a name and a description for your dashboard.

 

 
03-name-dashboard.png

 

 

Add a chart

Now, add rour first chart. You'll want a “big number” chart that shows the current load that Apache is putting on your host CPU. First, click Add a chart.

 
04-add-chart-button.png
 

Name your chart “Current Load on CPU”.

 

The Query Builder in Cloud Observability lets you build your query using a helpful UI of drop-downs and options. If you’re familiar with our Unified Querying Language (UQL) (learn more here), then you can write your queries directly by using the Query Editor.

 

For now, use the Query Builder to keep this simple. Start by selecting the type of telemetry that you want to chart: Metric.

 

 
06-select-telemetry-type.png

 

Because Cloud Observability will display all the metrics (not just the ones from Apache) that are sent to your account, you might see a long list. By typing “Apache” in the search box, you can filter down to just those metrics related to your Apache server. Find and select the metric called apache.cpu.load.

07-choose-apache-metric-cpu-load.png

 

Initially, you're shown a line chart that shows Apache’s load on the CPU over the last 60 minutes.

 

 
08-line-chart-cpu-load-last-60-minutes.png

 

What you're interested in is the current load (or the latest value). For this, change the View as chart type, select Big number instead.

 

 
09-view-as-big-number.png

 

 

10-dashboard-one-chart.png

 

Additional charts

Next, add a line chart that shows current connections to your Apache server. For this chart, use the apache.current_connections metric. Again, you'll be shown a line chart with connection counts from the last 60 minutes.

 

11-chart-current-connections-60-minutes.png

 

You can adjust the time range for the chart if you're interested in specific windows of time. For example, you can adjust the chart to see just the last 10 minutes.

 
12-adjust-time-scale.png 

After you apply this change, the chart looks like this:

 

 
13-chart-current-connections-10-minutes.png

 

 

Save your chart to add it to the dashboard.

Now, create one more chart. For this one, you're interested in the apache.traffic metric. This is a cumulative metric as opposed to the metrics charted so far, which were both gauges. Because apache.traffic is a running count that’s always increasing, you're more interested in the rate of change.

Cloud Observability knows this, so it aggregates the data to display rate. Set your chart to be an Area type chart. The resulting view shows the traffic in terms of bytes per second at any given moment. Again, look at your chart with the “last 10 minutes” time range.
 

 
14-area-chart-traffic-10-minutes.png

 

 

After you save your charts (and even create a few more), your dashboard looks like this:

 

 
15-dashboard-five-charts.png

 

 

Set up alerts and notifications

Although viewing Apache metrics in real time is helpful, you'll benefit even more if you can set up alerts and automated notifications whenever a certain metric reaches a critical threshold. For example, if you set up an alert if the rate of incoming requests exceeds 100 requests per second, it might be indicative of a coordinated DDoS attack.

To do this, click on Alerts in the navbar on the left.

16-narvbar-arts.png

 

 

Next, click Create an alert.
 

The metric you’ll monitor is apache.requests. You’ll aggregate this metric to look at the rate over an input window of 30 seconds.

 

 
18-alert-metric-settings.png

 

For the alert configuration, set a critical threshold of 100.

 

 
19-alert-configuration-threshold.png
 

For this example, you'll run a load test script to hit your Apache server with lots of requests. Your alert setup page shows when the metric passes the critical threshold you set.

 

 
20-alert-chart-with-threshold.png

 

Finally, for notification rules, you can choose how you want to be notified whenever the alert is triggered. You can choose from ServiceNow, PagerDuty, Slack, BigPanda, or webhooks.

 

 
21-notification_rule.png

 

After we save our alert, Cloud Observability begins watching the apache.requests metric. Whenever the rate of requests spike past our threshold, we’ll receive a notification.

 

 

Conclusion

Apache web server metrics can give you early indicators of traffic spikes or excess load. Traffic spikes might be the result of valid user visits. Perhaps a new marketing campaign just dropped, and lots of interested users are clicking links to visit your site. Or, a spike in incoming requests could be a sign of a coordinated DDoS attack.

 

Monitoring your Apache metrics, especially those related to connections, requests, and traffic, can ensure your team is always aware of what’s going on with your web server. By using the OpenTelemetry Collector to gather metrics in conjunction with Cloud Observability for dashboards and alerts, you set your team up for quick and efficient incident response.

 

0 Helpfuls
  • 7,260 Views