Better Logs in Production

Why Logging Matters

Logging is a fundamental aspect of observability. In production environments, effective logging is critical to diagnose problems, monitor system health, and gain insight into application behavior. However, without proper structure and practices, logs can become noisy, incomplete, or even useless. When combined with OpenTelemetry, structured logging is a powerful ally for root cause analysis.

Common Pitfalls in Production Logging

• Too much noise: Excessive logs at debug level that overwhelm storage and make analysis difficult.
• Lack of structure: Free-text logs that cannot be parsed or indexed properly by log aggregation tools.
• Missing context: Logs that don’t include request IDs, user identifiers, or environment metadata.
• Inconsistent formats: Varying styles across teams or services make correlation difficult.

Principles of Better Logging

1. Use structured logs: Log in JSON or a format easily consumed by logging systems like ELK, Loki, or Datadog.
2. Include context: Always include trace IDs, span IDs, request URLs, user sessions, and environment tags.
3. Log at the right level: Use error/warning for failures, info for business events, and debug only in dev environments.
4. Avoid sensitive data: Never log passwords, secrets, or personal data (PII).
5. Be consistent: Define log schemas and naming conventions across all services.

Code Examples

Node.js (Winston + OpenTelemetry)

const { createLogger, transports, format } = require('winston');
const logger = createLogger({
  level: 'info',
  format: format.json(),
  defaultMeta: { service: 'payments-service' },
  transports: [
    new transports.Console()
  ]
});

// Example enriched log
logger.info('Order created', {
  orderId: '12345',
  traceId: 'abc-xyz',
  userId: 'user42'
});

Python (structlog)

import structlog

log = structlog.get_logger()

log.info("user_login", user_id="u123", trace_id="abc-xyz")
log.error("payment_failed", error="timeout", order_id="987")

For more examples, see the OpenTelemetry Logging Documentation.

Log Enrichment Techniques

Enriching logs means appending contextual metadata automatically. You can use log middleware or wrappers to add:

• Trace and span IDs from your tracing system (e.g., OpenTelemetry).
• Application version and deployment information.
• Container or node identifiers in distributed environments.

Centralized Logging and Querying

Centralized log aggregation is key to managing production logs. Tools like Elasticsearch, Loki, Datadog Logs, or CloudWatch Logs help you:

• Search and filter logs in real time.
• Create alerts based on log patterns or anomalies.
• Correlate logs with metrics and traces for root cause analysis.

Hands-on Labs and Tools

• KillerCoda Labs: Logging with OpenShift
• Grafana Loki Tutorial
• Azure Log Analytics Hands-on

Conclusion

Production logs are your first line of defense when things go wrong. Investing in structured, enriched, and centralized logging practices dramatically improves your ability to detect, debug, and prevent incidents.

Logs are not just artifacts—they are signals. Make them count.