maint: prepare for 2.9.6 release (#1588)

## Which problem is this PR solving?

Prepare for 2.9.6 release

## Short description of the changes

- update CHANGELOG.md
- update release notes
- regenerate doc

---------

Co-authored-by: Mike Goldsmith <goldsmith.mike@gmail.com>
Co-authored-by: Alex Boten <223565+codeboten@users.noreply.github.com>

Author: 3 people

CHANGELOG.md

@@ -1,5 +1,43 @@
 # Refinery Changelog
 
+## 2.9.6 2025-06-20
+
+This release aims to improve user experience by adding new metrics and making default values more sensible.
+It also includes several bug fixes and maintenance updates.
+
+### Features
+
+- feat: add incoming_router_otlp_events and incoming_router_batch_events metrics (#1566) | [Yingrong Zhao](https://github.com/vinozzZ)
+
+### Fixes
+
+- fix: wait for all goroutines inside collector to finish during shutdown (#1589) | [Yingrong Zhao](https://github.com/vinozzZ)
+- fix: close redis pubsub connection on shutdown (#1578) | [Yingrong Zhao](https://github.com/vinozzZ)
+- fix: use value from GetCollectionConfigVal for HealthCheckTimeout in mock config (#1577) | [Yingrong Zhao](https://github.com/vinozzZ)
+- fix: TraceKey used in dynamic sampler should not create a new key if no key fields are found (#1575) | [Yingrong Zhao](https://github.com/vinozzZ)
+- fix: Docs about the `meta` fields were wrong; fix them. (#1576) | [Kent Quirk](https://github.com/kentquirk)
+- fix: use sensible default value for HealthCheckTimeout and MaxExpiredTraces (#1573) | [Yingrong Zhao](https://github.com/vinozzZ)
+- fix: don't reset SendBy if trace has already expired (#1570) | [Yingrong Zhao](https://github.com/vinozzZ)
+- fix: TestWhichShard don't need to include self in peer list (#1571) | [Yingrong Zhao](https://github.com/vinozzZ)
+- fix: Add valuetype to rules datatype (#1562) | [Mike Goldsmith](https://github.com/MikeGoldsmith)
+- fix: use configured identifier for host address resolution in file peer (#1548) | [Yingrong Zhao](https://github.com/vinozzZ)
+- fix: only execute Gosched in tests (#1549) | [Yingrong Zhao](https://github.com/vinozzZ)
+- fix: update refinery to report effective config on update (#1545) | [Tyler Helmuth](https://github.com/TylerHelmuth)
+- fix: panic caused by empty traces after redistribution (#1546) | [Yingrong Zhao](https://github.com/vinozzZ)
+- fix: maintain original TraceTimeout value on redistribution (#1543) | [Yingrong Zhao](https://github.com/vinozzZ)
+- fix: decision spans should not be processed through stress relief (#1544) | [Yingrong Zhao](https://github.com/vinozzZ)
+
+### Maintenance
+
+- docs: update router_otlp description (#1565) | [Jamie Danielson](https://github.com/JamieDanielson)
+- maint: Add autogenerated comment to config templates (#1564) | [Mike Goldsmith](https://github.com/MikeGoldsmith)
+- maint(deps): bump the minor-patch group across 1 directory with 17 updates (#1572) | [dependabot[bot]](https://github.com/dependabot[bot])
+- maint: Bump dependencies (#1579) | [Kent Quirk](https://github.com/KentQuirk)
+- maint: use hpsf for strings (#1581) | [Tyler Helmuth](https://github.com/TylerHelmuth)
+- maint: Add stronger warning about TraceLocalityMode (#1568) | [Jessica Parsons](https://github.com/verythorough)
+- maint: update husky to v0.36.0 (#1559) | [Mike Goldsmith](https://github.com/MikeGoldsmith)
+- maint: publish images to sippycup (#1557) | [Tyler Helmuth](https://github.com/TylerHelmuth)
+
 ## 2.9.5 2025-04-24
 
 This release brings a experimental feature for OpAMP. This feature is intended for internal use only and is not supported or recommended for production.

RELEASE_NOTES.md

@@ -2,6 +2,32 @@
 
 While [CHANGELOG.md](./CHANGELOG.md) contains detailed documentation and links to all the source code changes in a given release, this document is intended to be aimed at a more comprehensible version of the contents of the release from the point of view of users of Refinery.
 
+## Version 2.9.6
+
+This release focuses on improving observability and making Refinery easier to operate with better default values and new metrics. It also includes several important bug fixes for edge cases and configuration handling.
+
+### Enhanced Observability
+
+Two new metrics have been added to help monitor OTLP traffic patterns:
+- `incoming_router_otlp_events` - tracks the number of events received via OTLP endpoints
+- `incoming_router_batch_events` - tracks the number of events received via /batch endpoints
+
+These metrics complement existing router metrics and provide better visibility into how your telemetry data is flowing through Refinery.
+
+### Improved Default Configuration
+
+Several default values have been updated to be more practical for real-world deployments:
+- `HealthCheckTimeout` now defaults to a more reasonable value instead of requiring manual configuration
+- `MaxExpiredTraces` has a sensible default to prevent collect loop issues out of the box
+
+### Key Bug Fixes
+
+This release addresses several important edge cases:
+- **Redis connection cleanup**: Fixed a bug where Refinery would not properly clean up Redis connections, potentially leading to Redis connection errors during shutdown.
+- **Dynamic sampler key generation**: Use an empty string for the sampler key when no fields are present, clarifying the behavior when no key fields are found in a trace.
+- **Peer management**: Fixed a bug where Refinery would not properly handle peer management when using file-based peer management.
+- **Trace timeout during redistribution**: Propagate the trace timeout to the new owner during redistribution, ensuring that traces are handled correctly even when ownership changes.
+
 ## Version 2.9.5
 
 This release includes improvements to trace cache documentation, internal metric flushing on shutdown, and enhanced metric visibility such as sampler key cardinality. It also introduces environment variable substitution in config strings and experimental support for OpAMP. The OpAMP feature is intended for *internal use only* and is *not supported or recommended* for production.

refinery_rules.md

@@ -98,6 +98,9 @@ Including an error field, or something like `HTTP status code`, is an excellent
 Using fields with very high cardinality, like `k8s.pod.id`, is a bad choice.
 If the combination of fields essentially makes each trace unique, then the Dynamic Sampler will sample everything.
 If the combination of fields is not unique enough, then you will not be guaranteed samples of the most interesting traces.
+If a trace does not contain any of the fields specified in the FieldList, it will still be evaluated by the sampler.
+However, since it lacks all key fields, it will be grouped under a single empty (blank) key.
+This means that all such traces will share the same sample rate, determined by that one shared key.
 As an example, consider as a good set of fields: the combination of `HTTP endpoint` (high-frequency and boring), `HTTP method`, and `status code` (normally boring but can become interesting when indicating an error) since it will allowing proper sampling of all endpoints under normal traffic and call out when there is failing traffic to any endpoint.
 As of Refinery 2.8.0, the `root.` prefix can be used to limit the field value to that of the root span.
 For example, `root.http.response.status_code` will only consider the `http.response.status_code` field from the root span rather than a combination of all the spans in the trace.
@@ -203,6 +206,9 @@ Including an error field, or something like `HTTP status code`, is an excellent
 Using fields with very high cardinality, like `k8s.pod.id`, is a bad choice.
 If the combination of fields essentially makes each trace unique, then the Dynamic Sampler will sample everything.
 If the combination of fields is not unique enough, then you will not be guaranteed samples of the most interesting traces.
+If a trace does not contain any of the fields specified in the FieldList, it will still be evaluated by the sampler.
+However, since it lacks all key fields, it will be grouped under a single empty (blank) key.
+This means that all such traces will share the same sample rate, determined by that one shared key.
 As an example, consider as a good set of fields: the combination of `HTTP endpoint` (high-frequency and boring), `HTTP method`, and `status code` (normally boring but can become interesting when indicating an error) since it will allowing proper sampling of all endpoints under normal traffic and call out when there is failing traffic to any endpoint.
 As of Refinery 2.8.0, the `root.` prefix can be used to limit the field value to that of the root span.
 For example, `root.http.response.status_code` will only consider the `http.response.status_code` field from the root span rather than a combination of all the spans in the trace.
@@ -320,6 +326,9 @@ Including an error field, or something like `HTTP status code`, is an excellent
 Using fields with very high cardinality, like `k8s.pod.id`, is a bad choice.
 If the combination of fields essentially makes each trace unique, then the Dynamic Sampler will sample everything.
 If the combination of fields is not unique enough, then you will not be guaranteed samples of the most interesting traces.
+If a trace does not contain any of the fields specified in the FieldList, it will still be evaluated by the sampler.
+However, since it lacks all key fields, it will be grouped under a single empty (blank) key.
+This means that all such traces will share the same sample rate, determined by that one shared key.
 As an example, consider as a good set of fields: the combination of `HTTP endpoint` (high-frequency and boring), `HTTP method`, and `status code` (normally boring but can become interesting when indicating an error) since it will allowing proper sampling of all endpoints under normal traffic and call out when there is failing traffic to any endpoint.
 As of Refinery 2.8.0, the `root.` prefix can be used to limit the field value to that of the root span.
 For example, `root.http.response.status_code` will only consider the `http.response.status_code` field from the root span rather than a combination of all the spans in the trace.
@@ -408,6 +417,9 @@ Including an error field, or something like `HTTP status code`, is an excellent
 Using fields with very high cardinality, like `k8s.pod.id`, is a bad choice.
 If the combination of fields essentially makes each trace unique, then the Dynamic Sampler will sample everything.
 If the combination of fields is not unique enough, then you will not be guaranteed samples of the most interesting traces.
+If a trace does not contain any of the fields specified in the FieldList, it will still be evaluated by the sampler.
+However, since it lacks all key fields, it will be grouped under a single empty (blank) key.
+This means that all such traces will share the same sample rate, determined by that one shared key.
 As an example, consider as a good set of fields: the combination of `HTTP endpoint` (high-frequency and boring), `HTTP method`, and `status code` (normally boring but can become interesting when indicating an error) since it will allowing proper sampling of all endpoints under normal traffic and call out when there is failing traffic to any endpoint.
 As of Refinery 2.8.0, the `root.` prefix can be used to limit the field value to that of the root span.
 For example, `root.http.response.status_code` will only consider the `http.response.status_code` field from the root span rather than a combination of all the spans in the trace.
@@ -624,6 +636,9 @@ Including an error field, or something like `HTTP status code`, is an excellent
 Using fields with very high cardinality, like `k8s.pod.id`, is a bad choice.
 If the combination of fields essentially makes each trace unique, then the Dynamic Sampler will sample everything.
 If the combination of fields is not unique enough, then you will not be guaranteed samples of the most interesting traces.
+If a trace does not contain any of the fields specified in the FieldList, it will still be evaluated by the sampler.
+However, since it lacks all key fields, it will be grouped under a single empty (blank) key.
+This means that all such traces will share the same sample rate, determined by that one shared key.
 As an example, consider as a good set of fields: the combination of `HTTP endpoint` (high-frequency and boring), `HTTP method`, and `status code` (normally boring but can become interesting when indicating an error) since it will allowing proper sampling of all endpoints under normal traffic and call out when there is failing traffic to any endpoint.
 As of Refinery 2.8.0, the `root.` prefix can be used to limit the field value to that of the root span.
 For example, `root.http.response.status_code` will only consider the `http.response.status_code` field from the root span rather than a combination of all the spans in the trace.

rules.md

@@ -3,7 +3,7 @@
 # Honeycomb Refinery Rules Documentation
 
 This is the documentation for the rules configuration for Honeycomb's Refinery.
-It was automatically generated on 2025-05-14 at 16:44:26 UTC.
+It was automatically generated on 2025-06-17 at 14:29:18 UTC.
 
 ## The Rules file
 
@@ -119,6 +119,9 @@ Including an error field, or something like `HTTP status code`, is an excellent
 Using fields with very high cardinality, like `k8s.pod.id`, is a bad choice.
 If the combination of fields essentially makes each trace unique, then the Dynamic Sampler will sample everything.
 If the combination of fields is not unique enough, then you will not be guaranteed samples of the most interesting traces.
+If a trace does not contain any of the fields specified in the FieldList, it will still be evaluated by the sampler.
+However, since it lacks all key fields, it will be grouped under a single empty (blank) key.
+This means that all such traces will share the same sample rate, determined by that one shared key.
 As an example, consider as a good set of fields: the combination of `HTTP endpoint` (high-frequency and boring), `HTTP method`, and `status code` (normally boring but can become interesting when indicating an error) since it will allowing proper sampling of all endpoints under normal traffic and call out when there is failing traffic to any endpoint.
 As of Refinery 2.8.0, the `root.` prefix can be used to limit the field value to that of the root span.
 For example, `root.http.response.status_code` will only consider the `http.response.status_code` field from the root span rather than a combination of all the spans in the trace.
@@ -227,6 +230,9 @@ Including an error field, or something like `HTTP status code`, is an excellent
 Using fields with very high cardinality, like `k8s.pod.id`, is a bad choice.
 If the combination of fields essentially makes each trace unique, then the Dynamic Sampler will sample everything.
 If the combination of fields is not unique enough, then you will not be guaranteed samples of the most interesting traces.
+If a trace does not contain any of the fields specified in the FieldList, it will still be evaluated by the sampler.
+However, since it lacks all key fields, it will be grouped under a single empty (blank) key.
+This means that all such traces will share the same sample rate, determined by that one shared key.
 As an example, consider as a good set of fields: the combination of `HTTP endpoint` (high-frequency and boring), `HTTP method`, and `status code` (normally boring but can become interesting when indicating an error) since it will allowing proper sampling of all endpoints under normal traffic and call out when there is failing traffic to any endpoint.
 As of Refinery 2.8.0, the `root.` prefix can be used to limit the field value to that of the root span.
 For example, `root.http.response.status_code` will only consider the `http.response.status_code` field from the root span rather than a combination of all the spans in the trace.
@@ -347,6 +353,9 @@ Including an error field, or something like `HTTP status code`, is an excellent
 Using fields with very high cardinality, like `k8s.pod.id`, is a bad choice.
 If the combination of fields essentially makes each trace unique, then the Dynamic Sampler will sample everything.
 If the combination of fields is not unique enough, then you will not be guaranteed samples of the most interesting traces.
+If a trace does not contain any of the fields specified in the FieldList, it will still be evaluated by the sampler.
+However, since it lacks all key fields, it will be grouped under a single empty (blank) key.
+This means that all such traces will share the same sample rate, determined by that one shared key.
 As an example, consider as a good set of fields: the combination of `HTTP endpoint` (high-frequency and boring), `HTTP method`, and `status code` (normally boring but can become interesting when indicating an error) since it will allowing proper sampling of all endpoints under normal traffic and call out when there is failing traffic to any endpoint.
 As of Refinery 2.8.0, the `root.` prefix can be used to limit the field value to that of the root span.
 For example, `root.http.response.status_code` will only consider the `http.response.status_code` field from the root span rather than a combination of all the spans in the trace.
@@ -438,6 +447,9 @@ Including an error field, or something like `HTTP status code`, is an excellent
 Using fields with very high cardinality, like `k8s.pod.id`, is a bad choice.
 If the combination of fields essentially makes each trace unique, then the Dynamic Sampler will sample everything.
 If the combination of fields is not unique enough, then you will not be guaranteed samples of the most interesting traces.
+If a trace does not contain any of the fields specified in the FieldList, it will still be evaluated by the sampler.
+However, since it lacks all key fields, it will be grouped under a single empty (blank) key.
+This means that all such traces will share the same sample rate, determined by that one shared key.
 As an example, consider as a good set of fields: the combination of `HTTP endpoint` (high-frequency and boring), `HTTP method`, and `status code` (normally boring but can become interesting when indicating an error) since it will allowing proper sampling of all endpoints under normal traffic and call out when there is failing traffic to any endpoint.
 As of Refinery 2.8.0, the `root.` prefix can be used to limit the field value to that of the root span.
 For example, `root.http.response.status_code` will only consider the `http.response.status_code` field from the root span rather than a combination of all the spans in the trace.
@@ -668,6 +680,9 @@ Including an error field, or something like `HTTP status code`, is an excellent
 Using fields with very high cardinality, like `k8s.pod.id`, is a bad choice.
 If the combination of fields essentially makes each trace unique, then the Dynamic Sampler will sample everything.
 If the combination of fields is not unique enough, then you will not be guaranteed samples of the most interesting traces.
+If a trace does not contain any of the fields specified in the FieldList, it will still be evaluated by the sampler.
+However, since it lacks all key fields, it will be grouped under a single empty (blank) key.
+This means that all such traces will share the same sample rate, determined by that one shared key.
 As an example, consider as a good set of fields: the combination of `HTTP endpoint` (high-frequency and boring), `HTTP method`, and `status code` (normally boring but can become interesting when indicating an error) since it will allowing proper sampling of all endpoints under normal traffic and call out when there is failing traffic to any endpoint.
 As of Refinery 2.8.0, the `root.` prefix can be used to limit the field value to that of the root span.
 For example, `root.http.response.status_code` will only consider the `http.response.status_code` field from the root span rather than a combination of all the spans in the trace.