==> Building on glalie ==> Checking for remote environment... ==> Syncing package to remote host... sending incremental file list ./ .SRCINFO 700 37% 0.00kB/s 0:00:00 1,852 100% 1.10MB/s 0:00:00 (xfr#1, to-chk=11/13) .nvchecker.toml 79 100% 77.15kB/s 0:00:00 79 100% 77.15kB/s 0:00:00 (xfr#2, to-chk=10/13) PKGBUILD 700 18% 683.59kB/s 0:00:00 3,842 100% 3.66MB/s 0:00:00 (xfr#3, to-chk=9/13) go-1.19.patch 700 16% 683.59kB/s 0:00:00 4,225 100% 4.03MB/s 0:00:00 (xfr#4, to-chk=8/13) loki-3.5.1-1.log 628 100% 613.28kB/s 0:00:00 628 100% 613.28kB/s 0:00:00 (xfr#5, to-chk=7/13) loki.service 700 100% 683.59kB/s 0:00:00 700 100% 683.59kB/s 0:00:00 (xfr#6, to-chk=6/13) loki.sysusers 36 100% 35.16kB/s 0:00:00 36 100% 35.16kB/s 0:00:00 (xfr#7, to-chk=5/13) loki.tmpfiles 33 100% 32.23kB/s 0:00:00 33 100% 32.23kB/s 0:00:00 (xfr#8, to-chk=4/13) no-moving-gc-bump.patch 700 9% 683.59kB/s 0:00:00 7,094 100% 6.77MB/s 0:00:00 (xfr#9, to-chk=3/13) promtail.service 700 98% 683.59kB/s 0:00:00 711 100% 694.34kB/s 0:00:00 (xfr#10, to-chk=2/13) promtail.sysusers 75 100% 73.24kB/s 0:00:00 75 100% 73.24kB/s 0:00:00 (xfr#11, to-chk=1/13) promtail.tmpfiles 45 100% 43.95kB/s 0:00:00 45 100% 43.95kB/s 0:00:00 (xfr#12, to-chk=0/13) sent 1,297 bytes received 463 bytes 1,173.33 bytes/sec total size is 18,885 speedup is 10.73 ==> Patching arch to riscv64... ==> Running pkgctl build --arch riscv64 --repo extra on remote host... ==> WARNING: unsupported architecture: riscv64 ==> Building loki  -> repo: extra  -> arch: riscv64  -> worker: felix-2 ==> Building loki for [extra] (riscv64) ]2;🔵 Container arch-nspawn-624188 on glalie.felixc.at\[?25l:: Synchronizing package databases... core downloading... extra downloading... :: Starting full system upgrade... there is nothing to do [?25h==> Building in chroot for [extra] (riscv64)... ==> Locking clean chroot [/var/lib/archbuild/extra-riscv64/root]...done ==> Synchronizing chroot copy [/var/lib/archbuild/extra-riscv64/root] -> [felix-2]...done ==> Making package: loki 3.5.1-1 (Fri Jun 13 11:43:57 2025) ==> Retrieving sources...  -> Found loki-3.5.1.tar.gz  -> Found promtail.sysusers  -> Found promtail.service  -> Found promtail.tmpfiles  -> Found loki.sysusers  -> Found loki.service  -> Found loki.tmpfiles ==> Validating source files with sha512sums... loki-3.5.1.tar.gz ... Passed promtail.sysusers ... Passed promtail.service ... Passed promtail.tmpfiles ... Passed loki.sysusers ... Passed loki.service ... Passed loki.tmpfiles ... Passed ]2;🔵 Container arch-nspawn-630319 on glalie.felixc.at\==> Making package: loki 3.5.1-1 (Fri Jun 13 11:45:03 2025) ==> Checking runtime dependencies... ==> Checking buildtime dependencies... ==> Installing missing dependencies... [?25lresolving dependencies... looking for conflicting packages... Package (1) New Version Net Change extra/go 2:1.24.4-1 233.74 MiB Total Installed Size: 233.74 MiB :: Proceed with installation? [Y/n] checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing go... :: Running post-transaction hooks... (1/1) Arming ConditionNeedsUpdate... [?25h==> Retrieving sources...  -> Found loki-3.5.1.tar.gz  -> Found promtail.sysusers  -> Found promtail.service  -> Found promtail.tmpfiles  -> Found loki.sysusers  -> Found loki.service  -> Found loki.tmpfiles ==> WARNING: Skipping all source file integrity checks. ==> Extracting sources...  -> Extracting loki-3.5.1.tar.gz with bsdtar ==> Starting build()... ==> Starting check()... ok github.com/grafana/loki/v3/clients/cmd/docker-driver 4.332s ok github.com/grafana/loki/v3/clients/cmd/fluent-bit 5.456s ? github.com/grafana/loki/v3/clients/cmd/promtail [no test files] ok github.com/grafana/loki/v3/clients/pkg/logentry/logql 1.484s ok github.com/grafana/loki/v3/clients/pkg/logentry/metric 2.657s ok github.com/grafana/loki/v3/clients/pkg/logentry/stages 12.385s ok github.com/grafana/loki/v3/clients/pkg/promtail 29.746s ? github.com/grafana/loki/v3/clients/pkg/promtail/api [no test files] ok github.com/grafana/loki/v3/clients/pkg/promtail/client 10.644s ? github.com/grafana/loki/v3/clients/pkg/promtail/client/fake [no test files] ok github.com/grafana/loki/v3/clients/pkg/promtail/config 5.795s ok github.com/grafana/loki/v3/clients/pkg/promtail/discovery/consulagent 3.080s ? github.com/grafana/loki/v3/clients/pkg/promtail/limit [no test files] ok github.com/grafana/loki/v3/clients/pkg/promtail/positions 1.530s ok github.com/grafana/loki/v3/clients/pkg/promtail/scrapeconfig 4.342s ? github.com/grafana/loki/v3/clients/pkg/promtail/server [no test files] ? github.com/grafana/loki/v3/clients/pkg/promtail/server/ui [no test files] ? github.com/grafana/loki/v3/clients/pkg/promtail/targets [no test files] ok github.com/grafana/loki/v3/clients/pkg/promtail/targets/azureeventhubs 5.632s ok github.com/grafana/loki/v3/clients/pkg/promtail/targets/cloudflare 5.491s ok github.com/grafana/loki/v3/clients/pkg/promtail/targets/docker 10.456s ok github.com/grafana/loki/v3/clients/pkg/promtail/targets/file 48.939s ok github.com/grafana/loki/v3/clients/pkg/promtail/targets/gcplog 7.609s ok github.com/grafana/loki/v3/clients/pkg/promtail/targets/gelf 6.372s ok github.com/grafana/loki/v3/clients/pkg/promtail/targets/heroku 25.373s ? github.com/grafana/loki/v3/clients/pkg/promtail/targets/journal [no test files] ok github.com/grafana/loki/v3/clients/pkg/promtail/targets/kafka 7.137s ok github.com/grafana/loki/v3/clients/pkg/promtail/targets/lokipush 11.103s ? github.com/grafana/loki/v3/clients/pkg/promtail/targets/serverutils [no test files] ok github.com/grafana/loki/v3/clients/pkg/promtail/targets/stdin 4.995s ok github.com/grafana/loki/v3/clients/pkg/promtail/targets/syslog 8.610s ok github.com/grafana/loki/v3/clients/pkg/promtail/targets/syslog/syslogparser 0.439s ? github.com/grafana/loki/v3/clients/pkg/promtail/targets/target [no test files] ? github.com/grafana/loki/v3/clients/pkg/promtail/targets/testutils [no test files] ? github.com/grafana/loki/v3/clients/pkg/promtail/targets/windows [no test files] ? github.com/grafana/loki/v3/clients/pkg/promtail/targets/windows/win_eventlog [no test files] ok github.com/grafana/loki/v3/clients/pkg/promtail/utils 6.479s ok github.com/grafana/loki/v3/clients/pkg/promtail/wal 23.209s ? github.com/grafana/loki/v3/cmd/logcli [no test files] ? github.com/grafana/loki/v3/cmd/logql-analyzer [no test files] ? github.com/grafana/loki/v3/cmd/loki [no test files] ? github.com/grafana/loki/v3/cmd/loki-canary [no test files] ? github.com/grafana/loki/v3/cmd/lokitool [no test files] ok github.com/grafana/loki/v3/cmd/migrate 6.103s ? github.com/grafana/loki/v3/cmd/querytee [no test files] ? github.com/grafana/loki/v3/integration/client [no test files] ? github.com/grafana/loki/v3/integration/cluster [no test files] ? github.com/grafana/loki/v3/integration/util [no test files] ok github.com/grafana/loki/v3/pkg/analytics 27.229s ok github.com/grafana/loki/v3/pkg/blockbuilder/builder 5.405s ok github.com/grafana/loki/v3/pkg/blockbuilder/scheduler 2.274s ? github.com/grafana/loki/v3/pkg/blockbuilder/types [no test files] ? github.com/grafana/loki/v3/pkg/blockbuilder/types/proto [no test files] ? github.com/grafana/loki/v3/pkg/bloombuild [no test files] --- FAIL: Test_BuilderLoop (35.30s) builder_test.go:145: Error Trace: /build/loki/src/loki-3.5.1/pkg/bloombuild/builder/builder_test.go:145 Error: Condition never satisfied Test: Test_BuilderLoop FAIL FAIL github.com/grafana/loki/v3/pkg/bloombuild/builder 95.782s ok github.com/grafana/loki/v3/pkg/bloombuild/common 5.700s ok github.com/grafana/loki/v3/pkg/bloombuild/planner 42.676s ? github.com/grafana/loki/v3/pkg/bloombuild/planner/plannertest [no test files] ok github.com/grafana/loki/v3/pkg/bloombuild/planner/queue 7.317s ok github.com/grafana/loki/v3/pkg/bloombuild/planner/strategies 7.420s ok github.com/grafana/loki/v3/pkg/bloombuild/protos 5.038s ok github.com/grafana/loki/v3/pkg/bloomgateway 20.244s ok github.com/grafana/loki/v3/pkg/canary/comparator 4.265s ? github.com/grafana/loki/v3/pkg/canary/reader [no test files] ok github.com/grafana/loki/v3/pkg/canary/writer 3.744s panic: test timed out after 10m0s running tests: TestGenerateDataSize (6m13s) TestGenerateDataSize/flate (19s) goroutine 12089 [running]: testing.(*M).startAlarm.func1() /usr/lib/go/src/testing/testing.go:2484 +0x304 created by time.goFunc /usr/lib/go/src/time/sleep.go:215 +0x58 goroutine 1 [chan receive, 6 minutes]: testing.(*T).Run(0x3f48602380, {0xd3a41a?, 0x27cef0cb485abb88?}, 0xdcddb8) /usr/lib/go/src/testing/testing.go:1859 +0x3b4 testing.runTests.func1(0x3f48602380) /usr/lib/go/src/testing/testing.go:2279 +0x4c testing.tRunner(0x3f48602380, 0x3f485abca8) /usr/lib/go/src/testing/testing.go:1792 +0xf8 testing.runTests(0x3f4800c168, {0x17ade40, 0x24, 0x24}, {0x1cee4?, 0x3f48026210?, 0x17c0b40?}) /usr/lib/go/src/testing/testing.go:2277 +0x440 testing.(*M).Run(0x3f48310820) /usr/lib/go/src/testing/testing.go:2142 +0x5bc main.main() _testmain.go:141 +0x94 goroutine 281 [chan receive, 9 minutes]: testing.(*T).Parallel(0x3f4a580380) /usr/lib/go/src/testing/testing.go:1577 +0x1c0 github.com/grafana/loki/v3/pkg/chunkenc.TestReadFormatV1(0x3f4a580380) /build/loki/src/loki-3.5.1/pkg/chunkenc/memchunk_test.go:363 +0x30 testing.tRunner(0x3f4a580380, 0xdcde10) /usr/lib/go/src/testing/testing.go:1792 +0xf8 created by testing.(*T).Run in goroutine 1 /usr/lib/go/src/testing/testing.go:1851 +0x3a0 goroutine 4486 [chan receive, 7 minutes]: testing.(*T).Parallel(0x3e6e56f180) /usr/lib/go/src/testing/testing.go:1577 +0x1c0 github.com/grafana/loki/v3/pkg/chunkenc.TestGZIPChunkTargetSize(0x3e6e56f180) /build/loki/src/loki-3.5.1/pkg/chunkenc/memchunk_test.go:624 +0x30 testing.tRunner(0x3e6e56f180, 0xdcddb0) /usr/lib/go/src/testing/testing.go:1792 +0xf8 created by testing.(*T).Run in goroutine 1 /usr/lib/go/src/testing/testing.go:1851 +0x3a0 goroutine 12169 [runnable]: github.com/Workiva/go-datastructures/rangetree.newNode(...) /build/loki/src/loki-3.5.1/vendor/github.com/Workiva/go-datastructures/rangetree/node.go:28 github.com/Workiva/go-datastructures/rangetree.(*orderedTree).add(0x3f4847fe40, {0xef7a00, 0x3c6a25ad60}) /build/loki/src/loki-3.5.1/vendor/github.com/Workiva/go-datastructures/rangetree/orderedtree.go:51 +0xa8 github.com/Workiva/go-datastructures/rangetree.(*orderedTree).Add(0x3f4847fe40, {0x3c6a26c680, 0x1, 0xffffffff3c607038?}) /build/loki/src/loki-3.5.1/vendor/github.com/Workiva/go-datastructures/rangetree/orderedtree.go:81 +0xb4 github.com/grafana/loki/v3/pkg/chunkenc.(*unorderedHeadBlock).Append(0x3f4847ebc0, 0xd50c7, {0xdbba7c, 0x13a}, {0x0?, 0xa2da88?, 0xcce01?}) /build/loki/src/loki-3.5.1/pkg/chunkenc/unordered.go:132 +0x108 github.com/grafana/loki/v3/pkg/chunkenc.(*MemChunk).Append(0x3c68e7af60, 0x3c6a25fec0?) /build/loki/src/loki-3.5.1/pkg/chunkenc/memchunk.go:878 +0x178 github.com/grafana/loki/v3/pkg/chunkenc.generateData(0x8, 0x1490fd8?, 0x40000, 0x177000) /build/loki/src/loki-3.5.1/pkg/chunkenc/util_test.go:37 +0x23c github.com/grafana/loki/v3/pkg/chunkenc.TestGenerateDataSize.func1(0x3ed1cc0e00) /build/loki/src/loki-3.5.1/pkg/chunkenc/memchunk_test.go:1072 +0x48 testing.tRunner(0x3ed1cc0e00, 0x3f48294770) /usr/lib/go/src/testing/testing.go:1792 +0xf8 created by testing.(*T).Run in goroutine 12394 /usr/lib/go/src/testing/testing.go:1851 +0x3a0 goroutine 12394 [chan receive]: testing.(*T).Run(0x3e6a87a540, {0xd27dbc?, 0x8c2bc?}, 0x3f48294770) /usr/lib/go/src/testing/testing.go:1859 +0x3b4 github.com/grafana/loki/v3/pkg/chunkenc.TestGenerateDataSize(0x3e6a87a540) /build/loki/src/loki-3.5.1/pkg/chunkenc/memchunk_test.go:1071 +0x78 testing.tRunner(0x3e6a87a540, 0xdcddb8) /usr/lib/go/src/testing/testing.go:1792 +0xf8 created by testing.(*T).Run in goroutine 1 /usr/lib/go/src/testing/testing.go:1851 +0x3a0 goroutine 4487 [chan receive, 7 minutes]: testing.(*T).Parallel(0x3e6e56f340) /usr/lib/go/src/testing/testing.go:1577 +0x1c0 github.com/grafana/loki/v3/pkg/chunkenc.TestMemChunk_AppendOutOfOrder(0x3e6e56f340) /build/loki/src/loki-3.5.1/pkg/chunkenc/memchunk_test.go:677 +0x30 testing.tRunner(0x3e6e56f340, 0xdcddd8) /usr/lib/go/src/testing/testing.go:1792 +0xf8 created by testing.(*T).Run in goroutine 1 /usr/lib/go/src/testing/testing.go:1851 +0x3a0 FAIL github.com/grafana/loki/v3/pkg/chunkenc 603.107s ok github.com/grafana/loki/v3/pkg/compactor 37.979s ? github.com/grafana/loki/v3/pkg/compactor/client [no test files] ? github.com/grafana/loki/v3/pkg/compactor/client/grpc [no test files] ok github.com/grafana/loki/v3/pkg/compactor/deletion 12.014s ok github.com/grafana/loki/v3/pkg/compactor/deletionmode 0.390s ok github.com/grafana/loki/v3/pkg/compactor/generationnumber 2.109s level=info ts=2025-06-13T04:35:20.862982202Z caller=marker.go:177 msg="mark processor started" workers=10 delay=0s level=debug ts=2025-06-13T04:35:20.865138424Z caller=retention.go:139 msg="starting to process table" level=info ts=2025-06-13T04:35:20.877996099Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionnothing_is_expiring153836336/002/retention/markers/1749789320865648265 level=debug ts=2025-06-13T04:35:20.883916845Z caller=marker.go:207 msg="processing mark file" path=/tmp/Test_Retentionnothing_is_expiring153836336/002/retention/markers/1749789320865648265 level=debug ts=2025-06-13T04:35:20.89649256Z caller=retention.go:137 msg="finished to process table" duration=31.409055ms level=debug ts=2025-06-13T04:35:20.89694928Z caller=retention.go:139 msg="starting to process table" level=info ts=2025-06-13T04:35:20.901558385Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionnothing_is_expiring153836336/002/retention/markers/1749789320897282280 level=debug ts=2025-06-13T04:35:20.913725379Z caller=retention.go:137 msg="finished to process table" duration=16.756479ms level=info ts=2025-06-13T04:35:20.98637112Z caller=marker.go:177 msg="mark processor started" workers=10 delay=0s level=debug ts=2025-06-13T04:35:20.988553222Z caller=retention.go:139 msg="starting to process table" level=info ts=2025-06-13T04:35:20.99548693Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionone_global_expiration2185484036/002/retention/markers/1749789320989200263 level=debug ts=2025-06-13T04:35:20.994806689Z caller=marker.go:207 msg="processing mark file" path=/tmp/Test_Retentionone_global_expiration2185484036/002/retention/markers/1749789320989200263 level=debug ts=2025-06-13T04:35:21.002234778Z caller=retention.go:137 msg="finished to process table" duration=13.697136ms level=debug ts=2025-06-13T04:35:21.002557438Z caller=retention.go:139 msg="starting to process table" level=info ts=2025-06-13T04:35:21.00468162Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionone_global_expiration2185484036/002/retention/markers/1749789321002765518 level=debug ts=2025-06-13T04:35:21.008984205Z caller=retention.go:137 msg="finished to process table" duration=6.422227ms level=info ts=2025-06-13T04:35:21.992379845Z caller=marker.go:202 msg="no marks file found" level=info ts=2025-06-13T04:35:22.064776906Z caller=marker.go:177 msg="mark processor started" workers=10 delay=0s level=debug ts=2025-06-13T04:35:22.06877191Z caller=retention.go:139 msg="starting to process table" level=info ts=2025-06-13T04:35:22.070436932Z caller=marker.go:202 msg="no marks file found" level=info ts=2025-06-13T04:35:22.074046816Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionone_global_expiration_and_stream3065787090/002/retention/markers/1749789322069266291 level=debug ts=2025-06-13T04:35:22.081201364Z caller=retention.go:137 msg="finished to process table" duration=12.436694ms level=debug ts=2025-06-13T04:35:22.081666065Z caller=retention.go:139 msg="starting to process table" level=info ts=2025-06-13T04:35:22.085541269Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionone_global_expiration_and_stream3065787090/002/retention/markers/1749789322082143105 level=debug ts=2025-06-13T04:35:22.091225635Z caller=retention.go:137 msg="finished to process table" duration=9.55481ms level=debug ts=2025-06-13T04:35:22.091634296Z caller=retention.go:139 msg="starting to process table" level=info ts=2025-06-13T04:35:22.09552336Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionone_global_expiration_and_stream3065787090/002/retention/markers/1749789322092085656 level=debug ts=2025-06-13T04:35:22.106821993Z caller=retention.go:137 msg="finished to process table" duration=15.145797ms level=debug ts=2025-06-13T04:35:22.107175913Z caller=retention.go:139 msg="starting to process table" level=info ts=2025-06-13T04:35:22.111605318Z caller=marker.go:77 msg="mark file created" file=/tmp/Test_Retentionone_global_expiration_and_stream3065787090/002/retention/markers/1749789322107579734 level=debug ts=2025-06-13T04:35:22.116982704Z caller=retention.go:137 msg="finished to process table" duration=9.780351ms level=debug ts=2025-06-13T04:35:23.076440537Z caller=marker.go:207 msg="processing mark file" path=/tmp/Test_Retentionone_global_expiration_and_stream3065787090/002/retention/markers/1749789322069266291 level=debug ts=2025-06-13T04:35:23.121771828Z caller=marker.go:207 msg="processing mark file" path=/tmp/Test_Retentionone_global_expiration_and_stream3065787090/002/retention/markers/1749789322092085656 level=info ts=2025-06-13T04:35:24.072963311Z caller=marker.go:202 msg="no marks file found" level=error ts=2025-06-13T04:35:24.145017292Z caller=retention.go:385 msg="error deleting chunk" chunkID=1/3fff2c2d7595e046:1916fa8c4bd:1916fdfb33d:bd55fc5 err="chunk deletion for chunkID:1/3fff2c2d7595e046:1916fa8c4bd:1916fdfb33d:bd55fc5 is failed by mockChunkClient" level=warn ts=2025-06-13T04:35:33.042774761Z caller=retention.go:288 msg="Timed out while running delete" --- FAIL: TestDeleteTimeout (2.35s) retention_test.go:1075: Error Trace: /build/loki/src/loki-3.5.1/pkg/compactor/retention/retention_test.go:1075 Error: Not equal: expected: 1 actual : 2 Test: TestDeleteTimeout level=info ts=2025-06-13T04:35:33.583524865Z caller=retention.go:530 msg="found markers in retention dir /tmp/TestMigrateMarkersmigrate_markers_dir403602723/001/markers, moving them to period specific dir: /tmp/TestMigrateMarkersmigrate_markers_dir403602723/001/store-1_2023-10-19/markers" FAIL FAIL github.com/grafana/loki/v3/pkg/compactor/retention 36.404s ok github.com/grafana/loki/v3/pkg/compression 2.748s ok github.com/grafana/loki/v3/pkg/dataobj 12.959s ? github.com/grafana/loki/v3/pkg/dataobj/config [no test files] --- FAIL: TestIdleFlushWithActiveProcessing (0.96s) partition_processor_test.go:270: Error Trace: /build/loki/src/loki-3.5.1/pkg/dataobj/consumer/partition_processor_test.go:270 Error: Should be true Test: TestIdleFlushWithActiveProcessing Messages: expected idle flush to occur while processor is running FAIL FAIL github.com/grafana/loki/v3/pkg/dataobj/consumer 11.051s ? github.com/grafana/loki/v3/pkg/dataobj/explorer [no test files] ok github.com/grafana/loki/v3/pkg/dataobj/internal/dataset 1.784s ok github.com/grafana/loki/v3/pkg/dataobj/internal/encoding 1.430s ? github.com/grafana/loki/v3/pkg/dataobj/internal/metadata/datasetmd [no test files] ? github.com/grafana/loki/v3/pkg/dataobj/internal/metadata/filemd [no test files] ? github.com/grafana/loki/v3/pkg/dataobj/internal/metadata/logsmd [no test files] ? github.com/grafana/loki/v3/pkg/dataobj/internal/metadata/streamsmd [no test files] ? github.com/grafana/loki/v3/pkg/dataobj/internal/result [no test files] ok github.com/grafana/loki/v3/pkg/dataobj/internal/sections/logs 1.449s ok github.com/grafana/loki/v3/pkg/dataobj/internal/sections/streams 1.244s ? github.com/grafana/loki/v3/pkg/dataobj/internal/streamio [no test files] ok github.com/grafana/loki/v3/pkg/dataobj/internal/util/bitmask 0.950s ok github.com/grafana/loki/v3/pkg/dataobj/internal/util/bufpool 0.811s ok github.com/grafana/loki/v3/pkg/dataobj/internal/util/sliceclear 0.600s ok github.com/grafana/loki/v3/pkg/dataobj/metastore 24.747s ok github.com/grafana/loki/v3/pkg/dataobj/querier 36.249s ? github.com/grafana/loki/v3/pkg/dataobj/tools [no test files] ? github.com/grafana/loki/v3/pkg/dataobj/uploader [no test files] ok github.com/grafana/loki/v3/pkg/distributor 33.293s ? github.com/grafana/loki/v3/pkg/distributor/clientpool [no test files] ? github.com/grafana/loki/v3/pkg/distributor/shardstreams [no test files] ok github.com/grafana/loki/v3/pkg/distributor/writefailures 5.961s ok github.com/grafana/loki/v3/pkg/engine 6.260s ? github.com/grafana/loki/v3/pkg/engine/internal/types [no test files] ok github.com/grafana/loki/v3/pkg/engine/planner/internal/tree 0.501s ok github.com/grafana/loki/v3/pkg/engine/planner/logical 0.499s ok github.com/grafana/loki/v3/pkg/engine/planner/physical 6.191s ? github.com/grafana/loki/v3/pkg/engine/planner/schema [no test files] ok github.com/grafana/loki/v3/pkg/framedstdcopy 0.381s ok github.com/grafana/loki/v3/pkg/indexgateway 9.749s --- FAIL: TestChunkFlushingIdle (0.68s) flush_test.go:561: Error Trace: /build/loki/src/loki-3.5.1/pkg/ingester/flush_test.go:561 /build/loki/src/loki-3.5.1/pkg/ingester/flush_test.go:62 Error: Not equal: expected: []push.Stream{push.Stream{Labels:"{job=\"testjob\", name=\"testmetric_0\"}", Entries:[]push.Entry{push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 1, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 2, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 3, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 27, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 28, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 29, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 30, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 31, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 32, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 33, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 34, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 35, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 36, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 37, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 38, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 39, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 40, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 41, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 42, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 43, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 44, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 45, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 46, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 47, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 48, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 49, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 50, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 51, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 52, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 53, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 54, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 55, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 56, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 57, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 58, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 59, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 0, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 1, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 2, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 3, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 27, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 28, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 29, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 30, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 31, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 32, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 33, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 34, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 35, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 36, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 37, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 38, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 39, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 40, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}}, Hash:0x0}, push.Stream{Labels:"{job=\"testjob\", name=\"testmetric_1\"}", Entries:[]push.Entry{push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 2, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 3, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 27, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 28, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 29, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 30, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 31, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 32, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 33, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 34, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 35, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 36, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 37, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 38, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 39, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 40, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 41, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 42, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 43, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 44, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 45, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 46, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 47, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 48, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 49, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 50, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 51, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 52, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 53, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 54, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 55, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 56, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 57, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 58, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 59, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 0, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 1, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 2, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 3, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 27, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 28, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 29, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 30, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 31, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 32, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 33, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 34, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 35, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 36, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 37, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 38, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 39, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 40, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 41, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}}, Hash:0x0}, push.Stream{Labels:"{job=\"testjob\", name=\"testmetric_2\"}", Entries:[]push.Entry{push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 3, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 27, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 28, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 29, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 30, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 31, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 32, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 33, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 34, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 35, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 36, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 37, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 38, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 39, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 40, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 41, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 42, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 43, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 44, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 45, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 46, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 47, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 48, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 49, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 50, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 51, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 52, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 53, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 54, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 55, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 56, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 57, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 58, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 59, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 0, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 1, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 2, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 3, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 27, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 28, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 29, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 30, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 31, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 32, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 33, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 34, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 35, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 36, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 37, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 38, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 39, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 40, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 41, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 42, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}}, Hash:0x0}, push.Stream{Labels:"{job=\"testjob\", name=\"testmetric_3\"}", Entries:[]push.Entry{push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 27, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 28, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 29, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 30, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 31, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 32, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 33, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 34, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 35, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 36, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 37, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 38, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 39, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 40, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 41, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 42, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 43, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 44, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 45, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 46, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 47, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 48, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 49, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 50, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 51, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 52, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 53, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 54, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 55, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 56, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 57, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 58, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 59, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 0, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 1, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 2, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 3, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry<... truncated> actual : []push.Stream{push.Stream{Labels:"{job=\"testjob\", name=\"testmetric_0\"}", Entries:[]push.Entry{push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 1, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 2, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 3, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 27, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 28, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 29, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 30, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 31, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 32, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 33, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 34, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 35, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 36, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 37, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 38, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 39, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 40, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 41, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 42, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 43, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 44, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 45, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 46, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 47, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 48, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 49, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 50, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 51, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 52, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 53, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 54, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 55, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 56, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 57, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 58, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 59, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 0, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 1, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 2, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 3, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 27, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 28, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 29, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 30, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 31, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 32, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 33, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 34, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 35, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 36, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 37, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 38, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 39, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 40, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}}, Hash:0x0}, push.Stream{Labels:"{job=\"testjob\", name=\"testmetric_1\"}", Entries:[]push.Entry{push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 2, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 3, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 27, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 28, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 29, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 30, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 31, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 32, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 33, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 34, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 35, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 36, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 37, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 38, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 39, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 40, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 41, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 42, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 43, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 44, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 45, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 46, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 47, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 48, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 49, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 50, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 51, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 52, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 53, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 54, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 55, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 56, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 57, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 58, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 59, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 0, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 1, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 2, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 3, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 27, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 28, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 29, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 30, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 31, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 32, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 33, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 34, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 35, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 36, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 37, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 38, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 39, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 40, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 41, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}}, Hash:0x0}, push.Stream{Labels:"{job=\"testjob\", name=\"testmetric_2\"}", Entries:[]push.Entry{push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 3, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 27, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 28, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 29, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 30, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 31, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 32, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 33, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 34, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 35, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 36, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 37, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 38, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 39, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 40, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 41, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 42, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 43, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 44, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 45, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 46, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 47, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 48, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 49, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 50, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 51, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 52, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 53, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 54, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 55, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 56, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 57, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 58, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 59, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 0, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 1, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 2, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 3, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 27, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 28, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 29, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 30, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 31, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 32, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 33, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 34, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 35, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 36, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 37, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 38, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 39, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 40, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 41, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 42, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}}, Hash:0x0}, push.Stream{Labels:"{job=\"testjob\", name=\"testmetric_3\"}", Entries:[]push.Entry{push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 27, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 28, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 29, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 30, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 31, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 32, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 33, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 34, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 35, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 36, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 37, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 38, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 39, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 40, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 41, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 42, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 43, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 44, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 45, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 46, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 47, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 48, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 49, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 50, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 51, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 52, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 53, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 54, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 55, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 56, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 57, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 58, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 0, 59, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 0, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 1, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 2, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 3, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 4, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 5, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 6, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 7, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 8, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 9, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 10, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 11, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 12, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 13, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 14, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 15, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 16, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 17, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 18, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 19, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 20, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 21, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 22, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 23, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 24, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 25, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry{Timestamp:time.Date(1970, time.January, 1, 0, 1, 26, 0, time.Local), Line:"line", StructuredMetadata:push.LabelsAdapter(nil), Parsed:push.LabelsAdapter(nil)}, push.Entry<... truncated> Diff: --- Expected +++ Actual @@ -1,2 +1,2 @@ -([]push.Stream) (len=10) { +([]push.Stream) (len=7) { (push.Stream) { @@ -12642,5420 +12642,2 @@ Hash: (uint64) 0 - }, - (push.Stream) { - Labels: (string) (len=36) "{job=\"testjob\", name=\"testmetric_7\"}", - Entries: ([]push.Entry) (len=100) { - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596808, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596809, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596810, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596811, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596812, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596813, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596814, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596815, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596816, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596817, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596818, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596819, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596820, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596821, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596822, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596823, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596824, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596825, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596826, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596827, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596828, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596829, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596830, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596831, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596832, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596833, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596834, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596835, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596836, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596837, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596838, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596839, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596840, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596841, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596842, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596843, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596844, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596845, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596846, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596847, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596848, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596849, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596850, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596851, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596852, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596853, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596854, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596855, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596856, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596857, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596858, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596859, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596860, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596861, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596862, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596863, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596864, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596865, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596866, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596867, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596868, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596869, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596870, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596871, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596872, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596873, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596874, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596875, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596876, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596877, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596878, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596879, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596880, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596881, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596882, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596883, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596884, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596885, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596886, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596887, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596888, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596889, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596890, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596891, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596892, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596893, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596894, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596895, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596896, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596897, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596898, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596899, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596900, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596901, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596902, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596903, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596904, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596905, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596906, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596907, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - } - }, - Hash: (uint64) 0 - }, - (push.Stream) { - Labels: (string) (len=36) "{job=\"testjob\", name=\"testmetric_8\"}", - Entries: ([]push.Entry) (len=100) { - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596809, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596810, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596811, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596812, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596813, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596814, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596815, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596816, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596817, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596818, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596819, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596820, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596821, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596822, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596823, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596824, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596825, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596826, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596827, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596828, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596829, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596830, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596831, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596832, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596833, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596834, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596835, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596836, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596837, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596838, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596839, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596840, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596841, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596842, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596843, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596844, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596845, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596846, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596847, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596848, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596849, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596850, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596851, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596852, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596853, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596854, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596855, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596856, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596857, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596858, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596859, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596860, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596861, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596862, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596863, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596864, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596865, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596866, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596867, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596868, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596869, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596870, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596871, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596872, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596873, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596874, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596875, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596876, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596877, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596878, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596879, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596880, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596881, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596882, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596883, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596884, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596885, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596886, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596887, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596888, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596889, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596890, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596891, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596892, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596893, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596894, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596895, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596896, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596897, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596898, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596899, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596900, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596901, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596902, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596903, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596904, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596905, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596906, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596907, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596908, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - } - }, - Hash: (uint64) 0 - }, - (push.Stream) { - Labels: (string) (len=36) "{job=\"testjob\", name=\"testmetric_9\"}", - Entries: ([]push.Entry) (len=100) { - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596810, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596811, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596812, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596813, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596814, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596815, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596816, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596817, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596818, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596819, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596820, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596821, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596822, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596823, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596824, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596825, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596826, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596827, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596828, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596829, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596830, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596831, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596832, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596833, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596834, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596835, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596836, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596837, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596838, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596839, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596840, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596841, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596842, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596843, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596844, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596845, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596846, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596847, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596848, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596849, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596850, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596851, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596852, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596853, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596854, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596855, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596856, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596857, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596858, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596859, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596860, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596861, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596862, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596863, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596864, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596865, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596866, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596867, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596868, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596869, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596870, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596871, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596872, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596873, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596874, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596875, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596876, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596877, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596878, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596879, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596880, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596881, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596882, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596883, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596884, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596885, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596886, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596887, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596888, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596889, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596890, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596891, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596892, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596893, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596894, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596895, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596896, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596897, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596898, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596899, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596900, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596901, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596902, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596903, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596904, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596905, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596906, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596907, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596908, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - }, - (push.Entry) { - Timestamp: (time.Time) { - wall: (uint64) 0, - ext: (int64) 62135596909, - loc: (*time.Location)({ - name: (string) "", - zone: ([]time.zone) , - tx: ([]time.zoneTrans) , - extend: (string) "", - cacheStart: (int64) 0, - cacheEnd: (int64) 0, - cacheZone: (*time.zone)() - }) - }, - Line: (string) (len=4) "line", - StructuredMetadata: (push.LabelsAdapter) , - Parsed: (push.LabelsAdapter) - } - }, - Hash: (uint64) 0 } Test: TestChunkFlushingIdle panic: test timed out after 10m0s running tests: Test_DedupeIngester (8m21s) goroutine 42017 [running]: testing.(*M).startAlarm.func1() /usr/lib/go/src/testing/testing.go:2484 +0x304 created by time.goFunc /usr/lib/go/src/time/sleep.go:215 +0x58 goroutine 1 [chan receive, 8 minutes]: testing.(*T).Run(0x3f74550a80, {0x268d28d?, 0x3670344174d0fb88?}, 0x2773a60) /usr/lib/go/src/testing/testing.go:1859 +0x3b4 testing.runTests.func1(0x3f74550a80) /usr/lib/go/src/testing/testing.go:2279 +0x4c testing.tRunner(0x3f74550a80, 0x3f74d0fca8) /usr/lib/go/src/testing/testing.go:1792 +0xf8 testing.runTests(0x3f7400d098, {0x46c02a0, 0x59, 0x59}, {0x1d8f4?, 0x3f74bddb00?, 0x46d9780?}) /usr/lib/go/src/testing/testing.go:2277 +0x440 testing.(*M).Run(0x3f7491e460) /usr/lib/go/src/testing/testing.go:2142 +0x5bc main.main() _testmain.go:239 +0x94 goroutine 3 [select, 10 minutes]: github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1() /build/loki/src/loki-3.5.1/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:375 +0x94 created by github.com/baidubce/bce-sdk-go/util/log.NewLogger in goroutine 1 /build/loki/src/loki-3.5.1/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:368 +0x124 goroutine 121 [select]: go.opencensus.io/stats/view.(*worker).start(0x3f744d0c80) /build/loki/src/loki-3.5.1/vendor/go.opencensus.io/stats/view/worker.go:292 +0x94 created by go.opencensus.io/stats/view.init.0 in goroutine 1 /build/loki/src/loki-3.5.1/vendor/go.opencensus.io/stats/view/worker.go:34 +0x98 goroutine 4883 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f76017f00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75828780) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4868 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4812 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27e800, {0x2c308c8, 0x3f74c01740}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4811 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4902 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc706280, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74449400) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4901 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5314 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc620820) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5232 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 247 [select]: github.com/grafana/dskit/kv/consul.(*mockKV).loop(0x3f74c076d0) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/kv/consul/mock.go:90 +0xac created by github.com/grafana/dskit/kv/consul.NewInMemoryClientWithConfig in goroutine 246 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/kv/consul/mock.go:61 +0x23c goroutine 4427 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 3561 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5025 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74502e00, {0x2c308c8, 0x3ebc6c52c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5067 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74db6e00, {0x2c308c8, 0x3ebc181100}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4523 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74925030) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5012 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcab22a0, {0x2c3c1f0, 0x3ebd3de000}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4285 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27e000, {0x2c308c8, 0x3f750ccbe0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4934 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc706580, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74d92e00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4933 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4513 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f755122b8, 0x3b1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f755122a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75512240, {0x3ebc152000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f74e1e5a0, {0x3f74a98200, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f74e1e5a0}, {0x3f74a98200, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a98200, 0x9, 0x3eaec63bc0?}, {0x2c04240?, 0x3f74e1e5a0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a981c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f747b2908, 0x3f74622000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4282 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4087 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4108 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4698 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743c14b8, 0x3b5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743c14a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f743c1440, {0x3ebc938000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f74427980, {0x3ebc7a8580, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f74427980}, {0x3ebc7a8580, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a8580, 0x9, 0x3eaed2a3f0?}, {0x2c04240?, 0x3f74427980?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a8540) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f747b2d88, 0x3f74623c00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4738 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 2200 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebcc94000, {0x2c308c8, 0x3f77110160}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4113 [select, 8 minutes]: github.com/grafana/loki/v3/pkg/ingester.(*Ingester).loop(0x3f747fb508) /build/loki/src/loki-3.5.1/pkg/ingester/ingester.go:773 +0x1fc created by github.com/grafana/loki/v3/pkg/ingester.(*Ingester).starting in goroutine 4111 /build/loki/src/loki-3.5.1/pkg/ingester/ingester.go:628 +0xd54 goroutine 4131 [select]: github.com/grafana/loki/v3/pkg/ingester.New.newRecalculateOwnedStreamsSvc.NewTimerService.func4({0x2c3c1f0, 0x3ebcd3fcc0}) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/services.go:31 +0xd4 github.com/grafana/dskit/services.(*BasicService).main(0x3ebc77ad20) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:193 +0x17c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 4111 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:122 +0xf0 goroutine 4130 [select]: github.com/grafana/dskit/ring.(*Lifecycler).loop(0x3ebc8d0008, {0x2c3c1f0, 0x3ebcd3fae0}) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/ring/lifecycler.go:540 +0x188 github.com/grafana/dskit/services.(*BasicService).main(0x3ebc77abe0) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:193 +0x17c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 4111 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:122 +0xf0 goroutine 4118 [select]: github.com/grafana/dskit/ring.(*Lifecycler).loop(0x3f74b39b08, {0x2c3c1f0, 0x3f74c072c0}) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/ring/lifecycler.go:540 +0x188 github.com/grafana/dskit/services.(*BasicService).main(0x3f749c50e0) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:193 +0x17c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 4115 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:122 +0xf0 goroutine 4496 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f752366b0, {0x2c3c1f0, 0x3ebca92190}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4480 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc04e460, {0x2c3c1f0, 0x3ebc469400}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4479 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc04e430, {0x2c3c1f0, 0x3ebc4693b0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5310 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f752374e0, {0x2c3c1f0, 0x3f7436cd20}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4946 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca024f8, 0x3ad) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca024e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca02480, {0x3ebd1e2000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc8fdf20, {0x3f743ca900, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc8fdf20}, {0x3f743ca900, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743ca900, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc8fdf20?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743ca8c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd19c000, {0x2c3c1b8, 0x3ebc075650}, 0x3ebc075680) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f749a7800, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd19c000}, {0x2c48448?, 0x3f746eb2e0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4831 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4472 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f768df580, {0x2c3c1f0, 0x3ebc469130}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4598 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc04f8a0, {0x2c3c1f0, 0x3ebc4689b0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4432 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4486 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4755 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f7499a000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4689 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4761 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743c1338, 0x3b8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743c1328) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f743c12c0, {0x3ebc980000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75998300, {0x3ebc2b4580, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75998300}, {0x3ebc2b4580, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b4580, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f75998300?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b4540) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f7499ad00, {0x2c3c1b8, 0x3f74f40ff0}, 0x3f74f41020) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74db6800, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f7499ad00}, {0x2c48448?, 0x3f758921c0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4758 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4765 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4837 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac02b8, 0x3b5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac02a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac0240, {0x3ebcb72000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc5a80c0, {0x3ebca9c3c0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc5a80c0}, {0x3ebca9c3c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9c3c0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc5a80c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9c380) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5aa000, {0x2c3c1b8, 0x3f75182540}, 0x3f75182570) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27e400, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5aa000}, {0x2c48448?, 0x3ebc67e060?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4834 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5417 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444d638, 0x3bb) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444d628) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444d5c0, {0x3ec2aba000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc175500, {0x3f74a99620, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc175500}, {0x3f74a99620, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a99620, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc175500?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a995e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc620b60, {0x2c3c1b8, 0x3ec2aa8450}, 0x3ec2aa8480) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc474e00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc620b60}, {0x2c48448?, 0x3f75b01500?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5414 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4079 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4278 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4880 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75637860, {0x2c3c1f0, 0x3ebca93130}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4581 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f748848c0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5091 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75e9b970, {0x2c3c1f0, 0x3f763eb4f0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4756 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444cdf8, 0x3ac) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444cde8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444cd80, {0x3ebc726000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75998240, {0x3ebc2b44a0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75998240}, {0x3ebc2b44a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b44a0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f75998240?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b4460) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f7499a000, {0x2c3c1b8, 0x3f74f40390}, 0x3f74f403c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc474400, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f7499a000}, {0x2c48448?, 0x3f74e0c3a0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4689 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5164 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444ceb8, 0x3bb) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444cea8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444ce40, {0x3f74d62000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc174120, {0x3f74a98f20, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc174120}, {0x3f74a98f20, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a98f20, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc174120?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a98ee0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc620680, {0x2c3c1b8, 0x3ebc173410}, 0x3ebc173440) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74db7a00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc620680}, {0x2c48448?, 0x3f75b00bc0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5161 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4848 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f749a7200, {0x2c308c8, 0x3ebc67ee00}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 2208 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74afaf50, {0x2c3c1f0, 0x3f75f8a190}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4671 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc620340) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4669 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5308 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc474a00, {0x2c308c8, 0x3ec18f34e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4712 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a3f30, {0x2c3c1f0, 0x3ebcd3e780}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4693 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fa340, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f759ae200) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4660 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 2310 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebcaa2000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 2308 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4844 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc5c2700, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f743e9000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4809 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4768 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7569f590, {0x2c3c1f0, 0x3ebc468eb0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4431 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7444bc00) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4486 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4903 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74032b60) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4901 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4614 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc0e4000, {0x2c308c8, 0x3f74e0d040}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4751 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74dbc2c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74448400) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5015 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5325 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444d4b8, 0x3b8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444d4a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444d440, {0x3f74e6a000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc174fc0, {0x3f74a99460, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc174fc0}, {0x3f74a99460, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a99460, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc174fc0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a99420) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc6209c0, {0x2c3c1b8, 0x3f74e376b0}, 0x3f74e376e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc474800, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc6209c0}, {0x2c48448?, 0x3f75b01320?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5322 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5315 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444d338, 0x3bc) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444d328) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444d2c0, {0x3ec2256000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc174c00, {0x3f74a992a0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc174c00}, {0x3f74a992a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a992a0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc174c00?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a99260) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc620820, {0x2c3c1b8, 0x3ec2244ba0}, 0x3ec2244bd0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27f800, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc620820}, {0x2c48448?, 0x3f75b01200?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5232 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4632 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc6a0300, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8c480) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4631 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4953 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74dbdb40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a52b00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4928 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5086 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca02af8, 0x3c0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca02ae8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca02a80, {0x3ec1b0e000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec11a12c0, {0x3ebc12b700, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec11a12c0}, {0x3ebc12b700, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12b700, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ec11a12c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12b6c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd19d040, {0x2c3c1b8, 0x3ebc8fa7b0}, 0x3ebc8fa7e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27f000, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd19d040}, {0x2c48448?, 0x3ebc661900?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5212 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4633 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74c4e4e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4631 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4713 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a3f60, {0x2c3c1f0, 0x3ebcd3e7d0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4686 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74341cf8, 0x3a8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74341ce8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74341c80, {0x3ebc598000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75998120, {0x3ebc2b43c0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75998120}, {0x3ebc2b43c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b43c0, 0x9, 0x3eaeddfe78?}, {0x2c04240?, 0x3f75998120?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b4380) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc390248, 0x3f74b0acb0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4611 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4915 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f746f6cb0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4936 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc882d38, 0x3c8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc882d28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc882cc0, {0x3ebd382000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebceda660, {0x3ebc2b4f20, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebceda660}, {0x3ebc2b4f20, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b4f20, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebceda660?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b4ee0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74032ea0, {0x2c3c1b8, 0x3ebcedc780}, 0x3ebcedc7b0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f749a7c00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74032ea0}, {0x2c48448?, 0x3f75b00d60?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4933 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4881 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75637890, {0x2c3c1f0, 0x3ebca93180}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4695 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f750d2df8, 0x3b7) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f750d2de8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f750d2d80, {0x3ebc23e000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebcc86660, {0x3f74a98ba0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebcc86660}, {0x3f74a98ba0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a98ba0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebcc86660?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a98b60) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc6201a0, {0x2c3c1b8, 0x3ebcd64270}, 0x3ebcd642a0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc474200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc6201a0}, {0x2c48448?, 0x3f75892080?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4660 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4762 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e16480, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74448b00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4738 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4855 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc6a1f40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74448f00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4854 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4503 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fa080, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74d92780) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4481 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4530 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc882138, 0x3c2) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc882128) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc8820c0, {0x3ebc884000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f750e6120, {0x3ebc4a8120, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f750e6120}, {0x3ebc4a8120, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc4a8120, 0x9, 0x3eaec63b48?}, {0x2c04240?, 0x3f750e6120?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc4a80e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc894008, 0x3f74978bd0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4481 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4612 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74b0a5b0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4682 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743c13f8, 0x3b1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743c13e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f743c1380, {0x3ebcada000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f74ed6360, {0x3ebc0904a0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f74ed6360}, {0x3ebc0904a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc0904a0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f74ed6360?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc090460) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f7499a340, {0x2c3c1b8, 0x3ebc695860}, 0x3ebc695890) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f743e6400, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f7499a340}, {0x2c48448?, 0x3ebc6613a0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4679 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4610 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f75e96040, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8c380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4594 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4426 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7444ae70) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 3561 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4969 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5347 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5317 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444d578, 0x3b9) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444d568) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444d500, {0x3f74e5a000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc174ea0, {0x3f74a99380, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc174ea0}, {0x3f74a99380, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a99380, 0x9, 0x3eaec63de8?}, {0x2c04240?, 0x3ebc174ea0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a99340) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc391688, 0x3f74a461c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5305 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4116 [sync.Cond.Wait, 8 minutes]: sync.runtime_notifyListWait(0x3f75a24850, 0x1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75a24840) /usr/lib/go/src/sync/cond.go:71 +0xa8 github.com/grafana/loki/v3/pkg/util.(*PriorityQueue).Dequeue(0x3f74c07270) /build/loki/src/loki-3.5.1/pkg/util/priority_queue.go:115 +0x94 github.com/grafana/loki/v3/pkg/ingester.(*Ingester).flushLoop(0x3f7491b508, 0x0) /build/loki/src/loki-3.5.1/pkg/ingester/flush.go:226 +0x114 created by github.com/grafana/loki/v3/pkg/ingester.(*Ingester).InitFlushQueues in goroutine 4115 /build/loki/src/loki-3.5.1/pkg/ingester/flush.go:109 +0x54 goroutine 4499 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4433 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4907 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74e0f038, 0x3b5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74e0f028) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74e0efc0, {0x3ebcede000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebceda120, {0x3ebc2b4d60, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebceda120}, {0x3ebc2b4d60, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b4d60, 0x9, 0x3eaec63cb0?}, {0x2c04240?, 0x3ebceda120?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b4d20) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc390b48, 0x3f748eea10) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4914 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4490 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc6a0440, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74448600) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4648 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 2980 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f756900c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75828680) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4618 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4805 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27e600, {0x2c308c8, 0x3f74c005a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4471 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f768df550, {0x2c3c1f0, 0x3ebc4690e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4117 [select]: github.com/grafana/loki/v3/pkg/ingester.(*Ingester).loop(0x3f7491b508) /build/loki/src/loki-3.5.1/pkg/ingester/ingester.go:788 +0x2dc created by github.com/grafana/loki/v3/pkg/ingester.(*Ingester).starting in goroutine 4115 /build/loki/src/loki-3.5.1/pkg/ingester/ingester.go:628 +0xd54 goroutine 4746 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f750d2678, 0x3ac) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f750d2668) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f750d2600, {0x3ebc2e6000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebd19e000, {0x3ebc12ae40, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebd19e000}, {0x3ebc12ae40, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12ae40, 0x9, 0x3eaf046780?}, {0x2c04240?, 0x3ebd19e000?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12ae00) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74849688, 0x3f7472c4d0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5015 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4659 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca021f8, 0x3b8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca021e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca02180, {0x3ebc33e000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f759981e0, {0x3f743ca4a0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f759981e0}, {0x3f743ca4a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743ca4a0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f759981e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743ca460) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f740324e0, {0x2c3c1b8, 0x3ebc5c69c0}, 0x3ebc5c69f0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f749a6200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f740324e0}, {0x2c48448?, 0x3f746eabe0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4727 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4653 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5074 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5aa1a0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4960 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4279 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f761200b0, {0x2c3c1f0, 0x3ebcd3e460}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4575 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74d8a820) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4573 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 2209 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74afafb0, {0x2c3c1f0, 0x3f75f8a1e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4590 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc474400, {0x2c308c8, 0x3f768f4a60}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4728 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7569a080, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f759ae080) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4727 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4477 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc475600, {0x2c308c8, 0x3f75e009a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4905 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc706240, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74d92d00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4875 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4696 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e160c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f759ae300) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4587 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4586 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74317ce0, {0x2c3c1f0, 0x3ebcd3edc0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5075 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca02378, 0x3b3) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca02368) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca02300, {0x3ebd2e6000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebd3d2600, {0x3ebca9cc80, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebd3d2600}, {0x3ebca9cc80, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9cc80, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebd3d2600?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9cc40) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5aa1a0, {0x2c3c1b8, 0x3ebc8cdb60}, 0x3ebc8cdb90) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74db6c00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5aa1a0}, {0x2c48448?, 0x3ebc660b00?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4960 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4654 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74db6800, {0x2c308c8, 0x3f75e99900}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4808 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74d8cbc0, {0x2c3c1f0, 0x3ebc4698b0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5092 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75e9b9a0, {0x2c3c1f0, 0x3f763eb540}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4744 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f761206c0, {0x2c3c1f0, 0x3f763ead20}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4577 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e9ad40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c2700) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4522 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4570 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc872100, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c2500) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4282 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5023 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f748e7420) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4583 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc474200, {0x2c308c8, 0x3ebc00d9e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4592 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a3080, {0x2c3c1f0, 0x3ebcd3eff0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4680 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc9fc840, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a52280) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4679 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4585 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74317ca0, {0x2c3c1f0, 0x3ebcd3ed70}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4502 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc882078, 0x3bd) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc882068) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc882000, {0x3ebc6ee000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75f97920, {0x3ebc12a200, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75f97920}, {0x3ebc12a200, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12a200, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f75f97920?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12a1c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74c4e9c0, {0x2c3c1b8, 0x3ebc6e9bc0}, 0x3ebc6e9bf0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc475600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74c4e9c0}, {0x2c48448?, 0x3ebc00c100?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4535 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 3559 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74935500) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4457 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4941 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74c4e820) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4939 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4584 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74317c70, {0x2c3c1f0, 0x3ebcd3ed20}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5525 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fa540, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74d93200) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5425 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 3013 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74372cb0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 3010 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4599 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc04f8d0, {0x2c3c1f0, 0x3ebc468a00}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4709 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f745ffc00) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4885 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc882df8, 0x3c2) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc882de8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc882d80, {0x3ebd234000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc450ae0, {0x3ebc12ac80, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc450ae0}, {0x3ebc12ac80, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12ac80, 0x9, 0x3eaf0460f0?}, {0x2c04240?, 0x3ebc450ae0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12ac40) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebd244008, 0x3f748e90a0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4967 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4707 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f745feaf0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5062 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc04fc90, {0x2c3c1f0, 0x3f763eb040}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5065 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74cb47e0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5066 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4596 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4670 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fa200, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8c200) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4669 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4593 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a30b0, {0x2c3c1f0, 0x3ebcd3f040}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4674 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a21230, {0x2c3c1f0, 0x3f74c06280}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4823 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca02078, 0x3b8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca02068) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca02000, {0x3ebc054000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc8fdaa0, {0x3f743ca660, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc8fdaa0}, {0x3f743ca660, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743ca660, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc8fdaa0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743ca620) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f744816c0, {0x2c3c1b8, 0x3ebc6249c0}, 0x3ebc6249f0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74502a00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f744816c0}, {0x2c48448?, 0x3f746eaf00?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4820 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4970 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f743e6600, {0x2c308c8, 0x3ebc6c4960}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5628 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75513938, 0x3be) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75513928) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f755138c0, {0x3eba51e000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75088de0, {0x3ebcc90200, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75088de0}, {0x3ebcc90200, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebcc90200, 0x9, 0x3eaf5a5e90?}, {0x2c04240?, 0x3f75088de0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebcc901c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc313448, 0x3f74a97880) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5730 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5450 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebaacc000, {0x2c308c8, 0x3f75212360}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5709 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca03b78, 0x3bd) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca03b68) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca03b00, {0x3eba2ce000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9ab0de0, {0x3ebc091380, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9ab0de0}, {0x3ebc091380, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc091380, 0x9, 0x3eaf796510?}, {0x2c04240?, 0x3eb9ab0de0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc091340) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74ed4b48, 0x3eba2ca2a0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5783 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5751 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75513ab8, 0x3b9) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75513aa8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75513a40, {0x3eba544000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75089200, {0x3ebcc903c0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75089200}, {0x3ebcc903c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebcc903c0, 0x9, 0x3eaf6d00a8?}, {0x2c04240?, 0x3f75089200?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebcc90380) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc313688, 0x3eba5420e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5744 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 2294 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd19dd40) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5506 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 2293 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec3b78ac0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75828f80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5506 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4947 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74dbd940, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a52900) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4921 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4501 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74c4e9c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4535 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 2207 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebcc94200, {0x2c308c8, 0x3f74b008e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4813 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74d8dc10, {0x2c3c1f0, 0x3ebc469a90}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4484 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74623500) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4483 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5165 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fa440, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f759ae380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5202 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 3565 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4495 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75236680, {0x2c3c1f0, 0x3ebca92140}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5307 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5111 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36a9c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c2d00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5110 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5043 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f768de0c0, {0x2c3c1f0, 0x3ebd3dee10}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5480 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc37f900, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8d480) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5683 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5483 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebcaa2340) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5481 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5484 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca03ab8, 0x3b9) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca03aa8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca03a40, {0x3ebaf2c000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb85f43c0, {0x3f74a99a80, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb85f43c0}, {0x3f74a99a80, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a99a80, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb85f43c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a99a40) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebcaa2340, {0x2c3c1b8, 0x3eb85f3530}, 0x3eb85f3560) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eba7ce400, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebcaa2340}, {0x2c48448?, 0x3ebc900d00?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5481 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4917 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f749a7800, {0x2c308c8, 0x3f76019400}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4645 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76121d20, {0x2c3c1f0, 0x3f763ea4b0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4847 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4789 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e16500, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f759ae580) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4763 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 2987 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc883038, 0x3b8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc883028) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc882fc0, {0x3f74c2c000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75809980, {0x3ebcc90120, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75809980}, {0x3ebcc90120, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebcc90120, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f75809980?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebcc900e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd3d0000, {0x2c3c1b8, 0x3ebd3a7020}, 0x3ebd3a7050) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f743e6600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd3d0000}, {0x2c48448?, 0x3f77110c60?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4608 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4840 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac0678, 0x3b8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac0668) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac0600, {0x3ebcbf0000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc5a85a0, {0x3ebca9c580, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc5a85a0}, {0x3ebca9c580, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9c580, 0x9, 0x3eae5b8ac8?}, {0x2c04240?, 0x3ebc5a85a0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9c540) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc57e6c8, 0x3f746ed110) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4809 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4114 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 3952 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4743 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76120670, {0x2c3c1f0, 0x3f763eacd0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5024 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4710 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5052 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444d0f8, 0x3b2) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444d0e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444d080, {0x3eb8220000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f751b40c0, {0x3ebc2b52a0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f751b40c0}, {0x3ebc2b52a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b52a0, 0x9, 0x3eaf514a20?}, {0x2c04240?, 0x3f751b40c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b5260) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc57e908, 0x3f74a468c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5425 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4597 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74db6600, {0x2c308c8, 0x3f75b00c60}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5081 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f765c7840, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74d93600) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5071 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4647 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76121d80, {0x2c3c1f0, 0x3f763ea550}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4858 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc6a1f00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74449000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4845 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4076 [select]: github.com/grafana/loki/v3/pkg/util.(*TickerWithJitter).start(0x3f74c00040, 0x5f5e100, 0x1312d00) /build/loki/src/loki-3.5.1/pkg/util/ticker.go:41 +0x114 created by github.com/grafana/loki/v3/pkg/util.NewTickerWithJitter in goroutine 4117 /build/loki/src/loki-3.5.1/pkg/util/ticker.go:63 +0x114 goroutine 4692 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743c10f8, 0x3ba) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743c10e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f743c1080, {0x3ebc2f6000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f7568d320, {0x3ebc7a82e0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f7568d320}, {0x3ebc7a82e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a82e0, 0x9, 0x3eaec63c08?}, {0x2c04240?, 0x3f7568d320?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a82a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f747b2b48, 0x3f743df880) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4587 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4652 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74020d20) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4814 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74d8dc50, {0x2c3c1f0, 0x3ebc469ae0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4815 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74d8dc80, {0x2c3c1f0, 0x3ebc469b30}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4864 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74e0eeb8, 0x3b7) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74e0eea8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74e0ee40, {0x3ebcea6000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc623da0, {0x3ebc2b4ba0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc623da0}, {0x3ebc2b4ba0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b4ba0, 0x9, 0x3eae5b9368?}, {0x2c04240?, 0x3ebc623da0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b4b60) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc390908, 0x3f741cde30) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4875 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4634 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f750d2978, 0x3b5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f750d2968) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f750d2900, {0x3ebc4c8000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f750e6060, {0x3ebc4a83c0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f750e6060}, {0x3ebc4a83c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc4a83c0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f750e6060?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc4a8380) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74c4e4e0, {0x2c3c1b8, 0x3ebc134c00}, 0x3ebc134c30) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74db6600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74c4e4e0}, {0x2c48448?, 0x3f74985300?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4631 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4548 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc475800, {0x2c308c8, 0x3f75e01600}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4681 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f7499a340) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4679 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4740 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4741 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74db6a00, {0x2c308c8, 0x3f750cc560}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 3011 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74372bd0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 3010 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4491 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7492c2a0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4833 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd19c000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4831 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5522 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fa580, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74d92f00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5057 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4500 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc6a1300, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0f680) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4535 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5324 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc6209c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5322 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4547 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 2201 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74700530, {0x2c3c1f0, 0x3ebcbfe050}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4700 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743c1638, 0x3ae) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743c1628) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f743c15c0, {0x3ebc990000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc316000, {0x3ebc7a8660, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc316000}, {0x3ebc7a8660, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a8660, 0x9, 0x3eae5b8a08?}, {0x2c04240?, 0x3ebc316000?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a8620) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f747b2fc8, 0x3f74934000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4763 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4792 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f748e6540) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4916 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4861 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74230680) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4776 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 3012 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74372c40) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 3010 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 2206 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4280 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f761200e0, {0x2c3c1f0, 0x3f763ea000}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4519 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743c11b8, 0x3c0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743c11a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f743c1140, {0x3ebc63e000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75a08f00, {0x3ebc7a84a0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75a08f00}, {0x3ebc7a84a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a84a0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f75a08f00?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a8460) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc620000, {0x2c3c1b8, 0x3ebc617110}, 0x3ebc617140) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc475400, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc620000}, {0x2c48448?, 0x3f750cc3a0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4516 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4998 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f750d25b8, 0x3a9) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f750d25a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f750d2540, {0x3ebc3b0000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebcc86780, {0x3ebc2b50e0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebcc86780}, {0x3ebc2b50e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b50e0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebcc86780?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b50a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74032820, {0x2c3c1b8, 0x3f74bedaa0}, 0x3f74bedad0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27ea00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74032820}, {0x2c48448?, 0x3f750cc160?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4995 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 2986 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd3d0000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4608 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4849 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75237a30, {0x2c3c1f0, 0x3ebca92be0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5162 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fa480, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74449e80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5161 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5073 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5090 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74db7000, {0x2c308c8, 0x3f74b011c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5072 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7494d730) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5253 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75513338, 0x3b6) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75513328) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f755132c0, {0x3ec2146000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebfbcfaa0, {0x3ebc090ac0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebfbcfaa0}, {0x3ebc090ac0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc090ac0, 0x9, 0x3eaf5145a0?}, {0x2c04240?, 0x3ebfbcfaa0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc090a80) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebd244908, 0x3f74a6e070) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5352 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5070 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75e9ac60, {0x2c3c1f0, 0x3f763eb310}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4919 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74083bd0, {0x2c3c1f0, 0x3ebca933b0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4807 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74d8cb90, {0x2c3c1f0, 0x3ebc469860}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4927 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74871ee0, {0x2c3c1f0, 0x3ebca93680}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4904 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74e0edf8, 0x3b4) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74e0ede8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74e0ed80, {0x3ebceb6000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc623ec0, {0x3ebc2b4c80, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc623ec0}, {0x3ebc2b4c80, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b4c80, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc623ec0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b4c40) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74032b60, {0x2c3c1b8, 0x3ebce9bd10}, 0x3ebce9bd40) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f749a7400, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74032b60}, {0x2c48448?, 0x3f74c01b40?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4901 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5000 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74e0f338, 0x3ae) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74e0f328) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74e0f2c0, {0x3ebd36c000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebcc86c60, {0x3ebc2b51c0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebcc86c60}, {0x3ebc2b51c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b51c0, 0x9, 0x3eaf1ec708?}, {0x2c04240?, 0x3ebcc86c60?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b5180) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc57e488, 0x3f748fcaf0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5071 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4555 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc6a0100, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74afc480) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4554 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4997 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74032820) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4995 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4924 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f749a7a00, {0x2c308c8, 0x3f74a8a240}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4951 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f742311e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4890 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4694 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc6201a0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4660 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4922 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f746f7c70) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4923 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5044 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f768de100, {0x2c3c1f0, 0x3ebd3dee60}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5045 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36a8c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f743e8e00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5094 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4706 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7487b5c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a52380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4640 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5233 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fa840, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8cc80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5232 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5079 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5aa340) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5077 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4109 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f748ab650) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4077 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4943 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36a800, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a52500) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4753 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4968 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74881880) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5795 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e9be40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f759afe00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5853 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 2309 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebcaa8000, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74448100) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 2308 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4966 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75514d40, {0x2c3c1f0, 0x3ebca93900}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 2205 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7494e230) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4787 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebcaa24e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4705 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4786 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e16540, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f759ae480) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4705 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4796 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75e9a090, {0x2c3c1f0, 0x3ebc9b29b0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4788 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743c1578, 0x3b5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743c1568) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f743c1500, {0x3ebc9d8000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc316120, {0x3ebc7a8740, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc316120}, {0x3ebc7a8740, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a8740, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc316120?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a8700) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebcaa24e0, {0x2c3c1b8, 0x3ebcd64b40}, 0x3ebcd64b70) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74db6a00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebcaa24e0}, {0x2c48448?, 0x3f758922c0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4705 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4795 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75e9a060, {0x2c3c1f0, 0x3ebc9b2960}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4498 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f748aa770) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4433 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4478 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc04e400, {0x2c3c1f0, 0x3ebc469360}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4569 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f755121f8, 0x3b6) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f755121e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75512180, {0x3ebc1a6000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f74e1e7e0, {0x3f74a982e0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f74e1e7e0}, {0x3f74a982e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a982e0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f74e1e7e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a982a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74d8a680, {0x2c3c1b8, 0x3ebc5dc330}, 0x3ebc5dc360) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc474000, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74d8a680}, {0x2c48448?, 0x3f74319860?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4566 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5448 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f765c89c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74449980) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5737 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5326 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fa980, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8ce80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5305 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5445 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec18f0000, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f759af900) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5444 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5323 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fa9c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8cd80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5322 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4794 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc0e4200, {0x2c308c8, 0x3f75892f40}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4920 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74083c00, {0x2c3c1f0, 0x3ebca93400}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5598 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74c6f278, 0x3b4) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74c6f268) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74c6f200, {0x3eb9a2e000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9a2a120, {0x3ebc4a91c0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9a2a120}, {0x3ebc4a91c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc4a91c0, 0x9, 0x3eaf7962e8?}, {0x2c04240?, 0x3eb9a2a120?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc4a9180) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebd244fc8, 0x3f74c0dab0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5737 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4836 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5aa000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4834 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5163 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc620680) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5161 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4806 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74d8cb60, {0x2c3c1f0, 0x3ebc4696d0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5621 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7500a880, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c3600) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5591 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5656 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebfbc0d38, 0x3b1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebfbc0d28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebfbc0cc0, {0x3eb96b4000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec3a79740, {0x3f743cb700, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec3a79740}, {0x3f743cb700, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743cb700, 0x9, 0x3eaf6d0030?}, {0x2c04240?, 0x3ec3a79740?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743cb6c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc894fc8, 0x3f7494d260) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5614 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5618 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75513638, 0x3b4) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75513628) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f755135c0, {0x3eb9612000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75088840, {0x3ebc12bc40, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75088840}, {0x3ebc12bc40, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12bc40, 0x9, 0x3eaf1edfc8?}, {0x2c04240?, 0x3f75088840?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12bc00) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc312b48, 0x3f74a97030) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5591 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 2640 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca03578, 0x3c1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca03568) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca03500, {0x3ebc1de000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75088540, {0x3ebc12b8c0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75088540}, {0x3ebc12b8c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12b8c0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f75088540?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12b880) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5aa680, {0x2c3c1b8, 0x3ebc0e0030}, 0x3ebc0e0060) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74503600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5aa680}, {0x2c48448?, 0x3eb826b000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 2637 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 2639 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5aa680) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 2637 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 2638 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7500a740, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c3500) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 2637 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4870 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5346 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f749c2c40) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4918 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74083b90, {0x2c3c1f0, 0x3ebca93360}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4797 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75e9a0c0, {0x2c3c1f0, 0x3ebc9b2a00}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4769 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7569f5c0, {0x2c3c1f0, 0x3ebc468f00}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4842 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebcaa2680) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4790 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4803 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7401aa80) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4733 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac0378, 0x3b4) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac0368) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac0300, {0x3ebcb62000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc5a8000, {0x3ebca9c2e0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc5a8000}, {0x3ebca9c2e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9c2e0, 0x9, 0x3eae5b8a50?}, {0x2c04240?, 0x3ebc5a8000?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9c2a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc57e248, 0x3f746ec380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4802 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4843 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac05b8, 0x3b7) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac05a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac0540, {0x3ebccf4000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc316360, {0x3ebc7a8820, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc316360}, {0x3ebc7a8820, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a8820, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc316360?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a87e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebcaa2680, {0x2c3c1b8, 0x3f75182d80}, 0x3f75182db0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27e600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebcaa2680}, {0x2c48448?, 0x3ebc67e2e0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4790 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4600 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc04f900, {0x2c3c1f0, 0x3ebc468a50}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4276 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74934e70) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4448 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4549 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc04f1d0, {0x2c3c1f0, 0x3ebc4695e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4785 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74230d00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4783 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4657 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a21350, {0x2c3c1f0, 0x3f763ea8c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4793 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4739 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74021c00) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5078 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f765c9340, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74d93500) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5077 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4929 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f748809a0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4962 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4963 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f749a7c00, {0x2c308c8, 0x3f74a8bae0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4964 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75514ce0, {0x2c3c1f0, 0x3ebca93860}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4729 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f740324e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4727 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 2204 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebfbc0078, 0x3b8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebfbc0068) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebfbc0000, {0x3ebcf5e000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebcf266c0, {0x3ebc7a8ac0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebcf266c0}, {0x3ebc7a8ac0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a8ac0, 0x9, 0x3eaf046138?}, {0x2c04240?, 0x3ebcf266c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a8a80) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f747b3208, 0x3f7494e000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4801 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4112 [sync.Cond.Wait, 8 minutes]: sync.runtime_notifyListWait(0x3f7569b510, 0x0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7569b500) /usr/lib/go/src/sync/cond.go:71 +0xa8 github.com/grafana/loki/v3/pkg/util.(*PriorityQueue).Dequeue(0x3ebcd3fa90) /build/loki/src/loki-3.5.1/pkg/util/priority_queue.go:115 +0x94 github.com/grafana/loki/v3/pkg/ingester.(*Ingester).flushLoop(0x3f747fb508, 0x0) /build/loki/src/loki-3.5.1/pkg/ingester/flush.go:226 +0x114 created by github.com/grafana/loki/v3/pkg/ingester.(*Ingester).InitFlushQueues in goroutine 4111 /build/loki/src/loki-3.5.1/pkg/ingester/flush.go:109 +0x54 goroutine 5076 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f765c8a40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f743e8c00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5064 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 2312 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc6a1380, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74448200) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4552 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4288 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76120fa0, {0x2c3c1f0, 0x3f763ea2d0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4287 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76120f70, {0x2c3c1f0, 0x3f763ea280}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4492 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4286 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76120f40, {0x2c3c1f0, 0x3f763ea230}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5059 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4589 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4757 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc9fc680, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74448900) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4611 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4546 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f746ff8f0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4557 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75512078, 0x3ae) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75512068) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75512000, {0x3ebc000000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75f97e00, {0x3ebc4a8200, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75f97e00}, {0x3ebc4a8200, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc4a8200, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f75f97e00?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc4a81c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74c4e340, {0x2c3c1b8, 0x3ebc91f860}, 0x3ebc91f890) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27ee00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74c4e340}, {0x2c48448?, 0x3f743194c0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4554 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4623 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca02138, 0x3b8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca02128) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca020c0, {0x3ebc034000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc8fda40, {0x3f743ca580, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc8fda40}, {0x3f743ca580, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743ca580, 0x9, 0x3eaec63c68?}, {0x2c04240?, 0x3ebc8fda40?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743ca540) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc312248, 0x3f74373880) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4651 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5021 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcab3080, {0x2c3c1f0, 0x3ebd3debe0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5020 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcab3050, {0x2c3c1f0, 0x3ebd3deaf0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4832 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74dbd980, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a52800) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4831 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4784 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f76017f40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74449100) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4783 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4613 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4878 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f749a7600, {0x2c308c8, 0x3f760187a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 2203 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f747005a0, {0x2c3c1f0, 0x3ebcbfe0f0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4935 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74032ea0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4933 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4088 [select, 8 minutes]: github.com/grafana/loki/v3/pkg/ingester.(*Ingester).running(0x3f74a4a008, {0x2c3c1f0?, 0x3f7436d040?}) /build/loki/src/loki-3.5.1/pkg/ingester/ingester.go:671 +0xb4 github.com/grafana/dskit/services.(*BasicService).main(0x3ebc7a4960) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:193 +0x17c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 4108 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:122 +0xf0 goroutine 4754 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc9fc8c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74448800) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4689 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4760 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f7499ad00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4758 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4485 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4483 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4859 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743c1ab8, 0x3b8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743c1aa8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f743c1a40, {0x3ebcf2c000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebcf26360, {0x3ebc2b4ac0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebcf26360}, {0x3ebc2b4ac0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b4ac0, 0x9, 0x3eae5b8b28?}, {0x2c04240?, 0x3ebcf26360?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b4a80) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc3906c8, 0x3f741cd420) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4791 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4804 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4764 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74923960) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 3953 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74cb5340) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 3952 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5085 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd19d040) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5212 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4111 [select, 8 minutes]: github.com/grafana/loki/v3/pkg/ingester.(*Ingester).running(0x3f747fb508, {0x2c3c1f0?, 0x3ebcd3fa40?}) /build/loki/src/loki-3.5.1/pkg/ingester/ingester.go:671 +0xb4 github.com/grafana/dskit/services.(*BasicService).main(0x3ebc77ac80) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:193 +0x17c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 4077 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:122 +0xf0 goroutine 4115 [select, 8 minutes]: github.com/grafana/loki/v3/pkg/ingester.(*Ingester).running(0x3f7491b508, {0x2c3c1f0?, 0x3f74c07220?}) /build/loki/src/loki-3.5.1/pkg/ingester/ingester.go:671 +0xb4 github.com/grafana/dskit/services.(*BasicService).main(0x3f749c5360) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:193 +0x17c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 3952 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:122 +0xf0 goroutine 4874 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75636ac0, {0x2c3c1f0, 0x3ebca92f00}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4641 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7494cc40) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4767 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7569f560, {0x2c3c1f0, 0x3ebc468e60}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5251 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fa800, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c3100) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5298 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4742 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76120630, {0x2c3c1f0, 0x3f763eab40}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5300 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4965 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75514d10, {0x2c3c1f0, 0x3ebca938b0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4766 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27e400, {0x2c308c8, 0x3ebc00d980}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5068 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75e9ac00, {0x2c3c1f0, 0x3f763eb270}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4690 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac01f8, 0x3ad) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac01e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac0180, {0x3ebcaca000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f751bc7e0, {0x3ebca9c200, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f751bc7e0}, {0x3ebca9c200, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9c200, 0x9, 0x3eae5b8990?}, {0x2c04240?, 0x3f751bc7e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9c1c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc57e008, 0x3f74373490) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4640 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4913 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc706400, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74449500) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4912 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4932 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc7063c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74449600) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4914 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4826 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca025b8, 0x3ad) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca025a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca02540, {0x3ebd194000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc8fde60, {0x3f743ca820, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc8fde60}, {0x3f743ca820, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743ca820, 0x9, 0x3eaec63cc8?}, {0x2c04240?, 0x3ebc8fde60?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743ca7e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc3126c8, 0x3f7472c4d0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4921 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5063 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc04fcc0, {0x2c3c1f0, 0x3f763eb090}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5019 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcab3020, {0x2c3c1f0, 0x3ebd3de8c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 2199 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4978 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74afaff0, {0x2c3c1f0, 0x3f75f8a230}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4567 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc872140, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c2400) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4566 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4656 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a21320, {0x2c3c1f0, 0x3f763ea7d0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4524 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5080 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74e0f278, 0x3b0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74e0f268) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74e0f200, {0x3ebf8ec000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebd3d2ba0, {0x3ebca9cd60, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebd3d2ba0}, {0x3ebca9cd60, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9cd60, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebd3d2ba0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9cd20) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5aa340, {0x2c3c1b8, 0x3ebc8fa060}, 0x3ebc8fa090) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74db6e00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5aa340}, {0x2c48448?, 0x3f746eaec0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5077 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4646 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76121d50, {0x2c3c1f0, 0x3f763ea500}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4856 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f740329c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4854 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4635 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc6a02c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74afc680) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4289 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4642 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74935b20) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4643 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4644 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27e200, {0x2c308c8, 0x3f750cd840}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 2202 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74700560, {0x2c3c1f0, 0x3ebcbfe0a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4568 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74d8a680) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4566 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4655 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a212f0, {0x2c3c1f0, 0x3f763ea780}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5349 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75636610, {0x2c3c1f0, 0x3ebc9b3ef0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4940 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f75690040, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a52700) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4939 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4494 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75236650, {0x2c3c1f0, 0x3ebca920f0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4283 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74cb5b20) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4576 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc882378, 0x3ad) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc882368) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc882300, {0x3ebc2a0000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f74e1e960, {0x3f74a983c0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f74e1e960}, {0x3f74a983c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a983c0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f74e1e960?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a98380) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74d8a820, {0x2c3c1b8, 0x3ebc5dc990}, 0x3ebc5dc9c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27e000, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74d8a820}, {0x2c48448?, 0x3ebc00cdc0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4573 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4591 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a3050, {0x2c3c1f0, 0x3ebcd3efa0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4996 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f765c7700, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8c780) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4995 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4658 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5172 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74952d90) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4860 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f76017e80, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74449200) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4776 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4537 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74340678, 0x3a1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74340668) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74340600, {0x3ebc8d2000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f74f52a20, {0x3ebc2b4040, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f74f52a20}, {0x3ebc2b4040, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b4040, 0x9, 0x3eae5b88b8?}, {0x2c04240?, 0x3f74f52a20?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b4000) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f746b7688, 0x3f7491cbd0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4594 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4615 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f768de750, {0x2c3c1f0, 0x3f749923c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4518 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc620000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4516 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4520 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e9ab40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f743e8980) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4474 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4517 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e9ab80, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c3d00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4516 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4882 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc882bb8, 0x3b6) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc882ba8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc882b40, {0x3ebd0bc000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc450420, {0x3ebc12aba0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc450420}, {0x3ebc12aba0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12aba0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc450420?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12ab60) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74230d00, {0x2c3c1b8, 0x3ebc460060}, 0x3ebc460090) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc0e4200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74230d00}, {0x2c48448?, 0x3f75b005a0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4783 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5084 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f75e97a80, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f743e9a00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5212 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4926 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74871eb0, {0x2c3c1f0, 0x3ebca93630}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4091 [select]: github.com/grafana/dskit/ring.(*Lifecycler).loop(0x3f7437b688, {0x2c3c1f0, 0x3f7436d0e0}) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/ring/lifecycler.go:540 +0x188 github.com/grafana/dskit/services.(*BasicService).main(0x3ebc7a48c0) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:193 +0x17c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 4088 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:122 +0xf0 goroutine 4846 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f746edea0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4925 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74871e80, {0x2c3c1f0, 0x3ebca935e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5196 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75514f60, {0x2c3c1f0, 0x3f743c8c80}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4556 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74c4e340) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4554 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4284 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5069 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75e9ac30, {0x2c3c1f0, 0x3f763eb2c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5221 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fa780, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c2e00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5178 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4838 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc5c2400, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f743e8b80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4802 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4810 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f741cc1c0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4582 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4876 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74819b20) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4505 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f750d24f8, 0x3b5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f750d24e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f750d2480, {0x3ebcb0a000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75f97c20, {0x3ebc12a2e0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75f97c20}, {0x3ebc12a2e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12a2e0, 0x9, 0x3eae8b5f98?}, {0x2c04240?, 0x3f75f97c20?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12a2a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74849208, 0x3f748aad90) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4552 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 2311 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444cd38, 0x3b5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444cd28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444ccc0, {0x3ebca8a000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f74a5a060, {0x3ebca9c040, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f74a5a060}, {0x3ebca9c040, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9c040, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f74a5a060?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9c000) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebcaa2000, {0x2c3c1b8, 0x3ebca801b0}, 0x3ebca801e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc475800, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebcaa2000}, {0x2c48448?, 0x3f74985220?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 2308 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4078 [select, 8 minutes]: github.com/prometheus/prometheus/tsdb/wlog.(*WL).run(0x3f74471a20) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/tsdb/wlog/wlog.go:416 +0x94 created by github.com/prometheus/prometheus/tsdb/wlog.NewSize in goroutine 4278 /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/tsdb/wlog/wlog.go:375 +0x5f0 goroutine 4961 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f765c8ac0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f743e8800) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4960 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4110 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4077 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4862 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743c19f8, 0x3b4) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743c19e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f743c1980, {0x3ebcf80000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc450240, {0x3ebc12a9e0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc450240}, {0x3ebc12a9e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12a9e0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc450240?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12a9a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74230680, {0x2c3c1b8, 0x3ebce9b6e0}, 0x3ebce9b710) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f749a7200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74230680}, {0x2c48448?, 0x3f758931a0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4776 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5156 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444cf78, 0x3c1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444cf68) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444cf00, {0x3f74d52000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc174000, {0x3f74a98e40, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc174000}, {0x3f74a98e40, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a98e40, 0x9, 0x3eaf0467e0?}, {0x2c04240?, 0x3ebc174000?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a98e00) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc390d88, 0x3f7498c150) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5202 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4090 [select]: github.com/grafana/loki/v3/pkg/ingester.(*Ingester).loop(0x3f74a4a008) /build/loki/src/loki-3.5.1/pkg/ingester/ingester.go:788 +0x2dc created by github.com/grafana/loki/v3/pkg/ingester.(*Ingester).starting in goroutine 4088 /build/loki/src/loki-3.5.1/pkg/ingester/ingester.go:628 +0xd54 goroutine 4857 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74e0ec78, 0x3b6) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74e0ec68) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74e0ec00, {0x3ebcdfa000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc6235c0, {0x3ebc2b49e0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc6235c0}, {0x3ebc2b49e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b49e0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc6235c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b49a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f740329c0, {0x2c3c1b8, 0x3ebccd0ed0}, 0x3ebccd0f00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27e800, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f740329c0}, {0x2c48448?, 0x3f74c019c0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4854 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4711 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74502a00, {0x2c308c8, 0x3f75e01860}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4759 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc9fca40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74448a00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4758 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4493 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f743e6400, {0x2c308c8, 0x3ebc660de0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4877 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5042 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f768de080, {0x2c3c1f0, 0x3ebd3dedc0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4467 [runnable]: bytes.(*Buffer).WriteString(0x3f76ac90d8, {0x2663a17, 0x3}) /usr/lib/go/src/bytes/buffer.go:187 +0x18 github.com/prometheus/prometheus/model/labels.Labels.String.func1(...) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels_common.go:57 github.com/prometheus/prometheus/model/labels.Labels.Range(...) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:333 github.com/prometheus/prometheus/model/labels.Labels.String({0x3ebc5c4f80, 0x2, 0x15a0b4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels_common.go:49 +0x18c github.com/grafana/loki/v3/pkg/ingester.buildPushRequest(0x166, {0x3f74bf0a88, 0x14, 0x0?}) /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1459 +0x9c github.com/grafana/loki/v3/pkg/ingester.Test_DedupeIngester(0x3f7670a700) /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1021 +0x86c testing.tRunner(0x3f7670a700, 0x2773a60) /usr/lib/go/src/testing/testing.go:1792 +0xf8 created by testing.(*T).Run in goroutine 1 /usr/lib/go/src/testing/testing.go:1851 +0x3a0 goroutine 4277 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4448 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 2985 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36a600, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75828900) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4608 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4579 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc882438, 0x3b4) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc882428) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc8823c0, {0x3ebc280000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75a091a0, {0x3ebc12a4a0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75a091a0}, {0x3ebc12a4a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12a4a0, 0x9, 0x3eaec63bd8?}, {0x2c04240?, 0x3f75a091a0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12a460) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74849448, 0x3f74979d50) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4522 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4952 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca02678, 0x3b1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca02668) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca02600, {0x3ebd24c000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc450d20, {0x3ebc12ad60, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc450d20}, {0x3ebc12ad60, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12ad60, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc450d20?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12ad20) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f742311e0, {0x2c3c1b8, 0x3ebd311ec0}, 0x3ebd311ef0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f749a7a00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f742311e0}, {0x2c48448?, 0x3f746eb4c0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4890 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4822 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f744816c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4820 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4821 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74dbd640, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a52180) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4820 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4824 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74dbd600, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a52480) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4651 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4455 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f749229a0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4454 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4595 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7493e850) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4465 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc474000, {0x2c308c8, 0x3ebc00cc20}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4464 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4463 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74979730) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4637 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f750d2bb8, 0x3bd) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f750d2ba8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f750d2b40, {0x3ebc554000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75891da0, {0x3ebc4a84a0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75891da0}, {0x3ebc4a84a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc4a84a0, 0x9, 0x3eae5b8930?}, {0x2c04240?, 0x3f75891da0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc4a8460) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc8946c8, 0x3f74923ea0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4648 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4461 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74d8c920, {0x2c3c1f0, 0x3ebcd3e0a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4460 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74d8c8f0, {0x2c3c1f0, 0x3ebcd3e050}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4459 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74d8c8c0, {0x2c3c1f0, 0x3ebcd3e000}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4458 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27ee00, {0x2c308c8, 0x3f77111f00}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4456 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4454 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4487 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc6a0480, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74448380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4639 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4488 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74c4e680) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4639 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4489 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f750d2af8, 0x3bb) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f750d2ae8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f750d2a80, {0x3ebc564000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f750c8180, {0x3ebc4a8580, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f750c8180}, {0x3ebc4a8580, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc4a8580, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f750c8180?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc4a8540) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74c4e680, {0x2c3c1b8, 0x3ebc4900c0}, 0x3ebc4900f0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27e200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74c4e680}, {0x2c48448?, 0x3f74985460?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4639 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5083 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca02bb8, 0x3bc) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca02ba8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca02b40, {0x3ec18c4000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebd3d3320, {0x3ebca9cf20, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebd3d3320}, {0x3ebca9cf20, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9cf20, 0x9, 0x3eaec63da0?}, {0x2c04240?, 0x3ebd3d3320?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9cee0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc312fc8, 0x3f748afdc0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5185 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4994 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27ea00, {0x2c308c8, 0x3f74c01e80}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4897 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4896 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f748efce0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5121 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f750d3cf8, 0x3b0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f750d3ce8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f750d3c80, {0x3ec1ba4000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebfbcf440, {0x3ebc0909e0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebfbcf440}, {0x3ebc0909e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc0909e0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebfbcf440?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc0909a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74d8aea0, {0x2c3c1b8, 0x3ec1ba2720}, 0x3ec1ba2750) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27f200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74d8aea0}, {0x2c48448?, 0x3f752120a0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5118 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4879 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75637830, {0x2c3c1f0, 0x3ebca930e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4666 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444cc78, 0x3da) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444cc68) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444cc00, {0x3ebc11c000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f758096e0, {0x3f74a98ac0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f758096e0}, {0x3f74a98ac0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a98ac0, 0x9, 0x3eaed2ac30?}, {0x2c04240?, 0x3f758096e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a98a80) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc390008, 0x3f7492c690) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4895 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5416 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc620b60) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5414 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4588 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74885570) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5018 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74502800, {0x2c308c8, 0x3ebc65ddc0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5017 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5016 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f748e9c00) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5061 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc04fc60, {0x2c3c1f0, 0x3f763eaff0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5014 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcab2300, {0x2c3c1f0, 0x3ebd3de0a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5013 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcab22d0, {0x2c3c1f0, 0x3ebd3de050}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5011 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444cbb8, 0x3d7) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444cba8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444cb40, {0x3ebc266000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75809800, {0x3f74a98c80, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75809800}, {0x3f74a98c80, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a98c80, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f75809800?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a98c40) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc620340, {0x2c3c1b8, 0x3ebc5021b0}, 0x3ebc5021e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebcc94200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc620340}, {0x2c48448?, 0x3f75b00100?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4669 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5010 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fa1c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74d92380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4895 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 2988 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7495e000, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75828a00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4801 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4607 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74d8a9c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4604 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4606 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f75690080, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f743e8200) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4604 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4716 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc882af8, 0x3bb) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc882ae8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc882a80, {0x3ebcc5e000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f747cdb00, {0x3ebc12a660, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f747cdb00}, {0x3ebc12a660, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12a660, 0x9, 0x3eaec63c50?}, {0x2c04240?, 0x3f747cdb00?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12a620) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f746b78c8, 0x3f7494db20) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4708 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4778 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc882c78, 0x3b9) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc882c68) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc882c00, {0x3ebd0ac000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc450300, {0x3ebc12aac0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc450300}, {0x3ebc12aac0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12aac0, 0x9, 0x3eaf0fe558?}, {0x2c04240?, 0x3ebc450300?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12aa80) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f746b7b08, 0x3f748e85b0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4868 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4771 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f76017cc0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74d92900) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4708 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4720 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f76017d00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74d92800) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4719 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4721 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74230340) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4719 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4770 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc882a38, 0x3bf) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc882a28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc8829c0, {0x3ebcc70000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f747cdb60, {0x3ebc12a900, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f747cdb60}, {0x3ebc12a900, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12a900, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f747cdb60?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12a8c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74230340, {0x2c3c1b8, 0x3ebcd1f830}, 0x3ebcd1f860) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebcc94000, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74230340}, {0x2c48448?, 0x3f75b001c0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4719 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4107 [select]: github.com/grafana/loki/v3/pkg/ingester.New.newRecalculateOwnedStreamsSvc.NewTimerService.func4({0x2c3c1f0, 0x3ebcd3e280}) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/services.go:31 +0xd4 github.com/grafana/dskit/services.(*BasicService).main(0x3f749c5ae0) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:193 +0x17c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 4115 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:122 +0xf0 goroutine 2981 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f750d2eb8, 0x3b2) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f750d2ea8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f750d2e40, {0x3ebc1ec000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f758098c0, {0x3ebc4a8820, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f758098c0}, {0x3ebc4a8820, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc4a8820, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f758098c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc4a87e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74d8a9c0, {0x2c3c1b8, 0x3f74bdcea0}, 0x3f74bddb90) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc0e4000, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74d8a9c0}, {0x2c48448?, 0x3ebc65c360?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4604 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4617 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f768de7b0, {0x2c3c1f0, 0x3f74992460}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4616 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f768de780, {0x2c3c1f0, 0x3f74992410}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4545 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743404f8, 0x3a4) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743404e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74340480, {0x3ebc8e2000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f74f52ba0, {0x3ebc2b4120, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f74f52ba0}, {0x3ebc2b4120, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b4120, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f74f52ba0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b40e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebcaa21a0, {0x2c3c1b8, 0x3ebc8b0720}, 0x3ebc8b0750) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f749a6000, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebcaa21a0}, {0x2c48448?, 0x3f74e0c2a0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4542 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4544 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebcaa21a0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4542 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4543 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f75e96080, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8c180) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4542 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4619 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7569a040, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8c700) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4580 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5309 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f752374a0, {0x2c3c1f0, 0x3f7436ccd0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4955 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca02438, 0x3b6) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca02428) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca023c0, {0x3ebd120000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebd3d24e0, {0x3ebca9c4a0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebd3d24e0}, {0x3ebca9c4a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9c4a0, 0x9, 0x3eaf1ec6c0?}, {0x2c04240?, 0x3ebd3d24e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9c460) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc312488, 0x3f748aea10) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5064 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4724 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca022b8, 0x3ba) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca022a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca02240, {0x3ebc31c000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75998180, {0x3f743ca3c0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75998180}, {0x3f743ca3c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743ca3c0, 0x9, 0x3eae5b89d8?}, {0x2c04240?, 0x3f75998180?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743ca380) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc312008, 0x3f740204d0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4580 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5301 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc474800, {0x2c308c8, 0x3ec18f28a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4950 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f765c6180, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a52a00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4890 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4089 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75e972d0, 0x1391) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75e972c0) /usr/lib/go/src/sync/cond.go:71 +0xa8 github.com/grafana/loki/v3/pkg/util.(*PriorityQueue).Dequeue(0x3f7436d090) /build/loki/src/loki-3.5.1/pkg/util/priority_queue.go:115 +0x94 github.com/grafana/loki/v3/pkg/ingester.(*Ingester).flushLoop(0x3f74a4a008, 0x0) /build/loki/src/loki-3.5.1/pkg/ingester/flush.go:226 +0x114 created by github.com/grafana/loki/v3/pkg/ingester.(*Ingester).InitFlushQueues in goroutine 4088 /build/loki/src/loki-3.5.1/pkg/ingester/flush.go:109 +0x54 goroutine 4871 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f749a7400, {0x2c308c8, 0x3ebc67fa60}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4867 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75237a90, {0x2c3c1f0, 0x3ebca92c80}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4872 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75636a60, {0x2c3c1f0, 0x3ebca92e60}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4873 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75636a90, {0x2c3c1f0, 0x3ebca92eb0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4086 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f743dea80) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4108 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4558 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc872080, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74afc580) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4462 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4553 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f743df650) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4551 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc04f230, {0x2c3c1f0, 0x3ebc469680}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4550 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc04f200, {0x2c3c1f0, 0x3ebc469630}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5089 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f75e97f00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f743e9c00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5286 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4714 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74882010, {0x2c3c1f0, 0x3ebcd3e820}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5523 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc620ea0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5057 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5524 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444d038, 0x3b3) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444d028) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444cfc0, {0x3eb8350000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f751b41e0, {0x3ebc2b5ee0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f751b41e0}, {0x3ebc2b5ee0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b5ee0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f751b41e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b5ea0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc620ea0, {0x2c3c1b8, 0x3f751186f0}, 0x3f75118720) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74503000, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc620ea0}, {0x2c48448?, 0x3f75b00240?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5057 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4817 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74e0ed38, 0x3b5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74e0ed28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74e0ecc0, {0x3ebcdea000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc623500, {0x3ebc2b4900, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc623500}, {0x3ebc2b4900, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b4900, 0x9, 0x3eae5b8ae0?}, {0x2c04240?, 0x3ebc623500?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b48c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc390488, 0x3f741ccbd0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4845 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4949 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca02738, 0x3af) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca02728) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca026c0, {0x3ebd312000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebd19e060, {0x3f743ca9e0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebd19e060}, {0x3f743ca9e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743ca9e0, 0x9, 0x3eae5b9e00?}, {0x2c04240?, 0x3ebd19e060?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743ca9a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc312908, 0x3f7472cfc0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4928 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4281 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76120110, {0x2c3c1f0, 0x3f763ea050}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4473 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f768df5b0, {0x2c3c1f0, 0x3ebc469180}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4841 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e167c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f743e8f00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4790 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5415 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fabc0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8cf80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5414 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5304 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7495bd00, {0x2c3c1f0, 0x3f7436ca00}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4866 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75237a60, {0x2c3c1f0, 0x3ebca92c30}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4937 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f765c6140, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74449700) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4967 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5302 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7495bc90, {0x2c3c1f0, 0x3f7436c640}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5306 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f749b7ab0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 3560 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4457 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4626 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f750d2a38, 0x3b5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f750d2a28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f750d29c0, {0x3ebc480000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75949c20, {0x3ebc4a82e0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75949c20}, {0x3ebc4a82e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc4a82e0, 0x9, 0x3eaeddf230?}, {0x2c04240?, 0x3f75949c20?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc4a82a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc894488, 0x3f74923570) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4289 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5303 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7495bcd0, {0x2c3c1f0, 0x3f7436c9b0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4528 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a20f50, {0x2c3c1f0, 0x3f74c06230}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4527 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a20f20, {0x2c3c1f0, 0x3f74c061e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4507 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75512138, 0x3ab) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75512128) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f755120c0, {0x3ebc0e6000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f74e1e2a0, {0x3f74a98040, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f74e1e2a0}, {0x3f74a98040, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a98040, 0x9, 0x3eaec63b78?}, {0x2c04240?, 0x3f74e1e2a0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a98000) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f747b26c8, 0x3f743dfb90) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4462 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4526 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a20ef0, {0x2c3c1f0, 0x3f74c06190}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4525 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f749a6000, {0x2c308c8, 0x3f75e98480}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 2984 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc706080, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74449300) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4791 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 4869 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74818f50) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4605 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f750d2f78, 0x3ac) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f750d2f68) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f750d2f00, {0x3ebc100000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75809860, {0x3ebc4a8040, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75809860}, {0x3ebc4a8040, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc4a8040, 0x9, 0x3eaee27ec0?}, {0x2c04240?, 0x3f75809860?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc4a8000) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc894248, 0x3f7493f030) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4618 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4676 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a21290, {0x2c3c1f0, 0x3f74c06410}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4675 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a21260, {0x2c3c1f0, 0x3f74c063c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4476 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4468 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7492dce0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4574 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc8722c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c2600) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4573 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4470 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc475400, {0x2c308c8, 0x3f768f5ca0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4119 [select]: github.com/grafana/loki/v3/pkg/util.(*TickerWithJitter).start(0x3ebc67e1c0, 0x5f5e100, 0x1312d00) /build/loki/src/loki-3.5.1/pkg/util/ticker.go:41 +0x114 created by github.com/grafana/loki/v3/pkg/util.NewTickerWithJitter in goroutine 4090 /build/loki/src/loki-3.5.1/pkg/util/ticker.go:63 +0x114 goroutine 3823 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743c1278, 0x3bf) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743c1268) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f743c1200, {0x3ebc62e000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75a08ea0, {0x3ebc7a83c0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75a08ea0}, {0x3ebc7a83c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a83c0, 0x9, 0x3eaee27668?}, {0x2c04240?, 0x3f75a08ea0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a8380) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f746b7d48, 0x3f7491cee0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4474 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4469 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 4416 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7492ce70) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4278 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4092 [select]: github.com/grafana/loki/v3/pkg/ingester.New.newRecalculateOwnedStreamsSvc.NewTimerService.func4({0x2c3c1f0, 0x3f7436d310}) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/services.go:31 +0xd4 github.com/grafana/dskit/services.(*BasicService).main(0x3ebc7a4a00) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:193 +0x17c created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 4088 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:122 +0xf0 goroutine 4529 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f749a6200, {0x2c308c8, 0x3f74985ee0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4475 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f746fed90) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4835 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc5c2440, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f743e8a80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4834 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 4930 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74032d00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4912 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4931 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74e0ef78, 0x3b4) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74e0ef68) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74e0ef00, {0x3ebcef6000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebceda240, {0x3ebc2b4e40, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebceda240}, {0x3ebc2b4e40, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b4e40, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebceda240?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b4e00) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74032d00, {0x2c3c1b8, 0x3ebcedc330}, 0x3ebcedc360) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f749a7600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74032d00}, {0x2c48448?, 0x3f74c01ca0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4912 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 4971 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75515ab0, {0x2c3c1f0, 0x3ebca93ae0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4972 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75515ae0, {0x2c3c1f0, 0x3ebca93b30}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 4973 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75515b10, {0x2c3c1f0, 0x3ebca93b80}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5060 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74db6c00, {0x2c308c8, 0x3ebc1804a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 4975 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75512eb8, 0x3ac) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75512ea8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75512e40, {0x3ebc608000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebfbce600, {0x3ebc090200, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebfbce600}, {0x3ebc090200, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc090200, 0x9, 0x3eaed2b278?}, {0x2c04240?, 0x3ebfbce600?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc0901c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebd244248, 0x3f7401d7a0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5022 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 4752 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36a040, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74afc180) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5022 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5132 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac13f8, 0x3b9) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac13e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac1380, {0x3ec1100000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc4516e0, {0x3f743cae40, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc4516e0}, {0x3f743cae40, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743cae40, 0x9, 0x3eaec63d58?}, {0x2c04240?, 0x3ebc4516e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743cae00) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f747b38c8, 0x3f74895dc0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5147 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5058 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7472df10) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 4945 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac04f8, 0x3b7) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac04e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac0480, {0x3ec0c12000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc4511a0, {0x3f743caba0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc4511a0}, {0x3f743caba0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743caba0, 0x9, 0x3eaf0467c8?}, {0x2c04240?, 0x3ebc4511a0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743cab60) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f747b3448, 0x3f74895490) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5101 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5029 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36a080, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74afc280) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5028 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5030 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74d8ab60) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5028 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5031 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75512df8, 0x3af) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75512de8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75512d80, {0x3ebc844000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebfbce720, {0x3ebc090660, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebfbce720}, {0x3ebc090660, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc090660, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebfbce720?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc090620) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74d8ab60, {0x2c3c1b8, 0x3ebc4c6720}, 0x3ebc4c6750) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74502800, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74d8ab60}, {0x2c48448?, 0x3f75892060?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5028 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5033 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75513038, 0x3ca) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75513028) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75512fc0, {0x3ebc8b2000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebfbce7e0, {0x3ebc090740, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebfbce7e0}, {0x3ebc090740, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc090740, 0x9, 0x3eaec63d10?}, {0x2c04240?, 0x3ebfbce7e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc090700) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebd244488, 0x3f7494e7e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 4753 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5113 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444d1b8, 0x3b2) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444d1a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444d140, {0x3ec09d6000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebfbcede0, {0x3ebc090900, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebfbcede0}, {0x3ebc090900, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc090900, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebfbcede0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc0908c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74d8ad00, {0x2c3c1b8, 0x3ec09d5c50}, 0x3ec09d5c80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27ec00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74d8ad00}, {0x2c48448?, 0x3f75b01080?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5110 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5039 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f755131b8, 0x3a8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f755131a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75513140, {0x3ebfb90000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebfbcea80, {0x3ebc090820, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebfbcea80}, {0x3ebc090820, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc090820, 0x9, 0x3eaf0feba0?}, {0x2c04240?, 0x3ebfbcea80?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc0907e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebd2446c8, 0x3f7494ebd0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5094 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5527 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444dab8, 0x3bd) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444daa8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444da40, {0x3eb8200000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f751b4540, {0x3ebc7a8c80, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f751b4540}, {0x3ebc7a8c80, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a8c80, 0x9, 0x3eaed2bd40?}, {0x2c04240?, 0x3f751b4540?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a8c40) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc57f448, 0x3f74a47dc0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5544 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5112 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74d8ad00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5110 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 4942 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75512f78, 0x3c9) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75512f68) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75512f00, {0x3ebca10000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc450a80, {0x3f743ca740, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc450a80}, {0x3f743ca740, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743ca740, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc450a80?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743ca700) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74c4e820, {0x2c3c1b8, 0x3ebcee7800}, 0x3ebcee7830) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74502e00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74c4e820}, {0x2c48448?, 0x3f75892220?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4939 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5004 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f765c7980, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74d93700) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5003 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5093 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75e9b9d0, {0x2c3c1f0, 0x3f763eb590}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5050 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f768debe0, {0x2c3c1f0, 0x3ebd3df2c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5108 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f765c7ac0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c2c00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5140 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5120 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74d8aea0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5118 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5095 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74979ab0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5096 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5097 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74db7400, {0x2c308c8, 0x3f74b01e40}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5098 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76398d90, {0x2c3c1f0, 0x3f763eb770}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5099 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76398dc0, {0x2c3c1f0, 0x3f763eb7c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5100 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76398df0, {0x2c3c1f0, 0x3f763eb810}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5005 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74033040) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5003 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5006 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f755130f8, 0x3ab) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f755130e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75513080, {0x3ec08f0000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebcc871a0, {0x3ebc2b5380, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebcc871a0}, {0x3ebc2b5380, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b5380, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebcc871a0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b5340) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74033040, {0x2c3c1b8, 0x3ebd2f8900}, 0x3ebd2f8930) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74db7000, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74033040}, {0x2c48448?, 0x3f75892400?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5003 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5191 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac1578, 0x3ba) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac1568) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac1500, {0x3ec1120000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc451980, {0x3f743cb000, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc451980}, {0x3f743cb000, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743cb000, 0x9, 0x3eaf1ec7b0?}, {0x2c04240?, 0x3ebc451980?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743cafc0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f747b3b08, 0x3f7499e310) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5171 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5048 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f768deb80, {0x2c3c1f0, 0x3ebd3df220}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5047 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74503000, {0x2c308c8, 0x3ebc6c5640}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5049 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f768debb0, {0x2c3c1f0, 0x3ebd3df270}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5130 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f75690280, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74449a80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5101 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5127 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f756902c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74d93800) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5126 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5128 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74c4eb60) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5126 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5129 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac0438, 0x3b3) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac0428) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac03c0, {0x3ec0c80000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc4512c0, {0x3f743cac80, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc4512c0}, {0x3f743cac80, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743cac80, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc4512c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743cac40) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74c4eb60, {0x2c3c1b8, 0x3ebcee7ec0}, 0x3ebcee7ef0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74db7400, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74c4eb60}, {0x2c48448?, 0x3f771108c0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5126 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5102 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74373880) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5103 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5104 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74db7600, {0x2c308c8, 0x3ebc900c00}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5105 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76399b60, {0x2c3c1f0, 0x3f763eb9f0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5138 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76399b90, {0x2c3c1f0, 0x3f763eba40}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5139 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76399bc0, {0x2c3c1f0, 0x3f763eba90}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5109 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74e0f578, 0x3d0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74e0f568) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74e0f500, {0x3ec0e90000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75809ce0, {0x3f74a98d60, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75809ce0}, {0x3f74a98d60, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a98d60, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f75809ce0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a98d20) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc6204e0, {0x2c3c1b8, 0x3ec09bd5f0}, 0x3ec09bd620) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74db7600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc6204e0}, {0x2c48448?, 0x3f746eb360?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 4672 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5008 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74e0f638, 0x3c9) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74e0f628) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74e0f5c0, {0x3ec0916000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebcc874a0, {0x3ebc2b5460, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebcc874a0}, {0x3ebc2b5460, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b5460, 0x9, 0x3eaf0ff5c0?}, {0x2c04240?, 0x3ebcc874a0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b5420) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc57eb48, 0x3f748fda40) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5140 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5009 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fa3c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74449b80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4672 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5170 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc6204e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 4672 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5141 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74950a80) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5142 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5143 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74db7800, {0x2c308c8, 0x3ebc901860}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5144 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74afbc60, {0x2c3c1f0, 0x3f763ebc70}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5145 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74afbc90, {0x2c3c1f0, 0x3f763ebcc0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5146 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74afbcc0, {0x2c3c1f0, 0x3f763ebd10}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5197 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75514f90, {0x2c3c1f0, 0x3f743c8cd0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5174 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27ec00, {0x2c308c8, 0x3ebc67e1e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5173 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5175 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a2590, {0x2c3c1f0, 0x3ebcbff270}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5189 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f75690480, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74449d80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5147 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5186 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f756904c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74449c80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5137 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5187 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74c4ed00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5137 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5188 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac1338, 0x3bd) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac1328) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac12c0, {0x3ec1110000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc4517a0, {0x3f743caf20, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc4517a0}, {0x3f743caf20, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743caf20, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc4517a0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743caee0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74c4ed00, {0x2c3c1b8, 0x3ebcee86c0}, 0x3ebcee86f0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74db7800, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74c4ed00}, {0x2c48448?, 0x3f77110d00?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5137 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5148 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74951c70) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5149 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5150 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74db7a00, {0x2c308c8, 0x3f75e018c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5151 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcd1cc70, {0x2c3c1f0, 0x3f763ebef0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5152 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcd1cca0, {0x2c3c1f0, 0x3ebc9b2000}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5153 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcd1ccd0, {0x2c3c1f0, 0x3ebc9b2190}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5219 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444d278, 0x3af) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444d268) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444d200, {0x3ec1880000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc174900, {0x3f74a990e0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc174900}, {0x3f74a990e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a990e0, 0x9, 0x3eaf1ec9d8?}, {0x2c04240?, 0x3ebc174900?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a990a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc391208, 0x3f7498d1f0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5178 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5203 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74954850) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5204 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5205 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74db7c00, {0x2c308c8, 0x3f74e0d500}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5206 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcd1dad0, {0x2c3c1f0, 0x3ebc9b27d0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5207 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcd1db00, {0x2c3c1f0, 0x3ebc9b2af0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5208 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcd1db30, {0x2c3c1f0, 0x3ebc9b2b40}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5195 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f75691100, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a53180) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5171 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5192 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f75e97a00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8cb80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5209 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5193 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd19cea0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5209 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5194 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac14b8, 0x3bc) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac14a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac1440, {0x3ec11d6000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec11a11a0, {0x3ebc12b620, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec11a11a0}, {0x3ebc12b620, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12b620, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ec11a11a0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12b5e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd19cea0, {0x2c3c1b8, 0x3ebcee8a50}, 0x3ebcee8a80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74db7c00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd19cea0}, {0x2c48448?, 0x3f77110ea0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5209 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5176 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a25c0, {0x2c3c1f0, 0x3ebcbff2c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5177 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a2610, {0x2c3c1f0, 0x3ebcbff310}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5227 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444d3f8, 0x3bc) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444d3e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444d380, {0x3ec2246000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc174b40, {0x3f74a991c0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc174b40}, {0x3f74a991c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a991c0, 0x9, 0x3eaf1ec9f0?}, {0x2c04240?, 0x3ebc174b40?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a99180) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc391448, 0x3f7498d7a0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5298 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5087 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f765c9740, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f743e9b00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5185 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5255 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75513278, 0x3b5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75513268) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75513200, {0x3ec2a32000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec25d79e0, {0x3ebca9d460, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec25d79e0}, {0x3ebca9d460, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9d460, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ec25d79e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9d420) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5aab60, {0x2c3c1b8, 0x3ec2156150}, 0x3ec2156180) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f743e7400, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5aab60}, {0x2c48448?, 0x3f75893040?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5338 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5179 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74953b20) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5180 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5181 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27f000, {0x2c308c8, 0x3f76018880}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5182 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a3e70, {0x2c3c1f0, 0x3ebcbff4f0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5183 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a3ea0, {0x2c3c1f0, 0x3ebcbff540}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5184 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a3ed0, {0x2c3c1f0, 0x3ebcbff590}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5119 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36ac00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c2f00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5118 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5250 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f75e97b00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c3000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5240 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5392 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec1ee8a80, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0e600) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5396 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5234 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f749a8930) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5235 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5236 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27f200, {0x2c308c8, 0x3f74a8bc20}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5237 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a21560, {0x2c3c1f0, 0x3ebcbff720}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5238 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a21600, {0x2c3c1f0, 0x3ebcbff770}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5239 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a21640, {0x2c3c1f0, 0x3ebcbff7c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5254 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36af00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c3200) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5352 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5214 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f750d3db8, 0x3b0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f750d3da8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f750d3d40, {0x3f74dd6000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec11a1440, {0x3ebc12b7e0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec11a1440}, {0x3ebc12b7e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12b7e0, 0x9, 0x3eaf046828?}, {0x2c04240?, 0x3ec11a1440?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12b7a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f748498c8, 0x3f74955420) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5240 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5348 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f743e7400, {0x2c308c8, 0x3f752130a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5266 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743c1cf8, 0x3bc) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743c1ce8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f743c1c80, {0x3ec1e5e000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec11a1800, {0x3ebc12b9a0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec11a1800}, {0x3ebc12b9a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12b9a0, 0x9, 0x3eaf40d068?}, {0x2c04240?, 0x3ec11a1800?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12b960) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74849d48, 0x3f749558f0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5247 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5241 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f749a95e0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5242 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5243 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27f400, {0x2c308c8, 0x3f75820ac0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5244 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7569eb80, {0x2c3c1f0, 0x3ebcbff9a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5245 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7569ebc0, {0x2c3c1f0, 0x3ebcbff9f0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5246 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7569ebf0, {0x2c3c1f0, 0x3ebcbffa40}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5275 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743c1e78, 0x3c2) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743c1e68) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f743c1e00, {0x3ec2198000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec11a1da0, {0x3ebc12bb60, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec11a1da0}, {0x3ebc12bb60, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12bb60, 0x9, 0x3eaf40d6c8?}, {0x2c04240?, 0x3ec11a1da0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12bb20) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ec21a8008, 0x3f749c2150) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5286 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5299 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f749b6ee0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5273 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f75e97cc0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74afd600) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5247 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5270 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f75e97d00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74afd500) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5269 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5271 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd19d1e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5269 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5272 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743c1c38, 0x3be) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743c1c28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f743c1bc0, {0x3ec1e6e000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec11a1920, {0x3ebc12ba80, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec11a1920}, {0x3ebc12ba80, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12ba80, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ec11a1920?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12ba40) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd19d1e0, {0x2c3c1b8, 0x3f74dd5200}, 0x3f74dd5230) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27f400, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd19d1e0}, {0x2c48448?, 0x3f75212280?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5269 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5248 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f749bc2a0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5249 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5282 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27f600, {0x2c308c8, 0x3f75821700}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5283 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7569fc60, {0x2c3c1f0, 0x3ebcbffc20}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5284 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7569fca0, {0x2c3c1f0, 0x3ebcbffc70}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5285 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7569fce0, {0x2c3c1f0, 0x3ebcbffcc0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5329 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444d6f8, 0x3ba) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444d6e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444d680, {0x3ec2aaa000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc175440, {0x3f74a99540, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc175440}, {0x3f74a99540, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a99540, 0x9, 0x3eaec63e78?}, {0x2c04240?, 0x3ebc175440?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a99500) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc3918c8, 0x3f74a46a10) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5403 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5278 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f765c98c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f743e9d00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5088 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5279 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5aa4e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5088 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5280 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f743c1db8, 0x3c3) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f743c1da8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f743c1d40, {0x3ec18d4000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebd3d3740, {0x3ebca9d000, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebd3d3740}, {0x3ebca9d000, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9d000, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebd3d3740?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9cfc0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5aa4e0, {0x2c3c1b8, 0x3ec2197a70}, 0x3ec2197aa0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27f600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5aa4e0}, {0x2c48448?, 0x3f75212400?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5088 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5287 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f749bd2d0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5288 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5289 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27f800, {0x2c308c8, 0x3f775e4800}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5290 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f747013d0, {0x2c3c1f0, 0x3ebcbffea0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5291 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74701420, {0x2c3c1f0, 0x3ebcbffef0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5292 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74701450, {0x2c3c1f0, 0x3f75f8a000}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5311 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75237530, {0x2c3c1f0, 0x3f7436cd70}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5418 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fab80, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8d100) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5403 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5313 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca03038, 0x3b4) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca03028) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca02fc0, {0x3ec2a12000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec25d7560, {0x3ebca9d2a0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec25d7560}, {0x3ebca9d2a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9d2a0, 0x9, 0x3eaf514360?}, {0x2c04240?, 0x3ec25d7560?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9d260) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc313208, 0x3f74a50380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5281 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5327 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec2a040c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f759aed80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5281 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5420 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36b040, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c3400) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5198 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5423 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fadc0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75828e80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5461 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5384 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec1ee8800, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f759aef80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5359 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5335 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec2a04100, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75828280) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5334 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5336 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5aa9c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5334 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5337 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca02f78, 0x3ac) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca02f68) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca02f00, {0x3ec2a22000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec25d7680, {0x3ebca9d380, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec25d7680}, {0x3ebca9d380, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9d380, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ec25d7680?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9d340) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5aa9c0, {0x2c3c1b8, 0x3ec2a039e0}, 0x3ec2a03a10) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc474a00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5aa9c0}, {0x2c48448?, 0x3ec18f3720?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5334 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5350 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75636650, {0x2c3c1f0, 0x3ebc468050}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5351 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75636750, {0x2c3c1f0, 0x3ebc4680a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5541 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3abef00, {0x2c3c1f0, 0x3f75f8a8c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5386 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74c6e378, 0x3ac) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74c6e368) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74c6e300, {0x3ec3420000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec1ee7ce0, {0x3ebc2b5c40, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec1ee7ce0}, {0x3ebc2b5c40, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b5c40, 0x9, 0x3eaed2b2f0?}, {0x2c04240?, 0x3ec1ee7ce0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b5c00) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc57efc8, 0x3f74a7e5b0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5341 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5296 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74c6e1f8, 0x3b1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74c6e1e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74c6e180, {0x3ec3200000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec1ee7920, {0x3ebc2b5a80, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec1ee7920}, {0x3ebc2b5a80, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b5a80, 0x9, 0x3eaec63e30?}, {0x2c04240?, 0x3ec1ee7920?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b5a40) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc57ed88, 0x3f749bdd50) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5359 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5339 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec2a04200, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75828500) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5338 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5340 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5aab60) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5338 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5353 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f749c3880) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5354 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5355 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f743e7600, {0x2c308c8, 0x3ec2d424c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5356 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f756378f0, {0x2c3c1f0, 0x3ebc469c20}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5357 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75637920, {0x2c3c1f0, 0x3ebc469ef0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5358 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75637950, {0x2c3c1f0, 0x3f7446fae0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5376 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5360 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74a76620) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5361 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5362 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f743e7800, {0x2c308c8, 0x3ec2d43120}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5363 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74871c40, {0x2c3c1f0, 0x3f7446fcc0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5364 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74871c70, {0x2c3c1f0, 0x3f7446fd10}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5365 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74871ca0, {0x2c3c1f0, 0x3f7446fd60}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5342 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74a512d0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5343 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5375 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74960070) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5381 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec1ee8840, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f759aee80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5256 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5382 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74033ba0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5256 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5383 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74c6e138, 0x3a8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74c6e128) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74c6e0c0, {0x3ec3210000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec1ee79e0, {0x3ebc2b5b60, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec1ee79e0}, {0x3ebc2b5b60, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b5b60, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ec1ee79e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b5b20) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74033ba0, {0x2c3c1b8, 0x3ec24fa390}, 0x3ec24fa3c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f743e7600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74033ba0}, {0x2c48448?, 0x3f775e4ae0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5256 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5374 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec1ee8a00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0e400) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5341 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5371 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec2d6ac00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74afdf00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5370 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5372 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd19d860) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5370 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5373 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74c6e2b8, 0x3ab) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74c6e2a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74c6e240, {0x3ec2fd0000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec2fa4480, {0x3ebc7a8d60, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec2fa4480}, {0x3ebc7a8d60, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a8d60, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ec2fa4480?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a8d20) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd19d860, {0x2c3c1b8, 0x3ec2fa0b70}, 0x3ec2fa0ba0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f743e7800, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd19d860}, {0x2c48448?, 0x3f775e4c40?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5370 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5344 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc474c00, {0x2c308c8, 0x3ec2a662c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5345 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74cc89f0, {0x2c3c1f0, 0x3f7436d950}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5394 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74cc8a20, {0x2c3c1f0, 0x3f7436d9a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5395 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74cc8a50, {0x2c3c1f0, 0x3f7436dbd0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5432 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd19c1a0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5430 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5433 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebfbc02b8, 0x3c0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebfbc02a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebfbc0240, {0x3eb8a40000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75007800, {0x3ebca9d8c0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75007800}, {0x3ebca9d8c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9d8c0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f75007800?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9d880) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd19c1a0, {0x2c3c1b8, 0x3ebc3119e0}, 0x3ebc311a10) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74502600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd19c1a0}, {0x2c48448?, 0x3ec3b742a0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5430 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5388 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74c6e4f8, 0x3bc) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74c6e4e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74c6e480, {0x3ec3688000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec1ee7e60, {0x3ebc2b5d20, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec1ee7e60}, {0x3ebc2b5d20, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b5d20, 0x9, 0x3eaed2b320?}, {0x2c04240?, 0x3ec1ee7e60?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b5ce0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc57f208, 0x3f74a7e930) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5396 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5264 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fad40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c3300) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5419 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5263 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f755134b8, 0x3b1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f755134a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75513440, {0x3ec3d80000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec2170360, {0x3ebc090ba0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec2170360}, {0x3ebc090ba0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc090ba0, 0x9, 0x3eaf1edc50?}, {0x2c04240?, 0x3ec2170360?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc090b60) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebd244b48, 0x3f74a6e4d0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5198 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5422 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444d878, 0x3ab) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444d868) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444d800, {0x3ec3ac0000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc175980, {0x3f74a997e0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc175980}, {0x3f74a997e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a997e0, 0x9, 0x3eaf1edc98?}, {0x2c04240?, 0x3ebc175980?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a997a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc391b08, 0x3f74a47420) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5461 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5265 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc620d00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5419 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5389 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec1ee8ac0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0e500) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5257 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5390 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74033d40) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5257 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5391 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74c6e438, 0x3bc) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74c6e428) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74c6e3c0, {0x3ec3698000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec1ee7f20, {0x3ebc2b5e00, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec1ee7f20}, {0x3ebc2b5e00, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc2b5e00, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ec1ee7f20?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc2b5dc0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74033d40, {0x2c3c1b8, 0x3ec24fabd0}, 0x3ec24fac00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc474c00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74033d40}, {0x2c48448?, 0x3f775e4d40?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5257 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5397 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74a51d50) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5398 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5399 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc474e00, {0x2c308c8, 0x3ec2a66f20}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5400 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74cc9850, {0x2c3c1f0, 0x3f7436de50}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5401 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74cc9880, {0x2c3c1f0, 0x3f7436dea0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5402 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74cc98b0, {0x2c3c1f0, 0x3f7436def0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5540 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27fa00, {0x2c308c8, 0x3f775e5340}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5404 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74a93030) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5405 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5406 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc475000, {0x2c308c8, 0x3ec2a67b60}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5407 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3992640, {0x2c3c1f0, 0x3ec3478190}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5539 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5199 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7499f570) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5442 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f755133f8, 0x3b2) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f755133e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75513380, {0x3ec3aae000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc1758c0, {0x3f74a99700, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc1758c0}, {0x3f74a99700, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a99700, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebc1758c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a996c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc620d00, {0x2c3c1b8, 0x3ec2175800}, 0x3ec2175830) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc475000, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc620d00}, {0x2c48448?, 0x3f75893200?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5419 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5200 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5201 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc0e4400, {0x2c308c8, 0x3ec3a70000}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5458 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3a76000, {0x2c3c1f0, 0x3f743c8fa0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5459 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3a76030, {0x2c3c1f0, 0x3f743c9090}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5460 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3a76060, {0x2c3c1f0, 0x3f743c90e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5538 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74a7ed20) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5476 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec347ac00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75828d80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5475 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5479 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec3a19840, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75829100) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5493 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5477 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5ab1e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5475 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5478 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444d7b8, 0x3ab) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444d7a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444d740, {0x3ec39d8000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec347d980, {0x3ebca9d7e0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec347d980}, {0x3ebca9d7e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9d7e0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ec347d980?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9d7a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5ab1e0, {0x2c3c1b8, 0x3ec39d7d40}, 0x3ec39d7d70) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc0e4400, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5ab1e0}, {0x2c48448?, 0x3f75b016c0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5475 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5377 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f743e7a00, {0x2c308c8, 0x3ec2d43e40}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5490 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7601ae00, {0x2c3c1f0, 0x3f763c6640}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5491 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7601ae50, {0x2c3c1f0, 0x3f763c6690}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5492 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7601ae80, {0x2c3c1f0, 0x3f763c66e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5488 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebcaa2820) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5486 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5489 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75513b78, 0x3af) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75513b68) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75513b00, {0x3ebb246000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb85f47e0, {0x3f74a99b60, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb85f47e0}, {0x3f74a99b60, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a99b60, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb85f47e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a99b20) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebcaa2820, {0x2c3c1b8, 0x3ebb2441e0}, 0x3ebb244210) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebcc94600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebcaa2820}, {0x2c48448?, 0x3ec3a70940?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5486 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5463 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac19f8, 0x3ba) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac19e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac1980, {0x3f74ecc000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec3a780c0, {0x3f743cb2a0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec3a780c0}, {0x3f743cb2a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743cb2a0, 0x9, 0x3eaf4b40a8?}, {0x2c04240?, 0x3ec3a780c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743cb260) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74ed4008, 0x3f74a96070) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5493 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5554 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebfbc01f8, 0x3b3) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebfbc01e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebfbc0180, {0x3ebc064000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec3a786c0, {0x3f743cad60, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec3a786c0}, {0x3f743cad60, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743cad60, 0x9, 0x3eaf4b40f0?}, {0x2c04240?, 0x3ec3a786c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743cad20) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc894908, 0x3f74960b60) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5551 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5424 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74a479d0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5046 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 2295 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac1938, 0x3b6) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac1928) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac18c0, {0x3eb8262000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb8260000, {0x3ebc7a9000, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb8260000}, {0x3ebc7a9000, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a9000, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb8260000?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a8fc0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd19dd40, {0x2c3c1b8, 0x3eb83940c0}, 0x3eb83940f0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f743e7a00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd19dd40}, {0x2c48448?, 0x3ec3a703c0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5506 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5542 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3abef30, {0x2c3c1f0, 0x3f75f8a910}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5543 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3abef60, {0x2c3c1f0, 0x3f75f8a960}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5431 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb839a200, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a53300) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5430 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5472 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fac00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0e780) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5544 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5731 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7499e540) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5531 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444dc38, 0x3b5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444dc28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444dbc0, {0x3eb9130000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f751b46c0, {0x3ebc7a8e40, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f751b46c0}, {0x3ebc7a8e40, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a8e40, 0x9, 0x3eaf1edf20?}, {0x2c04240?, 0x3f751b46c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a8e00) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc57f688, 0x3f748aab60) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5588 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5469 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36a640, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0e680) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5468 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5470 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f742301a0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5468 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5471 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444d9f8, 0x3bb) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444d9e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444d980, {0x3ebc00e000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec3a78480, {0x3f743caac0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec3a78480}, {0x3f743caac0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743caac0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ec3a78480?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743caa80) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f742301a0, {0x2c3c1b8, 0x3f74ce2210}, 0x3f74ce2240) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27fa00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f742301a0}, {0x2c48448?, 0x3f75b00e80?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5468 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5545 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74a7f880) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5546 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5547 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc27fc00, {0x2c308c8, 0x3ec2a66120}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5548 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3abfcd0, {0x2c3c1f0, 0x3f75f8ab40}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5549 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3abfd00, {0x2c3c1f0, 0x3f75f8ab90}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5550 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3abfd30, {0x2c3c1f0, 0x3f75f8abe0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5438 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74dbc5c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0ec80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5581 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5427 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36abc0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a52c00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5551 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5564 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebfbc07f8, 0x3c0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebfbc07e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebfbc0780, {0x3eb8caa000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec3a78ae0, {0x3f743cb460, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec3a78ae0}, {0x3f743cb460, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743cb460, 0x9, 0x3eaf5a4378?}, {0x2c04240?, 0x3ec3a78ae0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743cb420) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc894b48, 0x3f74961420) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5574 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5562 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebfbc0138, 0x3b7) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebfbc0128) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebfbc00c0, {0x3eb8a00000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec3a787e0, {0x3f743cb0e0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec3a787e0}, {0x3f743cb0e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743cb0e0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ec3a787e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743cb0a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74230820, {0x2c3c1b8, 0x3f74ce2960}, 0x3f74ce2990) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc27fc00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74230820}, {0x2c48448?, 0x3ec3b74120?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5559 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5567 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36b440, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0ef00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5566 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5560 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36ac40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0e980) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5559 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5561 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74230820) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5559 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5552 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74a93dc0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5553 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5570 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74502600, {0x2c308c8, 0x3f75004600}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5571 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3992ee0, {0x2c3c1f0, 0x3f75f8ae60}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5572 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3992f10, {0x2c3c1f0, 0x3f75f8aeb0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5573 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3992f40, {0x2c3c1f0, 0x3f75f8af00}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5650 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7500a980, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0f000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5638 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5575 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74c4cf50) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5565 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36b100, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0ea80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5574 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5576 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5577 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74503200, {0x2c308c8, 0x3f75005260}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5578 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3993cb0, {0x2c3c1f0, 0x3f75f8b0e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5579 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3993ce0, {0x2c3c1f0, 0x3f75f8b130}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5580 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3993d10, {0x2c3c1f0, 0x3f75f8b180}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5499 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74503600, {0x2c308c8, 0x3ebc1805c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5500 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb8396440, {0x2c3c1f0, 0x3f74e128c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5495 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca028b8, 0x3a1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca028a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca02840, {0x3eb836e000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebd19e0c0, {0x3ebc090c80, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebd19e0c0}, {0x3ebc090c80, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc090c80, 0x9, 0x3eaf5a47c8?}, {0x2c04240?, 0x3ebd19e0c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc090c40) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74ed4488, 0x3f74ad6540) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5581 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5501 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb8396470, {0x2c3c1f0, 0x3f74e12910}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5502 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb83964a0, {0x2c3c1f0, 0x3f74e12960}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5435 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb839a300, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0eb80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5434 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5436 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd19c340) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5434 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5437 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca027f8, 0x3a5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca027e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca02780, {0x3eb8a5c000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75007b60, {0x3ebca9d9a0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75007b60}, {0x3ebca9d9a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9d9a0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f75007b60?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9d960) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd19c340, {0x2c3c1b8, 0x3f756f5110}, 0x3f756f5140) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74503200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd19c340}, {0x2c48448?, 0x3eb826a120?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5434 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5582 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74c4dab0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5583 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5584 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74503400, {0x2c308c8, 0x3f75005ec0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5585 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb8256b20, {0x2c3c1f0, 0x3f75f8b360}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5586 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb8256b50, {0x2c3c1f0, 0x3f75f8b3b0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5587 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb8256b80, {0x2c3c1f0, 0x3f75f8b400}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5769 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eba27c770) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5537 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fb080, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a53a00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5588 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5534 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc4fb0c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a53900) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5533 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5535 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc621040) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5533 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5536 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f7444db78, 0x3ba) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f7444db68) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f7444db00, {0x3eb9140000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f751b47e0, {0x3ebc7a90e0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f751b47e0}, {0x3ebc7a90e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a90e0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f751b47e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a90a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc621040, {0x2c3c1b8, 0x3ebc617cb0}, 0x3ebc617ce0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74503400, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc621040}, {0x2c48448?, 0x3f75b00fa0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5533 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5589 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74c0c850) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5590 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5612 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb8397590, {0x2c3c1f0, 0x3f74e12dc0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5504 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74ad6fc0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5505 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5602 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f743e7c00, {0x2c308c8, 0x3eb826ada0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5603 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb83971b0, {0x2c3c1f0, 0x3f74e12b40}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5604 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb83971e0, {0x2c3c1f0, 0x3f74e12b90}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5605 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb8397210, {0x2c3c1f0, 0x3f74e12be0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5613 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb83975c0, {0x2c3c1f0, 0x3f74e12e10}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5607 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca03638, 0x3c1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca03628) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca035c0, {0x3eb91bc000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9179c80, {0x3ebc090e40, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9179c80}, {0x3ebc090e40, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc090e40, 0x9, 0x3eaf1edf68?}, {0x2c04240?, 0x3eb9179c80?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc090e00) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74ed46c8, 0x3f74ad7650) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5503 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5623 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f755137b8, 0x3ac) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f755137a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75513740, {0x3eb9a00000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75088ae0, {0x3ebc12bd20, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75088ae0}, {0x3ebc12bd20, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc12bd20, 0x9, 0x3eaf0468a0?}, {0x2c04240?, 0x3f75088ae0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc12bce0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc312d88, 0x3f74a97500) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5638 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5635 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76121020, {0x2c3c1f0, 0x3f75f8b860}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5440 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb839a480, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a53b80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5439 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5441 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd19c4e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5439 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5634 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75513578, 0x3b3) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75513568) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75513500, {0x3eb9622000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75007e00, {0x3ebca9da80, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75007e00}, {0x3ebca9da80, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9da80, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f75007e00?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9da40) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd19c4e0, {0x2c3c1b8, 0x3eb9634150}, 0x3eb9634180) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f743e7c00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd19c4e0}, {0x2c48448?, 0x3ec3a70260?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5439 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5592 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74c0d260) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5611 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb8397560, {0x2c3c1f0, 0x3ec3478960}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5610 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f750d17c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75829200) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5503 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5593 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5594 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74503800, {0x2c308c8, 0x3f74b01560}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5636 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76121070, {0x2c3c1f0, 0x3ec3478730}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5637 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f761210a0, {0x2c3c1f0, 0x3ec3478780}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5657 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc37f740, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0f180) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5642 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5639 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74a6fc00) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5640 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5641 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc0e4600, {0x2c308c8, 0x3f75893aa0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5661 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f748950a0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5568 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74231380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5566 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5569 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f755136f8, 0x3ae) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f755136e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75513680, {0x3eb96a2000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ec3a79320, {0x3f743cb620, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ec3a79320}, {0x3f743cb620, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743cb620, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ec3a79320?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743cb5e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74231380, {0x2c3c1b8, 0x3eb96a0de0}, 0x3eb96a0e10) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74503800, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74231380}, {0x2c48448?, 0x3ec3a705a0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5566 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5662 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5658 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd19c820) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5642 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5660 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc36b600, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0f280) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5614 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5659 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebfbc0a38, 0x3b6) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebfbc0a28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebfbc09c0, {0x3eb9d08000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9633f20, {0x3ebca9dc40, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9633f20}, {0x3ebca9dc40, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9dc40, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb9633f20?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9dc00) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd19c820, {0x2c3c1b8, 0x3eb96a1350}, 0x3eb96a1380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc0e4600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd19c820}, {0x2c48448?, 0x3ec3b74520?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5642 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5615 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7472c930) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5616 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5617 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc474600, {0x2c308c8, 0x3eb826bc20}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5666 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75e9a480, {0x2c3c1f0, 0x3f74e12ff0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5667 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75e9a4b0, {0x2c3c1f0, 0x3f74e13040}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5668 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75e9a4e0, {0x2c3c1f0, 0x3f74e13090}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5482 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec3b78400, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8d980) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5481 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5645 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac1cf8, 0x3c9) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac1ce8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac1c80, {0x3eb9fa0000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb967e540, {0x3ebca9dd20, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb967e540}, {0x3ebca9dd20, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9dd20, 0x9, 0x3eaf0468e8?}, {0x2c04240?, 0x3eb967e540?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9dce0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ec21a8488, 0x3f748af570) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5683 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5663 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f749a6600, {0x2c308c8, 0x3ec3b74c40}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5664 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3a774f0, {0x2c3c1f0, 0x3f763c6dc0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5665 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3a77520, {0x2c3c1f0, 0x3f763c6e10}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5682 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ec3a77550, {0x2c3c1f0, 0x3f763c6e60}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5671 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca039f8, 0x3bc) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca039e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca03980, {0x3eb9ac0000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9ab0600, {0x3ebc091000, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9ab0600}, {0x3ebc091000, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc091000, 0x9, 0x3eaf5a5008?}, {0x2c04240?, 0x3eb9ab0600?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc090fc0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74ed4908, 0x3f74954150) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5669 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5643 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e9b900, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a53d80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5669 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5681 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e9ba40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8d380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5680 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5679 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca03938, 0x3b9) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca03928) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca038c0, {0x3eb9ad0000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9ab06c0, {0x3ebc0910e0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9ab06c0}, {0x3ebc0910e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc0910e0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb9ab06c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc0910a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74c4f380, {0x2c3c1b8, 0x3eb9aae9f0}, 0x3eb9aaea20) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc474600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74c4f380}, {0x2c48448?, 0x3ebc900580?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5676 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5859 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc37fc80, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74afd380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5858 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5677 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e9b940, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75829500) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5676 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5678 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74c4f380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5676 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5487 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec3b78500, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8da80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5486 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5684 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7498c770) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5685 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5686 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3f74db7200, {0x2c308c8, 0x3ec3b758a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5716 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e9bac0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0f800) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5700 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5715 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac1e78, 0x3bf) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac1e68) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac1e00, {0x3eb9fd4000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb967e8a0, {0x3ebca9de00, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb967e8a0}, {0x3ebca9de00, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9de00, 0x9, 0x3eaf5a5830?}, {0x2c04240?, 0x3eb967e8a0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9ddc0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ec21a86c8, 0x3f749bda40) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5690 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5759 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebb684200, {0x2c308c8, 0x3ec3a70be0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5717 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74c4f6c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5700 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5687 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76398500, {0x2c3c1f0, 0x3f763c7040}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5688 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76398540, {0x2c3c1f0, 0x3f763c7090}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5689 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76398580, {0x2c3c1f0, 0x3f763c70e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5698 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74c4f520) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5680 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5699 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac1c38, 0x3c1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac1c28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac1bc0, {0x3eb9ae0000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9ab0a80, {0x3ebc0911c0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9ab0a80}, {0x3ebc0911c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc0911c0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb9ab0a80?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc091180) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74c4f520, {0x2c3c1b8, 0x3eb85f2e70}, 0x3eb85f2ea0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f749a6600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74c4f520}, {0x2c48448?, 0x3f75893ce0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5680 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5797 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3ebb94a770) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5691 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f7494e7e0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5692 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5693 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebcc94400, {0x2c308c8, 0x3f746eb240}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5694 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f763996f0, {0x2c3c1f0, 0x3f763c7400}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5695 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76399720, {0x2c3c1f0, 0x3f763c7450}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5696 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f76399760, {0x2c3c1f0, 0x3f763c74a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5701 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc37f980, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a53e80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5690 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5718 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac1db8, 0x3bc) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac1da8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac1d40, {0x3eba288000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9ab0ae0, {0x3ebc0912a0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9ab0ae0}, {0x3ebc0912a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc0912a0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb9ab0ae0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc091260) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74c4f6c0, {0x2c3c1b8, 0x3eb9fc8360}, 0x3eb9fc8390) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3f74db7200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74c4f6c0}, {0x2c48448?, 0x3f75893de0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5700 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5749 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7500aa80, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c3800) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5730 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5732 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5733 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eba7ce000, {0x2c308c8, 0x3ebc67e0a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5734 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74afb300, {0x2c3c1f0, 0x3eb826c0f0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5735 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74afb350, {0x2c3c1f0, 0x3eb826c140}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5736 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74afb3e0, {0x2c3c1f0, 0x3eb826c190}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5746 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7500aac0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c3700) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5633 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5747 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5aa820) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5633 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5748 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75513878, 0x3ba) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75513868) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75513800, {0x3eba530000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75088ea0, {0x3ebcc902e0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75088ea0}, {0x3ebcc902e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebcc902e0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3f75088ea0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebcc902a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5aa820, {0x2c3c1b8, 0x3eba514d50}, 0x3eba514d80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebcc94400, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5aa820}, {0x2c48448?, 0x3ec3a706e0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5633 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5449 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5738 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74c707e0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5739 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5740 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eba7ce200, {0x2c308c8, 0x3ebc660e20}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5741 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f744229c0, {0x2c3c1f0, 0x3eb826c370}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5742 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74422a00, {0x2c3c1f0, 0x3eb826c3c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5743 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74422a30, {0x2c3c1f0, 0x3eb826c410}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5446 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74d8b040) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5444 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5447 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74c6f1b8, 0x3ad) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74c6f1a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74c6f140, {0x3ebaa8c000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebaa8a000, {0x3f74ce0120, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebaa8a000}, {0x3f74ce0120, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74ce0120, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebaa8a000?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74ce00e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74d8b040, {0x2c3c1b8, 0x3ebaa801b0}, 0x3ebaa801e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eba7ce000, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74d8b040}, {0x2c48448?, 0x3f75820040?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5444 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5766 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e9bb40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f759afa00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5783 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5796 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca03db8, 0x3bf) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca03da8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca03d40, {0x3ebb628000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb967f080, {0x3eb9ffa120, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb967f080}, {0x3eb9ffa120, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3eb9ffa120, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb967f080?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3eb9ffa0e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd19cb60, {0x2c3c1b8, 0x3ebbe106f0}, 0x3ebbe10720) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebcc94a00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd19cb60}, {0x2c48448?, 0x3ebc9016c0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5858 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5765 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7500ac80, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0fa00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5744 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5762 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebd128280, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0f900) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5697 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5763 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74231860) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5697 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5764 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f755139f8, 0x3ba) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f755139e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75513980, {0x3eba272000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eba21ec60, {0x3f743cba80, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eba21ec60}, {0x3f743cba80, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743cba80, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eba21ec60?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743cba40) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74231860, {0x2c3c1b8, 0x3eba271680}, 0x3eba2716b0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eba7ce200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74231860}, {0x2c48448?, 0x3ec3a70840?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5697 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5745 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74c71260) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5778 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5779 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eba7ce400, {0x2c308c8, 0x3ec18f39a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5780 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74423f00, {0x2c3c1f0, 0x3eb826c5f0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5781 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a20010, {0x2c3c1f0, 0x3eb826c640}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5782 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a20050, {0x2c3c1f0, 0x3eb826c690}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5828 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc8837b8, 0x3b7) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc8837a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc883740, {0x3ebb26e000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb85f4ae0, {0x3f74a99c40, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb85f4ae0}, {0x3f74a99c40, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a99c40, 0x9, 0x3eaf515c98?}, {0x2c04240?, 0x3eb85f4ae0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a99c00) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f747b3688, 0x3eb85f8af0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5846 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5713 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca03cf8, 0x3b5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca03ce8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca03c80, {0x3eba2f6000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9ab1020, {0x3ebc091460, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9ab1020}, {0x3ebc091460, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc091460, 0x9, 0x3eaf515c50?}, {0x2c04240?, 0x3eb9ab1020?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc091420) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74ed4d88, 0x3eba2ca700) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5790 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5784 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3f74c71e30) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5785 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5786 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eba7ce600, {0x2c308c8, 0x3ec2d42fc0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5787 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a21b60, {0x2c3c1f0, 0x3eb826c870}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5788 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a21b90, {0x2c3c1f0, 0x3eb826c8c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5789 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75a21bc0, {0x2c3c1f0, 0x3eb826c910}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5817 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca03e78, 0x3c3) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca03e68) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca03e00, {0x3ebbbda000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9ab1a40, {0x3ebc091620, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9ab1a40}, {0x3ebc091620, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc091620, 0x9, 0x3eaf047d40?}, {0x2c04240?, 0x3eb9ab1a40?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc0915e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74ed4fc8, 0x3eba2cb1f0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5853 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5791 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3ebaf4e930) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5810 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e9bbc0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75829600) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5790 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5752 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebd128440, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c3900) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5767 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5753 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74231a00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5767 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5754 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebca03c38, 0x3b7) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebca03c28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebca03bc0, {0x3ebb19c000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eba21f0e0, {0x3f743cbb60, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eba21f0e0}, {0x3f743cbb60, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743cbb60, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eba21f0e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743cbb20) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74231a00, {0x2c3c1b8, 0x3eba5152c0}, 0x3eba5152f0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eba7ce600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74231a00}, {0x2c48448?, 0x3ebc900e60?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5767 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5451 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a2e20, {0x2c3c1f0, 0x3f763eb360}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5452 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a2e50, {0x2c3c1f0, 0x3f763eb5e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5453 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a2eb0, {0x2c3c1f0, 0x3f763eb860}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5720 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb9fe8078, 0x3af) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb9fe8068) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb9fe8000, {0x3eb9fea000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb967eba0, {0x3ebca9dee0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb967eba0}, {0x3ebca9dee0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebca9dee0, 0x9, 0x3eaf6d0120?}, {0x2c04240?, 0x3eb967eba0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebca9dea0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ec21a8908, 0x3f74a77730) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5768 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5862 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb9ffee00) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5826 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7500adc0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8dc00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5775 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5832 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec3b786c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75829800) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5846 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5729 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc37fac0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74afc880) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5768 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5726 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc37fb00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74afc700) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5725 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5727 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd19c9c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5725 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5728 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebcac1f38, 0x3b0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebcac1f28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebcac1ec0, {0x3ebb600000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb967ecc0, {0x3eb9ffa040, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb967ecc0}, {0x3eb9ffa040, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3eb9ffa040, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb967ecc0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3eb9ffa000) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd19c9c0, {0x2c3c1b8, 0x3eb9fc88a0}, 0x3eb9fc88d0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebaacc000, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd19c9c0}, {0x2c48448?, 0x3f75893ee0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5725 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5770 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5771 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebcc94600, {0x2c308c8, 0x3eba27aac0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5772 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7569f6b0, {0x2c3c1f0, 0x3f763c78b0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5773 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7569f720, {0x2c3c1f0, 0x3f763c7900}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5774 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7569f760, {0x2c3c1f0, 0x3f763c7950}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5834 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc883938, 0x3aa) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc883928) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc8838c0, {0x3ebbf84000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb85f4cc0, {0x3f74a99d20, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb85f4cc0}, {0x3f74a99d20, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a99d20, 0x9, 0x3eaf6d0138?}, {0x2c04240?, 0x3eb85f4cc0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a99ce0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f747b3d48, 0x3eb85f8ee0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5854 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5756 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75513c38, 0x3b2) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75513c28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75513bc0, {0x3ebb686000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3f75089740, {0x3ebcc904a0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3f75089740}, {0x3ebcc904a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebcc904a0, 0x9, 0x3eaf515c80?}, {0x2c04240?, 0x3f75089740?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebcc90460) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc3138c8, 0x3eba5427e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5775 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5760 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f750011d0, {0x2c3c1f0, 0x3f743c9ea0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5776 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eba27d1f0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5777 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5842 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebcc94800, {0x2c308c8, 0x3eba27b720}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5843 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74e059f0, {0x2c3c1f0, 0x3f763c7b80}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5844 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74e05a20, {0x2c3c1f0, 0x3f763c7bd0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5845 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74e05a50, {0x2c3c1f0, 0x3f763c7d60}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5877 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7500b100, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c3a80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5876 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5829 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f74e9bcc0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75829700) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5813 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5830 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74c4f860) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5813 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5831 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc8836f8, 0x3b8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc8836e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc883680, {0x3ebbb80000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9ab1440, {0x3ebc091540, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9ab1440}, {0x3ebc091540, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc091540, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb9ab1440?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc091500) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74c4f860, {0x2c3c1b8, 0x3ebb26ca20}, 0x3ebb26ca50) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebcc94800, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74c4f860}, {0x2c48448?, 0x3ebc6c53a0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5813 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5847 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eba27dc70) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5848 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5849 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebcc94a00, {0x2c308c8, 0x3ebb920400}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5850 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74d8cfa0, {0x2c3c1f0, 0x3ebcbfec30}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5851 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74d8cfd0, {0x2c3c1f0, 0x3ebcbfec80}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5852 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74d8d000, {0x2c3c1f0, 0x3ebcbfecd0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5927 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebbf38378, 0x3ac) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebbf38368) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebbf38300, {0x3eb478c000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebbf4c300, {0x3ebc0919a0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebbf4c300}, {0x3ebc0919a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc0919a0, 0x9, 0x3eaf8d81f8?}, {0x2c04240?, 0x3ebbf4c300?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc091960) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74ed5688, 0x3eb478a1c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5901 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5758 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5860 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd19cb60) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5858 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5798 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5799 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebbe16000, {0x2c308c8, 0x3f75820d80}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5800 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcab33e0, {0x2c3c1f0, 0x3ebc9b2fa0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5801 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcab3420, {0x2c3c1f0, 0x3ebc9b2ff0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5802 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcab34f0, {0x2c3c1f0, 0x3ebc9b3040}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5881 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7500b380, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74afd900) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5880 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5855 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3ebb94aee0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5761 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75001200, {0x2c3c1f0, 0x3f743c9ef0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5874 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75001230, {0x2c3c1f0, 0x3f74c06640}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5875 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec3b78780, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8de80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5854 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5892 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7601b180, {0x2c3c1f0, 0x3ebd3dfd60}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5891 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7601b150, {0x2c3c1f0, 0x3ec3479220}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5893 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f7601b200, {0x2c3c1f0, 0x3ebd3dfdb0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5840 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec3b787c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74a8dd80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5839 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5841 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebcaa29c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5839 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5890 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebc883878, 0x3a8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebc883868) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebc883800, {0x3ebbf94000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb85f4d80, {0x3f74a99e00, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb85f4d80}, {0x3f74a99e00, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74a99e00, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb85f4d80?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74a99dc0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebcaa29c0, {0x2c3c1b8, 0x3ebbf834d0}, 0x3ebbf83500) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebbe16000, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebcaa29c0}, {0x2c48448?, 0x3ebc6c5520?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5839 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5922 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebbf381f8, 0x3bd) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebbf381e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebbf38180, {0x3ebbf50000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebbf4c0c0, {0x3ebc0918c0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebbf4c0c0}, {0x3ebc0918c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc0918c0, 0x9, 0x3eaf7965b8?}, {0x2c04240?, 0x3ebbf4c0c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc091880) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74ed5448, 0x3eba2cbb90) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5894 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5824 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebbe9d780, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75829c80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5861 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5857 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebfbc1878, 0x3bb) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebfbc1868) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebfbc1800, {0x3ebbede000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb971da0, {0x3f743cbee0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb971da0}, {0x3f743cbee0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f743cbee0, 0x9, 0x3eaf047db8?}, {0x2c04240?, 0x3ebb971da0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f743cbea0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc895208, 0x3ebb94b340) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5861 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5866 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebbf4a440, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75e9c080) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5940 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5909 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebbe9d7c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74b0ff00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5908 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5821 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebbf4a000, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75829b80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5820 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5822 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74c4fa00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5820 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5823 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebfbc17b8, 0x3c0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebfbc17a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebfbc1740, {0x3ebbf3a000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9ab1da0, {0x3ebc0917e0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9ab1da0}, {0x3ebc0917e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc0917e0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb9ab1da0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc0917a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74c4fa00, {0x2c3c1b8, 0x3ebbf10480}, 0x3ebbf104b0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebb684200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74c4fa00}, {0x2c48448?, 0x3ebb9216e0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5820 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5863 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5864 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc0e4800, {0x2c308c8, 0x3eb9ffcb00}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5932 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebbf384f8, 0x3b6) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebbf384e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebbf38480, {0x3eb479e000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebbf4c4e0, {0x3ebc091a80, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebbf4c4e0}, {0x3ebc091a80, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc091a80, 0x9, 0x3eaf74aeb8?}, {0x2c04240?, 0x3ebbf4c4e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc091a40) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74ed58c8, 0x3eb478a5b0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5940 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5976 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb47501f8, 0x3b8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb47501e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb4750180, {0x3eb5a00000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb6ad1a0, {0x3ebcc90ac0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb6ad1a0}, {0x3ebcc90ac0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebcc90ac0, 0x9, 0x3eaf6d0198?}, {0x2c04240?, 0x3ebb6ad1a0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebcc90a80) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3eb4f7e008, 0x3eba543ea0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5803 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5925 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebbf4a1c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f767c3b80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5894 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5878 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5aad00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5876 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5879 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebbf38138, 0x3c0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebbf38128) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebbf380c0, {0x3ebb6c6000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb6ac300, {0x3ebcc90660, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb6ac300}, {0x3ebcc90660, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebcc90660, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebb6ac300?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebcc90620) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5aad00, {0x2c3c1b8, 0x3ebb6c4a20}, 0x3ebb6c4a50) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc0e4800, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5aad00}, {0x2c48448?, 0x3ebc901a60?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5876 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5895 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb85f9b20) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5896 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5897 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eb46a8000, {0x2c308c8, 0x3ebbfe83c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5898 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75236150, {0x2c3c1f0, 0x3eb469a050}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5899 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75236190, {0x2c3c1f0, 0x3eb469a0a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5900 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f752363d0, {0x2c3c1f0, 0x3eb469a0f0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5889 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7500b500, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f744d0c00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5947 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 6194 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebbe7a480, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f76881380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6156 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5930 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebbf4a340, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75829d80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5901 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5910 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd3d0680) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5908 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5865 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebbf382b8, 0x3b1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebbf382a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebbf38240, {0x3ebbef6000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebbef0240, {0x3ebbefe040, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebbef0240}, {0x3ebbefe040, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebbefe040, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebbef0240?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebbefe000) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd3d0680, {0x2c3c1b8, 0x3ebbf7e3c0}, 0x3ebbf7e3f0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eb46a8000, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd3d0680}, {0x2c48448?, 0x3eb4788060?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5908 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5902 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb46c0620) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5903 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5904 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eb46a8200, {0x2c308c8, 0x3ebbfe9000}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5905 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74317540, {0x2c3c1f0, 0x3eb469a2d0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5938 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74317580, {0x2c3c1f0, 0x3eb469a320}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5939 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74317670, {0x2c3c1f0, 0x3eb469a370}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5935 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb478ad90) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5914 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb522c230) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 6075 [runnable]: runtime.Gosched(...) /usr/lib/go/src/runtime/proc.go:364 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3eb6554080) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:611 +0x1c8 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6113 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5936 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5882 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5aaea0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5880 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5883 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebbf38438, 0x3b6) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebbf38428) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebbf383c0, {0x3eb4724000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb6aca80, {0x3ebcc90820, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb6aca80}, {0x3ebcc90820, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebcc90820, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebb6aca80?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebcc907e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5aaea0, {0x2c3c1b8, 0x3ebb6c50e0}, 0x3ebb6c5110) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eb46a8200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5aaea0}, {0x2c48448?, 0x3eb47881a0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5880 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5941 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb46c10a0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5942 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5943 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eb46a8400, {0x2c308c8, 0x3ebbfe9c60}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5944 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74cc8e90, {0x2c3c1f0, 0x3eb469a550}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5945 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74cc8ec0, {0x2c3c1f0, 0x3eb469a5a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5946 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f74cc8ef0, {0x2c3c1f0, 0x3eb469a5f0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5960 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb4eb5200, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f76880780) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5954 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5963 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb4eb10a0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5871 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb9fe8678, 0x3b8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb9fe8668) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb9fe8600, {0x3eb5480000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb65b5c0, {0x3eb9ffa580, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb65b5c0}, {0x3eb9ffa580, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3eb9ffa580, 0x9, 0x3eaf796618?}, {0x2c04240?, 0x3ebb65b5c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3eb9ffa540) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ec21a8fc8, 0x3eb9fff6c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5913 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5885 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb4750078, 0x3b6) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb4750068) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb4750000, {0x3eb4752000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb6acc00, {0x3ebcc90900, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb6acc00}, {0x3ebcc90900, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebcc90900, 0x9, 0x3eaf8d8288?}, {0x2c04240?, 0x3ebb6acc00?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebcc908c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc313d48, 0x3eba543730) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5947 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5886 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebb67a340, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75829e80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5867 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5887 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd19d380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5867 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5888 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f75513f38, 0x3b9) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f75513f28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f75513ec0, {0x3eb4e94000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb65b2c0, {0x3eb9ffa3c0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb65b2c0}, {0x3eb9ffa3c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3eb9ffa3c0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebb65b2c0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3eb9ffa380) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd19d380, {0x2c3c1b8, 0x3eb474b710}, 0x3eb474b740) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eb46a8400, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd19d380}, {0x2c48448?, 0x3ec3a71080?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5867 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5948 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb46c1c00) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5949 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5950 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eb46a8600, {0x2c308c8, 0x3eb4c42940}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5951 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75514180, {0x2c3c1f0, 0x3eb469a7d0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5952 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f755141f0, {0x2c3c1f0, 0x3eb469a820}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5953 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f75514230, {0x2c3c1f0, 0x3eb469a870}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5961 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebb67a500, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3ebc552100) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5913 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5964 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 6102 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb4eb2678, 0x3ae) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb4eb2668) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb4eb2600, {0x3eb618c000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb4efdec0, {0x3eb46f2580, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb4efdec0}, {0x3eb46f2580, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3eb46f2580, 0x9, 0x3eaf4b4168?}, {0x2c04240?, 0x3eb4efdec0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3eb46f2540) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3eb4eda488, 0x3eb4eb1ce0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6082 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5956 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb4eb21f8, 0x3c1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb4eb21e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb4eb2180, {0x3eb4eca000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb4c574a0, {0x3eb46f22e0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb4c574a0}, {0x3eb46f22e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3eb46f22e0, 0x9, 0x3eaf74b4e8?}, {0x2c04240?, 0x3eb4c574a0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3eb46f22a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3eb4eda008, 0x3eb4eb05b0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5954 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5957 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7500b580, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f76880680) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5970 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5958 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5ab040) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5970 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5959 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb4eb2138, 0x3c3) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb4eb2128) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb4eb20c0, {0x3eb4f36000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb6acde0, {0x3ebcc909e0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb6acde0}, {0x3ebcc909e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebcc909e0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebb6acde0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebcc909a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5ab040, {0x2c3c1b8, 0x3eb4c750b0}, 0x3eb4c750e0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eb46a8600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5ab040}, {0x2c48448?, 0x3eb4c42fe0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5970 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5937 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebc475200, {0x2c308c8, 0x3eb47888a0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5986 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb4fcc070, {0x2c3c1f0, 0x3ebbf4e640}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5987 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb4fcc0a0, {0x2c3c1f0, 0x3ebbf4e690}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5988 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb4fcc0d0, {0x2c3c1f0, 0x3ebbf4e6e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6083 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb5f32830, {0x2c3c1f0, 0x3eb469b360}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6000 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebbf38af8, 0x3aa) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebbf38ae8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebbf38a80, {0x3eb7a00000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc5bd980, {0x3ebc091e00, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc5bd980}, {0x3ebc091e00, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc091e00, 0x9, 0x3eae8b4d68?}, {0x2c04240?, 0x3ebc5bd980?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc091dc0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebcbe8008, 0x3eb5afe4d0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6211 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 6003 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebb67a540, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3ebc552000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6002 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 6004 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd19d520) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6002 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 6005 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb9fe85b8, 0x3b3) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb9fe85a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb9fe8540, {0x3eb5490000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb65b6e0, {0x3eb9ffa660, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb65b6e0}, {0x3eb9ffa660, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3eb9ffa660, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebb65b6e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3eb9ffa620) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd19d520, {0x2c3c1b8, 0x3ebc52f710}, 0x3ebc52f740) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebc475200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd19d520}, {0x2c48448?, 0x3eb9ffcea0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 6002 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5915 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5916 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebcc94c00, {0x2c308c8, 0x3eb5232220}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5917 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb52582c0, {0x2c3c1f0, 0x3eb52080f0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5918 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb52582f0, {0x2c3c1f0, 0x3eb5208140}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5919 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb5258320, {0x2c3c1f0, 0x3eb5208190}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6077 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74d8b520) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6047 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5994 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebbf38738, 0x3b5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebbf38728) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebbf386c0, {0x3eb5a10000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebbf4df20, {0x3ebc091c40, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebbf4df20}, {0x3ebc091c40, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc091c40, 0x9, 0x3eaf6d0150?}, {0x2c04240?, 0x3ebbf4df20?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc091c00) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74ed5b08, 0x3eb478b420) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5962 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 6033 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb520de80, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75e9cc80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6092 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5965 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eb46a8800, {0x2c308c8, 0x3eb4c43ba0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5996 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebbf38978, 0x3b1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebbf38968) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebbf38900, {0x3eb5abe000, 0x8000, 0x3eaf9c1140?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebc5bd7a0, {0x3ebc091d20, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebc5bd7a0}, {0x3ebc091d20, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc091d20, 0x9, 0x3eaf9c1140?}, {0x2c04240?, 0x3ebc5bd7a0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc091ce0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3f74ed5d48, 0x3eb478bf10) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6113 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 5966 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb558c540, {0x2c3c1f0, 0x3eb469ad20}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5967 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb558c570, {0x2c3c1f0, 0x3eb469ad70}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5968 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb558c5a0, {0x2c3c1f0, 0x3eb469adc0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6020 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebbf4ad00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75e9c600) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5962 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 6078 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebbf388b8, 0x3b0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebbf388a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebbf38840, {0x3eb62f2000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb5eae600, {0x3f74ce0580, 0x9, 0x613ad4?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb5eae600}, {0x3f74ce0580, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74ce0580, 0x9, 0x3ebaae2428?}, {0x2c04240?, 0x3eb5eae600?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74ce0540) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74d8b520, {0x2c3c1b8, 0x3eb6552390}, 0x3eb65523c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eb46a8c00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74d8b520}, {0x2c48448?, 0x3eb4789320?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 6047 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 6103 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb4eb1d50) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5985 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7500b600, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3ebc552200) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 5803 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 5921 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb520db40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75e9c480) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5920 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 6018 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd3d09c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5920 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 6019 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebbf38678, 0x3b4) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebbf38668) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebbf38600, {0x3eb525c000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb522e7e0, {0x3ebbefe200, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb522e7e0}, {0x3ebbefe200, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebbefe200, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb522e7e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebbefe1c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd3d09c0, {0x2c3c1b8, 0x3eb524efc0}, 0x3eb524eff0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebcc94c00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd3d09c0}, {0x2c48448?, 0x3eb4788b00?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5920 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5982 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3f7500b640, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f76880b00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5981 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 5983 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5ab380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 5981 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 5984 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb4750138, 0x3af) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb4750128) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb47500c0, {0x3eb5a22000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb6ad260, {0x3ebcc90ba0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb6ad260}, {0x3ebcc90ba0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebcc90ba0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebb6ad260?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebcc90b60) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5ab380, {0x2c3c1b8, 0x3eb4f74c60}, 0x3eb4f74c90) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eb46a8800, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5ab380}, {0x2c48448?, 0x3ec3a71200?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 5981 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5454 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb478bb90) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5455 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 5456 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebaacc200, {0x2c308c8, 0x3f75212a20}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 5457 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a3800, {0x2c3c1f0, 0x3f763ebd60}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6034 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a3840, {0x2c3c1f0, 0x3ebaae2000}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6035 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3f754a3880, {0x2c3c1f0, 0x3ebaae2050}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5805 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74c6f7b8, 0x3b0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74c6f7a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74c6f740, {0x3eb54e0000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9a2b680, {0x3ebc4a9540, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9a2b680}, {0x3ebc4a9540, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc4a9540, 0x9, 0x3eaf796660?}, {0x2c04240?, 0x3eb9a2b680?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc4a9500) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebd245688, 0x3ebb39ecb0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6036 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 6021 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebbe7a200, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75e9c800) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6036 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 6063 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3eb7692000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6061 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 6062 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebbe7a4c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f76881280) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6061 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 6051 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebbe7a240, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3eb54fc080) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6050 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 6052 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f7499bba0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6050 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 6053 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74c6f6f8, 0x3b0) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74c6f6e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74c6f680, {0x3eb54f2000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9a2b740, {0x3ebc4a9620, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9a2b740}, {0x3ebc4a9620, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc4a9620, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb9a2b740?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc4a95e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f7499bba0, {0x2c3c1b8, 0x3eb54dfbf0}, 0x3eb54dfc20) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebaacc200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f7499bba0}, {0x2c48448?, 0x3f75821a80?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 6050 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5792 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3ebaf4ebd0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 5793 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 6066 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eba7ce800, {0x2c308c8, 0x3ebaf70740}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 6067 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcd1cf50, {0x2c3c1f0, 0x3eb826cb40}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6068 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcd1cf90, {0x2c3c1f0, 0x3eb826cb90}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6069 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcd1cfc0, {0x2c3c1f0, 0x3eb826cbe0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6104 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 6074 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb4ef9940, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74d93f00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6082 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 6071 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec2d33440, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f74d93e00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6070 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 6072 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc621a00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6070 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 6073 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb4eb25b8, 0x3b2) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb4eb25a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb4eb2540, {0x3eb5e36000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb1dca20, {0x3ebc7a9620, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb1dca20}, {0x3ebc7a9620, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a9620, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebb1dca20?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a95e0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc621a00, {0x2c3c1b8, 0x3eb5e2ecc0}, 0x3eb5e2ecf0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eba7ce800, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc621a00}, {0x2c48448?, 0x3eb4c43f00?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 6070 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 6105 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eb46a8a00, {0x2c308c8, 0x3eb619c220}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 6084 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb5f32860, {0x2c3c1f0, 0x3eb474e870}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6085 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb5f32890, {0x2c3c1f0, 0x3eb474e8c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6024 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebfbc1cf8, 0x3b2) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebfbc1ce8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebfbc1c80, {0x3eb6bda000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb522efc0, {0x3ebbefe3c0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb522efc0}, {0x3ebbefe3c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebbefe3c0, 0x9, 0x3eae5b8840?}, {0x2c04240?, 0x3eb522efc0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebbefe380) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc895688, 0x3eb522cfc0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6092 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 6038 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebaac6378, 0x3b3) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebaac6368) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebaac6300, {0x3eb629a000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb5eae060, {0x3f74ce03c0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb5eae060}, {0x3f74ce03c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74ce03c0, 0x9, 0x3eaf7966a8?}, {0x2c04240?, 0x3eb5eae060?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74ce0380) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3eb62ac008, 0x3f74c7f420) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6106 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 6022 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb5e87140, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3eb62c0000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6106 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 6056 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74c6fab8, 0x3c6) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74c6faa8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74c6fa40, {0x3eb6aee000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9a2be00, {0x3ebc4a97e0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9a2be00}, {0x3ebc4a97e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc4a97e0, 0x9, 0x3eaec63ae8?}, {0x2c04240?, 0x3eb9a2be00?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc4a97a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebd245b08, 0x3ebb39f570) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6156 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 6044 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb5e87180, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f743e9880) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6043 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 6045 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3f74d8b380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6043 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 6046 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebaac62b8, 0x3b6) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebaac62a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebaac6240, {0x3eb62ae000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb5eae120, {0x3f74ce04a0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb5eae120}, {0x3f74ce04a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3f74ce04a0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb5eae120?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3f74ce0460) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3f74d8b380, {0x2c3c1b8, 0x3eb62aa2a0}, 0x3eb62aa2d0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eb46a8a00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3f74d8b380}, {0x2c48448?, 0x3eb5e90580?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 6043 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 6107 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb61ae700) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 6108 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 6109 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eb46a8c00, {0x2c308c8, 0x3eb619cd80}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 6110 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb558d930, {0x2c3c1f0, 0x3eb469b540}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6111 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb558d960, {0x2c3c1f0, 0x3eb469b590}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6112 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb558d990, {0x2c3c1f0, 0x3eb469b5e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6076 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb5e87300, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3eb6554180) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6047 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 6114 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb61af180) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 6115 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 6116 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eb46a8e00, {0x2c308c8, 0x3eb619d9e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 6117 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc682780, {0x2c3c1f0, 0x3eb469b7c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6118 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc6827b0, {0x2c3c1f0, 0x3eb469b810}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6119 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc6827e0, {0x2c3c1f0, 0x3eb469b860}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 5998 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebb67a7c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3ebc552400) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6127 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 6080 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb5e301f8, 0x3ab) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb5e301e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb5e30180, {0x3eb6556000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb1dd260, {0x3ebc7a9700, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb1dd260}, {0x3ebc7a9700, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a9700, 0x9, 0x3eaee26d68?}, {0x2c04240?, 0x3ebb1dd260?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a96c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc57f8c8, 0x3ebaf4fa40) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6120 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 6081 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec2d33800, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f744d0e00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6054 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 6146 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc621ba0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6054 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 6147 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb5e30138, 0x3a7) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb5e30128) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb5e300c0, {0x3eb6566000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb1dd380, {0x3ebc7a97e0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb1dd380}, {0x3ebc7a97e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a97e0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebb1dd380?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a97a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc621ba0, {0x2c3c1b8, 0x3eb65528a0}, 0x3eb65528d0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eb46a8e00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc621ba0}, {0x2c48448?, 0x3ebaf70aa0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 6054 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 5997 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec2d337c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3eb6554280) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6120 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 6121 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb61afce0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 6122 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 6123 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eb46a9000, {0x2c308c8, 0x3ebc7ca6c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 6124 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc683550, {0x2c3c1f0, 0x3eb469ba40}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6125 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc683580, {0x2c3c1f0, 0x3eb469ba90}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6126 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc6835b0, {0x2c3c1f0, 0x3eb469bae0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6158 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 6007 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb9fe87f8, 0x3b1) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb9fe87e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb9fe8780, {0x3eb550e000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb65be60, {0x3eb9ffa740, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb65be60}, {0x3eb9ffa740, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3eb9ffa740, 0x9, 0x3eaee273c8?}, {0x2c04240?, 0x3ebb65be60?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3eb9ffa700) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ec21a9208, 0x3eb551e000) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6127 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 6008 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ec2d33900, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3ebc552300) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6150 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 6009 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc621d40) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6150 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 6010 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb9fe8738, 0x3b7) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb9fe8728) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb9fe86c0, {0x3eb6576000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3ebb1dd740, {0x3ebc7a98c0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3ebb1dd740}, {0x3ebc7a98c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a98c0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3ebb1dd740?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a9880) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc621d40, {0x2c3c1b8, 0x3eb5522180}, 0x3eb55221b0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eb46a9000, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc621d40}, {0x2c48448?, 0x3eb9ffcfe0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 6150 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 6086 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb5a21500) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 6087 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 6088 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebb684400, {0x2c308c8, 0x3eb6a38720}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 6089 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb5f334a0, {0x2c3c1f0, 0x3eb474ecd0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6090 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb5f334d0, {0x2c3c1f0, 0x3eb474ed20}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6091 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb5f33500, {0x2c3c1f0, 0x3eb474ed70}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6030 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb520dec0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75e9ca80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6029 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 6031 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebd3d0b60) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6029 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 6032 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebfbc1c38, 0x3b2) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebfbc1c28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebfbc1bc0, {0x3eb6bea000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb522f0e0, {0x3ebbefe4a0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb522f0e0}, {0x3ebbefe4a0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebbefe4a0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb522f0e0?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebbefe460) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebd3d0b60, {0x2c3c1b8, 0x3eb6bd9920}, 0x3eb6bd9950) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebb684400, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebd3d0b60}, {0x2c48448?, 0x3eb5232640?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 6029 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 6093 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb719a000) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 6094 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 6095 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3ebb684600, {0x2c308c8, 0x3eb6a39380}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 6128 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc683970, {0x2c3c1f0, 0x3eb474ef50}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6129 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc6839a0, {0x2c3c1f0, 0x3eb6ec6000}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6162 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebc6839d0, {0x2c3c1f0, 0x3eb6ec6050}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6187 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3ebc5c4d40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f744d0f00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6211 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 6157 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb6e52930) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 6159 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eba7cea00, {0x2c308c8, 0x3ebaf71680}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 6182 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb6a71e40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3ebc552500) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6163 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 6184 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb6a71f40, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3f75e9ce80) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6183 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 6178 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb5e30378, 0x3a5) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb5e30368) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb5e30300, {0x3eb71c6000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb7186c00, {0x3ebcc910e0, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb7186c00}, {0x3ebcc910e0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebcc910e0, 0x9, 0x3eaee275f0?}, {0x2c04240?, 0x3eb7186c00?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebcc910a0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3eb4f7e6c8, 0x3eb719a380) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6163 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 6179 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb6a71e00, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3eb5a7c580) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6096 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 6180 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5aba00) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6096 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 6181 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb5e302b8, 0x3a9) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb5e302a8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb5e30240, {0x3eb71b4000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb7186b40, {0x3ebcc91000, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb7186b40}, {0x3ebcc91000, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebcc91000, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb7186b40?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebcc90fc0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5aba00, {0x2c3c1b8, 0x3eb71a7c80}, 0x3eb71a7cb0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3ebb684600, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5aba00}, {0x2c48448?, 0x3ebaf70c00?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 6096 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 6164 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb6b62c40) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 6165 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 6166 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eb46a9200, {0x2c308c8, 0x3ebc7cb5c0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 6167 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb6ed4760, {0x2c3c1f0, 0x3eb6ec6230}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6168 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb6ed4790, {0x2c3c1f0, 0x3eb6ec6280}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6169 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3eb6ed47c0, {0x2c3c1f0, 0x3eb6ec62d0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6064 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3f74c6f9f8, 0x3cb) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3f74c6f9e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3f74c6f980, {0x3eb7680000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb9a2bf20, {0x3ebc4a98c0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb9a2bf20}, {0x3ebc4a98c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc4a98c0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb9a2bf20?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc4a9880) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3eb7692000, {0x2c3c1b8, 0x3eb6ac6cc0}, 0x3eb6ac6cf0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eb46a9200, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3eb7692000}, {0x2c48448?, 0x3eb54fa160?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 6061 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 6160 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcb00870, {0x2c3c1f0, 0x3eb826d310}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6161 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcb008a0, {0x2c3c1f0, 0x3eb826d360}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6210 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcb008d0, {0x2c3c1f0, 0x3eb826d3b0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6185 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebc5abba0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6183 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 6186 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3ebbf38a38, 0x3a8) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3ebbf38a28) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3ebbf389c0, {0x3eb7710000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb7186f00, {0x3ebcc911c0, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb7186f00}, {0x3ebcc911c0, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebcc911c0, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb7186f00?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebcc91180) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebc5abba0, {0x2c3c1b8, 0x3eb7706690}, 0x3eb77066c0) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eba7cea00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebc5abba0}, {0x2c48448?, 0x3eb47894c0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 6183 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 goroutine 6212 [select]: github.com/grafana/loki/v3/pkg/ingester.(*StreamRateCalculator).updateLoop(0x3eb6e533b0) /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:59 +0xb4 created by github.com/grafana/loki/v3/pkg/ingester.NewStreamRateCalculator in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/stream_rate_calculator.go:49 +0x16c goroutine 6213 [select, 8 minutes]: github.com/grafana/dskit/services.(*BasicService).AddListener.func2() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:355 +0xcc created by github.com/grafana/dskit/services.(*BasicService).AddListener in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/services/basic_service.go:352 +0x23c goroutine 6214 [select, 8 minutes]: google.golang.org/grpc/test/bufconn.(*Listener).Accept(0x2c52b60?) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:61 +0x70 google.golang.org/grpc.(*Server).Serve(0x3eba7cec00, {0x2c308c8, 0x3eb7652340}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:890 +0x3d0 github.com/grafana/loki/v3/pkg/ingester.createIngesterServer.func3() /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1433 +0x38 created by github.com/grafana/loki/v3/pkg/ingester.createIngesterServer in goroutine 4467 /build/loki/src/loki-3.5.1/pkg/ingester/ingester_test.go:1432 +0x6b4 goroutine 6215 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcb015e0, {0x2c3c1f0, 0x3eb826d590}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6216 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcb01610, {0x2c3c1f0, 0x3eb826d5e0}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6217 [select, 8 minutes]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x3ebcb01640, {0x2c3c1f0, 0x3eb826d630}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xec created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 4467 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x12c goroutine 6219 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb5e307f8, 0x3a3) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb5e307e8) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb5e30780, {0x3eb7d00000, 0x8000, 0x0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb7651500, {0x3ebc7a9b60, 0x9, 0x5f3bf8?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb7651500}, {0x3ebc7a9b60, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a9b60, 0x9, 0x3eaee27638?}, {0x2c04240?, 0x3eb7651500?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a9b20) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x3ebc57fb08, 0x3eb6e53a40) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1640 +0x19c created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6229 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:414 +0x1684 goroutine 6244 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb767a0c0, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3eb6554680) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:472 +0xbc created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 6229 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0x1ac8 goroutine 6225 [select]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x3eb767a100, 0x1) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x114 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x3eb6554580) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x80 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:335 +0xc8 created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6224 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:333 +0x1300 goroutine 6242 [select, 8 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x3ebcafa340) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1180 +0x17c created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 6224 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:356 +0x1340 goroutine 6243 [sync.Cond.Wait]: sync.runtime_notifyListWait(0x3eb5e30738, 0x3a4) /usr/lib/go/src/runtime/sema.go:597 +0x168 sync.(*Cond).Wait(0x3eb5e30728) /usr/lib/go/src/sync/cond.go:71 +0xa8 google.golang.org/grpc/test/bufconn.(*pipe).Read(0x3eb5e306c0, {0x3eb7d10000, 0x8000, 0x300060000?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/test/bufconn/bufconn.go:173 +0xc0 bufio.(*Reader).Read(0x3eb7651620, {0x3ebc7a9c40, 0x9, 0x614468?}) /usr/lib/go/src/bufio/bufio.go:245 +0x1c8 io.ReadAtLeast({0x2c04240, 0x3eb7651620}, {0x3ebc7a9c40, 0x9, 0x9}, 0x9) /usr/lib/go/src/io/io.go:335 +0xd0 io.ReadFull(...) /usr/lib/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x3ebc7a9c40, 0x9, 0xc0f80?}, {0x2c04240?, 0x3eb7651620?}) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:242 +0x88 golang.org/x/net/http2.(*Framer).ReadFrame(0x3ebc7a9c00) /build/loki/src/loki-3.5.1/vendor/golang.org/x/net/http2/frame.go:506 +0x80 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x3ebcafa340, {0x2c3c1b8, 0x3eb7678750}, 0x3eb7678780) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/internal/transport/http2_server.go:640 +0xe8 google.golang.org/grpc.(*Server).serveStreams(0x3eba7cec00, {0x2c3c068?, 0x46fdfa0?}, {0x2c3d4f0, 0x3ebcafa340}, {0x2c48448?, 0x3eb76525c0?}) /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:1029 +0x324 google.golang.org/grpc.(*Server).handleRawConn.func1() /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:964 +0x60 created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 6224 /build/loki/src/loki-3.5.1/vendor/google.golang.org/grpc/server.go:963 +0x1f4 FAIL github.com/grafana/loki/v3/pkg/ingester 619.711s ? github.com/grafana/loki/v3/pkg/ingester/client [no test files] ok github.com/grafana/loki/v3/pkg/ingester/index 6.469s ok github.com/grafana/loki/v3/pkg/ingester/wal 5.422s ok github.com/grafana/loki/v3/pkg/iter 4.186s ok github.com/grafana/loki/v3/pkg/iter/v2 0.686s ok github.com/grafana/loki/v3/pkg/kafka 4.533s ok github.com/grafana/loki/v3/pkg/kafka/client 5.017s ok github.com/grafana/loki/v3/pkg/kafka/partition 9.348s ok github.com/grafana/loki/v3/pkg/kafka/partitionring 2.971s ok github.com/grafana/loki/v3/pkg/kafka/partitionring/consumer 35.004s ? github.com/grafana/loki/v3/pkg/kafka/testkafka [no test files] ok github.com/grafana/loki/v3/pkg/limits 5.131s ? github.com/grafana/loki/v3/pkg/limits/client [no test files] ok github.com/grafana/loki/v3/pkg/limits/frontend 5.681s ? github.com/grafana/loki/v3/pkg/limits/frontend/client [no test files] ok github.com/grafana/loki/v3/pkg/logcli/client 7.816s ? github.com/grafana/loki/v3/pkg/logcli/detected [no test files] ? github.com/grafana/loki/v3/pkg/logcli/index [no test files] ? github.com/grafana/loki/v3/pkg/logcli/labelquery [no test files] ok github.com/grafana/loki/v3/pkg/logcli/output 6.449s ok github.com/grafana/loki/v3/pkg/logcli/print 7.449s ok github.com/grafana/loki/v3/pkg/logcli/query 23.682s ? github.com/grafana/loki/v3/pkg/logcli/seriesquery [no test files] ? github.com/grafana/loki/v3/pkg/logcli/util [no test files] ? github.com/grafana/loki/v3/pkg/logcli/volume [no test files] ok github.com/grafana/loki/v3/pkg/loghttp 7.161s ? github.com/grafana/loki/v3/pkg/loghttp/legacy [no test files] ok github.com/grafana/loki/v3/pkg/loghttp/push 6.940s ok github.com/grafana/loki/v3/pkg/logproto 5.415s panic: test timed out after 10m0s running tests: TestApproxTopkSketches (8m34s) TestApproxTopkSketches/approx_topk(100,_sum_by_(a)_(sum_over_time_({a=~".+"}_|_logfmt_|_unwrap_value_[1s])))/100/100000 (7m13s) goroutine 1042 [running]: testing.(*M).startAlarm.func1() /usr/lib/go/src/testing/testing.go:2484 +0x304 created by time.goFunc /usr/lib/go/src/time/sleep.go:215 +0x58 goroutine 1 [chan receive, 8 minutes]: testing.(*T).Run(0x3f584d8540, {0x1231cc4?, 0x2f0db42058959b88?}, 0x12cdca8) /usr/lib/go/src/testing/testing.go:1859 +0x3b4 testing.runTests.func1(0x3f584d8540) /usr/lib/go/src/testing/testing.go:2279 +0x4c testing.tRunner(0x3f584d8540, 0x3f58959ca8) /usr/lib/go/src/testing/testing.go:1792 +0xf8 testing.runTests(0x3f58216a98, {0x21e6040, 0x45, 0x45}, {0x1d8f4?, 0x3fa279e5c0?, 0x21f7a00?}) /usr/lib/go/src/testing/testing.go:2277 +0x440 testing.(*M).Run(0x3f587ee460) /usr/lib/go/src/testing/testing.go:2142 +0x5bc main.main() _testmain.go:201 +0x94 goroutine 607 [chan receive, 7 minutes]: testing.(*T).Run(0x3f5b582380, {0x3ec02b0480?, 0x3f581a6da0?}, 0x3ec6dc5800) /usr/lib/go/src/testing/testing.go:1859 +0x3b4 github.com/grafana/loki/v3/pkg/logql.TestApproxTopkSketches(0x3f5b582380) /build/loki/src/loki-3.5.1/pkg/logql/downstream_test.go:351 +0xa8 testing.tRunner(0x3f5b582380, 0x12cdca8) /usr/lib/go/src/testing/testing.go:1792 +0xf8 created by testing.(*T).Run in goroutine 1 /usr/lib/go/src/testing/testing.go:1851 +0x3a0 goroutine 937 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d53542090, {0x15c3738, 0x3d53542008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d53542008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5c68540?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5c68540, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d57904130?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d52f00230}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cd0c8?, {0x15d0ec8?, 0x3d52f00230?}, {0x15bbee0?, 0x3d57964100?}, {0x15da6b0?, 0x3d57964040?}, {0x15db118?, 0x3f5b4a7380?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d52f00230?}, {0x15bbee0?, 0x3d57964100?}, 0x3d57964080, {0x15db118?, 0x3f5b4a7380?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d52f00230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57964080}, {0x15db118, 0x3f5b4a7380}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d52f00230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d579640c0, {0x15db118?, 0x3f5b4a7380?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d52f00230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d579640c0}, {0x15db118, 0x3f5b4a7380}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d52f001e0, {0x15d0ec8, 0x3d52f00230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d52f001e0, {0x15d0e90, 0x3d52f123f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x3a) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 954 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d58f5e090, {0x15c3738, 0x3d58f5e008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d58f5e008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec539a630?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec539a630, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d53a06050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000cc3a8?, {0x15d0ec8?, 0x3d53a06050?}, {0x15bbee0?, 0x3d53e14100?}, {0x15da6b0?, 0x3d53e14040?}, {0x15db118?, 0x3f5b4a6bd0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d53a06050?}, {0x15bbee0?, 0x3d53e14100?}, 0x3d53e14080, {0x15db118?, 0x3f5b4a6bd0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d53a06050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d53e14080}, {0x15db118, 0x3f5b4a6bd0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d53a06050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d53e140c0, {0x15db118?, 0x3f5b4a6bd0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d53a06050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d53e140c0}, {0x15db118, 0x3f5b4a6bd0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d53a06000, {0x15d0ec8, 0x3d53a06050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d53a06000, {0x15d0e90, 0x3d53a1c0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x19) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 952 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d3120a090, {0x15c3738, 0x3d3120a008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d3120a008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5622f60?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5622f60, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d55e00050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c3a28?, {0x15d0ec8?, 0x3d55e00050?}, {0x15bbee0?, 0x3d55e7e100?}, {0x15da6b0?, 0x3d55e7e040?}, {0x15db118?, 0x3f5b4a6f60?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d55e00050?}, {0x15bbee0?, 0x3d55e7e100?}, 0x3d55e7e080, {0x15db118?, 0x3f5b4a6f60?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d55e00050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d55e7e080}, {0x15db118, 0x3f5b4a6f60}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d55e00050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d55e7e0c0, {0x15db118?, 0x3f5b4a6f60?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d55e00050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d55e7e0c0}, {0x15db118, 0x3f5b4a6f60}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d55e00000, {0x15d0ec8, 0x3d55e00050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d55e00000, {0x15d0e90, 0x3d55e180c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x24) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 914 [sync.WaitGroup.Wait, 5 minutes]: sync.runtime_SemacquireWaitGroup(0x6d9210?) /usr/lib/go/src/runtime/sema.go:110 +0x38 sync.(*WaitGroup).Wait(0x3d30f185c8) /usr/lib/go/src/sync/waitgroup.go:118 +0x60 golang.org/x/sync/errgroup.(*Group).Wait(0x3d30f185c0) /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x34 github.com/grafana/dskit/concurrency.ForEachJob({0x15d0f38, 0x3f587c60e0}, 0x64, 0x64, 0x3d30f18580) /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:127 +0x290 github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream({0x0?}, {0x15d0f38, 0x3f587c60e0}, {0x3f58748a08, 0x64, 0x64}, {0x15c2cf0, 0x3f58713c08}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:245 +0x124 github.com/grafana/loki/v3/pkg/logql.DownstreamEvaluator.Downstream({{0x15bbe00?, 0x3f5825c0a0?}, {0x15c7ec0?, 0x3f587987e0?}}, {0x15d0f38, 0x3f587c60e0}, {0x3f58748a08?, 0x0?, 0x2?}, {0x15c2cf0?, ...}) /build/loki/src/loki-3.5.1/pkg/logql/downstream.go:452 +0xc4 github.com/grafana/loki/v3/pkg/logql.(*DownstreamEvaluator).NewStepEvaluator(0x3f58798800, {0x15d0f38, 0x3f587c60e0}, {0x15bbe20?, 0x3f58798800?}, {0x15da850?, 0x3f58a9d3e0}, {0x15db0a8, 0x3f587987c0}) /build/loki/src/loki-3.5.1/pkg/logql/downstream.go:679 +0x1aac github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0f38?, 0x3f587c60e0?}, {0x15bbe20?, 0x3f58798800?}, 0x3d4f8321c0, {0x15db0a8?, 0x3f587987c0?}, 0x0) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f587987e0, {0x15d0f38, 0x3f587c60e0}, {0x15bbe20?, 0x3f58798800?}, {0x15da7e8, 0x3d4f8321c0}, {0x15db0a8, 0x3f587987c0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*DownstreamEvaluator).NewStepEvaluator(0x3f58798800?, {0x15d0f38?, 0x3f587c60e0?}, {0x15bbe20?, 0x3f58798800?}, {0x15da7e8?, 0x3d4f8321c0?}, {0x15db0a8?, 0x3f587987c0?}) /build/loki/src/loki-3.5.1/pkg/logql/downstream.go:694 +0x388 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3f5825c7d0, {0x15d0f38, 0x3f587c60e0}, {0x15da7e8, 0x3d4f8321c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3f5825c7d0, {0x15d0e90, 0x3f5b4a6690}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.TestApproxTopkSketches.func1(0x3ec9800a80) /build/loki/src/loki-3.5.1/pkg/logql/downstream_test.go:401 +0xcac testing.tRunner(0x3ec9800a80, 0x3ec6dc5800) /usr/lib/go/src/testing/testing.go:1792 +0xf8 created by testing.(*T).Run in goroutine 607 /usr/lib/go/src/testing/testing.go:1851 +0x3a0 goroutine 933 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d437b5600, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d58cca090, {0x15c3738, 0x3d58cca008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d58cca008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4dd7bc0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4dd7bc0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d50d00050}, {0x29eec?, 0x10?}, {0x866a4?, 0x3d54e0a400?}, {0xc?, 0x2dc50?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c0000?, {0x15d0ec8?, 0x3d50d00050?}, {0x15bbee0?, 0x3d54650040?}, {0x15da6b0?, 0x3d531f3f00?}, {0x15db118?, 0x3f5b4a68a0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50d00050?}, {0x15bbee0?, 0x3d54650040?}, 0x3d531f3f40, {0x15db118?, 0x3f5b4a68a0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50d00050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d531f3f40}, {0x15db118, 0x3f5b4a68a0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50d00050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d54650000, {0x15db118?, 0x3f5b4a68a0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50d00050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d54650000}, {0x15db118, 0x3f5b4a68a0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d50d00000, {0x15d0ec8, 0x3d50d00050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d50d00000, {0x15d0e90, 0x3d50d120c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x9) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 939 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d532ec090, {0x15c3738, 0x3d532ec008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d532ec008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5a6a810?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5a6a810, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d50d00140}, {0x29eec?, 0x3aebc?}, {0x866a4?, 0x360807aa?}, {0x32788?, 0x221d060?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c2ab0?, {0x15d0ec8?, 0x3d50d00140?}, {0x15bbee0?, 0x3d448590c0?}, {0x15da6b0?, 0x3d44859000?}, {0x15db118?, 0x3f5b4a70e0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50d00140?}, {0x15bbee0?, 0x3d448590c0?}, 0x3d44859040, {0x15db118?, 0x3f5b4a70e0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50d00140}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d44859040}, {0x15db118, 0x3f5b4a70e0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50d00140?}, {0x3f54784410?, 0x3f58798040?}, 0x3d44859080, {0x15db118?, 0x3f5b4a70e0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50d00140}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d44859080}, {0x15db118, 0x3f5b4a70e0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57100000, {0x15d0ec8, 0x3d50d00140}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57100000, {0x15d0e90, 0x3d5342a390}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x2c) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 956 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d3120b590, {0x15c3738, 0x3d3120b508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d3120b508, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4c43230?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4c43230, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d53414050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c4c70?, {0x15d0ec8?, 0x3d53414050?}, {0x15bbee0?, 0x3d5347c100?}, {0x15da6b0?, 0x3d5347c040?}, {0x15db118?, 0x3f5b4a6b40?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d53414050?}, {0x15bbee0?, 0x3d5347c100?}, 0x3d5347c080, {0x15db118?, 0x3f5b4a6b40?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d53414050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5347c080}, {0x15db118, 0x3f5b4a6b40}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d53414050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d5347c0c0, {0x15db118?, 0x3f5b4a6b40?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d53414050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5347c0c0}, {0x15db118, 0x3f5b4a6b40}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d53414000, {0x15d0ec8, 0x3d53414050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d53414000, {0x15d0e90, 0x3d5342a0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x16) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 957 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d510cca90, {0x15c3738, 0x3d510cca08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d510cca08, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec56ef8f0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec56ef8f0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d53314050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000cebf0?, {0x15d0ec8?, 0x3d53314050?}, {0x15bbee0?, 0x3d5337c100?}, {0x15da6b0?, 0x3d5337c040?}, {0x15db118?, 0x3f5b4a6ae0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d53314050?}, {0x15bbee0?, 0x3d5337c100?}, 0x3d5337c080, {0x15db118?, 0x3f5b4a6ae0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d53314050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5337c080}, {0x15db118, 0x3f5b4a6ae0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d53314050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d5337c0c0, {0x15db118?, 0x3f5b4a6ae0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d53314050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5337c0c0}, {0x15db118, 0x3f5b4a6ae0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d53314000, {0x15d0ec8, 0x3d53314050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d53314000, {0x15d0e90, 0x3d533240c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x15) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 953 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d450a1900, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d40210a90, {0x15c3738, 0x3d40210a08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d40210a08, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec170e690?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec170e690, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d53084050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c86b8?, {0x15d0ec8?, 0x3d53084050?}, {0x15bbee0?, 0x3d530f0100?}, {0x15da6b0?, 0x3d530f0040?}, {0x15db118?, 0x3f5b4a6a80?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d53084050?}, {0x15bbee0?, 0x3d530f0100?}, 0x3d530f0080, {0x15db118?, 0x3f5b4a6a80?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d53084050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d530f0080}, {0x15db118, 0x3f5b4a6a80}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d53084050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d530f00c0, {0x15db118?, 0x3f5b4a6a80?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d53084050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d530f00c0}, {0x15db118, 0x3f5b4a6a80}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d53084000, {0x15d0ec8, 0x3d53084050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d53084000, {0x15d0e90, 0x3d530900c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x13) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 958 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d311b8090, {0x15c3738, 0x3d311b8008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d311b8008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5744db0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5744db0, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d5718a320?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3ec4a3e230}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000ce598?, {0x15d0ec8?, 0x3ec4a3e230?}, {0x15bbee0?, 0x3d571f0100?}, {0x15da6b0?, 0x3d571f0040?}, {0x15db118?, 0x3f5b4a6d80?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3ec4a3e230?}, {0x15bbee0?, 0x3d571f0100?}, 0x3d571f0080, {0x15db118?, 0x3f5b4a6d80?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3ec4a3e230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d571f0080}, {0x15db118, 0x3f5b4a6d80}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3ec4a3e230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d571f00c0, {0x15db118?, 0x3f5b4a6d80?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3ec4a3e230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d571f00c0}, {0x15db118, 0x3f5b4a6d80}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3ec4a3e1e0, {0x15d0ec8, 0x3ec4a3e230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3ec4a3e1e0, {0x15d0e90, 0x3d50946240}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x1a) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 955 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d40386090, {0x15c3738, 0x3d40386008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d40386008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec53807b0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec53807b0, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d51a8a050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000ca088?, {0x15d0ec8?, 0x3d51a8a050?}, {0x15bbee0?, 0x3d51af8100?}, {0x15da6b0?, 0x3d51af8040?}, {0x15db118?, 0x3f5b4a69c0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d51a8a050?}, {0x15bbee0?, 0x3d51af8100?}, 0x3d51af8080, {0x15db118?, 0x3f5b4a69c0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d51a8a050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d51af8080}, {0x15db118, 0x3f5b4a69c0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d51a8a050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d51af80c0, {0x15db118?, 0x3f5b4a69c0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d51a8a050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d51af80c0}, {0x15db118, 0x3f5b4a69c0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d51a8a000, {0x15d0ec8, 0x3d51a8a050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d51a8a000, {0x15d0e90, 0x3d51a980c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0xf) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 944 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d5123e090, {0x15c3738, 0x3d5123e008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d5123e008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec41e8ff0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec41e8ff0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d50e90050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c2160?, {0x15d0ec8?, 0x3d50e90050?}, {0x15bbee0?, 0x3d50efe100?}, {0x15da6b0?, 0x3d50efe040?}, {0x15db118?, 0x3f5b4a68d0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50e90050?}, {0x15bbee0?, 0x3d50efe100?}, 0x3d50efe080, {0x15db118?, 0x3f5b4a68d0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50e90050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d50efe080}, {0x15db118, 0x3f5b4a68d0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50e90050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d50efe0c0, {0x15db118?, 0x3f5b4a68d0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50e90050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d50efe0c0}, {0x15db118, 0x3f5b4a68d0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d50e90000, {0x15d0ec8, 0x3d50e90050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d50e90000, {0x15d0e90, 0x3d50ea80c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0xa) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 945 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d5123f590, {0x15c3738, 0x3d5123f508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d5123f508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec57b73b0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec57b73b0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d50e90230}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c2160?, {0x15d0ec8?, 0x3d50e90230?}, {0x15bbee0?, 0x3d543d0600?}, {0x15da6b0?, 0x3d543d0540?}, {0x15db118?, 0x3f5b4a6e10?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50e90230?}, {0x15bbee0?, 0x3d543d0600?}, 0x3d543d0580, {0x15db118?, 0x3f5b4a6e10?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50e90230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d543d0580}, {0x15db118, 0x3f5b4a6e10}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50e90230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d543d05c0, {0x15db118?, 0x3f5b4a6e10?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50e90230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d543d05c0}, {0x15db118, 0x3f5b4a6e10}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d50e901e0, {0x15d0ec8, 0x3d50e90230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d50e901e0, {0x15d0e90, 0x3d50ea83f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x1d) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 950 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d58716090, {0x15c3738, 0x3d58716008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d58716008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4bcaea0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4bcaea0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d57694530?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d5768e410}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cdb80?, {0x15d0ec8?, 0x3d5768e410?}, {0x15bbee0?, 0x3d57a132c0?}, {0x15da6b0?, 0x3d57a13200?}, {0x15db118?, 0x3f5b4a7aa0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5768e410?}, {0x15bbee0?, 0x3d57a132c0?}, 0x3d57a13240, {0x15db118?, 0x3f5b4a7aa0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5768e410}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57a13240}, {0x15db118, 0x3f5b4a7aa0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5768e410?}, {0x3f54784410?, 0x3f58798040?}, 0x3d57a13280, {0x15db118?, 0x3f5b4a7aa0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5768e410}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57a13280}, {0x15db118, 0x3f5b4a7aa0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d5768e3c0, {0x15d0ec8, 0x3d5768e410}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d5768e3c0, {0x15d0e90, 0x3d576986c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x5e) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 941 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57b89590, {0x15c3738, 0x3d57b89508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57b89508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec49cce70?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec49cce70, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d58aa8170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d58a80050}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000ceb90?, {0x15d0ec8?, 0x3d58a80050?}, {0x15bbee0?, 0x3d58b38100?}, {0x15da6b0?, 0x3d58b38040?}, {0x15db118?, 0x3f5b4a7710?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58a80050?}, {0x15bbee0?, 0x3d58b38100?}, 0x3d58b38080, {0x15db118?, 0x3f5b4a7710?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58a80050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d58b38080}, {0x15db118, 0x3f5b4a7710}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58a80050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d58b380c0, {0x15db118?, 0x3f5b4a7710?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58a80050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d58b380c0}, {0x15db118, 0x3f5b4a7710}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d58a80000, {0x15d0ec8, 0x3d58a80050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d58a80000, {0x15d0e90, 0x3d58ac20c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x4b) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 938 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d40706090, {0x15c3738, 0x3d40706008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d40706008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec58a2450?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec58a2450, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d50d000f0}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c2ab0?, {0x15d0ec8?, 0x3d50d000f0?}, {0x15bbee0?, 0x3d50d66100?}, {0x15da6b0?, 0x3d50d66040?}, {0x15db118?, 0x3f5b4a6900?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50d000f0?}, {0x15bbee0?, 0x3d50d66100?}, 0x3d50d66080, {0x15db118?, 0x3f5b4a6900?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50d000f0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d50d66080}, {0x15db118, 0x3f5b4a6900}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50d000f0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d50d660c0, {0x15db118?, 0x3f5b4a6900?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50d000f0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d50d660c0}, {0x15db118, 0x3f5b4a6900}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d50d000a0, {0x15d0ec8, 0x3d50d000f0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d50d000a0, {0x15d0e90, 0x3d50d121b0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0xb) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 940 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d43f33100, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d5365a090, {0x15c3738, 0x3d5365a008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d5365a008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec522bd70?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec522bd70, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d53798050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000cae08?, {0x15d0ec8?, 0x3d53798050?}, {0x15bbee0?, 0x3d537fe100?}, {0x15da6b0?, 0x3d537fe040?}, {0x15db118?, 0x3f5b4a6ba0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d53798050?}, {0x15bbee0?, 0x3d537fe100?}, 0x3d537fe080, {0x15db118?, 0x3f5b4a6ba0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d53798050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d537fe080}, {0x15db118, 0x3f5b4a6ba0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d53798050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d537fe0c0, {0x15db118?, 0x3f5b4a6ba0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d53798050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d537fe0c0}, {0x15db118, 0x3f5b4a6ba0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d53798000, {0x15d0ec8, 0x3d53798050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d53798000, {0x15d0e90, 0x3d537ac0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x18) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 951 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57740a90, {0x15c3738, 0x3d57740a08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57740a08, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4f656e0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4f656e0, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d57598560?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57580460}, {0x40?, 0x0?}, {0x38?, 0x0?}, {0x15da7e8?, 0x3d57774f00?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x221bd60?, {0x15d0ec8?, 0x3d57580460?}, {0x15bbee0?, 0x3d57774f80?}, {0x15da6b0?, 0x3d57774ec0?}, {0x15db118?, 0x3f5b4a7ad0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57580460?}, {0x15bbee0?, 0x3d57774f80?}, 0x3d57774f00, {0x15db118?, 0x3f5b4a7ad0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57580460}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57774f00}, {0x15db118, 0x3f5b4a7ad0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57580460?}, {0x3f54784410?, 0x3f58798040?}, 0x3d57774f40, {0x15db118?, 0x3f5b4a7ad0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57580460}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57774f40}, {0x15db118, 0x3f5b4a7ad0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57580410, {0x15d0ec8, 0x3d57580460}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57580410, {0x15d0e90, 0x3d575a06f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x5f) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 946 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d56db0090, {0x15c3738, 0x3d56db0008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1608 +0x5018 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d56db0008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec58e4cc0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec58e4cc0, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d50f00410}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000cab88?, {0x15d0ec8?, 0x3d50f00410?}, {0x15bbee0?, 0x3d56942d00?}, {0x15da6b0?, 0x3d56942c40?}, {0x15db118?, 0x3f5b4a7200?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50f00410?}, {0x15bbee0?, 0x3d56942d00?}, 0x3d56942c80, {0x15db118?, 0x3f5b4a7200?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50f00410}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d56942c80}, {0x15db118, 0x3f5b4a7200}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50f00410?}, {0x3f54784410?, 0x3f58798040?}, 0x3d56942cc0, {0x15db118?, 0x3f5b4a7200?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50f00410}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d56942cc0}, {0x15db118, 0x3f5b4a7200}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d50f003c0, {0x15d0ec8, 0x3d50f00410}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d50f003c0, {0x15d0e90, 0x3d50f106f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x32) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 943 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d3109ca90, {0x15c3738, 0x3d3109ca08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d3109ca08, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5a9b9e0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5a9b9e0, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d57694170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d5768e050}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cdb80?, {0x15d0ec8?, 0x3d5768e050?}, {0x15bbee0?, 0x3d57a12100?}, {0x15da6b0?, 0x3d57a12040?}, {0x15db118?, 0x3f5b4a7410?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5768e050?}, {0x15bbee0?, 0x3d57a12100?}, 0x3d57a12080, {0x15db118?, 0x3f5b4a7410?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5768e050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57a12080}, {0x15db118, 0x3f5b4a7410}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5768e050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d57a120c0, {0x15db118?, 0x3f5b4a7410?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5768e050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57a120c0}, {0x15db118, 0x3f5b4a7410}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d5768e000, {0x15d0ec8, 0x3d5768e050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d5768e000, {0x15d0e90, 0x3d576980c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x3d) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 942 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d40387590, {0x15c3738, 0x3d40387508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d40387508, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec467b2c0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec467b2c0, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3f581fe0f0}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c7f08?, {0x15d0ec8?, 0x3f581fe0f0?}, {0x15bbee0?, 0x3d508ba300?}, {0x15da6b0?, 0x3d508ba240?}, {0x15db118?, 0x3f5b4a66f0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f581fe0f0?}, {0x15bbee0?, 0x3d508ba300?}, 0x3d508ba280, {0x15db118?, 0x3f5b4a66f0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f581fe0f0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d508ba280}, {0x15db118, 0x3f5b4a66f0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f581fe0f0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d508ba2c0, {0x15db118?, 0x3f5b4a66f0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f581fe0f0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d508ba2c0}, {0x15db118, 0x3f5b4a66f0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3f581fe050, {0x15d0ec8, 0x3f581fe0f0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3f581fe050, {0x15d0e90, 0x3f58736d80}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x1) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 948 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d51060a90, {0x15c3738, 0x3d51060a08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d51060a08, 0x2d?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4ca6420?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4ca6420, 0x2d}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d50e900a0}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000cc940?, {0x15d0ec8?, 0x3d50e900a0?}, {0x15bbee0?, 0x3d44d1e180?}, {0x15da6b0?, 0x3d44d1e0c0?}, {0x15db118?, 0x3f5b4a7170?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50e900a0?}, {0x15bbee0?, 0x3d44d1e180?}, 0x3d44d1e100, {0x15db118?, 0x3f5b4a7170?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50e900a0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d44d1e100}, {0x15db118, 0x3f5b4a7170}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50e900a0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d44d1e140, {0x15db118?, 0x3f5b4a7170?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50e900a0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d44d1e140}, {0x15db118, 0x3f5b4a7170}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57480000, {0x15d0ec8, 0x3d50e900a0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57480000, {0x15d0e90, 0x3d537ac330}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x2f) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 949 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d51144090, {0x15c3738, 0x3d51144008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d51144008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5470060?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5470060, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d51706320}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c27b8?, {0x15d0ec8?, 0x3d51706320?}, {0x15bbee0?, 0x3d46472100?}, {0x15da6b0?, 0x3d46472040?}, {0x15db118?, 0x3f5b4a7a70?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d51706320?}, {0x15bbee0?, 0x3d46472100?}, 0x3d46472080, {0x15db118?, 0x3f5b4a7a70?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d51706320}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d46472080}, {0x15db118, 0x3f5b4a7a70}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d51706320?}, {0x3f54784410?, 0x3f58798040?}, 0x3d464720c0, {0x15db118?, 0x3f5b4a7a70?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d51706320}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d464720c0}, {0x15db118, 0x3f5b4a7a70}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d575803c0, {0x15d0ec8, 0x3d51706320}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d575803c0, {0x15d0e90, 0x3d51722540}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x5d) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 947 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57fa2a90, {0x15c3738, 0x3d57fa2a08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57fa2a08, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4480ae0?, 0x3d3d274008?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4480ae0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d56406230}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c7208?, {0x15d0ec8?, 0x3d56406230?}, {0x15bbee0?, 0x3d55ee0200?}, {0x15da6b0?, 0x3d55ee0140?}, {0x15db118?, 0x3f5b4a72c0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d56406230?}, {0x15bbee0?, 0x3d55ee0200?}, 0x3d55ee0180, {0x15db118?, 0x3f5b4a72c0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d56406230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d55ee0180}, {0x15db118, 0x3f5b4a72c0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d56406230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d55ee01c0, {0x15db118?, 0x3f5b4a72c0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d56406230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d55ee01c0}, {0x15db118, 0x3f5b4a72c0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d564061e0, {0x15d0ec8, 0x3d56406230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d564061e0, {0x15d0e90, 0x3d564203f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x36) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 936 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d51145590, {0x15c3738, 0x3d51145508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d51145508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec50c21e0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec50c21e0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d563f4090?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d53680230}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000ceeb8?, {0x15d0ec8?, 0x3d53680230?}, {0x15bbee0?, 0x3d563a3e80?}, {0x15da6b0?, 0x3d563a3dc0?}, {0x15db118?, 0x3f5b4a7350?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d53680230?}, {0x15bbee0?, 0x3d563a3e80?}, 0x3d563a3e00, {0x15db118?, 0x3f5b4a7350?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d53680230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d563a3e00}, {0x15db118, 0x3f5b4a7350}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d53680230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d563a3e40, {0x15db118?, 0x3f5b4a7350?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d53680230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d563a3e40}, {0x15db118, 0x3f5b4a7350}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d536801e0, {0x15d0ec8, 0x3d53680230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d536801e0, {0x15d0e90, 0x3d536b83f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x39) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 908 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57a6ea90, {0x15c3738, 0x3d57a6ea08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1041 +0x7b0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57a6ea08, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5a96bd0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5a96bd0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3f5825ccd0}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000ce458?, {0x15d0ec8?, 0x3f5825ccd0?}, {0x15bbee0?, 0x3d30f18700?}, {0x15da6b0?, 0x3d30f18640?}, {0x15db118?, 0x3f5b4a66c0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f5825ccd0?}, {0x15bbee0?, 0x3d30f18700?}, 0x3d30f18680, {0x15db118?, 0x3f5b4a66c0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f5825ccd0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d30f18680}, {0x15db118, 0x3f5b4a66c0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f5825ccd0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d30f186c0, {0x15db118?, 0x3f5b4a66c0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f5825ccd0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d30f186c0}, {0x15db118, 0x3f5b4a66c0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3f5825cc80, {0x15d0ec8, 0x3f5825ccd0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3f5825cc80, {0x15d0e90, 0x3f5b4a7d40}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x0) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 689 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d510ca090, {0x15c3738, 0x3d510ca008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d510ca008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec0d79e00?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec0d79e00, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3ec5300050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000cd3b8?, {0x15d0ec8?, 0x3ec5300050?}, {0x15bbee0?, 0x3d50ad4100?}, {0x15da6b0?, 0x3d50ad4040?}, {0x15db118?, 0x3f5b4a67e0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3ec5300050?}, {0x15bbee0?, 0x3d50ad4100?}, 0x3d50ad4080, {0x15db118?, 0x3f5b4a67e0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3ec5300050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d50ad4080}, {0x15db118, 0x3f5b4a67e0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3ec5300050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d50ad40c0, {0x15db118?, 0x3f5b4a67e0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3ec5300050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d50ad40c0}, {0x15db118, 0x3f5b4a67e0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3ec5300000, {0x15d0ec8, 0x3ec5300050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3ec5300000, {0x15d0e90, 0x3f58025290}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x6) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 930 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d3f9e5700, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d58d77590, {0x15c3738, 0x3d58d77508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d58d77508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5a0bb60?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5a0bb60, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d572b8090?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3f581fe2d0}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c2160?, {0x15d0ec8?, 0x3f581fe2d0?}, {0x15bbee0?, 0x3d572f2100?}, {0x15da6b0?, 0x3d572f2040?}, {0x15db118?, 0x3f5b4a6ed0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f581fe2d0?}, {0x15bbee0?, 0x3d572f2100?}, 0x3d572f2080, {0x15db118?, 0x3f5b4a6ed0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f581fe2d0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d572f2080}, {0x15db118, 0x3f5b4a6ed0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f581fe2d0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d572f20c0, {0x15db118?, 0x3f5b4a6ed0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f581fe2d0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d572f20c0}, {0x15db118, 0x3f5b4a6ed0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3f581fe280, {0x15d0ec8, 0x3f581fe2d0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3f581fe280, {0x15d0e90, 0x3f587370b0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x21) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 688 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d40487590, {0x15c3738, 0x3d40487508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d40487508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec442ca50?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec442ca50, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3f582c6550}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000ce9f8?, {0x15d0ec8?, 0x3f582c6550?}, {0x15bbee0?, 0x3d509d8100?}, {0x15da6b0?, 0x3d509d8040?}, {0x15db118?, 0x3f5b4a6780?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f582c6550?}, {0x15bbee0?, 0x3d509d8100?}, 0x3d509d8080, {0x15db118?, 0x3f5b4a6780?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f582c6550}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d509d8080}, {0x15db118, 0x3f5b4a6780}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f582c6550?}, {0x3f54784410?, 0x3f58798040?}, 0x3d509d80c0, {0x15db118?, 0x3f5b4a6780?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f582c6550}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d509d80c0}, {0x15db118, 0x3f5b4a6780}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3f582c6500, {0x15d0ec8, 0x3f582c6550}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3f582c6500, {0x15d0e90, 0x3f5893ca50}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x4) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 935 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d561f7590, {0x15c3738, 0x3d561f7508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d561f7508, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4f54540?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4f54540, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d50d002d0}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c2ab0?, {0x15d0ec8?, 0x3d50d002d0?}, {0x15bbee0?, 0x3d54460600?}, {0x15da6b0?, 0x3d54460540?}, {0x15db118?, 0x3f5b4a6de0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50d002d0?}, {0x15bbee0?, 0x3d54460600?}, 0x3d54460580, {0x15db118?, 0x3f5b4a6de0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50d002d0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d54460580}, {0x15db118, 0x3f5b4a6de0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50d002d0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d544605c0, {0x15db118?, 0x3f5b4a6de0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50d002d0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d544605c0}, {0x15db118, 0x3f5b4a6de0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d50d00280, {0x15d0ec8, 0x3d50d002d0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d50d00280, {0x15d0e90, 0x3d50d124e0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x1c) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 931 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d51db6090, {0x15c3738, 0x3d51db6008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d51db6008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec2483b30?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec2483b30, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d56406050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c7208?, {0x15d0ec8?, 0x3d56406050?}, {0x15bbee0?, 0x3d5647e100?}, {0x15da6b0?, 0x3d5647e040?}, {0x15db118?, 0x3f5b4a6fc0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d56406050?}, {0x15bbee0?, 0x3d5647e100?}, 0x3d5647e080, {0x15db118?, 0x3f5b4a6fc0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d56406050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5647e080}, {0x15db118, 0x3f5b4a6fc0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d56406050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d5647e0c0, {0x15db118?, 0x3f5b4a6fc0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d56406050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5647e0c0}, {0x15db118, 0x3f5b4a6fc0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d56406000, {0x15d0ec8, 0x3d56406050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d56406000, {0x15d0e90, 0x3d564200c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x26) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 932 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d40707590, {0x15c3738, 0x3d40707508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d40707508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec45ab170?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec45ab170, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d50c16050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000cb018?, {0x15d0ec8?, 0x3d50c16050?}, {0x15bbee0?, 0x3d5102a100?}, {0x15da6b0?, 0x3d5102a040?}, {0x15db118?, 0x3f5b4a6870?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50c16050?}, {0x15bbee0?, 0x3d5102a100?}, 0x3d5102a080, {0x15db118?, 0x3f5b4a6870?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50c16050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5102a080}, {0x15db118, 0x3f5b4a6870}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50c16050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d5102a0c0, {0x15db118?, 0x3f5b4a6870?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50c16050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5102a0c0}, {0x15db118, 0x3f5b4a6870}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d50c16000, {0x15d0ec8, 0x3d50c16050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d50c16000, {0x15d0e90, 0x3d50c4e0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x8) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 934 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d44b36200, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d578c2090, {0x15c3738, 0x3d578c2008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d578c2008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5269950?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5269950, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d50b8e050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c3a18?, {0x15d0ec8?, 0x3d50b8e050?}, {0x15bbee0?, 0x3d50be6100?}, {0x15da6b0?, 0x3d50be6040?}, {0x15db118?, 0x3f5b4a6750?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50b8e050?}, {0x15bbee0?, 0x3d50be6100?}, 0x3d50be6080, {0x15db118?, 0x3f5b4a6750?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50b8e050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d50be6080}, {0x15db118, 0x3f5b4a6750}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50b8e050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d50be60c0, {0x15db118?, 0x3f5b4a6750?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50b8e050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d50be60c0}, {0x15db118, 0x3f5b4a6750}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d50b8e000, {0x15d0ec8, 0x3d50b8e050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d50b8e000, {0x15d0e90, 0x3d50b960c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x3) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 959 [runnable]: github.com/prometheus/prometheus/promql/parser.yylex1({0x15c3738, 0x3d53cb8008}, 0x3d53cb8090) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:824 +0x18 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d53cb8090, {0x15c3738, 0x3d53cb8008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:911 +0x8bb4 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d53cb8008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec42241e0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec42241e0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d5782c170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57806050}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cf590?, {0x15d0ec8?, 0x3d57806050?}, {0x15bbee0?, 0x3d57a9a100?}, {0x15da6b0?, 0x3d57a9a040?}, {0x15db118?, 0x3f5b4a74d0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57806050?}, {0x15bbee0?, 0x3d57a9a100?}, 0x3d57a9a080, {0x15db118?, 0x3f5b4a74d0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57806050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57a9a080}, {0x15db118, 0x3f5b4a74d0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57806050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d57a9a0c0, {0x15db118?, 0x3f5b4a74d0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57806050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57a9a0c0}, {0x15db118, 0x3f5b4a74d0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57806000, {0x15d0ec8, 0x3d57806050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57806000, {0x15d0e90, 0x3d5782e0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x3f) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 960 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d4020f590, {0x15c3738, 0x3d4020f508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d4020f508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec2871500?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec2871500, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3ec4a3e050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000cc6a8?, {0x15d0ec8?, 0x3ec4a3e050?}, {0x15bbee0?, 0x3d5094e100?}, {0x15da6b0?, 0x3d5094e040?}, {0x15db118?, 0x3f5b4a6720?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3ec4a3e050?}, {0x15bbee0?, 0x3d5094e100?}, 0x3d5094e080, {0x15db118?, 0x3f5b4a6720?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3ec4a3e050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5094e080}, {0x15db118, 0x3f5b4a6720}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3ec4a3e050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d5094e0c0, {0x15db118?, 0x3f5b4a6720?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3ec4a3e050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5094e0c0}, {0x15db118, 0x3f5b4a6720}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3ec4a3e000, {0x15d0ec8, 0x3ec4a3e050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3ec4a3e000, {0x15d0e90, 0x3f58a9c5a0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x2) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 961 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57b88090, {0x15c3738, 0x3d57b88008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57b88008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec14995f0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec14995f0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d52f00050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c0508?, {0x15d0ec8?, 0x3d52f00050?}, {0x15bbee0?, 0x3d52f6c100?}, {0x15da6b0?, 0x3d52f6c040?}, {0x15db118?, 0x3f5b4a6a50?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d52f00050?}, {0x15bbee0?, 0x3d52f6c100?}, 0x3d52f6c080, {0x15db118?, 0x3f5b4a6a50?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d52f00050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d52f6c080}, {0x15db118, 0x3f5b4a6a50}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d52f00050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d52f6c0c0, {0x15db118?, 0x3f5b4a6a50?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d52f00050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d52f6c0c0}, {0x15db118, 0x3f5b4a6a50}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d52f00000, {0x15d0ec8, 0x3d52f00050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d52f00000, {0x15d0e90, 0x3d52f120c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x12) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 962 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d43549a00, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57fa1590, {0x15c3738, 0x3d57fa1508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57fa1508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec579fd70?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec579fd70, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d52f8c050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c92e8?, {0x15d0ec8?, 0x3d52f8c050?}, {0x15bbee0?, 0x3d53890100?}, {0x15da6b0?, 0x3d53890040?}, {0x15db118?, 0x3f5b4a6a20?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d52f8c050?}, {0x15bbee0?, 0x3d53890100?}, 0x3d53890080, {0x15db118?, 0x3f5b4a6a20?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d52f8c050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d53890080}, {0x15db118, 0x3f5b4a6a20}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d52f8c050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d538900c0, {0x15db118?, 0x3f5b4a6a20?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d52f8c050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d538900c0}, {0x15db118, 0x3f5b4a6a20}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d52f8c000, {0x15d0ec8, 0x3d52f8c050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d52f8c000, {0x15d0e90, 0x3d52f980c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x11) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 963 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d58aba090, {0x15c3738, 0x3d58aba008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d58aba008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec481cf30?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec481cf30, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d58aa8310?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d51706230}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000ceb90?, {0x15d0ec8?, 0x3d51706230?}, {0x15bbee0?, 0x3d58b38280?}, {0x15da6b0?, 0x3d58b381c0?}, {0x15db118?, 0x3f5b4a6ea0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d51706230?}, {0x15bbee0?, 0x3d58b38280?}, 0x3d58b38200, {0x15db118?, 0x3f5b4a6ea0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d51706230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d58b38200}, {0x15db118, 0x3f5b4a6ea0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d51706230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d58b38240, {0x15db118?, 0x3f5b4a6ea0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d51706230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d58b38240}, {0x15db118, 0x3f5b4a6ea0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d517061e0, {0x15d0ec8, 0x3d51706230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d517061e0, {0x15d0e90, 0x3d517223f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x20) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 964 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d5724c090, {0x15c3738, 0x3d5724c008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1608 +0x5018 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d5724c008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec6e63440?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec6e63440, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d51706050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c27b8?, {0x15d0ec8?, 0x3d51706050?}, {0x15bbee0?, 0x3d5177e100?}, {0x15da6b0?, 0x3d5177e040?}, {0x15db118?, 0x3f5b4a6960?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d51706050?}, {0x15bbee0?, 0x3d5177e100?}, 0x3d5177e080, {0x15db118?, 0x3f5b4a6960?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d51706050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5177e080}, {0x15db118, 0x3f5b4a6960}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d51706050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d5177e0c0, {0x15db118?, 0x3f5b4a6960?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d51706050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5177e0c0}, {0x15db118, 0x3f5b4a6960}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d51706000, {0x15d0ec8, 0x3d51706050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d51706000, {0x15d0e90, 0x3d517220c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0xd) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 965 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d51891590, {0x15c3738, 0x3d51891508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d51891508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4ae1d10?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4ae1d10, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d55d80050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c86c8?, {0x15d0ec8?, 0x3d55d80050?}, {0x15bbee0?, 0x3d5619a100?}, {0x15da6b0?, 0x3d5619a040?}, {0x15db118?, 0x3f5b4a6f30?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d55d80050?}, {0x15bbee0?, 0x3d5619a100?}, 0x3d5619a080, {0x15db118?, 0x3f5b4a6f30?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d55d80050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5619a080}, {0x15db118, 0x3f5b4a6f30}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d55d80050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d5619a0c0, {0x15db118?, 0x3f5b4a6f30?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d55d80050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5619a0c0}, {0x15db118, 0x3f5b4a6f30}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d55d80000, {0x15d0ec8, 0x3d55d80050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d55d80000, {0x15d0e90, 0x3d55daa0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x23) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 966 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d3109b590, {0x15c3738, 0x3d3109b508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d3109b508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec02df590?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec02df590, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d51b84050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c16b8?, {0x15d0ec8?, 0x3d51b84050?}, {0x15bbee0?, 0x3d51d8c100?}, {0x15da6b0?, 0x3d51d8c040?}, {0x15db118?, 0x3f5b4a69f0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d51b84050?}, {0x15bbee0?, 0x3d51d8c100?}, 0x3d51d8c080, {0x15db118?, 0x3f5b4a69f0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d51b84050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d51d8c080}, {0x15db118, 0x3f5b4a69f0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d51b84050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d51d8c0c0, {0x15db118?, 0x3f5b4a69f0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d51b84050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d51d8c0c0}, {0x15db118, 0x3f5b4a69f0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d51b84000, {0x15d0ec8, 0x3d51b84050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d51b84000, {0x15d0e90, 0x3d51b980c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x10) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 967 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d44c25d00, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3f5a4f4090, {0x15c3738, 0x3f5a4f4008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3f5a4f4008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec522af90?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec522af90, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d50f00230}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000cab88?, {0x15d0ec8?, 0x3d50f00230?}, {0x15bbee0?, 0x3d54a36280?}, {0x15da6b0?, 0x3d54a361c0?}, {0x15db118?, 0x3f5b4a6e40?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50f00230?}, {0x15bbee0?, 0x3d54a36280?}, 0x3d54a36200, {0x15db118?, 0x3f5b4a6e40?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50f00230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d54a36200}, {0x15db118, 0x3f5b4a6e40}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50f00230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d54a36240, {0x15db118?, 0x3f5b4a6e40?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50f00230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d54a36240}, {0x15db118, 0x3f5b4a6e40}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d50f001e0, {0x15d0ec8, 0x3d50f00230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d50f001e0, {0x15d0e90, 0x3d50f103f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x1e) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 968 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57f44090, {0x15c3738, 0x3d57f44008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57f44008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec45ffda0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec45ffda0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d50f00050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000cab88?, {0x15d0ec8?, 0x3d50f00050?}, {0x15bbee0?, 0x3d50f6a100?}, {0x15da6b0?, 0x3d50f6a040?}, {0x15db118?, 0x3f5b4a6930?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50f00050?}, {0x15bbee0?, 0x3d50f6a100?}, 0x3d50f6a080, {0x15db118?, 0x3f5b4a6930?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50f00050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d50f6a080}, {0x15db118, 0x3f5b4a6930}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50f00050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d50f6a0c0, {0x15db118?, 0x3f5b4a6930?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50f00050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d50f6a0c0}, {0x15db118, 0x3f5b4a6930}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d50f00000, {0x15d0ec8, 0x3d50f00050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d50f00000, {0x15d0e90, 0x3d50f100c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0xc) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 969 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d45392a00, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d58958090, {0x15c3738, 0x3d58958008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d58958008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4e67980?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4e67980, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d563f4620?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d563fc460}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000caf48?, {0x15d0ec8?, 0x3d563fc460?}, {0x15bbee0?, 0x3d310d40c0?}, {0x15da6b0?, 0x3d310d4000?}, {0x15db118?, 0x3f5b4a7260?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d563fc460?}, {0x15bbee0?, 0x3d310d40c0?}, 0x3d310d4040, {0x15db118?, 0x3f5b4a7260?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d563fc460}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d310d4040}, {0x15db118, 0x3f5b4a7260}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d563fc460?}, {0x3f54784410?, 0x3f58798040?}, 0x3d310d4080, {0x15db118?, 0x3f5b4a7260?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d563fc460}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d310d4080}, {0x15db118, 0x3f5b4a7260}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3ec4a3e280, {0x15d0ec8, 0x3d563fc460}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3ec4a3e280, {0x15d0e90, 0x3d54d22840}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x34) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 970 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57d88090, {0x15c3738, 0x3d57d88008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57d88008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4dae9c0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4dae9c0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d5300e050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000cae58?, {0x15d0ec8?, 0x3d5300e050?}, {0x15bbee0?, 0x3d5306e100?}, {0x15da6b0?, 0x3d5306e040?}, {0x15db118?, 0x3f5b4a6ab0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5300e050?}, {0x15bbee0?, 0x3d5306e100?}, 0x3d5306e080, {0x15db118?, 0x3f5b4a6ab0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5300e050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5306e080}, {0x15db118, 0x3f5b4a6ab0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5300e050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d5306e0c0, {0x15db118?, 0x3f5b4a6ab0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5300e050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5306e0c0}, {0x15db118, 0x3f5b4a6ab0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d5300e000, {0x15d0ec8, 0x3d5300e050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d5300e000, {0x15d0e90, 0x3d5301e0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x14) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 971 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d45612300, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d53be0090, {0x15c3738, 0x3d53be0008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d53be0008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec57e5770?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec57e5770, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d53680050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000ceeb8?, {0x15d0ec8?, 0x3d53680050?}, {0x15bbee0?, 0x3d53b8e100?}, {0x15da6b0?, 0x3d53b8e040?}, {0x15db118?, 0x3f5b4a6b70?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d53680050?}, {0x15bbee0?, 0x3d53b8e100?}, 0x3d53b8e080, {0x15db118?, 0x3f5b4a6b70?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d53680050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d53b8e080}, {0x15db118, 0x3f5b4a6b70}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d53680050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d53b8e0c0, {0x15db118?, 0x3f5b4a6b70?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d53680050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d53b8e0c0}, {0x15db118, 0x3f5b4a6b70}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d53680000, {0x15d0ec8, 0x3d53680050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d53680000, {0x15d0e90, 0x3d536b80c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x17) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 972 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d5105f590, {0x15c3738, 0x3d5105f508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d5105f508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec42e2ae0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec42e2ae0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d5198a050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000cc940?, {0x15d0ec8?, 0x3d5198a050?}, {0x15bbee0?, 0x3d51c86100?}, {0x15da6b0?, 0x3d51c86040?}, {0x15db118?, 0x3f5b4a6990?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5198a050?}, {0x15bbee0?, 0x3d51c86100?}, 0x3d51c86080, {0x15db118?, 0x3f5b4a6990?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5198a050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d51c86080}, {0x15db118, 0x3f5b4a6990}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5198a050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d51c860c0, {0x15db118?, 0x3f5b4a6990?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5198a050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d51c860c0}, {0x15db118, 0x3f5b4a6990}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d5198a000, {0x15d0ec8, 0x3d5198a050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d5198a000, {0x15d0e90, 0x3d519a60c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0xe) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 973 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d40488a90, {0x15c3738, 0x3d40488a08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d40488a08, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec51b8210?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec51b8210, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d573940c0?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3f5b4b0230}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c2ab0?, {0x15d0ec8?, 0x3f5b4b0230?}, {0x15bbee0?, 0x3d573f0100?}, {0x15da6b0?, 0x3d573f0040?}, {0x15db118?, 0x3f5b4a6e70?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f5b4b0230?}, {0x15bbee0?, 0x3d573f0100?}, 0x3d573f0080, {0x15db118?, 0x3f5b4a6e70?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f5b4b0230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d573f0080}, {0x15db118, 0x3f5b4a6e70}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f5b4b0230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d573f00c0, {0x15db118?, 0x3f5b4a6e70?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f5b4b0230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d573f00c0}, {0x15db118, 0x3f5b4a6e70}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3f5b4b01e0, {0x15d0ec8, 0x3f5b4b0230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3f5b4b01e0, {0x15d0e90, 0x3ec559a3f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x1f) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 974 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d5724ea90, {0x15c3738, 0x3d5724ea08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d5724ea08, 0x2d?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec402c210?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec402c210, 0x2d}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d5782c4f0?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3f5b4b02d0}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c7570?, {0x15d0ec8?, 0x3f5b4b02d0?}, {0x15bbee0?, 0x3d583d4200?}, {0x15da6b0?, 0x3d583d4140?}, {0x15db118?, 0x3f5b4a6f00?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f5b4b02d0?}, {0x15bbee0?, 0x3d583d4200?}, 0x3d583d4180, {0x15db118?, 0x3f5b4a6f00?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f5b4b02d0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d583d4180}, {0x15db118, 0x3f5b4a6f00}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f5b4b02d0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d583d41c0, {0x15db118?, 0x3f5b4a6f00?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f5b4b02d0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d583d41c0}, {0x15db118, 0x3f5b4a6f00}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3f5b4b0280, {0x15d0ec8, 0x3f5b4b02d0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3f5b4b0280, {0x15d0e90, 0x3ec559a4e0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x22) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 975 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d42a50600, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57a6d590, {0x15c3738, 0x3d57a6d508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57a6d508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec50ddda0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec50ddda0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d58d0a350?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d58d00230}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c9788?, {0x15d0ec8?, 0x3d58d00230?}, {0x15bbee0?, 0x3d58d4e280?}, {0x15da6b0?, 0x3d58d4e1c0?}, {0x15db118?, 0x3f5b4a7b60?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58d00230?}, {0x15bbee0?, 0x3d58d4e280?}, 0x3d58d4e200, {0x15db118?, 0x3f5b4a7b60?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58d00230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d58d4e200}, {0x15db118, 0x3f5b4a7b60}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58d00230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d58d4e240, {0x15db118?, 0x3f5b4a7b60?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58d00230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d58d4e240}, {0x15db118, 0x3f5b4a7b60}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d58d001e0, {0x15d0ec8, 0x3d58d00230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d58d001e0, {0x15d0e90, 0x3d58d0e3c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x62) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 976 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d51890090, {0x15c3738, 0x3d51890008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d51890008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4ca7620?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4ca7620, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3f5b4b0050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c9008?, {0x15d0ec8?, 0x3f5b4b0050?}, {0x15bbee0?, 0x3d50a58100?}, {0x15da6b0?, 0x3d50a58040?}, {0x15db118?, 0x3f5b4a67b0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f5b4b0050?}, {0x15bbee0?, 0x3d50a58100?}, 0x3d50a58080, {0x15db118?, 0x3f5b4a67b0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f5b4b0050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d50a58080}, {0x15db118, 0x3f5b4a67b0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f5b4b0050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d50a580c0, {0x15db118?, 0x3f5b4a67b0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f5b4b0050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d50a580c0}, {0x15db118, 0x3f5b4a67b0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3f5b4b0000, {0x15d0ec8, 0x3f5b4b0050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3f5b4b0000, {0x15d0e90, 0x3ec559a0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x5) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 977 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d41eee700, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d51cc7590, {0x15c3738, 0x3d51cc7508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d51cc7508, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec2425680?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec2425680, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d55da0660?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3f584ebbd0}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c86c8?, {0x15d0ec8?, 0x3f584ebbd0?}, {0x15bbee0?, 0x3d31389580?}, {0x15da6b0?, 0x3d313894c0?}, {0x15db118?, 0x3f5b4a6db0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f584ebbd0?}, {0x15bbee0?, 0x3d31389580?}, 0x3d31389500, {0x15db118?, 0x3f5b4a6db0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f584ebbd0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d31389500}, {0x15db118, 0x3f5b4a6db0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f584ebbd0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d31389540, {0x15db118?, 0x3f5b4a6db0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f584ebbd0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d31389540}, {0x15db118, 0x3f5b4a6db0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3f584ebb80, {0x15d0ec8, 0x3f584ebbd0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3f584ebb80, {0x15d0e90, 0x3ec90e5f50}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x1b) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 978 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d41bf8800, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d5773e090, {0x15c3738, 0x3d5773e008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d5773e008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5518660?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5518660, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d57598170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57580050}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cce88?, {0x15d0ec8?, 0x3d57580050?}, {0x15bbee0?, 0x3d575fa100?}, {0x15da6b0?, 0x3d575fa040?}, {0x15db118?, 0x3f5b4a73e0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57580050?}, {0x15bbee0?, 0x3d575fa100?}, 0x3d575fa080, {0x15db118?, 0x3f5b4a73e0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57580050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d575fa080}, {0x15db118, 0x3f5b4a73e0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57580050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d575fa0c0, {0x15db118?, 0x3f5b4a73e0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57580050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d575fa0c0}, {0x15db118, 0x3f5b4a73e0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57580000, {0x15d0ec8, 0x3d57580050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57580000, {0x15d0e90, 0x3d575a00c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x3c) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 979 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d40708a90, {0x15c3738, 0x3d40708a08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d40708a08, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec59aa900?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec59aa900, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d57598350?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57580230}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cce88?, {0x15d0ec8?, 0x3d57580230?}, {0x15bbee0?, 0x3d57774dc0?}, {0x15da6b0?, 0x3d57774d00?}, {0x15db118?, 0x3f5b4a7830?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57580230?}, {0x15bbee0?, 0x3d57774dc0?}, 0x3d57774d40, {0x15db118?, 0x3f5b4a7830?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57580230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57774d40}, {0x15db118, 0x3f5b4a7830}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57580230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d57774d80, {0x15db118?, 0x3f5b4a7830?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57580230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57774d80}, {0x15db118, 0x3f5b4a7830}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d575801e0, {0x15d0ec8, 0x3d57580230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d575801e0, {0x15d0e90, 0x3d575a03f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x51) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 980 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d510cb590, {0x15c3738, 0x3d510cb508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d510cb508, 0x2d?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec45da720?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec45da720, 0x2d}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d57694350?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d5768e230}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cdb80?, {0x15d0ec8?, 0x3d5768e230?}, {0x15bbee0?, 0x3d57a13080?}, {0x15da6b0?, 0x3d57a12fc0?}, {0x15db118?, 0x3f5b4a7950?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5768e230?}, {0x15bbee0?, 0x3d57a13080?}, 0x3d57a13000, {0x15db118?, 0x3f5b4a7950?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5768e230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57a13000}, {0x15db118, 0x3f5b4a7950}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5768e230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d57a13040, {0x15db118?, 0x3f5b4a7950?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5768e230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57a13040}, {0x15db118, 0x3f5b4a7950}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d5768e1e0, {0x15d0ec8, 0x3d5768e230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d5768e1e0, {0x15d0e90, 0x3d576983f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x57) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 981 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d31808090, {0x15c3738, 0x3d31808008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d31808008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec425cea0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec425cea0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d5782c350?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57806230}, {0x29eec?, 0x3f3c2498c8?}, {0x866a4?, 0x7e?}, {0x7e?, 0x29e80?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cf50c?, {0x15d0ec8?, 0x3d57806230?}, {0x15bbee0?, 0x3d583d4080?}, {0x15da6b0?, 0x3d57a9bf40?}, {0x15db118?, 0x3f5b4a7890?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57806230?}, {0x15bbee0?, 0x3d583d4080?}, 0x3d583d4000, {0x15db118?, 0x3f5b4a7890?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57806230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d583d4000}, {0x15db118, 0x3f5b4a7890}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57806230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d583d4040, {0x15db118?, 0x3f5b4a7890?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57806230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d583d4040}, {0x15db118, 0x3f5b4a7890}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d578061e0, {0x15d0ec8, 0x3d57806230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d578061e0, {0x15d0e90, 0x3d5782e3f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x53) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 982 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d538e7590, {0x15c3738, 0x3d538e7508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d538e7508, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec0e40930?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec0e40930, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d57904310?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d5790e190}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cd0c8?, {0x15d0ec8?, 0x3d5790e190?}, {0x15bbee0?, 0x3d57965240?}, {0x15da6b0?, 0x3d57965180?}, {0x15db118?, 0x3f5b4a78f0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5790e190?}, {0x15bbee0?, 0x3d57965240?}, 0x3d579651c0, {0x15db118?, 0x3f5b4a78f0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5790e190}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d579651c0}, {0x15db118, 0x3f5b4a78f0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5790e190?}, {0x3f54784410?, 0x3f58798040?}, 0x3d57965200, {0x15db118?, 0x3f5b4a78f0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5790e190}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57965200}, {0x15db118, 0x3f5b4a78f0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d5790e140, {0x15d0ec8, 0x3d5790e190}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d5790e140, {0x15d0e90, 0x3d579362d0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x55) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 983 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d58717590, {0x15c3738, 0x3d58717508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d58717508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec27e8030?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec27e8030, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d580a0350?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d58086230}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c8df8?, {0x15d0ec8?, 0x3d58086230?}, {0x15bbee0?, 0x3d314801c0?}, {0x15da6b0?, 0x3d31480100?}, {0x15db118?, 0x3f5b4a7b30?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58086230?}, {0x15bbee0?, 0x3d314801c0?}, 0x3d31480140, {0x15db118?, 0x3f5b4a7b30?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58086230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d31480140}, {0x15db118, 0x3f5b4a7b30}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58086230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d31480180, {0x15db118?, 0x3f5b4a7b30?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58086230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d31480180}, {0x15db118, 0x3f5b4a7b30}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d580861e0, {0x15d0ec8, 0x3d58086230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d580861e0, {0x15d0e90, 0x3d580a83f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x61) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 984 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d40388a90, {0x15c3738, 0x3d40388a08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d40388a08, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec2ab6930?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec2ab6930, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3f584ebef0}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c9d68?, {0x15d0ec8?, 0x3f584ebef0?}, {0x15bbee0?, 0x3d525e5440?}, {0x15da6b0?, 0x3d525e5380?}, {0x15db118?, 0x3f5b4a6ff0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f584ebef0?}, {0x15bbee0?, 0x3d525e5440?}, 0x3d525e53c0, {0x15db118?, 0x3f5b4a6ff0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f584ebef0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d525e53c0}, {0x15db118, 0x3f5b4a6ff0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f584ebef0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d525e5400, {0x15db118?, 0x3f5b4a6ff0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f584ebef0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d525e5400}, {0x15db118, 0x3f5b4a6ff0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3f584ebe50, {0x15d0ec8, 0x3f584ebef0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3f584ebe50, {0x15d0e90, 0x3d52d0a0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x27) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 985 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57d89590, {0x15c3738, 0x3d57d89508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57d89508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5c69b90?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5c69b90, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d5799c170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57986050}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c5b98?, {0x15d0ec8?, 0x3d57986050?}, {0x15bbee0?, 0x3d57ba2100?}, {0x15da6b0?, 0x3d57ba2040?}, {0x15db118?, 0x3f5b4a7500?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57986050?}, {0x15bbee0?, 0x3d57ba2100?}, 0x3d57ba2080, {0x15db118?, 0x3f5b4a7500?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57986050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57ba2080}, {0x15db118, 0x3f5b4a7500}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57986050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d57ba20c0, {0x15db118?, 0x3f5b4a7500?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57986050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57ba20c0}, {0x15db118, 0x3f5b4a7500}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57986000, {0x15d0ec8, 0x3d57986050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57986000, {0x15d0e90, 0x3d579a00c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x40) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 986 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d43ae8900, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d538e6090, {0x15c3738, 0x3d538e6008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d538e6008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec42be450?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec42be450, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d57b1a170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57b06050}, {0x29eec?, 0x40?}, {0x866a4?, 0x2a840?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c61b8?, {0x15d0ec8?, 0x3d57b06050?}, {0x15bbee0?, 0x3d57b7e100?}, {0x15da6b0?, 0x3d57b7e040?}, {0x15db118?, 0x3f5b4a7530?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57b06050?}, {0x15bbee0?, 0x3d57b7e100?}, 0x3d57b7e080, {0x15db118?, 0x3f5b4a7530?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57b06050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57b7e080}, {0x15db118, 0x3f5b4a7530}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57b06050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d57b7e0c0, {0x15db118?, 0x3f5b4a7530?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57b06050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57b7e0c0}, {0x15db118, 0x3f5b4a7530}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57b06000, {0x15d0ec8, 0x3d57b06050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57b06000, {0x15d0e90, 0x3d57b200c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x41) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 987 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57fa0090, {0x15c3738, 0x3d57fa0008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57fa0008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec42d58f0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec42d58f0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d57c12170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57c06050}, {0x29eec?, 0x40?}, {0x866a4?, 0x2a840?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c6bb8?, {0x15d0ec8?, 0x3d57c06050?}, {0x15bbee0?, 0x3d57c7a100?}, {0x15da6b0?, 0x3d57c7a040?}, {0x15db118?, 0x3f5b4a7560?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57c06050?}, {0x15bbee0?, 0x3d57c7a100?}, 0x3d57c7a080, {0x15db118?, 0x3f5b4a7560?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57c06050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57c7a080}, {0x15db118, 0x3f5b4a7560}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57c06050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d57c7a0c0, {0x15db118?, 0x3f5b4a7560?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57c06050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57c7a0c0}, {0x15db118, 0x3f5b4a7560}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57c06000, {0x15d0ec8, 0x3d57c06050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57c06000, {0x15d0e90, 0x3d57c140c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x42) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 988 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d53b54090, {0x15c3738, 0x3d53b54008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1608 +0x5018 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d53b54008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec13b7b90?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec13b7b90, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d57c9e170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57c96050}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cbd78?, {0x15d0ec8?, 0x3d57c96050?}, {0x15bbee0?, 0x3d57e86100?}, {0x15da6b0?, 0x3d57e86040?}, {0x15db118?, 0x3f5b4a7590?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57c96050?}, {0x15bbee0?, 0x3d57e86100?}, 0x3d57e86080, {0x15db118?, 0x3f5b4a7590?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57c96050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57e86080}, {0x15db118, 0x3f5b4a7590}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57c96050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d57e860c0, {0x15db118?, 0x3f5b4a7590?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57c96050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57e860c0}, {0x15db118, 0x3f5b4a7590}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57c96000, {0x15d0ec8, 0x3d57c96050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57c96000, {0x15d0e90, 0x3d57ca00c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x43) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 989 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57ece090, {0x15c3738, 0x3d57ece008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57ece008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4b6cb10?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4b6cb10, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d57d1a170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57d0e050}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cbf88?, {0x15d0ec8?, 0x3d57d0e050?}, {0x15bbee0?, 0x3d57d7e100?}, {0x15da6b0?, 0x3d57d7e040?}, {0x15db118?, 0x3f5b4a75c0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57d0e050?}, {0x15bbee0?, 0x3d57d7e100?}, 0x3d57d7e080, {0x15db118?, 0x3f5b4a75c0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57d0e050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57d7e080}, {0x15db118, 0x3f5b4a75c0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57d0e050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d57d7e0c0, {0x15db118?, 0x3f5b4a75c0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57d0e050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57d7e0c0}, {0x15db118, 0x3f5b4a75c0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57d0e000, {0x15d0ec8, 0x3d57d0e050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57d0e000, {0x15d0e90, 0x3d57d1c0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x44) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 990 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d43fc2f00, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d562b8090, {0x15c3738, 0x3d562b8008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d562b8008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec24f82a0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec24f82a0, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d508506e0?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3f581fe370}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c7f08?, {0x15d0ec8?, 0x3f581fe370?}, {0x15bbee0?, 0x3d568baac0?}, {0x15da6b0?, 0x3d568baa00?}, {0x15db118?, 0x3f5b4a75f0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f581fe370?}, {0x15bbee0?, 0x3d568baac0?}, 0x3d568baa40, {0x15db118?, 0x3f5b4a75f0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f581fe370}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d568baa40}, {0x15db118, 0x3f5b4a75f0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f581fe370?}, {0x3f54784410?, 0x3f58798040?}, 0x3d568baa80, {0x15db118?, 0x3f5b4a75f0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f581fe370}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d568baa80}, {0x15db118, 0x3f5b4a75f0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3f581fe320, {0x15d0ec8, 0x3f581fe370}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3f581fe320, {0x15d0e90, 0x3f587371a0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x45) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 991 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d452abe00, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d58946090, {0x15c3738, 0x3d58946008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d58946008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec56537d0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec56537d0, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d5802a170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d58014050}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cc8a8?, {0x15d0ec8?, 0x3d58014050?}, {0x15bbee0?, 0x3d585cc100?}, {0x15da6b0?, 0x3d585cc040?}, {0x15db118?, 0x3f5b4a7620?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58014050?}, {0x15bbee0?, 0x3d585cc100?}, 0x3d585cc080, {0x15db118?, 0x3f5b4a7620?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58014050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d585cc080}, {0x15db118, 0x3f5b4a7620}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58014050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d585cc0c0, {0x15db118?, 0x3f5b4a7620?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58014050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d585cc0c0}, {0x15db118, 0x3f5b4a7620}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d58014000, {0x15d0ec8, 0x3d58014050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d58014000, {0x15d0e90, 0x3d5802c0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x46) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 992 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d5773f590, {0x15c3738, 0x3d5773f508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d5773f508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4e66ff0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4e66ff0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d580a0170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d58086050}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c8df8?, {0x15d0ec8?, 0x3d58086050?}, {0x15bbee0?, 0x3d584be100?}, {0x15da6b0?, 0x3d584be040?}, {0x15db118?, 0x3f5b4a7650?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58086050?}, {0x15bbee0?, 0x3d584be100?}, 0x3d584be080, {0x15db118?, 0x3f5b4a7650?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58086050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d584be080}, {0x15db118, 0x3f5b4a7650}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58086050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d584be0c0, {0x15db118?, 0x3f5b4a7650?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58086050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d584be0c0}, {0x15db118, 0x3f5b4a7650}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d58086000, {0x15d0ec8, 0x3d58086050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d58086000, {0x15d0e90, 0x3d580a80c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x47) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 993 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d44e1f100, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d58634090, {0x15c3738, 0x3d58634008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d58634008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec44897a0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec44897a0, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d58216170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d58202050}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cff48?, {0x15d0ec8?, 0x3d58202050?}, {0x15bbee0?, 0x3d58446100?}, {0x15da6b0?, 0x3d58446040?}, {0x15db118?, 0x3f5b4a7680?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58202050?}, {0x15bbee0?, 0x3d58446100?}, 0x3d58446080, {0x15db118?, 0x3f5b4a7680?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58202050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d58446080}, {0x15db118, 0x3f5b4a7680}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58202050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d584460c0, {0x15db118?, 0x3f5b4a7680?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58202050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d584460c0}, {0x15db118, 0x3f5b4a7680}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d58202000, {0x15d0ec8, 0x3d58202050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d58202000, {0x15d0e90, 0x3d582220c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x48) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 994 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d4ac13590, {0x15c3738, 0x3d4ac13508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d4ac13508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec15b98f0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec15b98f0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d563f4290?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d563fc0f0}, {0x29eec?, 0x3f186fa408?}, {0x866a4?, 0x7e?}, {0x7e?, 0x29e80?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000caf0c?, {0x15d0ec8?, 0x3d563fc0f0?}, {0x15bbee0?, 0x3d31098080?}, {0x15da6b0?, 0x3d563a3f40?}, {0x15db118?, 0x3f5b4a71a0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d563fc0f0?}, {0x15bbee0?, 0x3d31098080?}, 0x3d31098000, {0x15db118?, 0x3f5b4a71a0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d563fc0f0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d31098000}, {0x15db118, 0x3f5b4a71a0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d563fc0f0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d31098040, {0x15db118?, 0x3f5b4a71a0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d563fc0f0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d31098040}, {0x15db118, 0x3f5b4a71a0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3f5b4b0320, {0x15d0ec8, 0x3d563fc0f0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3f5b4b0320, {0x15d0e90, 0x3d54d222a0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x30) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 995 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d40400090, {0x15c3738, 0x3d40400008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1608 +0x5018 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d40400008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4a02090?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4a02090, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d57792170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57786050}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c4728?, {0x15d0ec8?, 0x3d57786050?}, {0x15bbee0?, 0x3d577f6100?}, {0x15da6b0?, 0x3d577f6040?}, {0x15db118?, 0x3f5b4a7470?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57786050?}, {0x15bbee0?, 0x3d577f6100?}, 0x3d577f6080, {0x15db118?, 0x3f5b4a7470?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57786050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d577f6080}, {0x15db118, 0x3f5b4a7470}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57786050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d577f60c0, {0x15db118?, 0x3f5b4a7470?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57786050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d577f60c0}, {0x15db118, 0x3f5b4a7470}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57786000, {0x15d0ec8, 0x3d57786050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57786000, {0x15d0e90, 0x3d5779c0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x3e) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 996 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d3169e090, {0x15c3738, 0x3d3169e008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d3169e008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4fa8450?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4fa8450, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d59050170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d5900c050}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c4038?, {0x15d0ec8?, 0x3d5900c050?}, {0x15bbee0?, 0x3d3168e100?}, {0x15da6b0?, 0x3d3168e040?}, {0x15db118?, 0x3f5b4a7b00?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5900c050?}, {0x15bbee0?, 0x3d3168e100?}, 0x3d3168e080, {0x15db118?, 0x3f5b4a7b00?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5900c050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d3168e080}, {0x15db118, 0x3f5b4a7b00}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5900c050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d3168e0c0, {0x15db118?, 0x3f5b4a7b00?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5900c050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d3168e0c0}, {0x15db118, 0x3f5b4a7b00}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d5900c000, {0x15d0ec8, 0x3d5900c050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d5900c000, {0x15d0e90, 0x3d590540c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x60) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 997 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d439c9900, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d311baa90, {0x15c3738, 0x3d311baa08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d311baa08, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5410f30?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5410f30, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d55e10360?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d55e00230}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c3a28?, {0x15d0ec8?, 0x3d55e00230?}, {0x15bbee0?, 0x3d5726f540?}, {0x15da6b0?, 0x3d5726f480?}, {0x15db118?, 0x3f5b4a7a10?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d55e00230?}, {0x15bbee0?, 0x3d5726f540?}, 0x3d5726f4c0, {0x15db118?, 0x3f5b4a7a10?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d55e00230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5726f4c0}, {0x15db118, 0x3f5b4a7a10}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d55e00230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d5726f500, {0x15db118?, 0x3f5b4a7a10?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d55e00230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5726f500}, {0x15db118, 0x3f5b4a7a10}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d55e001e0, {0x15d0ec8, 0x3d55e00230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d55e001e0, {0x15d0e90, 0x3d55e18450}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x5b) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 998 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d561f6090, {0x15c3738, 0x3d561f6008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d561f6008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4f45b90?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4f45b90, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d579044f0?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d5790e370}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cd0c8?, {0x15d0ec8?, 0x3d5790e370?}, {0x15bbee0?, 0x3d57965400?}, {0x15da6b0?, 0x3d57965340?}, {0x15db118?, 0x3f5b4a7a40?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5790e370?}, {0x15bbee0?, 0x3d57965400?}, 0x3d57965380, {0x15db118?, 0x3f5b4a7a40?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5790e370}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57965380}, {0x15db118, 0x3f5b4a7a40}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5790e370?}, {0x3f54784410?, 0x3f58798040?}, 0x3d579653c0, {0x15db118?, 0x3f5b4a7a40?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5790e370}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d579653c0}, {0x15db118, 0x3f5b4a7a40}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d5790e320, {0x15d0ec8, 0x3d5790e370}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d5790e320, {0x15d0e90, 0x3d579365a0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x5c) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 999 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d31708090, {0x15c3738, 0x3d31708008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d31708008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4bdbc80?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4bdbc80, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d55da04c0?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d55d802d0}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c86c8?, {0x15d0ec8?, 0x3d55d802d0?}, {0x15bbee0?, 0x3d5619a400?}, {0x15da6b0?, 0x3d5619a340?}, {0x15db118?, 0x3f5b4a7290?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d55d802d0?}, {0x15bbee0?, 0x3d5619a400?}, 0x3d5619a380, {0x15db118?, 0x3f5b4a7290?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d55d802d0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5619a380}, {0x15db118, 0x3f5b4a7290}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d55d802d0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d5619a3c0, {0x15db118?, 0x3f5b4a7290?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d55d802d0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5619a3c0}, {0x15db118, 0x3f5b4a7290}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3f582c6910, {0x15d0ec8, 0x3d55d802d0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3f582c6910, {0x15d0e90, 0x3d55daa4e0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x35) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1000 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d42b64400, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d311b9590, {0x15c3738, 0x3d311b9508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d311b9508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec47729f0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec47729f0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d5718a500?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d571863c0}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000ce598?, {0x15d0ec8?, 0x3d571863c0?}, {0x15bbee0?, 0x3d311cc780?}, {0x15da6b0?, 0x3d311cc6c0?}, {0x15db118?, 0x3f5b4a7740?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d571863c0?}, {0x15bbee0?, 0x3d311cc780?}, 0x3d311cc700, {0x15db118?, 0x3f5b4a7740?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d571863c0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d311cc700}, {0x15db118, 0x3f5b4a7740}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d571863c0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d311cc740, {0x15db118?, 0x3f5b4a7740?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d571863c0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d311cc740}, {0x15db118, 0x3f5b4a7740}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57186370, {0x15d0ec8, 0x3d571863c0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57186370, {0x15d0e90, 0x3d5719e600}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x4c) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1001 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d517a2090, {0x15c3738, 0x3d517a2008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d517a2008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec42bede0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec42bede0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d56e962c0?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d56e90190}, {0x29eec?, 0x40?}, {0x866a4?, 0x4e1f0?}, {0x8dc50?, 0x12ce5d0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c0001?, {0x15d0ec8?, 0x3d56e90190?}, {0x15bbee0?, 0x3d56ef9a80?}, {0x15da6b0?, 0x3d56ef99c0?}, {0x15db118?, 0x3f5b4a7770?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d56e90190?}, {0x15bbee0?, 0x3d56ef9a80?}, 0x3d56ef9a00, {0x15db118?, 0x3f5b4a7770?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d56e90190}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d56ef9a00}, {0x15db118, 0x3f5b4a7770}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d56e90190?}, {0x3f54784410?, 0x3f58798040?}, 0x3d56ef9a40, {0x15db118?, 0x3f5b4a7770?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d56e90190}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d56ef9a40}, {0x15db118, 0x3f5b4a7770}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d56e90140, {0x15d0ec8, 0x3d56e90190}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d56e90140, {0x15d0e90, 0x3d56ea02a0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x4d) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1002 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57d8aa90, {0x15c3738, 0x3d57d8aa08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57d8aa08, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec596c090?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec596c090, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d5782c690?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d5198a230}, {0x29eec?, 0x40?}, {0x866a4?, 0x2a840?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cf590?, {0x15d0ec8?, 0x3d5198a230?}, {0x15bbee0?, 0x3d583d4e40?}, {0x15da6b0?, 0x3d583d4d80?}, {0x15db118?, 0x3f5b4a7320?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5198a230?}, {0x15bbee0?, 0x3d583d4e40?}, 0x3d583d4dc0, {0x15db118?, 0x3f5b4a7320?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5198a230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d583d4dc0}, {0x15db118, 0x3f5b4a7320}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d5198a230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d583d4e00, {0x15db118?, 0x3f5b4a7320?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d5198a230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d583d4e00}, {0x15db118, 0x3f5b4a7320}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d5198a1e0, {0x15d0ec8, 0x3d5198a230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d5198a1e0, {0x15d0e90, 0x3d519a63f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x38) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1003 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d43c5fb00, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d5105e090, {0x15c3738, 0x3d5105e008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d5105e008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec0846d50?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec0846d50, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d5852c170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d58520050}, {0x29eec?, 0x40?}, {0x866a4?, 0x2a840?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c3f08?, {0x15d0ec8?, 0x3d58520050?}, {0x15bbee0?, 0x3d588c8100?}, {0x15da6b0?, 0x3d588c8040?}, {0x15db118?, 0x3f5b4a76b0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58520050?}, {0x15bbee0?, 0x3d588c8100?}, 0x3d588c8080, {0x15db118?, 0x3f5b4a76b0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58520050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d588c8080}, {0x15db118, 0x3f5b4a76b0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58520050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d588c80c0, {0x15db118?, 0x3f5b4a76b0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58520050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d588c80c0}, {0x15db118, 0x3f5b4a76b0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d58520000, {0x15d0ec8, 0x3d58520050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d58520000, {0x15d0e90, 0x3d5853c0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x49) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1004 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d455b1d00, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d51cc6090, {0x15c3738, 0x3d51cc6008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d51cc6008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4ff2cf0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4ff2cf0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d586a0170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d58682050}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c19d8?, {0x15d0ec8?, 0x3d58682050?}, {0x15bbee0?, 0x3d5885a100?}, {0x15da6b0?, 0x3d5885a040?}, {0x15db118?, 0x3f5b4a76e0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58682050?}, {0x15bbee0?, 0x3d5885a100?}, 0x3d5885a080, {0x15db118?, 0x3f5b4a76e0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58682050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5885a080}, {0x15db118, 0x3f5b4a76e0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58682050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d5885a0c0, {0x15db118?, 0x3f5b4a76e0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58682050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5885a0c0}, {0x15db118, 0x3f5b4a76e0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d58682000, {0x15d0ec8, 0x3d58682050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d58682000, {0x15d0e90, 0x3d586b20c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x4a) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1005 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d31298090, {0x15c3738, 0x3d31298008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d31298008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4d07410?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4d07410, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d55e80050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c9f60?, {0x15d0ec8?, 0x3d55e80050?}, {0x15bbee0?, 0x3d55ee0100?}, {0x15da6b0?, 0x3d55ee0040?}, {0x15db118?, 0x3f5b4a6f90?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d55e80050?}, {0x15bbee0?, 0x3d55ee0100?}, 0x3d55ee0080, {0x15db118?, 0x3f5b4a6f90?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d55e80050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d55ee0080}, {0x15db118, 0x3f5b4a6f90}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d55e80050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d55ee00c0, {0x15db118?, 0x3f5b4a6f90?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d55e80050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d55ee00c0}, {0x15db118, 0x3f5b4a6f90}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d55e80000, {0x15d0ec8, 0x3d55e80050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d55e80000, {0x15d0e90, 0x3d55e9a0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x25) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1006 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d41e63200, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d53543590, {0x15c3738, 0x3d53543508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d53543508, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec05eea20?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec05eea20, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d572b8270?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57296190}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cd228?, {0x15d0ec8?, 0x3d57296190?}, {0x15bbee0?, 0x3d572f35c0?}, {0x15da6b0?, 0x3d572f3500?}, {0x15db118?, 0x3f5b4a77a0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57296190?}, {0x15bbee0?, 0x3d572f35c0?}, 0x3d572f3540, {0x15db118?, 0x3f5b4a77a0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57296190}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d572f3540}, {0x15db118, 0x3f5b4a77a0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57296190?}, {0x3f54784410?, 0x3f58798040?}, 0x3d572f3580, {0x15db118?, 0x3f5b4a77a0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57296190}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d572f3580}, {0x15db118, 0x3f5b4a77a0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57296140, {0x15d0ec8, 0x3d57296190}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57296140, {0x15d0e90, 0x3d572c22d0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x4e) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1007 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d40401590, {0x15c3738, 0x3d40401508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d40401508, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec1689650?, 0x3d302b6008?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec1689650, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d582a0170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d574800a0}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cdb78?, {0x15d0ec8?, 0x3d574800a0?}, {0x15bbee0?, 0x3d574fd540?}, {0x15da6b0?, 0x3d574fd480?}, {0x15db118?, 0x3f5b4a77d0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d574800a0?}, {0x15bbee0?, 0x3d574fd540?}, 0x3d574fd4c0, {0x15db118?, 0x3f5b4a77d0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d574800a0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d574fd4c0}, {0x15db118, 0x3f5b4a77d0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d574800a0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d574fd500, {0x15db118?, 0x3f5b4a77d0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d574800a0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d574fd500}, {0x15db118, 0x3f5b4a77d0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57480050, {0x15d0ec8, 0x3d574800a0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57480050, {0x15d0e90, 0x3d582b80c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x4f) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1008 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d43863c00, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d58c54090, {0x15c3738, 0x3d58c54008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d58c54008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4b0e630?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4b0e630, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d581b0170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d571000a0}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cc5d8?, {0x15d0ec8?, 0x3d571000a0?}, {0x15bbee0?, 0x3d58190f80?}, {0x15da6b0?, 0x3d58190ec0?}, {0x15db118?, 0x3f5b4a7800?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d571000a0?}, {0x15bbee0?, 0x3d58190f80?}, 0x3d58190f00, {0x15db118?, 0x3f5b4a7800?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d571000a0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d58190f00}, {0x15db118, 0x3f5b4a7800}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d571000a0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d58190f40, {0x15db118?, 0x3f5b4a7800?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d571000a0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d58190f40}, {0x15db118, 0x3f5b4a7800}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57100050, {0x15d0ec8, 0x3d571000a0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57100050, {0x15d0e90, 0x3d571280f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x50) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1009 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d4521f400, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d3109a090, {0x15c3738, 0x3d3109a008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d3109a008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec50c3e60?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec50c3e60, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d573942a0?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57384190}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cb038?, {0x15d0ec8?, 0x3d57384190?}, {0x15bbee0?, 0x3d573f1b40?}, {0x15da6b0?, 0x3d573f1a80?}, {0x15db118?, 0x3f5b4a7860?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57384190?}, {0x15bbee0?, 0x3d573f1b40?}, 0x3d573f1ac0, {0x15db118?, 0x3f5b4a7860?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57384190}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d573f1ac0}, {0x15db118, 0x3f5b4a7860}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57384190?}, {0x3f54784410?, 0x3f58798040?}, 0x3d573f1b00, {0x15db118?, 0x3f5b4a7860?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57384190}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d573f1b00}, {0x15db118, 0x3f5b4a7860}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57384140, {0x15d0ec8, 0x3d57384190}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57384140, {0x15d0e90, 0x3d573ae2d0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x52) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1010 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d411d5900, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d31809590, {0x15c3738, 0x3d31809508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d31809508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4620840?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4620840, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d5799c350?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57986230}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c5b98?, {0x15d0ec8?, 0x3d57986230?}, {0x15bbee0?, 0x3d57ba3c40?}, {0x15da6b0?, 0x3d57ba3b80?}, {0x15db118?, 0x3f5b4a78c0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57986230?}, {0x15bbee0?, 0x3d57ba3c40?}, 0x3d57ba3bc0, {0x15db118?, 0x3f5b4a78c0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57986230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57ba3bc0}, {0x15db118, 0x3f5b4a78c0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57986230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d57ba3c00, {0x15db118?, 0x3f5b4a78c0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57986230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d57ba3c00}, {0x15db118, 0x3f5b4a78c0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d579861e0, {0x15d0ec8, 0x3d57986230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d579861e0, {0x15d0e90, 0x3d579a03c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x54) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1011 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57e3c090, {0x15c3738, 0x3d57e3c008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57e3c008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec2747710?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec2747710, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d57792350?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57786230}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c4728?, {0x15d0ec8?, 0x3d57786230?}, {0x15bbee0?, 0x3d578f4800?}, {0x15da6b0?, 0x3d578f4740?}, {0x15db118?, 0x3f5b4a7920?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57786230?}, {0x15bbee0?, 0x3d578f4800?}, 0x3d578f4780, {0x15db118?, 0x3f5b4a7920?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57786230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d578f4780}, {0x15db118, 0x3f5b4a7920}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57786230?}, {0x3f54784410?, 0x3f58798040?}, 0x3d578f47c0, {0x15db118?, 0x3f5b4a7920?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57786230}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d578f47c0}, {0x15db118, 0x3f5b4a7920}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d577861e0, {0x15d0ec8, 0x3d57786230}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d577861e0, {0x15d0e90, 0x3d5779c3f0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x56) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1012 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d45b18d00, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d578c3590, {0x15c3738, 0x3d578c3508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d578c3508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5653380?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5653380, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d563f4470?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d563fc2d0}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000caf48?, {0x15d0ec8?, 0x3d563fc2d0?}, {0x15bbee0?, 0x3d310afac0?}, {0x15da6b0?, 0x3d310afa00?}, {0x15db118?, 0x3f5b4a7980?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d563fc2d0?}, {0x15bbee0?, 0x3d310afac0?}, 0x3d310afa40, {0x15db118?, 0x3f5b4a7980?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d563fc2d0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d310afa40}, {0x15db118, 0x3f5b4a7980}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d563fc2d0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d310afa80, {0x15db118?, 0x3f5b4a7980?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d563fc2d0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d310afa80}, {0x15db118, 0x3f5b4a7980}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d563fc280, {0x15d0ec8, 0x3d563fc2d0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d563fc280, {0x15d0e90, 0x3d54d225a0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x58) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1013 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d3130e090, {0x15c3738, 0x3d3130e008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d3130e008, 0x2d?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4fa8d50?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4fa8d50, 0x2d}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d58d0a170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d58d00050}, {0x29eec?, 0x40?}, {0x866a4?, 0x2a840?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c9788?, {0x15d0ec8?, 0x3d58d00050?}, {0x15bbee0?, 0x3d58d4e100?}, {0x15da6b0?, 0x3d58d4e040?}, {0x15db118?, 0x3f5b4a79b0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58d00050?}, {0x15bbee0?, 0x3d58d4e100?}, 0x3d58d4e080, {0x15db118?, 0x3f5b4a79b0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58d00050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d58d4e080}, {0x15db118, 0x3f5b4a79b0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58d00050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d58d4e0c0, {0x15db118?, 0x3f5b4a79b0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58d00050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d58d4e0c0}, {0x15db118, 0x3f5b4a79b0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d58d00000, {0x15d0ec8, 0x3d58d00050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d58d00000, {0x15d0e90, 0x3d58d0e0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x59) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1014 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d58d76090, {0x15c3738, 0x3d58d76008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1041 +0x7b0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d58d76008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4d54600?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4d54600, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d58d84170?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d58d80050}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000cd888?, {0x15d0ec8?, 0x3d58d80050?}, {0x15bbee0?, 0x3d58df0100?}, {0x15da6b0?, 0x3d58df0040?}, {0x15db118?, 0x3f5b4a79e0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58d80050?}, {0x15bbee0?, 0x3d58df0100?}, 0x3d58df0080, {0x15db118?, 0x3f5b4a79e0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58d80050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d58df0080}, {0x15db118, 0x3f5b4a79e0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d58d80050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d58df00c0, {0x15db118?, 0x3f5b4a79e0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d58d80050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d58df00c0}, {0x15db118, 0x3f5b4a79e0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d58d80000, {0x15d0ec8, 0x3d58d80050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d58d80000, {0x15d0e90, 0x3d58d880c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x5a) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1015 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d4404cf00, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57a6c090, {0x15c3738, 0x3d57a6c008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57a6c008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4961cb0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4961cb0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d56812050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000ca7c8?, {0x15d0ec8?, 0x3d56812050?}, {0x15bbee0?, 0x3d5687e100?}, {0x15da6b0?, 0x3d5687e040?}, {0x15db118?, 0x3f5b4a7020?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d56812050?}, {0x15bbee0?, 0x3d5687e100?}, 0x3d5687e080, {0x15db118?, 0x3f5b4a7020?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d56812050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5687e080}, {0x15db118, 0x3f5b4a7020}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d56812050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d5687e0c0, {0x15db118?, 0x3f5b4a7020?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d56812050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5687e0c0}, {0x15db118, 0x3f5b4a7020}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d56812000, {0x15d0ec8, 0x3d56812050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d56812000, {0x15d0e90, 0x3d568200c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x28) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1016 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d43c9d100, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d53e72090, {0x15c3738, 0x3d53e72008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d53e72008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec432c930?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec432c930, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d595f6000}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c9788?, {0x15d0ec8?, 0x3d595f6000?}, {0x15bbee0?, 0x3d583d55c0?}, {0x15da6b0?, 0x3d583d5500?}, {0x15db118?, 0x3f5b4a70b0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d595f6000?}, {0x15bbee0?, 0x3d583d55c0?}, 0x3d583d5540, {0x15db118?, 0x3f5b4a70b0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d595f6000}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d583d5540}, {0x15db118, 0x3f5b4a70b0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d595f6000?}, {0x3f54784410?, 0x3f58798040?}, 0x3d583d5580, {0x15db118?, 0x3f5b4a70b0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d595f6000}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d583d5580}, {0x15db118, 0x3f5b4a70b0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57010000, {0x15d0ec8, 0x3d595f6000}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57010000, {0x15d0e90, 0x3d570260c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x2b) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1017 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d57df6090, {0x15c3738, 0x3d57df6008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d57df6008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec23cedb0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec23cedb0, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d56e90050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000ca678?, {0x15d0ec8?, 0x3d56e90050?}, {0x15bbee0?, 0x3d56ef8100?}, {0x15da6b0?, 0x3d56ef8040?}, {0x15db118?, 0x3f5b4a7080?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d56e90050?}, {0x15bbee0?, 0x3d56ef8100?}, 0x3d56ef8080, {0x15db118?, 0x3f5b4a7080?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d56e90050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d56ef8080}, {0x15db118, 0x3f5b4a7080}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d56e90050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d56ef80c0, {0x15db118?, 0x3f5b4a7080?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d56e90050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d56ef80c0}, {0x15db118, 0x3f5b4a7080}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d56e90000, {0x15d0ec8, 0x3d56e90050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d56e90000, {0x15d0e90, 0x3d56ea00c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x2a) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1018 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d518d6090, {0x15c3738, 0x3d518d6008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d518d6008, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5410630?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5410630, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57186190}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000ce598?, {0x15d0ec8?, 0x3d57186190?}, {0x15bbee0?, 0x3d4692a100?}, {0x15da6b0?, 0x3d4692a040?}, {0x15db118?, 0x3f5b4a7230?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57186190?}, {0x15bbee0?, 0x3d4692a100?}, 0x3d4692a080, {0x15db118?, 0x3f5b4a7230?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57186190}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d4692a080}, {0x15db118, 0x3f5b4a7230}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57186190?}, {0x3f54784410?, 0x3f58798040?}, 0x3d4692a0c0, {0x15db118?, 0x3f5b4a7230?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57186190}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d4692a0c0}, {0x15db118, 0x3f5b4a7230}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57186140, {0x15d0ec8, 0x3d57186190}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57186140, {0x15d0e90, 0x3d5719e240}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x33) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1019 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d561f8a90, {0x15c3738, 0x3d561f8a08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d561f8a08, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec17980f0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec17980f0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d55da0840?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d55d805f0}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c86c8?, {0x15d0ec8?, 0x3d55d805f0?}, {0x15bbee0?, 0x3d31389e00?}, {0x15da6b0?, 0x3d31389d40?}, {0x15db118?, 0x3f5b4a7b90?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d55d805f0?}, {0x15bbee0?, 0x3d31389e00?}, 0x3d31389d80, {0x15db118?, 0x3f5b4a7b90?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d55d805f0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d31389d80}, {0x15db118, 0x3f5b4a7b90}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d55d805f0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d31389dc0, {0x15db118?, 0x3f5b4a7b90?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d55d805f0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d31389dc0}, {0x15db118, 0x3f5b4a7b90}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d55d805a0, {0x15d0ec8, 0x3d55d805f0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d55d805a0, {0x15d0e90, 0x3d55daa9c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x63) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1020 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d518d7590, {0x15c3738, 0x3d518d7508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d518d7508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec53d1350?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec53d1350, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d57186050}, {0x29eec?, 0x3aebc?}, {0x866a4?, 0x360807aa?}, {0x32788?, 0x221d060?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000ce598?, {0x15d0ec8?, 0x3d57186050?}, {0x15bbee0?, 0x3d444b2100?}, {0x15da6b0?, 0x3d444b2040?}, {0x15db118?, 0x3f5b4a7050?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57186050?}, {0x15bbee0?, 0x3d444b2100?}, 0x3d444b2080, {0x15db118?, 0x3f5b4a7050?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57186050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d444b2080}, {0x15db118, 0x3f5b4a7050}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d57186050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d444b20c0, {0x15db118?, 0x3f5b4a7050?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d57186050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d444b20c0}, {0x15db118, 0x3f5b4a7050}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57186000, {0x15d0ec8, 0x3d57186050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57186000, {0x15d0e90, 0x3d5719e0c0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x29) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1021 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d54ca0090, {0x15c3738, 0x3d54ca0008}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d54ca0008, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec250a6c0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec250a6c0, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3f5825c140}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000ce458?, {0x15d0ec8?, 0x3f5825c140?}, {0x15bbee0?, 0x3d56e00100?}, {0x15da6b0?, 0x3d56e00040?}, {0x15db118?, 0x3f5b4a72f0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f5825c140?}, {0x15bbee0?, 0x3d56e00100?}, 0x3d56e00080, {0x15db118?, 0x3f5b4a72f0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f5825c140}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d56e00080}, {0x15db118, 0x3f5b4a72f0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f5825c140?}, {0x3f54784410?, 0x3f58798040?}, 0x3d56e000c0, {0x15db118?, 0x3f5b4a72f0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f5825c140}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d56e000c0}, {0x15db118, 0x3f5b4a72f0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d570b6000, {0x15d0ec8, 0x3f5825c140}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d570b6000, {0x15d0e90, 0x3f5b290090}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x37) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1022 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d51db7590, {0x15c3738, 0x3d51db7508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d51db7508, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5e86de0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5e86de0, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d570100a0}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000cd3b8?, {0x15d0ec8?, 0x3d570100a0?}, {0x15bbee0?, 0x3d457c6100?}, {0x15da6b0?, 0x3d457c6040?}, {0x15db118?, 0x3f5b4a73b0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d570100a0?}, {0x15bbee0?, 0x3d457c6100?}, 0x3d457c6080, {0x15db118?, 0x3f5b4a73b0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d570100a0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d457c6080}, {0x15db118, 0x3f5b4a73b0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d570100a0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d457c60c0, {0x15db118?, 0x3f5b4a73b0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d570100a0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d457c60c0}, {0x15db118, 0x3f5b4a73b0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57010050, {0x15d0ec8, 0x3d570100a0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57010050, {0x15d0e90, 0x3d570261b0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x3b) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1023 [runnable]: github.com/prometheus/prometheus/model/labels.New({0x3d43e4f300, 0x5, 0x4?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/model/labels/labels.go:272 +0x50 github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d517a3590, {0x15c3738, 0x3d517a3508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1583 +0x4be0 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d517a3508, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5169560?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5169560, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d50c160a0}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000cb018?, {0x15d0ec8?, 0x3d50c160a0?}, {0x15bbee0?, 0x3d45742100?}, {0x15da6b0?, 0x3d45742040?}, {0x15db118?, 0x3f5b4a7110?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50c160a0?}, {0x15bbee0?, 0x3d45742100?}, 0x3d45742080, {0x15db118?, 0x3f5b4a7110?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50c160a0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d45742080}, {0x15db118, 0x3f5b4a7110}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d50c160a0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d457420c0, {0x15db118?, 0x3f5b4a7110?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d50c160a0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d457420c0}, {0x15db118, 0x3f5b4a7110}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57384000, {0x15d0ec8, 0x3d50c160a0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57384000, {0x15d0e90, 0x3d53090360}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x2d) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1024 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d40402a90, {0x15c3738, 0x3d40402a08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d40402a08, 0x2e?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec575a8a0?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec575a8a0, 0x2e}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3f5b4b00a0}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c9008?, {0x15d0ec8?, 0x3f5b4b00a0?}, {0x15bbee0?, 0x3d4413a140?}, {0x15da6b0?, 0x3d4413a080?}, {0x15db118?, 0x3f5b4a7140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f5b4b00a0?}, {0x15bbee0?, 0x3d4413a140?}, 0x3d4413a0c0, {0x15db118?, 0x3f5b4a7140?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f5b4b00a0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d4413a0c0}, {0x15db118, 0x3f5b4a7140}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f5b4b00a0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d4413a100, {0x15db118?, 0x3f5b4a7140?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f5b4b00a0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d4413a100}, {0x15db118, 0x3f5b4a7140}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d57296000, {0x15d0ec8, 0x3f5b4b00a0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d57296000, {0x15d0e90, 0x3d517225d0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x2e) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1025 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d5724d590, {0x15c3738, 0x3d5724d508}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d5724d508, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec5ebfbf0?, 0x3d3e0a7008?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec5ebfbf0, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x3d55da02f0?, 0x1151460?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3d517062d0}, {0x29eec?, 0x40?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x400006000c86c8?, {0x15d0ec8?, 0x3d517062d0?}, {0x15bbee0?, 0x3d5619a280?}, {0x15da6b0?, 0x3d5619a1c0?}, {0x15db118?, 0x3f5b4a71d0?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d517062d0?}, {0x15bbee0?, 0x3d5619a280?}, 0x3d5619a200, {0x15db118?, 0x3f5b4a71d0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d517062d0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5619a200}, {0x15db118, 0x3f5b4a71d0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3d517062d0?}, {0x3f54784410?, 0x3f58798040?}, 0x3d5619a240, {0x15db118?, 0x3f5b4a71d0?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3d517062d0}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d5619a240}, {0x15db118, 0x3f5b4a71d0}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3d51706280, {0x15d0ec8, 0x3d517062d0}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3d51706280, {0x15d0e90, 0x3d517224e0}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x31) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 goroutine 1026 [runnable]: github.com/prometheus/prometheus/promql/parser.(*yyParserImpl).Parse(0x3d51240a90, {0x15c3738, 0x3d51240a08}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/generated_parser.y.go:1603 +0x4e60 github.com/prometheus/prometheus/promql/parser.(*parser).parseGenerated(0x3d51240a08, 0x2f?) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:896 +0x78 github.com/prometheus/prometheus/promql/parser.ParseMetric({0x3ec4086d80?, 0x10ea7a0?}) /build/loki/src/loki-3.5.1/vendor/github.com/prometheus/prometheus/promql/parser/parse.go:180 +0xbc github.com/grafana/loki/v3/pkg/logql.mustParseLabels({0x3ec4086d80, 0x2f}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:304 +0x38 github.com/grafana/loki/v3/pkg/logql.MockQuerier.SelectSamples({0x1108b40?, {0x3d4ce00000?, 0x15db0a8?, 0xdce384?}}, {0x10?, 0x1108b40?}, {0x1?}) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:210 +0x250 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator.func1({0x15d0ec8, 0x3f584ea050}, {0x29eec?, 0x118b5e0?}, {0x866a4?, 0xa40784?}, {0x15bc800?, 0x3f58a9d140?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:339 +0x29c github.com/grafana/loki/v3/pkg/logql.SampleEvaluatorFunc.NewStepEvaluator(0x6000c9d68?, {0x15d0ec8?, 0x3f584ea050?}, {0x15bbee0?, 0x3d30e99b40?}, {0x15da6b0?, 0x3d30e99a80?}, {0x15db118?, 0x3f5b4a6810?}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:276 +0xb0 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f584ea050?}, {0x15bbee0?, 0x3d30e99b40?}, 0x3d30e99ac0, {0x15db118?, 0x3f5b4a6810?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f584ea050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d30e99ac0}, {0x15db118, 0x3f5b4a6810}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.newVectorAggEvaluator({0x15d0ec8?, 0x3f584ea050?}, {0x3f54784410?, 0x3f58798040?}, 0x3d30e99b00, {0x15db118?, 0x3f5b4a6810?}, 0x2710) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:406 +0xb8 github.com/grafana/loki/v3/pkg/logql.(*DefaultEvaluator).NewStepEvaluator(0x3f58798040, {0x15d0ec8, 0x3f584ea050}, {0x3f54784410?, 0x3f58798040?}, {0x15da7e8, 0x3d30e99b00}, {0x15db118, 0x3f5b4a6810}) /build/loki/src/loki-3.5.1/pkg/logql/evaluator.go:360 +0x614 github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0x3f584ea000, {0x15d0ec8, 0x3f584ea050}, {0x15da7e8, 0x3d4f832180}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:400 +0x2dc github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0x3f584ea000, {0x15d0e90, 0x3ec90e5c20}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:337 +0x500 github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _}) /build/loki/src/loki-3.5.1/pkg/logql/engine.go:286 +0xcfc github.com/grafana/loki/v3/pkg/logql.MockDownstreamer.Downstream.func1({0x15d0ec8, 0x3f5825cb40}, 0x7) /build/loki/src/loki-3.5.1/pkg/logql/test_utils.go:246 +0x1b0 github.com/grafana/dskit/concurrency.ForEachJob.func1() /build/loki/src/loki-3.5.1/vendor/github.com/grafana/dskit/concurrency/runner.go:118 +0x90 golang.org/x/sync/errgroup.(*Group).Go.func1() /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 914 /build/loki/src/loki-3.5.1/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xa8 FAIL github.com/grafana/loki/v3/pkg/logql 612.199s ok github.com/grafana/loki/v3/pkg/logql/bench 57.209s ? github.com/grafana/loki/v3/pkg/logql/bench/cmd/bench [no test files] ? github.com/grafana/loki/v3/pkg/logql/bench/cmd/bench/views [no test files] ? github.com/grafana/loki/v3/pkg/logql/bench/cmd/generate [no test files] ? github.com/grafana/loki/v3/pkg/logql/bench/cmd/stream [no test files] ok github.com/grafana/loki/v3/pkg/logql/log 35.658s ok github.com/grafana/loki/v3/pkg/logql/log/jsonexpr 0.700s ok github.com/grafana/loki/v3/pkg/logql/log/logfmt 0.836s ok github.com/grafana/loki/v3/pkg/logql/log/pattern 0.743s ok github.com/grafana/loki/v3/pkg/logql/sketch 66.737s ok github.com/grafana/loki/v3/pkg/logql/syntax 26.839s ? github.com/grafana/loki/v3/pkg/logql/vector [no test files] ok github.com/grafana/loki/v3/pkg/logqlanalyzer 4.991s ? github.com/grafana/loki/v3/pkg/logqlmodel [no test files] ok github.com/grafana/loki/v3/pkg/logqlmodel/metadata 3.867s ok github.com/grafana/loki/v3/pkg/logqlmodel/stats 3.337s ok github.com/grafana/loki/v3/pkg/loki 347.108s ? github.com/grafana/loki/v3/pkg/loki/common [no test files] ? github.com/grafana/loki/v3/pkg/lokifrontend [no test files] ? github.com/grafana/loki/v3/pkg/lokifrontend/frontend [no test files] ok github.com/grafana/loki/v3/pkg/lokifrontend/frontend/transport 7.748s ok github.com/grafana/loki/v3/pkg/lokifrontend/frontend/v1 14.472s ? github.com/grafana/loki/v3/pkg/lokifrontend/frontend/v1/frontendv1pb [no test files] ok github.com/grafana/loki/v3/pkg/lokifrontend/frontend/v2 16.749s ? github.com/grafana/loki/v3/pkg/lokifrontend/frontend/v2/frontendv2pb [no test files] ok github.com/grafana/loki/v3/pkg/pattern 9.263s ok github.com/grafana/loki/v3/pkg/pattern/aggregation 7.833s ? github.com/grafana/loki/v3/pkg/pattern/chunk [no test files] ? github.com/grafana/loki/v3/pkg/pattern/clientpool [no test files] ok github.com/grafana/loki/v3/pkg/pattern/drain 5.479s ok github.com/grafana/loki/v3/pkg/pattern/iter 4.781s ok github.com/grafana/loki/v3/pkg/querier 11.286s ? github.com/grafana/loki/v3/pkg/querier/astmapper [no test files] ? github.com/grafana/loki/v3/pkg/querier/deletion [no test files] ok github.com/grafana/loki/v3/pkg/querier/limits 4.411s ? github.com/grafana/loki/v3/pkg/querier/pattern [no test files] ok github.com/grafana/loki/v3/pkg/querier/plan 3.555s ok github.com/grafana/loki/v3/pkg/querier/queryrange 34.202s ? github.com/grafana/loki/v3/pkg/querier/queryrange/limits [no test files] ok github.com/grafana/loki/v3/pkg/querier/queryrange/queryrangebase 70.137s ? github.com/grafana/loki/v3/pkg/querier/queryrange/queryrangebase/definitions [no test files] ? github.com/grafana/loki/v3/pkg/querier/series [no test files] ? github.com/grafana/loki/v3/pkg/querier/stats [no test files] ok github.com/grafana/loki/v3/pkg/querier/tail 8.397s ? github.com/grafana/loki/v3/pkg/querier/testutil [no test files] ok github.com/grafana/loki/v3/pkg/querier/worker 17.876s --- FAIL: TestContextCond (0.01s) --- FAIL: TestContextCond/lots_of_goroutines_waiting_at_the_same_time,_none_of_them_misses_it's_broadcast_from_cancel (1.07s) queue_test.go:301: Error Trace: /build/loki/src/loki-3.5.1/pkg/queue/queue_test.go:301 Error: Condition never satisfied Test: TestContextCond/lots_of_goroutines_waiting_at_the_same_time,_none_of_them_misses_it's_broadcast_from_cancel FAIL FAIL github.com/grafana/loki/v3/pkg/queue 36.409s ok github.com/grafana/loki/v3/pkg/ruler 92.984s ok github.com/grafana/loki/v3/pkg/ruler/base 41.643s ? github.com/grafana/loki/v3/pkg/ruler/config [no test files] ? github.com/grafana/loki/v3/pkg/ruler/rulespb [no test files] ok github.com/grafana/loki/v3/pkg/ruler/rulestore 10.618s ok github.com/grafana/loki/v3/pkg/ruler/rulestore/bucketclient 14.639s ok github.com/grafana/loki/v3/pkg/ruler/rulestore/local 6.587s ok github.com/grafana/loki/v3/pkg/ruler/rulestore/objectclient 10.588s [no tests to run] ok github.com/grafana/loki/v3/pkg/ruler/storage/cleaner 8.829s ok github.com/grafana/loki/v3/pkg/ruler/storage/instance 10.841s ? github.com/grafana/loki/v3/pkg/ruler/storage/util [no test files] ok github.com/grafana/loki/v3/pkg/ruler/storage/wal 9.398s ? github.com/grafana/loki/v3/pkg/ruler/util [no test files] ? github.com/grafana/loki/v3/pkg/runtime [no test files] ok github.com/grafana/loki/v3/pkg/scheduler 10.566s ok github.com/grafana/loki/v3/pkg/scheduler/limits 0.909s ? github.com/grafana/loki/v3/pkg/scheduler/schedulerpb [no test files] ? github.com/grafana/loki/v3/pkg/server [no test files] ok github.com/grafana/loki/v3/pkg/storage 20.648s ok github.com/grafana/loki/v3/pkg/storage/bloom/v1 13.241s ok github.com/grafana/loki/v3/pkg/storage/bloom/v1/filter 3.235s ok github.com/grafana/loki/v3/pkg/storage/bucket 15.535s ok github.com/grafana/loki/v3/pkg/storage/bucket/azure 1.256s ? github.com/grafana/loki/v3/pkg/storage/bucket/bos [no test files] ? github.com/grafana/loki/v3/pkg/storage/bucket/filesystem [no test files] ? github.com/grafana/loki/v3/pkg/storage/bucket/gcs [no test files] ok github.com/grafana/loki/v3/pkg/storage/bucket/http 2.159s ? github.com/grafana/loki/v3/pkg/storage/bucket/oss [no test files] ok github.com/grafana/loki/v3/pkg/storage/bucket/s3 5.826s ? github.com/grafana/loki/v3/pkg/storage/bucket/swift [no test files] ok github.com/grafana/loki/v3/pkg/storage/chunk 6.751s ok github.com/grafana/loki/v3/pkg/storage/chunk/cache 26.872s ok github.com/grafana/loki/v3/pkg/storage/chunk/cache/resultscache 6.885s ok github.com/grafana/loki/v3/pkg/storage/chunk/client 5.209s ? github.com/grafana/loki/v3/pkg/storage/chunk/client/alibaba [no test files] ok github.com/grafana/loki/v3/pkg/storage/chunk/client/aws 11.154s ok github.com/grafana/loki/v3/pkg/storage/chunk/client/azure 6.671s ok github.com/grafana/loki/v3/pkg/storage/chunk/client/baidubce 5.263s ok github.com/grafana/loki/v3/pkg/storage/chunk/client/cassandra 7.331s ok github.com/grafana/loki/v3/pkg/storage/chunk/client/congestion 12.818s ok github.com/grafana/loki/v3/pkg/storage/chunk/client/gcp 13.355s ok github.com/grafana/loki/v3/pkg/storage/chunk/client/grpc 8.690s ok github.com/grafana/loki/v3/pkg/storage/chunk/client/hedging 3.434s ok github.com/grafana/loki/v3/pkg/storage/chunk/client/ibmcloud 7.295s ok github.com/grafana/loki/v3/pkg/storage/chunk/client/local 8.214s ok github.com/grafana/loki/v3/pkg/storage/chunk/client/openstack 9.131s ? github.com/grafana/loki/v3/pkg/storage/chunk/client/testutils [no test files] ok github.com/grafana/loki/v3/pkg/storage/chunk/client/util 4.414s ok github.com/grafana/loki/v3/pkg/storage/chunk/fetcher 12.828s ok github.com/grafana/loki/v3/pkg/storage/chunk/tests 46.815s ok github.com/grafana/loki/v3/pkg/storage/common/aws 3.898s ok github.com/grafana/loki/v3/pkg/storage/config 6.098s ok github.com/grafana/loki/v3/pkg/storage/detected 8.323s ? github.com/grafana/loki/v3/pkg/storage/errors [no test files] ? github.com/grafana/loki/v3/pkg/storage/hack [no test files] ok github.com/grafana/loki/v3/pkg/storage/stores 7.428s ? github.com/grafana/loki/v3/pkg/storage/stores/index [no test files] ok github.com/grafana/loki/v3/pkg/storage/stores/index/seriesvolume 6.355s ok github.com/grafana/loki/v3/pkg/storage/stores/index/stats 7.828s ok github.com/grafana/loki/v3/pkg/storage/stores/series 71.289s ok github.com/grafana/loki/v3/pkg/storage/stores/series/index 5.452s ok github.com/grafana/loki/v3/pkg/storage/stores/shipper/bloomshipper 8.168s ok github.com/grafana/loki/v3/pkg/storage/stores/shipper/bloomshipper/config 1.919s ok github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper 3.317s ok github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/boltdb 3.592s ok github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/boltdb/compactor 57.893s ok github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/downloads 7.063s ? github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/index [no test files] ok github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/storage 4.560s ? github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/testutil [no test files] ok github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb 349.752s ok github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb/index 91.473s ok github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb/sharding 2.720s ? github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb/testutil [no test files] ok github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/uploads 3.858s ok github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/util 4.571s ? github.com/grafana/loki/v3/pkg/storage/types [no test files] ok github.com/grafana/loki/v3/pkg/tool/audit 7.881s ok github.com/grafana/loki/v3/pkg/tool/client 1.553s ok github.com/grafana/loki/v3/pkg/tool/commands 6.751s ok github.com/grafana/loki/v3/pkg/tool/printer 6.772s ok github.com/grafana/loki/v3/pkg/tool/rules 6.840s ? github.com/grafana/loki/v3/pkg/tool/rules/rwrulefmt [no test files] ? github.com/grafana/loki/v3/pkg/tracing [no test files] ? github.com/grafana/loki/v3/pkg/ui [no test files] ok github.com/grafana/loki/v3/pkg/util 41.614s ok github.com/grafana/loki/v3/pkg/util/atomicfs 0.487s ? github.com/grafana/loki/v3/pkg/util/build [no test files] ok github.com/grafana/loki/v3/pkg/util/cfg 0.910s ? github.com/grafana/loki/v3/pkg/util/constants [no test files] ? github.com/grafana/loki/v3/pkg/util/deletion [no test files] ? github.com/grafana/loki/v3/pkg/util/discovery [no test files] ? github.com/grafana/loki/v3/pkg/util/encoding [no test files] ? github.com/grafana/loki/v3/pkg/util/extract [no test files] ? github.com/grafana/loki/v3/pkg/util/fakeauth [no test files] ? github.com/grafana/loki/v3/pkg/util/filter [no test files] ok github.com/grafana/loki/v3/pkg/util/flagext 2.858s ? github.com/grafana/loki/v3/pkg/util/httpgrpc [no test files] ok github.com/grafana/loki/v3/pkg/util/httpreq 1.468s ok github.com/grafana/loki/v3/pkg/util/instrument 1.540s ok github.com/grafana/loki/v3/pkg/util/jumphash 1.980s ? github.com/grafana/loki/v3/pkg/util/limiter [no test files] ok github.com/grafana/loki/v3/pkg/util/log 1.753s ok github.com/grafana/loki/v3/pkg/util/loser 0.314s ok github.com/grafana/loki/v3/pkg/util/marshal 5.839s ok github.com/grafana/loki/v3/pkg/util/marshal/legacy 3.594s ok github.com/grafana/loki/v3/pkg/util/mempool 2.281s ? github.com/grafana/loki/v3/pkg/util/net [no test files] ok github.com/grafana/loki/v3/pkg/util/pool 0.459s ok github.com/grafana/loki/v3/pkg/util/querylimits 5.874s ok github.com/grafana/loki/v3/pkg/util/ring 2.550s ok github.com/grafana/loki/v3/pkg/util/server 2.798s ? github.com/grafana/loki/v3/pkg/util/spanlogger [no test files] ? github.com/grafana/loki/v3/pkg/util/test [no test files] ok github.com/grafana/loki/v3/pkg/util/unmarshal 4.316s ok github.com/grafana/loki/v3/pkg/util/unmarshal/legacy 2.619s ok github.com/grafana/loki/v3/pkg/util/validation 2.308s ? github.com/grafana/loki/v3/pkg/util/wal [no test files] ok github.com/grafana/loki/v3/pkg/validation 4.033s ok github.com/grafana/loki/v3/production/helm/loki/test 0.549s [no tests to run] ? github.com/grafana/loki/v3/tools/bloom/inspector [no test files] ? github.com/grafana/loki/v3/tools/deprecated-config-checker [no test files] ok github.com/grafana/loki/v3/tools/deprecated-config-checker/checker 1.110s ? github.com/grafana/loki/v3/tools/doc-generator [no test files] ok github.com/grafana/loki/v3/tools/doc-generator/parse 6.138s ok github.com/grafana/loki/v3/tools/querytee 4.214s ? github.com/grafana/loki/v3/tools/tsdb/helpers [no test files] ? github.com/grafana/loki/v3/tools/tsdb/index-analyzer [no test files] ok github.com/grafana/loki/v3/tools/tsdb/migrate-versions 9.386s ok github.com/grafana/loki/v3/tools/tsdb/tsdb-map 3.989s FAIL ==> ERROR: A failure occurred in check().  Aborting... ==> ERROR: Build failed, check /var/lib/archbuild/extra-riscv64/felix-2/build [?25h[?25h[?25hreceiving incremental file list loki-3.5.1-1-riscv64-build.log loki-3.5.1-1-riscv64-check.log sent 62 bytes received 69,331 bytes 138,786.00 bytes/sec total size is 1,594,589 speedup is 22.98