GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Show connector errors in the profiler · feldera/feldera@4d3e529

File tree

Expand file treeCollapse file tree
Open diff view settings
Filter options
  • js-packages
    • common-ui
      • README.md
      • src/lib
        • ConnectorErrors.svelte
        • connectorMetrics.ts
        • index.ts
    • profiler-app/src
      • App.svelte
    • profiler-layout/src/lib
      • components
        • ConnectorErrors.svelte.spec.ts
        • SupportBundleViewerLayout.svelte.spec.ts
        • SupportBundleViewerLayout.svelte
        • tabs
          • ConnectorsTab.svelte.spec.ts
          • ConnectorsTab.svelte
          • MetricsTab.svelte
      • functions
        • countConnectorErrors.spec.ts
        • processZipBundle.ts
    • web-console/src
      • lib/components/pipelines/editor/performance
        • ConnectorErrors.svelte.spec.ts
        • ConnectorErrors.svelte
      • routes/(system)/(authenticated)/(authorized)/profile-viewer
        • +page.svelte
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎js-packages/common-ui/README.md‎

Copy file name to clipboardExpand all lines: js-packages/common-ui/README.md
+1-1Lines changed: 1 addition & 1 deletion
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ import { SegmentedControl, TabsPanel, PersistentContent, MonacoEditor } from 'co
2121
| Layout | `Drawer` shows a resizable side panel on wide screens (`InlineDrawer`) and an overlay on tablets and phones (`OverlayDrawer`). `InlineDropdown` toggles content under a header. `useIsMobile.svelte` has the screen-size media queries. |
2222
| Layout helpers | `PersistentContent` and `persistentRect.svelte` for content that must survive conditional re-renders (see those files for details). `ANSIDecoratedText` renders ANSI-colored log output. |
2323
| Buttons | `ClipboardCopyButton`. `ClickFeedback` shows a short confirmation state after a click. |
24-
| Pipeline views | `ConnectorMetricsTables` shows the connector statistics of each table and view. Use `aggregateConnectorMetrics` from `common-ui/connectorMetrics` to make its input from a `/stats` response. |
24+
| Pipeline views | `ConnectorMetricsTables` shows the connector statistics of each table and view. Use `aggregateConnectorMetrics` from `common-ui/connectorMetrics` to make its input from a `/stats` response. `ConnectorErrors` shows the error lists of one connector. The caller passes the connector status, from the API or from a support bundle. |
2525
| Health views | `StatusTimeline`, `EventLogList` and `HealthEventList` show a status history, its incidents and the events of one incident. `PipelineEventHistory` combines them for the monitor events of a pipeline, from the API or from a support bundle. |
2626

2727
Plain TypeScript modules that only the web console and the profiler use also
Collapse file

‎js-packages/common-ui/src/lib/ConnectorErrors.svelte‎

Copy file name to clipboard
+198Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
<script lang="ts">
2+
import { formatDateTime, formatServerTime } from 'common-lib/format'
3+
import Dayjs from 'dayjs'
4+
import { fade, slide } from 'svelte/transition'
5+
import type {
6+
ConnectorError,
7+
ConnectorErrorFilter,
8+
ConnectorErrorKind,
9+
InputEndpointStatus,
10+
OutputEndpointStatus
11+
} from './connectorMetrics'
12+
import { getCaseDependentName } from './felderaRelation'
13+
import InlineDropdown from './InlineDropdown.svelte'
14+
import SegmentedControl from './SegmentedControl.svelte'
15+
import Tooltip from './Tooltip.svelte'
16+
17+
const inputFilterOptions = ['all', 'parse', 'transport'] as const
18+
const outputFilterOptions = ['all', 'encode', 'transport'] as const
19+
20+
let filterOptionLabels: Record<string, string> = {
21+
transport: 'Transport',
22+
parse: 'Parse',
23+
encode: 'Encode',
24+
all: 'All'
25+
}
26+
27+
let tagsFilter: ConnectorErrorFilter = $state('all')
28+
29+
let {
30+
relationName,
31+
connectorName,
32+
direction,
33+
filter = 'all',
34+
status,
35+
loading = false,
36+
onClose
37+
}: {
38+
relationName: string
39+
/** Endpoint name, prefixed with the relation name. */
40+
connectorName: string
41+
direction: 'input' | 'output'
42+
filter?: ConnectorErrorFilter
43+
/** The connector status with its error lists. `null` while it is not available. */
44+
status: InputEndpointStatus | OutputEndpointStatus | null
45+
loading?: boolean
46+
onClose: () => void
47+
} = $props()
48+
49+
const filterOptions = $derived(direction === 'input' ? inputFilterOptions : outputFilterOptions)
50+
const filterItems = $derived(
51+
filterOptions.map((value) => ({ value, label: filterOptionLabels[value] ?? value }))
52+
)
53+
54+
$effect(() => {
55+
tagsFilter = filter
56+
})
57+
58+
const strippedConnectorName = $derived(
59+
connectorName.slice(getCaseDependentName(relationName).name.length + 1)
60+
)
61+
62+
/** One row of the error list: an error, or a gap. A gap stands for `count` errors that are
63+
* missing between two listed errors, by `ConnectorError.index`, e.g. older errors that the
64+
* pipeline dropped because it keeps only the most recent ones. */
65+
type DisplayItem = { kind: 'error'; error: ConnectorError } | { kind: 'gap'; count: number }
66+
67+
function withGaps(errors: ConnectorError[]): DisplayItem[] {
68+
const byTag = new Map<string, ConnectorError[]>()
69+
for (const e of errors) {
70+
const k = e.tag ?? 'other'
71+
const existing = byTag.get(k)
72+
if (existing) {
73+
existing.push(e)
74+
} else {
75+
byTag.set(k, [e])
76+
}
77+
}
78+
const result: DisplayItem[] = []
79+
for (const tagErrors of byTag.values()) {
80+
const sorted = [...tagErrors].sort((a, b) => a.index - b.index)
81+
for (let i = 0; i < sorted.length; i++) {
82+
if (i > 0) {
83+
const gap = sorted[i].index - sorted[i - 1].index - 1
84+
if (gap > 0) {
85+
result.push({ kind: 'gap', count: gap })
86+
}
87+
}
88+
result.push({ kind: 'error', error: sorted[i] })
89+
}
90+
}
91+
return result
92+
}
93+
94+
/** Returns the error lists of the connector by error kind, e.g. `{ parse: [...], transport:
95+
* [...] }` for an input connector. If no errors for an error kind are there the field remains
96+
* with an empty array. */
97+
function getErrorsByKind(
98+
s: InputEndpointStatus | OutputEndpointStatus
99+
): Partial<Record<ConnectorErrorKind, ConnectorError[]>> {
100+
if (direction === 'input') {
101+
return {
102+
parse: (s as InputEndpointStatus).parse_errors ?? [],
103+
transport: s.transport_errors ?? []
104+
}
105+
}
106+
return {
107+
encode: (s as OutputEndpointStatus).encode_errors ?? [],
108+
transport: s.transport_errors ?? []
109+
}
110+
}
111+
112+
const displayItems = $derived.by(() => {
113+
if (!status) {
114+
return [] as DisplayItem[]
115+
}
116+
const errorsByKind = getErrorsByKind(status)
117+
const combined =
118+
tagsFilter === 'all' ? Object.values(errorsByKind).flat() : (errorsByKind[tagsFilter] ?? [])
119+
return withGaps(combined).reverse()
120+
})
121+
</script>
122+
123+
<div class="bg-white-dark flex h-full flex-col gap-2 rounded px-4 pt-4">
124+
<div class="flex items-start justify-between">
125+
<div>
126+
<div class="font-medium">{relationName} · {strippedConnectorName}</div>
127+
</div>
128+
<button class="fd fd-x text-[20px]" onclick={onClose} aria-label="Close"></button>
129+
</div>
130+
<div class="flex flex-nowrap items-center gap-2">
131+
<SegmentedControl
132+
value={tagsFilter}
133+
onValueChange={(v) => (tagsFilter = v as ConnectorErrorFilter)}
134+
items={filterItems}
135+
class="-mt-2"
136+
/>
137+
<span class="text-surface-500">errors</span>
138+
</div>
139+
<div class="-mr-4 scrollbar flex-1 overflow-y-auto pr-4">
140+
{#if loading}
141+
<div class="p-2 text-surface-500">Loading…</div>
142+
{:else if displayItems.length === 0}
143+
<div class="p-2 text-surface-500">
144+
{#if tagsFilter === 'all'}
145+
No errors
146+
{:else}
147+
No errors of this type
148+
{/if}
149+
</div>
150+
{:else}
151+
{#each displayItems as item}
152+
{#if item.kind === 'gap'}
153+
<div class="px-2 py-1 text-center text-surface-500 italic">
154+
{#if item.count !== 1}
155+
{item.count} other errors skipped
156+
{:else}
157+
{item.count} other error skipped
158+
{/if}
159+
</div>
160+
{:else}
161+
<InlineDropdown>
162+
{#snippet header(isOpen, toggle)}
163+
<div
164+
class="flex w-full cursor-pointer items-center gap-2 py-2 pr-2"
165+
onclick={toggle}
166+
role="presentation"
167+
>
168+
<div
169+
class="fd fd-chevron-down text-[20px] transition-transform {isOpen
170+
? 'rotate-180'
171+
: ''}"
172+
></div>
173+
<span class="flex-1 overflow-hidden text-nowrap text-ellipsis">
174+
{#if !isOpen}
175+
<span out:fade={{ duration: 100 }}>{item.error.message}</span>
176+
{/if}
177+
</span>
178+
{#if status?.fatal_error != null && item.error.message === status.fatal_error}
179+
<span class="fd fd-circle-x text-error-500"></span>
180+
{/if}
181+
<span class="ml-auto">{formatServerTime(item.error.timestamp)}</span>
182+
<Tooltip placement="top">{formatDateTime(Dayjs(item.error.timestamp))}</Tooltip>
183+
</div>
184+
{/snippet}
185+
{#snippet content()}
186+
<div
187+
class="mr-16 -mb-7 ml-7 -translate-y-7 pb-2"
188+
transition:slide={{ duration: 150 }}
189+
>
190+
{item.error.message}
191+
</div>
192+
{/snippet}
193+
</InlineDropdown>
194+
{/if}
195+
{/each}
196+
{/if}
197+
</div>
198+
</div>
Collapse file

‎js-packages/common-ui/src/lib/connectorMetrics.ts‎

Copy file name to clipboardExpand all lines: js-packages/common-ui/src/lib/connectorMetrics.ts
+35-1Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,16 @@ export type OutputEndpointMetrics = {
3939
transmitted_records: number
4040
}
4141

42+
/** An error reported by a connector. */
43+
export type ConnectorError = {
44+
/** Sequence number of the error among the errors of one kind: parse, encode or transport. */
45+
index: number
46+
message: string
47+
tag?: string | null
48+
/** RFC3339 timestamp */
49+
timestamp: string
50+
}
51+
4252
export type InputEndpointStatus = {
4353
endpoint_name: string
4454
config: { stream: string }
@@ -47,6 +57,9 @@ export type InputEndpointStatus = {
4757
barrier: boolean
4858
health?: ConnectorHealth | null
4959
fatal_error?: string | null
60+
/** The most recent errors. Absent when the status was collected without connector errors. */
61+
parse_errors?: ConnectorError[] | null
62+
transport_errors?: ConnectorError[] | null
5063
}
5164

5265
export type OutputEndpointStatus = {
@@ -57,6 +70,9 @@ export type OutputEndpointStatus = {
5770
paused?: boolean
5871
health?: ConnectorHealth | null
5972
fatal_error?: string | null
73+
/** The most recent errors. Absent when the status was collected without connector errors. */
74+
encode_errors?: ConnectorError[] | null
75+
transport_errors?: ConnectorError[] | null
6076
}
6177

6278
/** The part of a `/stats` response that `aggregateConnectorMetrics` reads. */
@@ -70,8 +86,12 @@ export type ConnectorStatus = {
7086
}
7187
}
7288

89+
/** An input connector reports parse and transport errors. An output connector reports encode and
90+
* transport errors. */
91+
export type ConnectorErrorKind = 'parse' | 'transport' | 'encode'
92+
7393
/** The kind of connector errors to show when a connector is selected. */
74-
export type ConnectorErrorFilter = 'all' | 'parse' | 'transport' | 'encode'
94+
export type ConnectorErrorFilter = 'all' | ConnectorErrorKind
7595

7696
export type AggregatedMetrics<M, EndpointStatus = {}> = {
7797
aggregate: { metrics: M }
@@ -100,6 +120,20 @@ export type ConnectorMetrics = {
100120
views: Map<string, AggregatedOutputEndpointMetrics>
101121
}
102122

123+
/** Sum of the error counters that the connector tables show, over all connectors. */
124+
export const countConnectorErrors = ({ tables, views }: ConnectorMetrics): number => {
125+
let count = 0
126+
for (const { aggregate } of tables.values()) {
127+
count +=
128+
(aggregate.metrics.num_parse_errors ?? 0) + (aggregate.metrics.num_transport_errors ?? 0)
129+
}
130+
for (const { aggregate } of views.values()) {
131+
count +=
132+
(aggregate.metrics.num_encode_errors ?? 0) + (aggregate.metrics.num_transport_errors ?? 0)
133+
}
134+
return count
135+
}
136+
103137
/**
104138
* Higher of two connector latencies, ignoring connectors without samples.
105139
*
Collapse file

‎js-packages/common-ui/src/lib/index.ts‎

Copy file name to clipboardExpand all lines: js-packages/common-ui/src/lib/index.ts
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ export { default as ANSIDecoratedText } from './ANSIDecoratedText.svelte'
33
export { default as ClickFeedback } from './ClickFeedback.svelte'
44
export { default as ClipboardCopyButton } from './ClipboardCopyButton.svelte'
55
export { default as ConnectorMetricsTables } from './ConnectorMetricsTables.svelte'
6+
export { default as ConnectorErrors } from './ConnectorErrors.svelte'
67
export { default as Drawer } from './Drawer.svelte'
78
export { default as EventLogList } from './EventLogList.svelte'
89
export { default as HealthEventList } from './HealthEventList.svelte'
Collapse file

‎js-packages/profiler-app/src/App.svelte‎

Copy file name to clipboardExpand all lines: js-packages/profiler-app/src/App.svelte
+4-1Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
<script lang="ts">
2-
import type { ConnectorMetrics } from 'common-ui/connectorMetrics'
2+
import type { ConnectorMetrics, ConnectorStatus } from 'common-ui/connectorMetrics'
33
import type { PipelineMonitorEvent } from 'common-ui/pipelineHealth'
44
import {
55
createLoadGuard,
@@ -28,6 +28,7 @@
2828
logText: string | undefined
2929
globalMetrics: GlobalMetrics | undefined
3030
connectorMetrics: ConnectorMetrics | undefined
31+
connectorStatus: ConnectorStatus | undefined
3132
runtimeConfig: unknown
3233
pipelineEvents: PipelineMonitorEvent[] | undefined
3334
sourceFiles: SourceFile[] | undefined
@@ -54,6 +55,7 @@
5455
logText: data.logText,
5556
globalMetrics: data.globalMetrics,
5657
connectorMetrics: data.connectorMetrics,
58+
connectorStatus: data.connectorStatus,
5759
runtimeConfig: data.runtimeConfig,
5860
pipelineEvents: data.pipelineEvents,
5961
sourceFiles: data.sourceFiles
@@ -124,6 +126,7 @@
124126
logText={profileData.logText}
125127
globalMetrics={profileData.globalMetrics}
126128
connectorMetrics={profileData.connectorMetrics}
129+
connectorStatus={profileData.connectorStatus}
127130
runtimeConfig={profileData.runtimeConfig}
128131
pipelineEvents={profileData.pipelineEvents}
129132
sourceFiles={profileData.sourceFiles}

0 commit comments

Comments
0 (0)
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.