GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix: Bugfix and add retry logic to dynamodb as onlinestore by iamcodingcat · Pull Request #5476 · feast-dev/feast

Conversation

@iamcodingcat

@iamcodingcat iamcodingcat commented Jun 30, 2025

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

A few weeks ago, I made PR and merge it for integrating dynamodb as online store in Golang feature server. And before applying it in my production, I run load test with locust in EKS cluster. During load testing, One bug is found and I found the one thing that I will enhance when requesting to dynamodb.

Which issue(s) this PR fixes:

I made two features in Golang feature server. the explanation for them is below.

  1. Bugfix: Before revising the commit, unprocessedEntityIds variable maintains processed entity keys. But this variable is globally defined outside feature view loop. So, if some entity keys doesn't exist in A feature view, then delete them from unprocessedEntityIds. Because of this, although the entity keys exist in B feature view, Golang server requests to dynamodb not including the entity keys. At the result, the below bug occurs nil derefrence. The stack trace is below.

{"level":"error",
"stack_trace":"goroutine 121881 [running]:\ngithub.com/feast-dev/feast/go/internal/feast/server.logStackTrace()\n\t
/feast/go/internal/feast/server/http_server.go:304 +0x64\n
github.com/feast-dev/feast/go/internal/feast/server.(*httpServer).Serve.recoverMiddleware.func1.1()\n\t
/feast/go/internal/feast/server/http_server.go:333 +0x98\n
panic({0x1a3fd20?, 0x2db8d60?})\n\t
/usr/local/go/src/runtime/panic.go:792 +0x124\n
github.com/feast-dev/feast/go/internal/feast/onlineserving.TransposeFeatureRowsIntoColumns({0x4001662008, 0x1f3, 0x40010820a0?}, 0x4001196580, {0x4001148fc0, 0x5, 0x400567e800?}, {0x1f5c680, 0x2f1ede0}, 0x1f4)\n\t
/feast/go/internal/feast/onlineserving/serving.go:388 +0x424\ngithub.com/feast-dev/feast/go/internal/feast.(*FeatureStore).GetOnlineFeatures(0x4000976fc0, {0x1f61e98, 0x40010820a0}, {0x4001100008, 0x2f, 0x47}, 0x0, 0x4003968db0, 0x4003968de0, 0x0)\n\t
/feast/go/internal/feast/featurestore.go:173 +0x5d0\n
github.com/feast-dev/feast/go/internal/feast/server.(*httpServer).getOnlineFeatures(0x40004c5f08, {0x1f5cc20, 0x400087e1c0}, 0x40008deb40)\n\t
/feast/go/internal/feast/server/http_server.go:201 +0x6a8\n
net/http.HandlerFunc.ServeHTTP(0x8080808080806a36?, {0x1f5cc20?, 0x400087e1c0?}, 0x2ef5480?)\n\t
/usr/local/go/src/net/http/server.go:2294 +0x38\ngithub.com/feast-dev/feast/go/internal/feast/server.(*httpServer).Serve.recoverMiddleware.func1({0x1f5cc20?, 0x400087e1c0?}, 0x4004653b28?)\n\t
/feast/go/internal/feast/server/http_server.go:338 +0x6c\n
net/http.HandlerFunc.ServeHTTP(0x400088b200?, {0x1f5cc20?, 0x400087e1c0?}, 0x4004653b10?)\n\t
/usr/local/go/src/net/http/server.go:2294 +0x38\n
net/http.(*ServeMux).ServeHTTP(0x10?, {0x1f5cc20, 0x400087e1c0}, 0x40008deb40)\n\t
/usr/local/go/src/net/http/server.go:2822 +0x1b4\n
net/http.serverHandler.ServeHTTP({0x1f53028?}, {0x1f5cc20?, 0x400087e1c0?}, 0x6?)\n\t
/usr/local/go/src/net/http/server.go:3301 +0xbc\n
net/http.(*conn).serve(0x400241a240, {0x1f61e60, 0x40008b1230})\n\t
/usr/local/go/src/net/http/server.go:2102 +0x52c\ncreated by net/http.(*Server).Serve in goroutine 1\n\t
/usr/local/go/src/net/http/server.go:3454 +0x3d8\n","time":"2025-06-27T07:36:01Z"
}

So, I move defining unprocessedEntityIds(in the new commit, I rename unprocessedEntityIdsFeatureView variable) in feature view loop.

  1. Enhancement: I add dynamodb retrying related config and retry logic when request to dynamodb. Of course, the revision in dynamodb-side configuration(Read units per seconds, ...) will be needed but I think that client side retrty logic must be needed. So I add retry logic with exponential backoff and max retry config value is set to dynamodb online store config that can be defined in feature_store.yaml.

Misc

@shuchu @jyejare cc. @franciscojavierarceo

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@iamcodingcat
fix/enhancement: bugfix and add retry logic to dynamodb as onlinestore
bbca481
Signed-off-by: iamcodingcat <[email protected]>
@iamcodingcat
iamcodingcat requested a review from a team as a code owner June 30, 2025 06:07
ntkathole
ntkathole approved these changes Jun 30, 2025
View reviewed changes

@ntkathole ntkathole left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

iamcodingcat reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@ntkathole ntkathole added the ok-to-test label Jun 30, 2025
franciscojavierarceo
franciscojavierarceo approved these changes Jun 30, 2025
View reviewed changes
@ntkathole
ntkathole merged commit af99f0a into feast-dev:master Jun 30, 2025
@iamcodingcat
iamcodingcat deleted the fix/enhancement/go/onlinestore/dynamodb/bug-and-enhancement branch July 1, 2025 01:09
@iamcodingcat

iamcodingcat commented Jul 1, 2025

Copy link
Copy Markdown
Contributor Author

@ntkathole Thanks for your fast review!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@franciscojavierarceo franciscojavierarceo franciscojavierarceo approved these changes
@ntkathole ntkathole ntkathole approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@iamcodingcat @franciscojavierarceo @ntkathole
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.