GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — server: investigate pending HA work when executing in new MS session by shwstppr · Pull Request #10167 · apache/cloudstack

Conversation

@shwstppr

@shwstppr shwstppr commented Jan 8, 2025

Copy link
Copy Markdown
Contributor

Description

For HA work items that are created for host state change, checks must be done when execution is called in a new management server session.

A new column, reason, has been added in cloud.op_ha_work table to track the reason for HA work.

When HighAvailabilityManager starts it finds and puts all pending HA work items in Investigating state. During execution of the HA work if it is found in investigating state, checks are done to verify if the work is still valid. If the jobs is found to be invalid it is cancelled.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@shwstppr
server: investigate pending HA work when executing in new MS session
a3aa8ff 
For HA work items that are created for host state change, checks must be
done when execution is called in a new management server session.

A new column, reason, has been added in cloud.op_ha_work table to track
the reason for HA work.

When HighAvailabilityManager starts it finds and puts all pending HA
work items in Investigating state. During execution of the HA work if it
is found in investigating state, checks are done to verify if the work
is still valid. If the jobs is found to be invalid it is cancelled.

Signed-off-by: Abhishek Kumar <[email protected]>
@boring-cyborg boring-cyborg Bot added component:agent component:orchestration labels Jan 8, 2025
@codecov

codecov Bot commented Jan 8, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 71.76471% with 24 lines in your changes missing coverage. Please review.

Project coverage is 16.99%. Comparing base (bd488c4) to head (9c18920).
Report is 38 commits behind head on 4.20.

Files with missing lines Patch % Lines
...java/com/cloud/ha/HighAvailabilityManagerImpl.java 74.41% 10 Missing and 1 partial ⚠️
server/src/main/java/com/cloud/ha/HaWorkVO.java 25.00% 6 Missing ⚠️
...n/java/com/cloud/resource/ResourceManagerImpl.java 0.00% 5 Missing ⚠️
...java/com/cloud/agent/manager/AgentManagerImpl.java 0.00% 1 Missing ⚠️
...stack/ha/provider/host/HAAbstractHostProvider.java 0.00% 1 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               4.20   #10167      +/-   ##
============================================
+ Coverage     16.07%   16.99%   +0.92%     
- Complexity    12884    12977      +93     
============================================
  Files          5639     5245     -394     
  Lines        494184   461985   -32199     
  Branches      59924    54191    -5733     
============================================
- Hits          79420    78513     -907     
+ Misses       405934   374799   -31135     
+ Partials       8830     8673     -157
Flag Coverage Δ
uitests ?
unittests 16.99% <71.76%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@shwstppr

shwstppr commented Jan 8, 2025

Copy link
Copy Markdown
Contributor Author

@blueorangutan package
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@blueorangutan

blueorangutan commented Jan 8, 2025

Copy link
Copy Markdown

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
sureshanaparti
sureshanaparti approved these changes Jan 8, 2025
View reviewed changes

@sureshanaparti sureshanaparti left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@DaanHoogland DaanHoogland added this to the 4.20.1 milestone Jan 8, 2025
@blueorangutan

blueorangutan commented Jan 8, 2025

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12012
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@shwstppr

shwstppr commented Jan 8, 2025

Copy link
Copy Markdown
Contributor Author

@blueorangutan test
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@blueorangutan

blueorangutan commented Jan 8, 2025

Copy link
Copy Markdown

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@blueorangutan

blueorangutan commented Jan 9, 2025

Copy link
Copy Markdown

[SF] Trillian test result (tid-12060)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52979 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10167-t12060-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_11_isolated_network_with_dynamic_routed_mode Error 2.28 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 2.50 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 2.50 test_ipv4_routing.py
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@shwstppr
fix for one managment server shutdown
e2a6450 
Signed-off-by: Abhishek Kumar <[email protected]>
@shwstppr

shwstppr commented Jan 9, 2025

Copy link
Copy Markdown
Contributor Author

@blueorangutan package
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@blueorangutan

blueorangutan commented Jan 9, 2025

Copy link
Copy Markdown

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@blueorangutan

blueorangutan commented Jan 9, 2025

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12025
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@shwstppr
fix unused aurgument
04913a4 
Signed-off-by: Abhishek Kumar <[email protected]>
@sureshanaparti

sureshanaparti commented Jan 10, 2025

Copy link
Copy Markdown
Contributor

@blueorangutan package
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@blueorangutan

blueorangutan commented Jan 10, 2025

Copy link
Copy Markdown

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@blueorangutan

blueorangutan commented Jan 10, 2025

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12041
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@shwstppr
fix
9c18920 
Signed-off-by: Abhishek Kumar <[email protected]>
@shwstppr

shwstppr commented Jan 21, 2025

Copy link
Copy Markdown
Contributor Author

@blueorangutan package
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@blueorangutan

blueorangutan commented Jan 21, 2025

Copy link
Copy Markdown

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@blueorangutan

blueorangutan commented Jan 21, 2025

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12145
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@kiranchavala

kiranchavala commented Jan 27, 2025

Copy link
Copy Markdown
Member

@blueorangutan test
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@blueorangutan

blueorangutan commented Jan 27, 2025

Copy link
Copy Markdown

@kiranchavala a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@blueorangutan

blueorangutan commented Jan 27, 2025

Copy link
Copy Markdown

[SF] Trillian test result (tid-12189)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 49255 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10167-t12189-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_11_isolated_network_with_dynamic_routed_mode Error 0.24 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 2.70 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 2.70 test_ipv4_routing.py
test_02_list_cpvm_vm Failure 0.04 test_ssvm.py
test_04_cpvm_internals Failure 0.04 test_ssvm.py
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
kiranchavala
kiranchavala approved these changes Jan 28, 2025
View reviewed changes

@kiranchavala kiranchavala left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM , Tested the scenario of HostMaintenance, HostDown and HostDegraded.

In case of management server restart, found that cloudstack validates the entries in op_ha_work table and takes action based on the host state

Test Case Execution Result
Test host maintainence scenario Pass
Test host ha /host down scenario Pass

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@yadvr
yadvr marked this pull request as ready for review January 28, 2025 09:08
yadvr
yadvr approved these changes Jan 28, 2025
View reviewed changes
@yadvr
yadvr merged commit 33a37da into apache:4.20 Jan 28, 2025
@yadvr
yadvr deleted the investigate-pending-ha-420 branch January 28, 2025 09:09
DaanHoogland added a commit that referenced this pull request Jan 28, 2025
@DaanHoogland
Merge release branch 4.20 to main
048649d 
* 4.20:
  server: investigate pending HA work when executing in new MS session (#10167)
  extra null guard (#10264)
weizhouapache pushed a commit to shapeblue/cloudstack that referenced this pull request Mar 4, 2025
@shwstppr
FR86 - server: investigate pending HA work when executing in new MS s…
34c3218 
…ession (apache#519)

Fixes apache#511

For HA work items that are created for host state change, checks must be done when execution is called in a new management server session.

A new column, reason, has been added in cloud.op_ha_work table to track the reason for HA work.

When HighAvailabilityManager starts it finds and puts all pending HA work items in Investigating state. During execution of the HA work if it is found in investigating state, checks are done to verify if the work is still valid. If the jobs is found to be invalid it is cancelled.

Upstream PR: apache#10167

Signed-off-by: Abhishek Kumar <[email protected]>
@Pearl1594 Pearl1594 added this to ACS 4.20.1 Mar 17, 2025
@Pearl1594 Pearl1594 moved this to Done in ACS 4.20.1 Mar 17, 2025
dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Jun 19, 2025
@shwstppr @dhslove
server: investigate pending HA work when executing in new MS session (a…
2bc5769 
…pache#10167)

For HA work items that are created for host state change, checks must be
done when execution is called in a new management server session.

A new column, reason, has been added in cloud.op_ha_work table to track
the reason for HA work.

When HighAvailabilityManager starts it finds and puts all pending HA
work items in Investigating state. During execution of the HA work if it
is found in investigating state, checks are done to verify if the work
is still valid. If the jobs is found to be invalid it is cancelled.

Signed-off-by: Abhishek Kumar <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yadvr yadvr yadvr approved these changes
@kiranchavala kiranchavala kiranchavala approved these changes
@sureshanaparti sureshanaparti sureshanaparti approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

component:agent component:orchestration

Projects

ACS 4.20.1
Status: Done

Milestone

4.20.1

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@shwstppr @blueorangutan @sureshanaparti @kiranchavala @yadvr @DaanHoogland
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.