GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — Bugfix/cstackex 327-Concurrent iSCSI VM deployment fails intermittently: post-clone LUN resize returns 404 by sp16743 · Pull Request #118 · NetApp/cloudstack

Conversation

sp16743 commented Oct 9, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Collaborator

Description

After an iSCSI template clone, CloudStack grows the new LUN when the service offering disk is larger than the template. ONTAP returns the LUN UUID from the clone, but the immediate resize PATCH /api/storage/luns/{uuid} can return HTTP 404 (entry doesn't exist, target uuid) before that LUN is visible.

resizeCloudStackVolume now retries only that PATCH, using the same UUID. A 404 is retried for up to 30 attempts, one second apart. Any other error fails on the first attempt.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

BLOCKER
Critical
Major
Minor
Trivial

Screenshots (if appropriate):

Before the fix:

How Has This Been Tested?

Live concurrent deployment:
-Created an ONTAP iSCSI primary storage pool.
-Deployed ten CentOS VMs concurrently using the ISCSI compute offering.
-Nine deployments reached LUN cloning; one failed earlier due to insufficient host capacity.
-Nine clone POST requests returned LUN UUIDs.
-Five cloned LUNs were visible on the first GET.
-Four cloned LUNs returned 404 on the first GET.
-For all four, CloudStack waited and retried; the LUN became visible on attempt 2/5.
-CloudStack then resized each LUN to 10 GiB using PATCH and continued VM deployment successfully.
-No clone failed with Cloned Lun uuid was not visible.

How did you try to break this feature and the system with this change?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

sp16743 requested review from piyush5netapp, rajiv-jain-netapp, sandeeplocharla and suryag1201 as code owners

October 9, 2026 16:49

github-actions Bot commented Oct 9, 2026

Copy link

Copy Markdown

🔴 Test Coverage Grade: `D` — Marginal

Metric	Value
Line coverage	24.82%
Branch coverage	18.97%

Grade Scale

Grade	Line Coverage	Meaning
🟢 A	≥ 80%	Excellent - this code sleeps well at night 😴
🟡 B	60-79%	Good - almost there, don't stop now 😉
🟠 C	40-59%	Acceptable - your code is wearing a seatbelt, but no airbags 😬
🔴 D	20-39%	Marginal - boldly shipping where no test has gone before 🖖
⛔ F	< 20%	Failing - tests? what tests? 🔥

Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

github-actions Bot commented Oct 12, 2026

Copy link

Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

github-actions Bot added the status:has-conflicts label

Oct 12, 2026

suryag1201 reviewed

Oct 12, 2026

View reviewed changes

Comment thread

...age/volume/ontap/src/main/java/org/apache/cloudstack/storage/service/UnifiedSANStrategy.java Outdated

		if (sizeInBytes <= 0) {
		throw new CloudRuntimeException("Failed to resize Lun, invalid size " + sizeInBytes);
		}
		String authHeader = OntapStorageUtils.generateAuthHeader(storage.getUsername(), storage.getPassword());

suryag1201 Oct 12, 2026

Copy link

Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of adding the retry logic here, during lun clone return the job response and poll for that job to get complete and perform the resize later

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

suryag1201 requested changes

Oct 12, 2026

View reviewed changes


          Fix ONTAP LUN clone visibility using GET retry

182b07b

sp16743 force-pushed the bugfix/CSTACKEX-327 branch from 62c8ec7 to 182b07b Compare

October 26, 2026 14:00

github-actions Bot removed the status:has-conflicts label

Oct 26, 2026

github-actions Bot commented Oct 26, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

🔴 Test Coverage Grade: `D` — Marginal

Metric	Value
Line coverage	25.83%
Branch coverage	20.12%

Grade Scale

Grade	Line Coverage	Meaning
🟢 A	≥ 80%	Excellent - this code sleeps well at night 😴
🟡 B	60-79%	Good - almost there, don't stop now 😉
🟠 C	40-59%	Acceptable - your code is wearing a seatbelt, but no airbags 😬
🔴 D	20-39%	Marginal - boldly shipping where no test has gone before 🖖
⛔ F	< 20%	Failing - tests? what tests? 🔥

Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

sandeeplocharla reviewed

Nov 4, 2026

View reviewed changes

Comment thread

...age/volume/ontap/src/main/java/org/apache/cloudstack/storage/service/UnifiedSANStrategy.java Outdated

		}
		}
		}
		throw new CloudRuntimeException("Cloned Lun uuid was not visible for: " + lunUuid);

sandeeplocharla Nov 4, 2026

Copy link

Copy Markdown

Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If lun uuid itself is not visible, what's the point of logging lunUuid? 😆

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

sp16743 Nov 4, 2026

Copy link

Copy Markdown

Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there is no point. I have changed it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions


          Change the cloned LUN read failure message

a0586de

sandeeplocharla approved these changes

Nov 7, 2026

View reviewed changes

sandeeplocharla left a comment

Copy link

Copy Markdown

Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

suryag1201 reviewed

Nov 12, 2026

View reviewed changes

Comment thread

...age/volume/ontap/src/main/java/org/apache/cloudstack/storage/service/UnifiedSANStrategy.java Outdated

		String authHeader = OntapStorageUtils.generateAuthHeader(storage.getUsername(), storage.getPassword());
		OntapResponse<Lun> clonedLun = sanFeignClient.createLun(authHeader, true, lunRequest);
		if (clonedLun == null \|\| CollectionUtils.isEmpty(clonedLun.getRecords())) {
		OntapResponse<Lun> created = sanFeignClient.createLun(authHeader, true, lunRequest);

suryag1201 Oct 29, 2026

Copy link

Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change the variable name to clonedLun

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

sp16743 Nov 12, 2026

Copy link

Copy Markdown

Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

...age/volume/ontap/src/main/java/org/apache/cloudstack/storage/service/UnifiedSANStrategy.java Outdated

		OntapResponse<Lun> clonedLun = sanFeignClient.createLun(authHeader, true, lunRequest);
		if (clonedLun == null \|\| CollectionUtils.isEmpty(clonedLun.getRecords())) {
		OntapResponse<Lun> created = sanFeignClient.createLun(authHeader, true, lunRequest);
		if (created == null \|\| created.getRecords() == null \|\| created.getRecords().isEmpty()

suryag1201 Oct 29, 2026

Copy link

Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use collectionUtils/Stringutils for null or empty check

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

sp16743 Nov 12, 2026

Copy link

Copy Markdown

Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed it now

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

...age/volume/ontap/src/main/java/org/apache/cloudstack/storage/service/UnifiedSANStrategy.java

		}
		Lun lun = clonedLun.getRecords().get(0);
		String lunUuid = created.getRecords().get(0).getUuid();
		Lun lun = getLunByUuidUntilVisible(authHeader, lunUuid, lunRequest.getName());

suryag1201 Nov 12, 2026

Copy link

Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if the get lun call failed, cloned lun will. still exists and retry will see the duplicate lun

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

sp16743 Nov 12, 2026

Copy link

Copy Markdown

Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added it.The cloned LUN is deleted ,so a subsequent CloudStack retry does not encounter a duplicate LUN name.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

...age/volume/ontap/src/main/java/org/apache/cloudstack/storage/service/UnifiedSANStrategy.java Outdated

		for (int attempt = 1; attempt <= OntapStorageConstants.ONTAP_LUN_CLONE_GET_MAX_RETRIES; attempt++) {
		try {
		Lun lun = sanFeignClient.getLunByUUID(authHeader, lunUuid);
		if (lun != null) {

suryag1201 Nov 12, 2026

Copy link

Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can not decode based on lun != null condition that lun is visible.
return the get response record instead

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

sp16743 Nov 12, 2026

Copy link

Copy Markdown

Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed it.The code now uses the collection GET with the LUN UUID, verifies that the response contains a record, and returns that record instead of relying on lun != null.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

...ge/volume/ontap/src/main/java/org/apache/cloudstack/storage/utils/OntapStorageConstants.java Outdated

		public static final int ONTAP_FLEXVOL_JOB_POLL_INTERVAL_MS = 2000;
		public static final int ONTAP_FLEXVOL_RESOLVE_MAX_RETRIES = 30;
		/** GET-by-UUID retries after LUN clone response. */
		public static final int ONTAP_LUN_CLONE_GET_MAX_RETRIES = 5;

suryag1201 Nov 12, 2026

Copy link

Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this interval and no of retries are less for clone

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

sp16743 Nov 12, 2026

Copy link

Copy Markdown

Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the retries to 10

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

sp16743 added 2 commits

November 12, 2026 18:45


          Addressing Comments

99194b7


          Restore unused LUN endpoint

e68f745

Reviewers

suryag1201

suryag1201 requested changes

sandeeplocharla

sandeeplocharla approved these changes

rajiv-jain-netapp

Awaiting requested review from rajiv-jain-netapp

rajiv-jain-netapp is a code owner

piyush5netapp

Awaiting requested review from piyush5netapp

piyush5netapp is a code owner

Requested changes must be addressed to merge this pull request.

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

Conversation

sp16743 commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

github-actions Bot commented Oct 9, 2026

🔴 Test Coverage Grade: D — Marginal

Grade Scale

Uh oh!

github-actions Bot commented Oct 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔴 Test Coverage Grade: D — Marginal

Grade Scale

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sandeeplocharla left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sp16743 commented Oct 9, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

🔴 Test Coverage Grade: `D` — Marginal

github-actions Bot commented Oct 26, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

🔴 Test Coverage Grade: `D` — Marginal