You may refer to #7572 and #9625

i'm not sure, what you are refering to. esx 8U3 is supported from version 4.20.1

i'm not sure, what you are refering to. esx 8U3 is supported from version 4.20.1

if you go through the issue #7572, you might know that we face a similar issue with cloudstack systemvm template since VMware 8.0.1. The root cause is still unknown but we have addressed the issue by #9625. To be clear

• add fsck.mode=force fsck.repair=yes to boot args
• add depmod -a in init.sh

The same issue might impact user templates too. This is why I asked you to have a look.

I assumed if these issues are closed and compatibility matrix is ok with ESX 8.0.3, it will be ok. Anyway, all other templates are ok with 8.0.3, except Ubuntu 20.04. I added fsck.mode, fsck.repair and depmod -a. I'm now little step closer, but it is still not working:

I assumed if these issues are closed and compatibility matrix is ok with ESX 8.0.3, it will be ok. Anyway, all other templates are ok with 8.0.3, except Ubuntu 20.04. I added fsck.mode, fsck.repair and depmod -a. I'm now little step closer, but it is still not working:

You may need to run "depmod -a" when create the template.
CloudStack uses packer, it is one of the last steps in the process, see #11128

I did, of course. Now I tried to change the way i ran it, and i ran it at the end, and it looks promising now. I will let you tomorrow.

Well, no luck. i run depmod -a, set fsck in the grub. Fsck is running during booting process, fix a lot of things, but VM is obviously still corrupted:

@rastopato We fixed similar file system errors with the below options mentioned by @weizhouapache already, in this PR: #9625. Maybe, it's hitting different issue now. Can you try to create a VM in vCenter/ESXi directly using this template and check.

• add fsck.mode=force fsck.repair=yes to boot args
• add depmod -a in init.sh

Hi guys, i've done a little bit more troubleshooting based on your comment and I found out some issues. When i prepared Template VM i export it from VMware using vmware ovftool.

When I import this ova to the target vmware environment, it is working with no problems.

When I import this ova to the ACS as template it is saved on NFS storage in separate directory with some files in it. There is whole .ova file with different name (3ff4f989-6751-3607-9fd1-63c1429da88d.ova) and most probably the same .ova but unpacked to vmdk, ovf, nvram and mf file.

When I try deploy OVF template directly on VMware from these unpacked 4 files, i got error: "Failed to deploy OVF package. Cause: A specified parameter was not correct: The checksum(s) from the provided manifest file do not match the content of file(s): ubuntu2004.ovf."

When I copy whole .ova file from NFS storage (for example 3ff4f989-6751-3607-9fd1-63c1429da88d.ova), unpack it manually and Deploy it as OVF directly on vmware, it is working

Then i tried to delete 4 files from directory on NFS and unpack there that .ova file, delete all templates from vCenter and also from primary storage. Then i tried deploy VM again from ACS, but still no success - corrupted filesystem.

I just realize that all Templates which were working on previous versions are now broken. I was able to repair some of them by simply regenerate and import new template (Ubuntu 22.04, 24.04, RHEL9.x), some of them are broken with corrupted filesystems also after regenerating it (Ubuntu 20.04, Rocky Linux 9.x, CentOS Stream 9)

It is definitely corrupting filesystem when VMware Templates are creating. During the process of creating instance, multiple VMware templates are created by deploying .ovf template from NFS storage to Primary NTFS storage. This templates are already corrupted, when i try to deploy Vm from it directly in VMware.

@rastopato, I have tested this and was not able to reproduce the issue.

Environment

Templates Tested

(localcloud) 🐱 > list templates templatefilter=all filter=name,deployasis,hypervisor,format,isready
{
  "count": 5,
  "template": [
    {
      "deployasis": false,
      "format": "OVA",
      "hypervisor": "VMware",
      "isready": true,
      "name": "Ubuntu 20.04"
    },
    {
      "deployasis": true,
      "format": "OVA",
      "hypervisor": "VMware",
      "isready": true,
      "name": "Ubuntu 20.04 deploy-as-is"
    },
    {
      "deployasis": true,
      "format": "OVA",
      "hypervisor": "VMware",
      "isready": true,
      "name": "Ubuntu 20.04 from lab deploy as is"
    },
    {
      "deployasis": false,
      "format": "OVA",
      "hypervisor": "VMware",
      "isready": true,
      "name": "CentOS 5.3(64-bit) no GUI (vSphere)"
    },
    {
      "deployasis": false,
      "format": "OVA",
      "hypervisor": "VMware",
      "isready": true,
      "name": "SystemVM Template (vSphere)"
    }
  ]
}

Storage

(localcloud) 🐱 >  list storagepools
{
  "count": 2,
  "storagepool": [
    {
      "clusterid": "42a694eb-1ed5-479e-bfca-04389400e770",
      "clustername": "p1-c1",
      "created": "2026-02-05T10:35:55+0000",
      "details": {},
      "disksizeallocated": 70852298752,
      "disksizetotal": 160792838144,
      "disksizeused": 15406727168,
      "hasannotations": false,
      "hypervisor": "VMware",
      "id": "d173e0a1-40db-3053-8caf-f20f437151cf",
      "ipaddress": "VMFS datastore: /Trillian/ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri1",
      "istagarule": false,
      "managed": false,
      "name": "ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri1",
      "overprovisionfactor": "2.0",
      "path": "/Trillian/ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri1",
      "podid": "8ed4917e-5f74-44f6-b395-891580cf7c47",
      "podname": "Pod1",
      "provider": "DefaultPrimary",
      "scope": "CLUSTER",
      "state": "Up",
      "storagecapabilities": {
        "VOLUME_SNAPSHOT_QUIESCEVM": "false"
      },
      "type": "VMFS",
      "zoneid": "ece6ad13-731c-4dc9-a052-102c801e87e9",
      "zonename": "ref-trl-10863-v-Mol8-rositsa-kyuchukova"
    },
    {
      "clusterid": "42a694eb-1ed5-479e-bfca-04389400e770",
      "clustername": "p1-c1",
      "created": "2026-02-05T10:35:57+0000",
      "details": {},
      "disksizeallocated": 31960596480,
      "disksizetotal": 160792838144,
      "disksizeused": 6465519616,
      "hasannotations": false,
      "hypervisor": "VMware",
      "id": "a72b7e2f-47aa-3c24-9d23-2d2468ab16f0",
      "ipaddress": "VMFS datastore: /Trillian/ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri2",
      "istagarule": false,
      "managed": false,
      "name": "ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri2",
      "overprovisionfactor": "2.0",
      "path": "/Trillian/ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri2",
      "podid": "8ed4917e-5f74-44f6-b395-891580cf7c47",
      "podname": "Pod1",
      "provider": "DefaultPrimary",
      "scope": "CLUSTER",
      "state": "Up",
      "storagecapabilities": {
        "VOLUME_SNAPSHOT_QUIESCEVM": "false"
      },
      "type": "VMFS",
      "zoneid": "ece6ad13-731c-4dc9-a052-102c801e87e9",
      "zonename": "ref-trl-10863-v-Mol8-rositsa-kyuchukova"
    }
  ]
}

VMs Deployed

list virtualmachines filter=name,state,templatename

(localcloud) 🐱 > list virtualmachines filter=name,state,templatename
{
  "count": 3,
  "virtualmachine": [
    {
      "name": "VM-07ef5514-382c-4014-b1b9-6041f60ea63a",
      "state": "Running",
      "templatename": "Ubuntu 20.04"
    },
    {
      "name": "test-deploy-as-is",
      "state": "Running",
      "templatename": "Ubuntu 20.04 deploy-as-is"
    },
    {
      "name": "test-lab-dai",
      "state": "Running",
      "templatename": "Ubuntu 20.04 from lab deploy as is"
    }
  ]
}

All three VMs boot successfully and keyboard input works at console login prompt.

Checksum Verification on Secondary Storage

Verified that extracted OVA files on secondary storage match the manifest:

cat ubuntu-focal-20.04-cloudimg.mf
SHA256(ubuntu-focal-20.04-cloudimg.vmdk)= 192b5c869b99a0db91f6567fe29f89d9b98984a3b16220ef06ab7320de779411
SHA256(ubuntu-focal-20.04-cloudimg.ovf)= 603242f28820af7fe2afb621990ce130b6c6a1cb10ed481b9254cadb83fd4547

sha256sum ubuntu-focal-20.04-cloudimg.ovf ubuntu-focal-20.04-cloudimg.vmdk
603242f28820af7fe2afb621990ce130b6c6a1cb10ed481b9254cadb83fd4547  ubuntu-focal-20.04-cloudimg.ovf ✓
192b5c869b99a0db91f6567fe29f89d9b98984a3b16220ef06ab7320de779411  ubuntu-focal-20.04-cloudimg.vmdk ✓

Checksums match — no corruption detected during OVA extraction.

The OVA extraction and VM deployment work correctly.

To help us investigate further, could you please provide:

1. Your problematic Ubuntu 20.04 OVA template (or a download link) so we can test with the exact same template
2. Template registration settings - was "Read Instance settings from OVA" enabled?

If you can share the OVA that consistently fails, we can compare it against templates that work and try identifying what triggers the corruption.

I have run into the same problem running 4.22.1 on esxi 8.0.3 with Ubuntu 24.4. Have you guys been able to figure out the issue ?

Hey,

Based on the details provided, I would not treat this as an Ubuntu 20.04/ext4 issue alone.

The most useful finding here is the comparison between the two VMware clusters: the same CloudStack template works with ESXi 8.0.0 but becomes corrupted with ESXi 8.0.3. You also mentioned that the problem can be reproduced with other templates, which points more toward the VMware/CloudStack template deployment path than a specific Ubuntu template.

I would suggest one simple isolation test before rebuilding the template again:

• Take the same template/image and deploy it directly through vCenter/ESXi on the affected 8.0.3 host.
• If it is already corrupted there, the issue is occurring during the VMware template/clone/storage operation.
• If it works correctly when deployed directly through VMware but fails only through CloudStack, then the CloudStack VMware deployment path should be investigated.

The fsck.mode=force fsck.repair=yes and depmod -a workaround is worth trying because a similar VMware-related filesystem issue was previously addressed this way in CloudStack. However, it does not appear to solve this particular case, since you already tested both and the filesystem was still corrupted.

I would therefore collect the following from a failed deployment:

1. CloudStack management-server log covering the deployment.
2. ESXi vmkernel.log, hostd.log and the VM vmware.log.
3. The exact ESXi 8.0.3 build number.
4. Datastore/VMFS version and storage configuration.
5. VMDK/controller type and VM hardware version.
6. The exact EXT4 errors from the first boot.

If the VMDK is already corrupted when the intermediate VMware template is created, as reported here, that is the most important clue and should be included in a CloudStack bug report.

For production, I would avoid deploying this template on the affected ESXi 8.0.3 hosts until the storage/template conversion path is isolated. If ESXi 8.0.0 is still available and stable, using it as a temporary workaround is safer than repeatedly repairing newly deployed filesystems.

Regards,

Ryan P.
Cloud & DevOps Engineer
AccuWeb.Cloud

@rastopato
could you test #13977 ?

@rastopato rastopato check the PR in 1h = there should be packages available
#13977 (comment)