GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Ubuntu 20.04 Template Problem · Issue #12036 · apache/cloudstack

Ubuntu 20.04 Template Problem #12036

New issue
Copy link
New issue
Copy link
Closed
Bug
Closed
Ubuntu 20.04 Template Problem#12036
Bug
Copy link
Assignees
RosiKyu
Labels
Severity:CriticalCritical bugCritical bugstatus:needs-investigation
Milestone
4.20.3

Description

@rastopato
rastopato
opened on Nov 9, 2025
Issue body actions

problem

Deployment of VMs from Ubuntu 20.04.06 LTS stopped working after upgrading ACS to 4.21 and VMware to 8.0.3. Every VM deployed from this template is stuck at boot with corrupted ext4 filesystem errors.

I tried every possible combination of preparing new ubuntu template, disk adapters, bios/uefi, etc, etc, nothing worked.
Ubuntu 22.04 and 24.04 has no problem.

I tried it on environment where are previously deployed Vms from Ubuntu 20.04 template without issues. Now, after upgrade, it is the same error.
I got this error in all my production and testing environments.

Finally i tried it on environment on ACS 4.20.1 and VMware 8.0.0 and there is no problem with Ubuntu 20.04.

Image

management-server.log.zip

versions

ACS: 4.21
VMware vCenter 8.0.3
Vmware ESXi 8.0.3
Ubuntu 20.04.06 LTS
Primary storage is VMFS datastores on SAN storage

The steps to reproduce the bug

  1. Create Template with Ubuntu 20.04.06 LTS
  2. Deploy new VM from it
  3. It should be stack at the boot with filesystem corruption error.
  4. If you not prepare any /etc/cloud/cloud.cfg.d/ files, it maybe start, but you cant even write username/password because of corrupted filesytem

What to do about it?

Please resolve it, or suggest workaround.

Activity

  1. weizhouapache commented on Nov 9, 2025

    @weizhouapache
    weizhouapache
    on Nov 9, 2025
    Member
    More actions

    You may refer to #7572 and #9625

  2. rastopato commented on Nov 9, 2025

    @rastopato
    rastopato
    on Nov 9, 2025
    Author
    More actions

    i'm not sure, what you are refering to. esx 8U3 is supported from version 4.20.1

  3. weizhouapache commented on Nov 9, 2025

    @weizhouapache
    weizhouapache
    on Nov 9, 2025
    Member
    More actions

    i'm not sure, what you are refering to. esx 8U3 is supported from version 4.20.1

    if you go through the issue #7572, you might know that we face a similar issue with cloudstack systemvm template since VMware 8.0.1. The root cause is still unknown but we have addressed the issue by #9625. To be clear

    • add fsck.mode=force fsck.repair=yes to boot args
    • add depmod -a in init.sh

    The same issue might impact user templates too. This is why I asked you to have a look.

  4. rastopato commented on Nov 9, 2025

    @rastopato
    rastopato
    on Nov 9, 2025
    Author
    More actions

    I assumed if these issues are closed and compatibility matrix is ok with ESX 8.0.3, it will be ok. Anyway, all other templates are ok with 8.0.3, except Ubuntu 20.04. I added fsck.mode, fsck.repair and depmod -a. I'm now little step closer, but it is still not working:

    Image
  5. weizhouapache commented on Nov 9, 2025

    @weizhouapache
    weizhouapache
    on Nov 9, 2025
    Member
    More actions

    I assumed if these issues are closed and compatibility matrix is ok with ESX 8.0.3, it will be ok. Anyway, all other templates are ok with 8.0.3, except Ubuntu 20.04. I added fsck.mode, fsck.repair and depmod -a. I'm now little step closer, but it is still not working:

    Image

    You may need to run "depmod -a" when create the template.
    CloudStack uses packer, it is one of the last steps in the process, see #11128

  6. rastopato commented on Nov 9, 2025

    @rastopato
    rastopato
    on Nov 9, 2025
    Author
    More actions

    I did, of course. Now I tried to change the way i ran it, and i ran it at the end, and it looks promising now. I will let you tomorrow.

  7. rastopato commented on Nov 10, 2025

    @rastopato
    rastopato
    on Nov 10, 2025
    Author
    More actions

    Well, no luck. i run depmod -a, set fsck in the grub. Fsck is running during booting process, fix a lot of things, but VM is obviously still corrupted:

    Image
  8. sureshanaparti commented on Nov 12, 2025

    @sureshanaparti
    sureshanaparti
    on Nov 12, 2025
    Contributor
    More actions

    @rastopato We fixed similar file system errors with the below options mentioned by @weizhouapache already, in this PR: #9625. Maybe, it's hitting different issue now. Can you try to create a VM in vCenter/ESXi directly using this template and check.

    • add fsck.mode=force fsck.repair=yes to boot args
    • add depmod -a in init.sh
  9. rastopato commented on Nov 12, 2025

    @rastopato
    rastopato
    on Nov 12, 2025
    Author
    More actions

    Hi guys, i've done a little bit more troubleshooting based on your comment and I found out some issues. When i prepared Template VM i export it from VMware using vmware ovftool.

    When I import this ova to the target vmware environment, it is working with no problems.

    When I import this ova to the ACS as template it is saved on NFS storage in separate directory with some files in it. There is whole .ova file with different name (3ff4f989-6751-3607-9fd1-63c1429da88d.ova) and most probably the same .ova but unpacked to vmdk, ovf, nvram and mf file.

    When I try deploy OVF template directly on VMware from these unpacked 4 files, i got error: "Failed to deploy OVF package. Cause: A specified parameter was not correct: The checksum(s) from the provided manifest file do not match the content of file(s): ubuntu2004.ovf."

    When I copy whole .ova file from NFS storage (for example 3ff4f989-6751-3607-9fd1-63c1429da88d.ova), unpack it manually and Deploy it as OVF directly on vmware, it is working

    Then i tried to delete 4 files from directory on NFS and unpack there that .ova file, delete all templates from vCenter and also from primary storage. Then i tried deploy VM again from ACS, but still no success - corrupted filesystem.

    I just realize that all Templates which were working on previous versions are now broken. I was able to repair some of them by simply regenerate and import new template (Ubuntu 22.04, 24.04, RHEL9.x), some of them are broken with corrupted filesystems also after regenerating it (Ubuntu 20.04, Rocky Linux 9.x, CentOS Stream 9)

  10. rastopato commented on Nov 13, 2025

    @rastopato
    rastopato
    on Nov 13, 2025
    Author
    More actions

    It is definitely corrupting filesystem when VMware Templates are creating. During the process of creating instance, multiple VMware templates are created by deploying .ovf template from NFS storage to Primary NTFS storage. This templates are already corrupted, when i try to deploy Vm from it directly in VMware.

  11. DaanHoogland
    added
    status:needs-investigation
    on Nov 13, 2025
  12. DaanHoogland
    added this to the 4.20.3 milestone on Nov 13, 2025
  13. DaanHoogland
    added
    Severity:CriticalCritical bugCritical bug
    on Nov 13, 2025
  14. DaanHoogland
    added the
    Bug
    issue type on Jan 5, 2026
  15. DaanHoogland
    added this to Apache CloudStack BugFest - Issueson Jan 5, 2026
  16. DaanHoogland
    moved this to Todo in Apache CloudStack BugFest - Issueson Jan 6, 2026
  17. abh1sar
    moved this from Todo to Dev In Progress in Apache CloudStack BugFest - Issueson Jan 14, 2026
  18. abh1sar
    self-assigned this
    on Jan 14, 2026
  19. abh1sar
    removed their assignment
    on Jan 28, 2026
  20. abh1sar
    moved this from Dev In Progress to Todo in Apache CloudStack BugFest - Issueson Jan 28, 2026
  21. RosiKyu
    self-assigned this
    on Feb 4, 2026
  22. RosiKyu commented on Feb 9, 2026

    @RosiKyu
    RosiKyu
    on Feb 9, 2026
    Collaborator
    More actions

    @rastopato, I have tested this and was not able to reproduce the issue.

    Environment

    Component Version
    CloudStack 4.21.0.0
    VMware ESXi 8.0.3.0
    Primary Storage VMFS
    Management Server Oracle Linux 8

    Templates Tested

    Template Name Deploy-as-is Status
    Ubuntu 20.04 false Ready
    Ubuntu 20.04 deploy-as-is true Ready
    Ubuntu 20.04 from lab deploy as is true Ready
    (localcloud) 🐱 > list templates templatefilter=all filter=name,deployasis,hypervisor,format,isready
    {
      "count": 5,
      "template": [
        {
          "deployasis": false,
          "format": "OVA",
          "hypervisor": "VMware",
          "isready": true,
          "name": "Ubuntu 20.04"
        },
        {
          "deployasis": true,
          "format": "OVA",
          "hypervisor": "VMware",
          "isready": true,
          "name": "Ubuntu 20.04 deploy-as-is"
        },
        {
          "deployasis": true,
          "format": "OVA",
          "hypervisor": "VMware",
          "isready": true,
          "name": "Ubuntu 20.04 from lab deploy as is"
        },
        {
          "deployasis": false,
          "format": "OVA",
          "hypervisor": "VMware",
          "isready": true,
          "name": "CentOS 5.3(64-bit) no GUI (vSphere)"
        },
        {
          "deployasis": false,
          "format": "OVA",
          "hypervisor": "VMware",
          "isready": true,
          "name": "SystemVM Template (vSphere)"
        }
      ]
    }
    
    

    Storage

    (localcloud) 🐱 >  list storagepools
    {
      "count": 2,
      "storagepool": [
        {
          "clusterid": "42a694eb-1ed5-479e-bfca-04389400e770",
          "clustername": "p1-c1",
          "created": "2026-02-05T10:35:55+0000",
          "details": {},
          "disksizeallocated": 70852298752,
          "disksizetotal": 160792838144,
          "disksizeused": 15406727168,
          "hasannotations": false,
          "hypervisor": "VMware",
          "id": "d173e0a1-40db-3053-8caf-f20f437151cf",
          "ipaddress": "VMFS datastore: /Trillian/ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri1",
          "istagarule": false,
          "managed": false,
          "name": "ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri1",
          "overprovisionfactor": "2.0",
          "path": "/Trillian/ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri1",
          "podid": "8ed4917e-5f74-44f6-b395-891580cf7c47",
          "podname": "Pod1",
          "provider": "DefaultPrimary",
          "scope": "CLUSTER",
          "state": "Up",
          "storagecapabilities": {
            "VOLUME_SNAPSHOT_QUIESCEVM": "false"
          },
          "type": "VMFS",
          "zoneid": "ece6ad13-731c-4dc9-a052-102c801e87e9",
          "zonename": "ref-trl-10863-v-Mol8-rositsa-kyuchukova"
        },
        {
          "clusterid": "42a694eb-1ed5-479e-bfca-04389400e770",
          "clustername": "p1-c1",
          "created": "2026-02-05T10:35:57+0000",
          "details": {},
          "disksizeallocated": 31960596480,
          "disksizetotal": 160792838144,
          "disksizeused": 6465519616,
          "hasannotations": false,
          "hypervisor": "VMware",
          "id": "a72b7e2f-47aa-3c24-9d23-2d2468ab16f0",
          "ipaddress": "VMFS datastore: /Trillian/ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri2",
          "istagarule": false,
          "managed": false,
          "name": "ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri2",
          "overprovisionfactor": "2.0",
          "path": "/Trillian/ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri2",
          "podid": "8ed4917e-5f74-44f6-b395-891580cf7c47",
          "podname": "Pod1",
          "provider": "DefaultPrimary",
          "scope": "CLUSTER",
          "state": "Up",
          "storagecapabilities": {
            "VOLUME_SNAPSHOT_QUIESCEVM": "false"
          },
          "type": "VMFS",
          "zoneid": "ece6ad13-731c-4dc9-a052-102c801e87e9",
          "zonename": "ref-trl-10863-v-Mol8-rositsa-kyuchukova"
        }
      ]
    }
    

    VMs Deployed

    VM Name State Template
    VM-07ef5514... Running Ubuntu 20.04 (deployasis=false)
    test-deploy-as-is Running Ubuntu 20.04 deploy-as-is (deployasis=true)
    test-lab-dai Running Ubuntu 20.04 from lab (deployasis=true)
    list virtualmachines filter=name,state,templatename
    
    (localcloud) 🐱 > list virtualmachines filter=name,state,templatename
    {
      "count": 3,
      "virtualmachine": [
        {
          "name": "VM-07ef5514-382c-4014-b1b9-6041f60ea63a",
          "state": "Running",
          "templatename": "Ubuntu 20.04"
        },
        {
          "name": "test-deploy-as-is",
          "state": "Running",
          "templatename": "Ubuntu 20.04 deploy-as-is"
        },
        {
          "name": "test-lab-dai",
          "state": "Running",
          "templatename": "Ubuntu 20.04 from lab deploy as is"
        }
      ]
    }
    
    

    All three VMs boot successfully and keyboard input works at console login prompt.

    Image

    Checksum Verification on Secondary Storage

    Verified that extracted OVA files on secondary storage match the manifest:

    cat ubuntu-focal-20.04-cloudimg.mf
    SHA256(ubuntu-focal-20.04-cloudimg.vmdk)= 192b5c869b99a0db91f6567fe29f89d9b98984a3b16220ef06ab7320de779411
    SHA256(ubuntu-focal-20.04-cloudimg.ovf)= 603242f28820af7fe2afb621990ce130b6c6a1cb10ed481b9254cadb83fd4547
    
    sha256sum ubuntu-focal-20.04-cloudimg.ovf ubuntu-focal-20.04-cloudimg.vmdk
    603242f28820af7fe2afb621990ce130b6c6a1cb10ed481b9254cadb83fd4547  ubuntu-focal-20.04-cloudimg.ovf ✓
    192b5c869b99a0db91f6567fe29f89d9b98984a3b16220ef06ab7320de779411  ubuntu-focal-20.04-cloudimg.vmdk ✓
    

    Checksums match — no corruption detected during OVA extraction.

    Image

    The OVA extraction and VM deployment work correctly.

    To help us investigate further, could you please provide:

    1. Your problematic Ubuntu 20.04 OVA template (or a download link) so we can test with the exact same template
    2. Template registration settings - was "Read Instance settings from OVA" enabled?

    If you can share the OVA that consistently fails, we can compare it against templates that work and try identifying what triggers the corruption.

  23. apache
    locked and limited conversation to collaborators on Feb 9, 2026
  24. DaanHoogland
    converted this issue into a discussion #12610 on Feb 9, 2026
  25. github-project-automation
    moved this from Todo to Done in Apache CloudStack BugFest - Issueson Feb 9, 2026
  26. DaanHoogland
    removed this from Apache CloudStack BugFest - Issueson Mar 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

  • @RosiKyu
    RosiKyu

Labels

Severity:CriticalCritical bugCritical bugstatus:needs-investigation

Type

Bug

Projects

No projects

    Milestone

    • 4.20.3

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    You are visitor 01402520 — GeezerVison does not store the pages you look up.
    Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.