Ready, Set, Virtual!

Blog

  • Getting network on Linux servers operational after migrating hypervisors

    Getting network on Linux servers operational after migrating hypervisors

    Recently I made the move to migrate all my VMs from running on VMware ESXi to running on Hyper-V. This was the first time that I have shifted to another platform away from ESXi as my main hypervisor platform.

    As it was a single server, I made a backup of all my VMs with Veeam, taking also a copy of my config file and built a new Veeam Backup and Replication server on Hyper-V with importing the config file, allowing to restore all VMs across to Hyper-V.

    Whilst all VMs booted up, the one issue I did run into was the ethernet adapter had changed from a VMXnet3 to a Hyper-V Virtual Network Adapter.

    For Windows VMs, this was no issue and they all connected as if nothing had changed.

    For Linux servers (Ubuntu in my case), the adapter name did change and the the Netplan .yaml file did not update. Thus, there was no connectivity to the outside world.

    While the adapter in Hyper-V manager looked to have a connection, it was unable to display the IP Address as it was not associated with it. (This would be relevant to any hypervisor migration where the adapters aren’t exactly the same).

    There is a very simple fix, as I mentioned above,  the. netplan yaml file does not automatically update to reflect a new adapter name as there is no detection that runs like it does when installing the OS.

    To find your new adapter, you can run: ip link show – You will be able to match the mac address on the adapter to the results in the ip link output.

    Once you have your new adapters name, you can then update the .yaml file. Using the below commands, you will first search and confirm the name of the .yaml file and then edit with your preferred editor (in my case I will use vi). You can use tab completion to fill in the file name when you have the netplan folder specified

    ls /etc/netplan/
    sudo vi /etc/netplan/00-installer-config.yaml

    Inside the .yaml file, you will need to find the line underneath “ethernets” – this will be your old adapter name and using the correct key combination, enter the insert (i key) function.

    Using my screenshots, update the “ens34” to “eth0”

    Press esc to escape the insert mode, then use :wq to write and quit vi.

    Lastly, make sure to apply the netplan configuration. This is easily achieved by sudo netplan apply – Once applied, run through a ping test and make sure your network is functioning correctly.

    My understanding, and experience so far, is that with Hyper-V the adapter name is going to be the same followed by ascending numbers (eth0, eth1, eth2, etc.) – However, it is always best to confirm first.

    Hopefully that will save you some time when migrating between various hypervisors where the adapter changes.

  • Veeam 365 – Missing Application Permissions

    Veeam 365 – Missing Application Permissions

    With technology ever-changing, and businesses continuously facing security challenges, services such as Azure and AWS need to keep on adapting and improving their security posture to ensure that they continue to protect their customers and services. While this is certainly the correct thing to do, it does sometime create issues for the end users or other applications that are making use of those services.

    If you are a Veeam Backup for M365 customer or user, you may come across the below warning messages when running a backup job. This is because Microsoft have made some additional security changes with different graph API access permissions. This isn’t really all that bad, minus the bits that have been skipped during that backup window, but it is extremely easy to resolve.

    12/02/2025 9:07:25 PM :: Missing application permissions: Exchange.ManageAsApp. Missing application roles: Global Reader. Public folder and discovery search mailboxes will be skipped from processing, and a shared mailbox type will not be identified.

    12/02/2025 9:07:25 PM :: Missing application permissions: ChannelMember.Read.All. Private and shared team channels will be skipped from processing

     

    To resolve this, all you need to do is run through editing your organization once again, and allowing to. update the application registration.

    You do need to make sure you still have the certificate in the cert store for which you used to create the application or recently updated with

    1. Right click on the organization and select Edit Organization


    2. Click Next until you get to the “Microsoft 365 Connection Settings” – Here you can select Use an existing Azure AD Application


    3. Your Username and Application ID will appear in the boxes. Here you will need to then select Install and select your certificate that you used to connect to the application when you either created or last updated.

      Ensure to check the  Grant this application required permissions and register its certificate in Azure AD box. Click Next

    4.  You will then be request to copy the code and go to https://microsoft.com/devicelogin and log in with a GA account to authorize the change

    5. Once Authorized you will then see the progress of the update.

    6. Once all updated click Finish. You can then go ahead and start the job and wait for it to complete.

    7. Once the job has completed, confirm that all was successful and no further warnings.

    This should be resolve any issues Veeam Backup for Microsoft had for backing up using the Graph API.

     

     

  • How to automate certificate update in Veeam Backup for M365

    How to automate certificate update in Veeam Backup for M365

    One of the most tedious operational tasks you can do in IT is manage domains and certificates – these chores can become quite tiresome when replacing many over a year and then starting again as most certificates only have a 1 year (13 month) life cycle. 

    But then… you have certificate providers like Let’s Encrypt and ZeroSSL who provide 3 month free certificates (some with additional costs for repeated use) that with the help with additional toolsets can become automated certificate renewals. 

    in this article I want to cover how to set up and use Let’s Encrypt with Certify the Web to automate the renewal of the certificates for Veeam Backup for M365. 

    As Let’s encrypt needs to use either DNS or HTML to check the legitimacy of the domain to be ale to provide a valid certificate, it is helpful to use an application like CertifyTheWeb to automate the process to update the DNS record when required. CertifyTheWeb has the ability to update a number of DNS providers using the APIs provided along with the API token. 
    Use the attached link here for Setting up CloudFlare API Token for your required DNS zone – this will then allow CertifyTheWeb to make those changes on the fly. 

    To set up CertifyTheWeb to just generate certificate: 

    1. Download and install CertifyTheWeb’s application (Best on the server you require the certificates on) 
    2. Create a new Certificate
    3. Set the Challenge type to DNS and select your DNS Update Method /Manage

    4. Add your credentials/API Token for the DNS manager and put in your DNS Zone ID
    5. Run a Test and confirm that the challenge is working and that the certificate is ready to be deployed. 

    While certify the web will handle applying the certificate to a number of web host systems, for applications like VB365, you need to utilise the APIs/PowerShell commands to install the certificates. 

    This will be applied to the Tasks section of CertifyTheWeb where you can run post-scripts to your recently registered certificates. 

     

    When a certificate is downloaded, it is added to the default folder of C:\ProgramData\certify\assets\<_.domainName>\  – Here you will find the .pfx files. 

     

    The below script will install the certificate to the below parts of Veeam Backup for M365. It will also enable the features.

    • RestAPI
    • Tenant and Operator Authentication
    • VBO Restore Portal

    In a nutshell the script will perform the below actions: 

    1. Import required modules from Veeam Backup for M365
    2. Create archive folder for certificate if it doesn’t already exist
    3. Set Certificate path
    4. Enable RestAPI setting and install certificate 
    5. Enable Operator Authentication setting and install certificate
    6. Enable Tenant Authentication setting and install certificate
    7. Enable Restore Portal Setting and install Certificate while setting Region, AppID and PortalURI
    8. Move certificate to the archive folder

    Save the below script as a .ps1 and map it through the Tasks in CertifyTheWeb – this will then call during deployment. 

    # Import VB365 PowerShell Modules

Import-Module "C:\Program Files\Veeam\Backup365\Veeam.Archiver.PowerShell\Veeam.Archiver.PowerShell.psd1"
Import-Module "C:\Program Files\Veeam\Backup and Replication\Explorers\Exchange\Veeam.Exchange.PowerShell\Veeam.Exchange.PowerShell.psd1"
Import-Module "C:\Program Files\Veeam\Backup and Replication\Explorers\SharePoint\Veeam.SharePoint.PowerShell\Veeam.SharePoint.PowerShell.psd1"
Import-Module "C:\Program Files\Veeam\Backup and Replication\Explorers\Teams\Veeam.Teams.PowerShell\Veeam.Teams.PowerShell.psd1"

$Region = #Default is = Worldwide
$AppID = #AppID
$PortalURI = #RestorePortal URI

# Set the archive folder path

$archive = "C:\ProgramData\certify\archive\"
# Check if folder not exists, and create it
if (-not(Test-Path $archive -PathType Container)) {
    New-Item -path $archive -ItemType Directory
}

# Grab new certificate
# -Path must be set to the assets folder domain with the wilcard for the .pfx 

$Certificate = Get-ChildItem -Path "C:\ProgramData\certify\assets\_.readysetvirtual.com\*.pfx"

# Install RestAPI certificate and enable service

write-host -foreground Yellow "Setting VBO365 Certificate"
Set-VBORestAPISettings -EnableService -AuthTokenLifeTime 4800 -CertificateFilePath $Certificate

# Install Tenant and Operator Authentication Certificate

Set-VBOOperatorAuthenticationSettings -EnableAuthentication -CertificateFilePath $Certificate
Set-VBOTenantAuthenticationSettings -EnableAuthentication -CertificateFilePath $Certificate

# Install Restore Portal Certificate

Set-VBORestorePortalSettings -EnableService -ApplicationId $ApplicationId -CertificateFilePath $Certificate -Region Worldwide -PortalUri $PortalURI

sleep 5

# Move Certificate to Archive C:\ProgramData\certify\archive\

write-host -foreground yellow "Moving certificate to C:\certs\certify_archive\"
Move-Item -Path $Certificate -Destination C:\ProgramData\certify\archive\


write-host -foreground green "Certificate Sucessfully Applied"

    Just like that, the certificates are updated, and in the final few weeks CertifyTheWeb will run and generate a new set of certificates and apply as required when the existing are about to expire.

  • VMCE Notes: Explain Backup Data Platform Components – Part 1

    VMCE Notes: Explain Backup Data Platform Components – Part 1

    Backup Server:

    The backup server contains several components that the backup operator can interact with through the console.
    The console is just one of the components that can be installed on the Backup Server, this is where you can create, run and manage backup jobs, as well as configure backup infrastructure.

    The backup Server allows for coordination of backup, replication and restore jobs as well as running backup and SureBackup verification tasks.

    The backup server coordinates the resource scheduling for attaching disks to proxies and managing the streams to the repository, honouring the configuration for how mange tasks can be ran against certain infrastructure.

    When first deployed, the backup server is also preconfigured as a VMware Proxy as well as the default repository. The first repository is configured as C drive by default

    Backup and Replication Console:

    The backup and replication console can operate from any windows computer that has access to the Backup server. By default, the console is installed alongside the backup services on the backup server, however, a separate installation can be made.

    The console use Simplified and Protected GSS-API Negotiation Mechenism (SPNEGO) that is the Windows Built-in authentication mechanism.

    The console can only access the backup infrastructure to interact with it via credential login. When the console is first opened, a login prompt is displayed to create the connect to the server – You can also point the login to any of the servers you may have deployed – Allowing you to have a dedicated console server and backup server.

    The Console can be deployed to multiple machines or opened several times on the same machine and connecting to the same server as this can help with opening different windows.

    You CANNOT use the same console version to connect to different versions of Veeam BR. You need to use the matching version. If you are using 2 different installations of Veeam Server, you will need to run 2 different versions of the console to connect.

    If the console connects to a server that is updated, the console will check and then update accordingly. This is only supported on the GA version, and not Preview, Beta or RTM.

    ** Downgrade of the console is not possible.

    When installed the remote console, multiple components are installed alongside.

    • Veeam Backup PowerShell Module
    • Veeam Explorers for;
      • Microsoft Active Directory
      • Exchange
      • Onedrive for Business
      • Sharepoint
      • SQL Server
      • Teams
      • Oracle
      • PostgreSQL
      • SAP HANA
    • Data Mover Servive
      • Used to run data processing tasks for VBR
    • Recovery Service
      • Used to perform recovery tasks
    • Veeam Installer Service
      • Used to install agents, transport services and other components on remote servers (Repositories, Proxies, etc.)
    • Veeam mount Service.
      • Used to mount backups during the restore process.

     

    Things to keep in mind:

    If you are deploying the console remotely, you can deloy it behind a NAT, but the Backup Server MUST be outside of the NAT. You also cannot remotely install the console out of the NAT while the backup server is in behind it.

     

    Veeam Backup and Configuration Database

    There are two options for running the Veeam BR configuration database that holds data for the Backup Infrastructure, Jobs, sessions and other Config data.

    You can install Veeam with either Microsoft SQL or PostgreSQL database. Each can be install remotely on a dedicated SQL server, however by default it will be installed locally.  Veeam includes either Microsoft SQL Express or PostgreSQL installers as per of the installation process. Be aware that using Microsoft SQL Express will bring in limitations with how much data can be installed and how that data can be accessed.

    *Some other Veeam software still support Microsoft SQL only – Keep this in mind when designing you backup infrastructure.

    Once a week and when the VBR Service is restarted, Veeam will run a database clean up and maintenance tasks for the database internal statistics. The maintenance task will also defragment indexes and clear unused data.

    The maintenance task will log any changes and tasks performed in %ProgramData%\Veeam\Backup\Job.DatabaseMaintenance.log. file.

     

    Veeam Backup Powershell Module

    Powershell has become almost the default command line tool for interacting with Windows applications. Veeam Backup and Replication includes a PowerShell module that is extending the native Windows Powershell.

    The Powershell snap-in adds a new set of cmdlets that are specifically built to interact with the Veeam Backup Service. This allows the ability to create customer scripts to pull reports or to automate management of backups jobs and restores.

    There is a full Powershell reference guide available.

    Virtualization Servers and Hosts

    There are 4 different types of servers that can been added to the backup infrastructure of Veeam Backup and Replication. Each carry a different role in which they play within the entire infrastructure.

    • VMware vSphere Server
      • Can be a source host or a target host for backups and restores.
    • VMware Cloud Director
      • Source Host and Target (when using Cloud Director Replication and CDP).
    • Microsoft Windows Server
      • Can be used as a Backup Proxy or Backup Repository (Using NTFS or ReFS format)
    • Linux Server
      • Can be used as a Backup Proxy or Backup Repository (Using XFS Format)

    It is ideal to only have 1 instance of the server in the backup infrastructure – Add it via Hostname or IP only, not both.

    Physical servers can also been added into the infrastructure as well as cloud servers.

    Additional server types that can be added, however some require additional plug-ins. :

     

    VMware Backup Proxies

    The VMware Proxy works just like a proxy, it sits between the backup server and the backup infrastructure components, but it is also the connection point for attaching and processing host items (e.g. Using Hot-Add to backup data from a VMware VM Disk)

    The general tasks a VMware Backup Proxy:

    • Retrieving VM data from the Production Storage
    • Compressing
    • Deduplicating
    • Encrypting
    • Passing to another Repository like running a replication job. 

    Usage Scenarios 

    • Backup
    • Replication
    • Quick Migration

    VMware Backup Proxy Transport Modes

    There are a couple of different ways that a proxy can be configured in order to backup if certain techniques are unavailable.

    • Direct Storage Access (Use of Hot Add from VMs – Proxy must be virtual)
    • Virtual Appliance
    • Network (Connect through the VMware hosts to backup)

    When the VM disks are running on a storage system with access to the Backup infrastructure, your proxy can also use the backup from Storage Snapshot mode.

    In most cases, Letting VBR automatically select the proxy mode is perfectly fine – If one method is not available due to communication issues or other limitations, then the next viable mode will be selected.  Failing over to a different method does not remove CBT.  You can also manually select the preferred option for your infrastructure.

     

    VMware Backup proxy Deployment:

     In most cases, you will need to deploy a dedicated proxy server that will be able to handle the load, but by default the VBR server will be preconfigured as the proxy until another is deployed. Deploying multiple backup proxies will allow Veeam to distribute the load across the proxies to provider better backup performance and greater redundancy.

    Proxies are now supported on both Windows and Linux, there are some requirements around configuring for Linux.

    VMware Backup Proxy Services and Components:

    There are 2 services that are required to be installed on the VMware Backup Proxy to operate the required tasks.
    Veeam Installer Service: This service is used to analyse the system and install and upgrades necessary components for the server, this is installed on Windows Servers.

    Veeam Data Mover:  This service does most of the heavy lifting by performing such tasks from the Veeam Backup and Replication Server – Retrieving Source VM Data, Data Deduplication and Compression and moving the data to the target storage as the backup location.

     

    VMware CDP Proxies

    The VMware CDP Proxy performs tasks of moving data between source and target hosts. There are a number of tasks that the Data Mover performs differently to just a regular proxy.

    • Receives VM data from the production Storage
    • Aggregates Changed Data
    • Prepares data for a short-term restore point
    • Compresses and deuplicates data
    • Encrypts and decrypts data
    • Sends data to the storage in the DR site or to another VMware CDP Proxy

    The VMware CDP Proxy Is required for use with the CDP component of Veeam Backup and Replication. This a different use case to the standard VMware proxies.

    VMware CDP Proxy

    The CDP Proxy can be installed on either Windows or Linux based servers that have been add to the the Veeam Backup and Replication infrastructure section. In order for the CDP service to operate correctly, there must be a source and target proxy configured.

    To optimise the performance of the CDP jobs, the VBR server will take into account the load across all proxies that are pooled together and assigning each tasks as required.

    As a design choice for better performance, having a set of proxies (source and target) for one direction is recommended. One source proxy to a target proxy from site A to B, and another set of proxies for going from site B to A.

    The CDP Proxy services are fairly similar to those used on the VMware Backup Proxy, however there is an additional service in use.

    Veeam CDP Proxy Service: Manages all CDP activities such as data aggregation, data compression and decompression, data transfer and other tasks.

    Veeam Installer Service: <Same as VMware Backup proxy>

    Veeam Data Mover: Handles traffic sent during failback

     

    VMware CDP proxy RAM and cache.

     

    CDP proxies use intelligence to allocate RAM to ensure data is processed as efficiently as possible. If the RAM is configured as 16GB or LESS then CDP will split the resource usage 50% each way – this means that 50% is used for the OS and 50% is used for data processing. If there is more than 16GB allocated to the proxy server, then 8GB will be assigned to the OS and the remaining will utilised by the CDP data processes to ensure the most efficient processing available. When a disk has been processed, CDP will allocate 1MB of RAM to ensure data processing will not stop even if some disks cause issues or process too much data.

    As a fail safe, Data is only removed from the cache or memory on the source once the proxy receives notice that the target proxy has successfully received the data.

    Requirements

    • A CDP Proxy must be Windows or Linux – Can be Physical or Virtual.
    • CDP Proxies are not available to deploy unless there is a vCenter or Cloud Director server configured in the backup infrastructure in Veeam Backup and Replication.
    • When using a physical server – Must have a fast network link between hosts and CDP Proxies.

     

    Backup Repositories.

    Direct Attached Storage – Virtual and Physical. 

    There are 3 types of Backup repositories that can be used for Veeam Backup and Replication that are directly attached to the server.

    There are where the disk is attached to the server (I.e. VMDK attached to the VM used as a repository or physical disk installed in the physical server)

    Microsoft Windows Server:
    Several different ways to use a MS Windows Server as a Backup Repository:

    • Local/Direct (USB Drive) attached storage
    • iSCSI/FC SAN if server is connected to a SAN

    There are 2 Data Mover servers that are in play during a backup. There is a DM on the Proxy, and one on the Repository, these both will talk to each other to transfer data over WAN or LAN efficiently. The Data Mover is installed automatically when the server is added to Veeam BR.

    A Windows based repository can also be configured to run the vPower NFS Server function Allowing Veeam BR to provide ESXi transparent access to backed-up VM images that are sitting on the backup repository.

    Requirements:

    • Must meet all system requirements
    • Server must be added as a managed server inside Veeam Backup and Replication.
    • To be able to utilise Fast Clone, must use ReFS on the target disk and meet any additional requirements for this function.

    Linux Server:

    Like Windows, you can connect to the disks on the backup repository in several ways.

    • Local / Direct (USB Drive) attached storage
    • iSCSI/FC SAN if Server is connect to a SAN
    • NFS

    A Linux repository provides additional security measures as well, including: Hardened Repository utilising immutability and single-use credentials.

    Again, similar to the Windows repository, there are 2 Veeam Data Movers that are in play to communicate and transfer data. These are located on both the Proxy and the Repository.

    Requirements:

    • Must meet all system requirements
    • Must add machine to Veeam Backup and Replication as managed Server
    • SSH Daemon must be properly configured and SCP utility is available on Linux host to enable the installation of the Veeam Data Mover.
    • To enable Fast Clone function, need to meet the FC Requirements – min. XFS with reflink=1 enabled.
    • Open required ports on the firewall

     

    To utilise both Hardened and standard repositories on the same Linux server, you need to use Single-Use Credentials when adding the host. – Standard repository will disable immutability and use persistent credentials.

    Hardened Repository:

    A hardended repository enables a series of additional security measures to ensure that backed up data is secure and unable to be tampered with. A hardened repository is only available when using a Linux based server.

    A hardened repository provides the below additional security measures:

    • Immutability = Backed up files can have a time limit assigned to them for how long they are locked, providing the ability to protect from modification and deletion during this time period.
    • Single-Use Credentials = This is a set of credentials that are only used once to deploy the Veeam services (Veeam Data Mover and/or Transport Service). These credentials are only added once to the Veeam Backup and Replication in order to run the install. These are not added and saved to the credential store, providing an additional layer of security and not allowing the credentials to become compromised.

    ** For security reasons, not additional roles can be assigned to the hardened repository except for the use of the VMware Backup Proxy running in Network mode (NBD). Hardened Repository as VMware Backup Proxy.

     

    Network Attached Storage

    SMB:

    To communicate with SMB Backup Repository, Veeam utilises two Veeam Data Movers. Not to be confused with the direct attached repositories, the two used are Veeam Data Mover on the VMware Backup Proxy & Veeam Data Mover on the Gateway Server

    Veeam Data Movers are unable to be installed on an SMB share and a Gateway Server is required to be deployed to connect both the Proxy running Veeam Data Mover and the target SMB share to enable efficient data transfers over LAN or WAN.

    When targeting an off-site SMB share, it is recommended to deploy an additional gateway server on the remote site that is close to the SMB repository.

    Requirements:

    An SMB repository can be assigned to a Microsoft Windows machine that meets the requirements.

    NFS:

    As is the case with SMB, both the Veeam Data Movers for Proxy and gateway servers are required as a VDM is unable to be installed on the NFS share. The processing and communication sequence is the same in that the Gateway server established the connection with the proxy server to move the data efficiently.

    ** Windows Gateway servers cannon be used on NFS shares with krb5i and krb5p support

    Requirements and Limitations:

    • The NFS repository must provided R/W access to the Gateway server
    • Both Windows and Linux servers, and NAS Storage that supports the NFS Protocol
    • The server must also meet all of the System Requirements

    Requirements for Gateway Server:

    A machine installed with the gateway Server role and used to communicate with NFS backup repositories required the following:

    • Both Windows and Linux can be used as gateway servers – meeting the System Requirements
    • The server must be a managed server within the Veeam Backup and Replication console.
    • Backup server must have R/W access to the NFS repository to allow automatic gateway selection
    • When using automatic gateway selection for NFS, the backup jobs can use the same machine as gateway for the repository and proxy. Ensure the below is configured for the backup proxies:
      • All required proxies have R/W access on the NFS repository
      • When configuring for automatic proxy selection, provide R/W to all procies in the VBR Infrastructure
      • Ensure the NFS Client package is installed on the Linux Proxy server

    While there are general requirements above, a Linux Gateway server has some additional requirements:

    • The NFS Client package must be installed
    • Must provide either root or elevated root credentials in order to authenticate with the Linux gateway server
    • Veeam Backup and Replication will only use the highest version of the NFS protocol that is enabled on the NFS repository.

    If the NFS repository has a newer version, then VBR will require the repository to be edited. Running through the edit wizard (without making changes) will run the DB update process after collecting the repositories information.

    Object Storage:

    Object storage is intended for long term retention of data that can be placed in either the cloud (AWS S3/Azure Blob) or an S3-compatible solution running locally/on-premises; such as MinIO.

    Starting in Veeam BR 12, Backups are now able to store direct to object as the primary repository, this is where the data will first back stored before going to a second copy (in most cases, off site.)

    There are a number of cloud object storage providers supported available;

    • Amazon S3, Amazon S3 Glacier or AWS Snowball Edge
    • S3 Compatible, S3 Compatible with Data Archiving
    • Google Cloud
    • IBM Cloud
    • Wasabi Cloud Storage
    • Microsft Azure Blob, Azure Archive Storage and Azure Data Box
    • (Veeam Data Cloud Vault – VBR 12.1.2.172 or higher)

    Object Storage is available to be used in multiple ways:

    • Target Repository as for backup and backup copy jobs
    • Object stoage source from which backup copies will copy restore points from
    • Target repository for file backup jobs
      • Files cannot be backed up to an object repository if the repo is part of a performance extent.
    • Target repository for Cloud Director Virtual Machines
    • Target repository for virual and physical machines by using the Veeam Agent for Windows and Linux.
    • Target repository for backups using the MacOS Client
    • Target Repository for Nutanix AHV
    • Target Repository for oVirt by Veeam Backup for Oracle Linux Virtualization Mnager and RedHat Virtualization
    • Target repository for Applications running on Kubernetes persistent volumes created by Kastan K10 Plugin
    • Target Repository for the configuration backup for Veeam Backup and Replication.

    Object Storage can form part of the SOBR (Scale-out Backup Repository) where it can be used on each of the tiers.

    • Performance Tier: For quickly accessing stored backups
    • Capacity Tier: Available for offloading backups to cloud storage
    • Archive Tier: Infrequently accessed backups, mainly considered cold storage

    Veeam BR will use a VMware backup proxy to transfer data and a mount server to process guest OS application and perform item recovery.

    VMware Backup Proxy will connect to the object storage using one of the below methods – This is all dependent on the type of job

    • Directly: VMware Backup Proxy will transfer data direct to the Object Storage Repository
    • VMware Backup Proxy will transfer data to the object storage repository utilising a gateway server (IF backing up multiple VMs in a job, then a gateway server pool can be used)

    Considerations and Limitations

    • Make sure all required ports are open to and from the object storage.
    • Backup Server and Gateway server require internet connection to validate certificates
    • A second backup server can be attached to the same Object Storage, however it must be read only to ensure that there is no split brain between backup servers where data is mismatched
  • VeeamOn 24 – Day 1 Keynote Announcements

    VeeamOn 24 – Day 1 Keynote Announcements

    VeeamOn is upon us once more and is loaded with many great announcements for new innovations and technology in the backup and cyber resiliency space. There were recaps of announcements made earlier in the year, such as; Veeam Data Cloud, and Coveware. But there was some secrets that were kept very quiet.

    Veeam, in recent years, has had a strong focus on Cyber Security, and protecting not only your backups, but also detecting anomalies in your data, capturing malicious content before it becomes a problem. There are many ways in which your backups can be protected using technologies like; Immutability, Encryption and in-line malware detection. Each one plays a critical role.

    At VeeamOn 24, Veeam highlighted the Veeam Cyber Secure Program detailing the workflow for securing your backups, detecting anomalies and performing the subsequent action required to ensure your business data is secure and protected or getting the business back on their feet after a cyber incident.


    Before getting into announcements, taking a look back at the history of innovation from Veeam is a great way show the progression over the years for how the product(s) have transformed and become what they are today, and where the product is heading.


    But now it is time, the future of Veeam products is laid out, what new versions and what are they going to entail?

    First up, the small updates and additions to existing products on the market. Some highlight

    Kubernetes Backup – V7 (Available Now):

    • FIPS- Enable Cluster
    • Azure Blob Immutability
    • OpenShift Support

    Veeam Backup for M365 – V8

    • Immutability for Primary Backups
    • Linux Proxys
    • Proxy Pools
    • MFA for Console access

    Backup for AWS – V8

    • AWS RedShift Support
    • ASW FSx

    Backup for Azure – V7 

    • CosmosDB Support

    Backup for Salesforce – V3

    • Data Encryption
    • Data Archiving
    • Data Pipeline

     

    Moving to the longest standing Veeam product, there is always room for improvement or other technologies out there that just aren’t being backed up yet. Anton dove straight in and presented backup for both MongoDB and Microsoft Entra ID. These are both built into Veeam Backup and Replication, extending the feature set to do more.

    Some will wonder why Entra ID is going to Veeam BR rather than VB365, and there are going to be a few answers, but it comes down to the majority that will benefit from being able to backup their users in Azure and restore user properties. Those customers may also not use the full Azure suite and may only have it for authentication into their environment. EntraID becomes the backbone, the replacement for Active Directory on-premises.

    And finally, the most exciting announcement I thought was worth highlighting is Veeam Backup and Replication V13 will be coming with support to run on Linux! This is something that Veeam has fallen behind on, but it has certainly been assumed that it was on the road map, as each new version of VBR brought in another component running on Linux.  There are possibly a few components that might not support running on Linux, but I’m sure these will come over time.

    Running on Linux allows for greater control, security, and performance. This now brings the flexibility that could be used to build Veeam appliances that MSPs and Service Providers can supply to customer sites to create a primary backup copy and then using cloud connect, store a backup copy offsite with the Service Provider – This is just one of the possibilities – Sure, you could do that with Windows, but there are more limitations and additional licensing.

    All this is due for release Q3 2024. Watch out for further announcements on day 2, along with product demoes.

  • VMCE: Core Concepts – Define RTO and RPO

    VMCE: Core Concepts – Define RTO and RPO

    Core Concepts: Define RTO and RPO (Veeam KB link)

     

    Defining the protection scope:

    • Find out first how many machines and how much disk space is currently used
    • Utilising the above information, add it with the calculation of the daily change rate
    • This information is critical to ensure that there is enough space available to protect the backup. Veeam will create a full backup file on first run, then each backup job after will run an incremental. The incremental backups will only backup the changed blocks, If the changed blocks is only small, then on a short and small backup will take place, if there are a large number of files changed, then the backup will take significantly longer and will take up additional space.

    Veeam Calculators

    RPO (Recovery point Objective)

    RPO and RTO are absolute requirements for a DR plan.
    RPO is the point in time of which the latest backup is available, this is the accepted risk of amount of time/data that may be lost since the last backup.
    This will also set how many backups will need to be taken to ensure an available copy within the window


    RTO (Recovery Time Objective):

    RTO is the time between the time of the incident to the time the environment or systems are available for use again.  If you have a RTO that is 24hrs in your Disaster Recovery plan, this is the agreed upon time that the systems should take to be online again and available. This time can be calculated by running failover tests or recovery scenarios in which the steps to recovery involved would be tested and timed.


    Planning RTO and RPO

    There are different recovery strategies that can be used, all of which can range from a short downtime to a longer recovery. Achieving a short downtime may require additional features and services to be available.

    In order to have an RTO and RPO within seconds, you can utilise Veeam CDP, however, this would require having two separate sites, Prod and DR, to live replicate and failover to in near real-time, the data would be synchronised instantly to ensure the recovery point objective is meter.

    For a RPO of minutes but an RTO of seconds, Veeam Replication will give you the ability to have have an RPO within minutes, but your recovery time can be within seconds from failing over to the environment starting up. This is due to way replications are created as these first create the snapshot like a backup, and then run a job to applying the changes to the target. This can run every few minutes.

    For a recovery time within minutes, the Veeam Snapshot Orchestration (within VMware vSphere) creating a chain of application consistent array-based snapshots which is then able to be mirror/replicated to a secondary array.

    Moving into the having an RPO of less than 24 hours and a recovery time that is generally within minutes or within a day is the straight up backup. Generally, a backup is taken every 24 hrs at the end of the day, some may be configured to take a backup every hour or every 2 hours depending on the requirement, however, this will creates a much longer RPO to go back to and thus more changes that occurred during the day that may not be backed up.

    If you find yourself in a situation where your backups are unavailable, then you will be looking at Backup Copies where your RPO is going to be within the 24-48 hour range, and your RTO could come within minutes, but generally a number of hours depending on where your backup copies are stored. This allows you to keep a copy offsite, in another data centre. As this is a copy of the backup, the calculation would first involve the frequency of the backup + the time for the backup copy to complete its copy process to the next location.

    To really get into the 3-2-1 rule, although it is going to put both the RPO and RTO into the hours with significant recovery operator involvement, Tape is a great way to air-gap your backups and have an offsite/disconnected copy. The reason that this is the slowest solution is the speed of which tapes read and write at. Tapes generally also only hold upwards of a few TB and thus require swapping, and if you are using GFS (Grandfather-father-son) method, depending on where in the chain the recovery is being processed may require a number of tapes to be swapped.

    Summarising RTO and RPO planning:

    Depending on the budget and resources available, the RTO and RPO can differ significantly, and with less equipment, the RTO can be much higher. For instance, without a failover site, both the RTO and RPO could take significantly longer times as the best option, if supported by the array, is Snapshot Orchestration where the data is available on the array already.  If not support, then you do not have a second site, you would then be recovering for a backup, and if those backups have been affected by the disaster as well, the recovery from Backup Copies or Tape are your real remaining options.

    With planning and budget, RTO and RPOs can be lowered and almost require only small amounts of restore operator assistance to get production back up and running.

    Keep in mind, the longer the RPO, then the bigger the gap of data that isn’t backed up and available when the environment is recovered.

  • Configure Object Backup Copy in Veeam Backup for Microsoft 365

    Configure Object Backup Copy in Veeam Backup for Microsoft 365

    In early 2023, Veeam released their next realease of Veeam Backup for Microsoft 365, with v7.  This brought a load of new features, allowing it provide faster and more resilient backups for Microsoft 365.  One of the biggest features was the ability to perform backup copies of your tenancy, allowing you to keep a second copy whether that be on another datastore, another data centre or off up into the mighty cloud.

    Backup copies aren’t new, they have been around in Veeam Backup and Replication for over 10 years, and it only made sense to extend this feature to 365 backups.

    A backup copy is a separate job from the primary job, this allows more flexibility and ease of use – so it is important to name the backup copy with something that distinguishes it from the primary job.

    Before we get into the configuration side of things, there are a couple of pre-requisites for being able to run a backup copy:

    • Only Object Repository to Object Repository is supported. You cannot perform a backup copy of the original JetDB – If you want to backup your JetDB, you could use Veeam Agent Based backup to take a copy of the JetDB files or veeam Backup and Replication to take a backup of the VM hosting those JetDB files.
    • Your Object target must have it’s own Proxy/Repository attached, you cannot share with Object targets. You will receive an error if you try to use a proxy folder that already contains data.
    • If you want to use Immutability, Object Lock must be enabled on the bucket before configuring the job.

    That’s pretty much all there is to watch out for and consider. The rest of the steps should be fairly familiar if you have already gone through and set up Object Repositories for your existing jobs.

    Config

    1. If you are using an on-prem solution for your object storage, like MinIO or Object First, you will need make sure your storage is pre-confgiured and accessible from your 365 server.
    2. Create your bucket on your Backup Copy target storage and confirm that you can access the location. Below you can see that minio-001 (left) contains my primary backup that i hav already configured and taken an initial full backup of my 365 account. My Backup Copy target, minio-002 (right) currently shows no backed up data for 365,


      You will also note that i currently only have 1 backup job configured.

    3. Navigate to Backup Infrastructure -> Object Storage and  select Add Object Storage.  This will open up the Object Storage connection wizard. Here you can start by giving your object storage a Name and Description

    4. Select the correct object storage solution to meet your requirements. If you are using something like MinIO, Ceph or another S3 Compatible object storage, select S3 Compatible, otherwise select the matching cloud target.

    5. In the next screen, you will need to enter your service point, Data Center Region and specify your Account Credentials for your target repository. These will be saved into the Veeam DB. The service point will be that of your backup copy target.

    6. Ensure that you have already prepped your repository with a bucket to connect to. Depending on the number of buckets that you have, the drop down menu will display all available buckets – Select the correct bucket for your target. Once you have selected your bucket, click Browse  and select the bucket name – Click New Folder and name your Backup Copy target folder.

    7. Another great feature brought into v7 is the ability to create Immutable Backup Copies. However, please consider and understand the use of this feature, whilst it is always recommended to have immutable backups, in Veeam Backup for Microsoft 365, the retention period you select for the job is also the retention period of the immutable backup. In other words, If you select to retain 2 years worth of backups before they age out and have applied immutable backups, if the customer leaves and you are required to delete the customers data off your system, you will need to wait until the last backup has aged out over 2 years before it can be removed.
      https://helpcenter.veeam.com/docs/vbo365/guide/immutability.html?ver=70

      Click Finish to create your object storage

    8. You will need to create and Object Repository that attached the Object Storage to a Proxy and a caching folder for the database. Select Backup Repository > Add Repository (You can also right click to select Add backup repository). Once again a wizard will open up and you can give the repository a Name and Description


    9. Select Backup to object storage – this will select the next few windows applicable to object storage. If you select the second option, this will allow you to create a JetDB repository – which unfortunately won’t work with what we’re trying to achieve here.

    10. Depending on your infrastructure design, you may have multiple proxy servers, and they may be in different locations. Select the right proxy server that connects to your object storage target. Here you can then create the local cache path that will reside on your proxy server. You should have a drive preconfigured to contain your cache files. Select Browse and then select the drive and path then New Folder to create the target cache folder,

    11. Select the target object storage. If the object storage is already in use by another repository, it will not show up in the list.

      You can configure an encryption password to ensure that the data is encrypted at the target. This is different from immutability, encryption will prevent someone from reading the data without the encryption password, but will not prevent them from deleting the data.


      During the validation process, if the selected cache folder already contains an existing database in it, you will receive an error message advising of this. You will either need to clear the folder or create a new folder.

    12. Select your retention policy and the type of backup you want to take, whether it be as an image or at the item level, make sure you read carefully the different options available.
      By selecting Advanced you have the ability to choose when you want the retention policy applied – make sure you understand how this works, otherwise you may end up paying additional egress charges.
      https://helpcenter.veeam.com/docs/vbo365/guide/new_repository_4.html?ver=70 

    13. Lastly, once the targets have been configure, it is now just a case of creating the backup copy job. Head back over to Organizations > Select your existing primary backup job and click Backup Copy  – this will open a new wizard that looks similar to the primary backup job creation wizard.

    14. Here you will be able to select your Target Backup Repository – take note that this is the Backup repository and not the Object Storage directly.

    15. You can choose when you want to run the backup copy job. You can select for it to occur immediately as the primary backup job runs, you can set a specific time of day or on a repeated schedule.
      There is also the option to run the backup job within a pre-defined window.

    16. Once the job has been configured, if you did not select the “Immediate” option to run the job, you can go ahead and run it for the first time. You will note that the job type is shown as Copy and the the start and last backup information is avialable.

    You have now configured a backup copy of your primary Microsoft 365 backup.

    For more information, please check out Veeam’s KB articles related to backup copies. 

  • Updating Veeam Backup for Microsoft 365 v7

    Updating Veeam Backup for Microsoft 365 v7

    One of the most common tasks we do as either a Systems Administrator or NOC (Network Operations Centre) is to ensure that your software that you and your customers use is kept up to date to ensure that you get the most out of the application, whether that be stability, performance or security. This is particularly important for your backup applications, without having a stable and secure backup can leave your business vulnerable to being unable to restore in the event of a disaster.

    Here in this article, I’ll take you through how easy it is to keep your Veeam Backup for Microsoft 365 up-to-date.

    ** Before we get started: If you are running your backup in a virtual machine make sure to take a snapshot of the VM before updating. This will allow you to roll-back in the event something goes wrong **
    …and don’t forget to remove the snapshot once confirmed the update is a success!!

    Another thing to check before getting into the upgrade is to check the system requirements to make sure that there have not been any changes. If there have been requirement changes and your VBO server isn’t running the right amount, you will receive a warning during the install to advise you that the server does not meet the minimum requirements.

    Update Process

    One of the great time savers Veeam have built into their Veeam Backup for Office 365 is the online update function. Here you just need to run a “Check for Updates”and proceed as per the wizard.

    1. First thing we need to do is take note of the version of Veeam Backup for M365 we have installed. Although in the next few steps we cover checking for updates, this creates our baseline to be able to confrim a successful install at the end.To check the current version. Select the Hamburger Menu -> Help and Support -> About

      Just click anywhere on the ‘About’ popup to close
    2. From here, if you haven’t done it yet, make sure you take your snapshot before proceeding.
    3. Once you’re ready to proceed, it’s time to kick off the update:Select the Hamburger Menu again -> Upgrade
    4. A wizard will open up. Here in the first screen, you will see Veeam running a quick check for updates, then open up another screen to show the update that is available.  If there is an update available, you will then get a link for “What’s New” where you will be able to see a summary of what has been added and resolved in the update.If there are updates for the Veeam Explorers for Sharepoint, Exchange, Onedrive and/or Teams, then updates will be displayed here also.


    5. Once you have confimed that you are happy with the changes, you can click “Ok” and proceed with click “Install” – The wizard will remain on the update screen until the updates have downloaded. Once the download has complete the wizard will continue with the installation.
    6. All the above steps could be performed while your jobs are still running, but sensibly, you would have stopped them as part of your change control before proceeding with the update. However, if you had not stopped your jobs, or one accidently started, the wizard will advise you that it is ready to stop all jobs and confirm that you are ready to install the update.
    7. The wizard will close along with the Veeam Backup for M365 console and it will open a new installation wizard from the downloaded update file. From here, it is just a series of clicking next until the install takes place.
      Regardless of the operating system is that you are installing onto, a pop-up will show up to advise you that Microsoft Windows 7 SP1, Microsoft Windows 8.x and Microsoft Windows Service 2008 R2 SP1 are no longer supported and you MUST upgrade your Windows installation before proceeding.
    8. Read thoroughly through the EULA – You will need to tick the boxes and agree to the EULA before you can proceed with the install.
    9. Confirm that allthe components that you want to upgrade have been selected. All components are selected by default, however, you may want to only update certain ones, you can deselect if needed.
    10. The final screen for the installation is to confirm your’re ready to install. Once you click Install the installation will take place. This can take anywhere between 1 – 5 minutes to completed.
    11. The install will also start your Veeam services as the final steps, this will confirm that the installation was successful and there are no issues that arose from the install process.
    12. Once the installation has complete, you will see the final screen where you can click Finish to complete and close the install.

    13. As we did earlier at the start, it is now ideal to open up Veeam console and go back to the About section to confirm the update was indeed successful and you are running the latest version.

    14. Once confirmed, you can then move over and check if there are any additional updates/upgrades that need to be applied to the the Backup Proxies or Backup Repositories. These will show up under the Backup Infrastructure “Backup Repository” and “Backup Proxies”  – If you are running the proxy local on the same server that you just updated Veeam Backup for Microsoft 365, then it will update at the same time.

    15. Finally as part of your Post Checks  for your change control, run a test job and confirm that everything is connecting and running as it should. If there are any issues, you always have your snapshot to fall back on.

    Previously I had created a video on how to peform a major version upgrade from Veeam Backup for M365 v5 to v6, in this video I also cover the repository and proxy upgrade process.

    P.s. Don’t forget to remove your snapshot after running your tests

     

  • Stepping up and taking on my Imposter Syndrome with the ISN Podcast

    Stepping up and taking on my Imposter Syndrome with the ISN Podcast

    It was July 13, the day was only just starting as I was coming up the elevator to my work’s office floor, and just as I usually do, I flicked open my phone for any notification badges. Although I had checked earlier, there weren’t really any notifications, but one caught my eye, it was a LinkedIn email informing me I had a new direct message from Chris Grundemann, whom I was very lucky to have met In Austin, TX at Tech Field Day 18 back in February 2019.

    Chris is a fantastic dude and together with Zoe Rose, they run the Imposter Syndrome Network Podcast, a podcast I was a huge supporter of before the first episode with Ethan Banks dropped. They really started promoting the idea early on, and it got my excited for what was to come.

    My mind is always running at 130% all the time, there is so much going on that there usually isn’t ever time to finish thinking through the last thought. The one place where I find that my thoughts can slow down and process more thoroughly is in the car, this is where I am usually listening to podcasts or thinking through a problem (but it is also the place where the solution will get lost because I can’t write it down). That being said, I’ve listened to every episode of the Imposter Syndrome Network Podcast, each and every week. Just like any other podcast, I start thinking of what I could contribute to that episode, what my opinion of the topic might be, or the answer to a quiz. I had also thought on many car trips that I would like to be asked on the Imposter Syndrome Podcast – of course, I had all the topics and answer right there in head, however I would forget it all once the car engine was turned off – although I didn’t actually think I would end up on an episode, especially as time went on and the guest just became ever so much more amazing with their long lists of achievements.

    However, on this particular day, that message arrived. Here was a message from Chris asking me to come onto the show and tell my story. Immediately my imposter syndrome kicked in and my mind started racing in all of the thoughts of how could I just say, “I really appreciate the offer, but I’m not the person you’re looking for..” I didn’t think I had any of the chops to be on the show. But the strange thing is, I actually wrote back “..My imposter syndrome is yelling at me to say “No” but the one thing I’ve taught myself is to just answer “Yes” and be terrible to be successful.” – that last bit was key. “Be terrible to be successful.”

    So, from that point, I was signed up and I booked in for my recording session with both Chris and Zoe for 2 weeks time, Friday at 3:30am in the morning. That’s right, 3:30am. The 3 of us together were able to split the earth in 3 even sections with our locations. The recording went well, and it was quite clear at the start that the cogs hadn’t really started turning and my answer to Chris’ first question was restarted 3 times. We spoke a lot about my 10 year history as a vExpert and what it takes to join and be a part of those community programs and the motiviation to produce content. As all three of us have been delegates with Tech Field Day, we also talked about that small close-knit family that community groups create, and the different levels of experience and opinions of technology. 

    I’ve done some podcasts in the past, when I did Nerd Journey Pod, I was nervous, the only thing going through my head was “What if I say the wrong thing publicly – what is everyone going to think?” and it really got to me, you can hear it throughout. My next podcast was with The On-Premise IT Roundtable with Gestalt IT, but was joined by Pure Storage – whilst I was there for Storage Field Day, talk about being thrown into the deep end of having to discuss a product I barely knew anything about – but I managed to get through it by focusing on the business side rather than the specific technology (phew!). I then made several appearances over the last few years on the Tech Breakfast Podcast just joining in on the conversation which had no real agenda except to discuss current technology events.

    So you can see my podcasting history is fairly short, and one thing I’ve learnt about myself over the recent years is that I am just 1 person in among 7 billion others, if I sink like a lead balloon doing a podcast or a video, then it won’t be long and I’ll be forgotten. It has taken me a VERY long time to learn that. Sure, there are people who are close to me, but they seem to move on pretty quick, too many other things that are happening, it’s just myself who dwells on my own past.

    Immediately after I said my first opening line on the show, I squirmed in my seat, I started focusing on that thinking it was terrible and very cringe worthy, although I had originally thought it would be a nice little joke, because really, I was an imposter being there – this continued to play on my mind all the way up to me hearing it when the recording was released. But, it was one of my friends who heard the episode and his first reply was “That first line reminded me of Fight Club” and yeah, that was exactly what I needed to hear, it tied it together nicely.

    I am always hesitant to listen to myself after any recordings I’ve done, but I do it anyway, I use it to help learn and make adjustments on what to do different next time. I was extremely nervous to listen back to the episode, the only things I could remember saying were the bits where I started to focus on where I felt I said the wrong thing. In the end, I actually enjoyed the episode and started to promote it in some of my channels with friends across the world – this is something I usually would be very strategic with and would wait until the episode was released out of my control, but on this occasion, I couldn’t wait.

    What happened over the next 24 hours was certainly a first from any of the other podcasts I had done, several people I had no direct connection with started reaching out on LinkedIn telling me how much they enjoyed the episode and they wanted to connect. This truly created some highlights, like anything I produce, as long as someone gets something out of it, then I’ve done something worthwhile, I just didn’t expect the response I got.

    Earlier this year, I accepted an offer to do a co-presentation with a friend at an event next year, it was a whole year away, and it still took some time to accept the offer, but the thing is, if I don’t take the opportunity, I’ve only got myself to blame for not taking the leap.

    While it might seem obvious that I process every word in real-time, it isn’t the voice that the audience is there listening to, they are there for the content and to learn something. You can be anxious, but don’t let that stop you because there is always a brighter outcome – it just takes 1 person.

    It doesn’t matter if someone has a longer list of credentials than you, it doesn’t matter if someone has written a post on a topic you were planning to. It doesn’t matter how you present. Step up, take the challenge and say “Yes”. There is at least someone out there who will appreciate it very much that you did.

    You can check out my Imposter Syndrome Network episode here:
    ISN Linkedin group

     

  • VCSA 8.x stuck in update staging loop

    VCSA 8.x stuck in update staging loop

    I ran into an issue with where my VCSA was consistantly throwing me an error regarding “Staging in Progress” and “You have reached maximum number of retries to resume the patching. Please restore the vCenter using the backup”

    As this is my home lab, the system had been turned on and off numerous times, and thus a restart does not resolve this issue.  It also stood out as a bit of an issue as no matter who URL I used to log in, it would popup immediate under https://<vcsa>:5480/ui/update/progress and I had to redirect to another page.

    I was unable to also load any new updates (which I knew I was a few behind by this stage) and so off I went to do some research where I found KB 87238 – This was pretty straight forward and just needed some files removed from the VCSA. While this article is for 7.x I did find that the “Software-pakages” folder did not exist and was unable to cop the json file as required (No issues appeared to occur)

    First make sure you take a snapshot or backup of your VCSA.

    Enabled SSH so that you can get shell access.

    Follow the below commands

    ssh root@vcsaadress

    Command> shell

    # service-control --stop applmgmt
    # rm -rf /storage/core/software-update/updates/*
    # rm -rf /storage/core/software-update/stage/*
    # rm -rf /storage/db/patching.db

    Depending on the version, the next file may not exist – this did not seem to be a problem for v8.x

    # mv /storage/core/software-packages/staged-configuration.json /storage/core
    # mv /etc/applmgmt/appliance/software_update_state.conf /storage/core/
    # service-control --start applmgmt

    Operation not cancellable. Please wait for it to finish...
    Performing start operation on service applmgmt...
    Successfully started service applmgmt

    Once These steps are completed, I was able to then log back into VCSA and run a scan for updates. This found 3 available updates:

    I was able to select and start staging the updates, there was a successful progress bar running.

    Although the Validation did complete successfully after staging, the update did not install and I was unable to scan for anything more, in fact, I received another few errors. I decided to give it a reboot and see what happens.

    After a reboot I was given the option to install (without the need for staging) and the VCSA was able to update as it should.