How to fix template transfer in CloudStack

While trying to transfer templates within Citrix CloudPlatform (Apache CloudStack) between different zones I received the below error:

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This post covers how to fix this.

Investigation

After some quick Google searching, I found this support article on the Citrix website which outlined the problem that I was having: http://support.citrix.com/article/CTX137428

Essentially the problem is that the secondary storage virtual machine (SSVM – where the templates are actually stored) doesn’t have the custom certificates from the other SSVMs in the key store, so we need to import them. This only started happening after I uploaded a new SSL certificate to cover my own domain rather than using the discontinued realhostip.com service.

The support article suggests getting InstallCert onto the SSVM and using it to import the certificates from the other SSVMs in other zones to the key store.

The links from that support article allow you to get a copy of InstallCert.java, however I had the problem of not being able to easily get ‘javac’ from the JDK onto my SSVM due to a number of mirror errors, and too many dependency problems trying to manually get the required .deb files.

Solution

In this particular instance I was running Citrix CloudPlatform 3.0.7 patch C, I have been advised by Citrix support that from version 3.0.7 patch F the problem has been resolved so you should not have this problem if you’re on a newer version. You can either apply the patch to fix the problem, or you can use the InstallCert work around.

I opted for the InstallCert work around as to patch CloudPlatform requires restarting the system VMs which was not an option and would have resulted in down time. If you continue through with this keep in mind that if you stop, reboot, or recreate the SSVM you will need to do this again and update the java key store with the custom certificate again or you will receive the same error when transferring or downloading templates.

In the previously mentioned support article, I was not able to get InstallCert working, instead I downloaded the InstallCert.zip file from the link below:

http://download.cnet.com/Java-InstallCert/3000-2085_4-75852342.html

I then placed this onto the first SSVM using SCP and ran the below command to unzip it.

unzip InstallCert-bin.zip

Once that’s exported, you can run the below command within the same directory.

java InstallCert :443

After entering this command you may see a list of the certificates found at that IP address, if there are multiple they will be numbered, enter the number corresponding to the certificate that you wish to enter. This will add the certificate to the trusted keystore.

Once complete, run the below command.

keytool -importkeystore -noprompt -srckeystore jssecacerts -destkeystore /usr/local/cloud/systemvm/certs/realhostip.keystore

You will be prompted to enter the destination keystore password, by default this is “vmops.com”. For the source password you can simply leave it blank and press enter.

You can go through the same process for any other SSVMs that you may have in different zones, ideally you should do this on all SSVMs specifying the public IP address of all others out there so that they all trust each other.

When you’re done making changes, restart the cloud service on each SSVM as below.

/etc/init.d/cloud restart

You can inspect the keystore to confirm that the certificates were added with this command.

keytool -list -keystore /usr/local/cloud/systemvm/certs/realhostip.keystore

You should now be able to copy templates between zones, or download templates successfully.

Conclusion

It’s important to note that even if you get this working, unless you patch the system VM you can lose these changes when the SSVM is rebuilt. The SSVM is designed to be replaced as needed, if you delete it, it should automatically recreate itself as needed from the system template. Ideally you should patch to permanently resolve the problem, however this work around allows us to continue working and transferring templates in the mean time.

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>