This is yet another problem that I faced during the setup of puppet. Puppet is a configuration management solution that I used in my infrastructure. I chose this over CFengine because I found it easier to understand and setup. While CFengine is very nimble and flexible, I would need to hire some very fine engineers to manage the setup in the long run.
Anyway back to the problem. Puppet setup was a breeze and I did some simple test on my staging servers. Works great. Configurations were pushed down nicely to the staging servers. However, the same did not happen on the servers that I kickstarted subsequently. These were the errors I got.
Sep 23 08:21:56 host2 puppetd: Could not retrieve catalog from remote server: certificate verify failed
Sep 23 08:31:51 host2 puppetd: Could not retrieve catalog from remote server: certificate verify failed
Naturally, I looked at the certificates from the puppetmaster and client end. Did several rounds of regenerating certificates, but ended up with the same results. The usual culprits were hostnames which SSL is particular about, but I was very sure I got that right. Googled and results still suggested that it was my hostnames and certificates.
Anyway I found the problem after a few painful rounds of reinstalling and regenerating certificates. The problem was a simple one – my server time. Eventually a quick ntpdate solved the issue. The irony was that the first configuration that my puppet server was configured to check and install – is NTP and timezone.