AWS Opsworks Troubleshooting Cheat-Sheet

When working with the chef-driven AWS Opsworks service, there are often times when waiting for logs from an instance to be processed can be a painful and slow process.  It can sometimes be very useful to watch things running through while connected to the command line.

Opsworks has already installed all you need in order to achieve this, and there are some useful places to look in order to find the resources you want.

First, the logs of the output from the chef-solo run can be found in the following location:

/var/lib/aws/opsworks/chef/

In that location you will find both a json and a log file for each opsworks command that has bee sent to your instance (setup, configure, deploy and so on).  If you find neither of these files are being generated and your opsworks runs are failing, there are a few possible causes – perhaps you’ve added an authentication system (LDAP for example) to the server or removed the aws user from the host.

The json files here contain the attributes provided to your chef cookbooks for the action triggered at the time for which they are named.  The log files are, more obviously, the log output from that same action.

The AWS Opsworks code and binaries are installed to /opt/aws/opsworks (which will be unaccessible unless you are root).  Inside this folder you will find executables (as usual) in bin and the cookbooks used in merged-cookbooks.

If you want to re-run a custom cookbook which failed so you can monitor the system during the run you will need to know the time the failed run happened and the name of the recipe(s) you want to repeat.  For the purposes of our example, we assume you are running the appserver::deploy recipe (which might be associated with the deploy stage of your OpsWorks tier).

First, locate the correct filename for the attribute json created during the run you are hoping to repeat from /var/lib/aws/opsworks/chef.

You will need to tell chef-solo where to find its cookbooks, so create a file /root/solo.rb and give it the following content:

Finally, execute the repeat run using chef-solo from the OpsWorks codebase: