TIL that you need to grant permissions to more than just your elasticbeanstalk-- S3 bucket when deploying to Elastic Beanstalk; you also need permissions on elasticbeanstalk-*.