I dug into it tonight and got it working. Here's the CloudFormation setup I'm using. I ordered the properties in the same order you'd find them on the console GUI if you were setting it up that way:
Thank you to https://github.com/yvele, this is essentially his set up with a few added details.
Note 1: The ForwardedValues property is deprecated and it seems that eventually we'll have to change to a CachePolicy and OriginRequestPolicy. I tried a few different setups but unfortunately I couldn't get it to work using those.
Note 2: After you get the CloudFront Distribution up and running, don't forget to change your app to switch to the new endpoint! i.e. fetching from https://something.appsync-api.region.amazonaws.com/graphql -> https://something.cloudfront.net/graphql
AWSTemplateFormatVersion: 2010-09-09