I think I figured it out. Apparently there's a really low default if you don't explicitly set a max size via client_max_body_size. Setting that explicitly seemed to work.
Thanks folks. Sometimes you just need to bounce stuff off people to get to the right place.