Currently, I'm working with Pulumi; here's a list of downsides.
- One thing that I noticed is a problem compared to a static configuration file like Terraform is that the destruction of resources is not in order. For instance, if I wanted to get rid of AWS Route53 records but some already existed, Pulumi would first create rather than destroy. In Route53, a single record type can only be defined once per name/host. This became a problem where I had to manually destroy the records first.
- Support is lacking due to a lack of a community
- Things don't work sometimes, and it's hard to debug (I'm basically facing the same issues as https://aws.plainenglish.io/why-we-abandoned-pulumi-after-3-months-and-went-back-to-terraform-03539d1859c1)
Terraform comparison:
- Multi-tenancy of terraform: Using a lot of for each(especially with dependencies) and having tenants loop(all the tenants in one state) makes terraform slow down a lot. I had experience with the system which i am describing and it was not very good. If i were to remade it, I would probably try to break down this terraform more, and try to have tenants in separate states. Or as you described having tenant management be it’s own application which manages all tenants related stuff, but there is a lot of reinventing the wheels, so also not ideal.
- Cross-account/provider churn. Assuming roles into many AWS accounts adds latency and increases retries/throttling. I originally did that and I'm now having issues with assuming roles in the first place
Upsides:
- I like the Pulumi Automation API for multi tenant purposes. The stack exports with the automation API is definitely more helpful than using OS stuff for Terraform/Terragrunt