Transcript

Rich Ziade This podcast is brought to you by Cisco. [Rich laughs]
Corey Quinn Really? It was a pretty good podcast! I haven’t gotten around to ruining it yet.
[music ramps up, plays alone, fades out]
RZ You know what’s the worst kind of creep?
Paul Ford Oh yeah, I read the New York Times.
RZ Scope creep.
PF Ohhhhh, different topic. Yeah, that is awful. That is bad stuff. 
RZ Yes. 
PF Well Rich, who knows something about scope creep?
RZ We are experts at defending the scope of a project, making sure it doesn’t get out of hand such that you can’t deliver. And we’re going to share a lot of the tools and ideas that we use–
PF Are we gonna have a conversation that people can tune into via Zoom?
RZ We are. We’re having a special edition of The Postlight Podcast. It’s gonna be live streamed September 16th at 1pm.
PF Okay, where do people go?
RZ Eastern Standard Time. They go to Postlight.com/events. We’ve done these in the past. They’re always kind of fun. Very lively.
PF Don’t think webinar. Think lively, fun goofball experience.
RZ Please join us. Postlight.com/events.
[music fades in and out]
PF Richard!
RZ Hi Paul.
PF I was talking to somebody a couple months ago, and it’s somebody who works with lots of young engineers, kind of a mentor, organization teaching organization.
RZ Okay. 
PF And I said, you know, I’m always trying to find out what’s happening in our misbegotten industry. And I said, look, what are the answers? What’s the new language? You know, I’m not expecting Julia, I’m expecting maybe some TypeScript or something like that. And my friend goes, ‘Oh, yeah, they all just want to get their AWS certification. They think that that’s programming.’
RZ Eyes rolling.
PF That kind of blew my mind, right, because it meant that the new framework for thinking about the world, for engineers, is like DevOps and deployment and building things and giant cloud frameworks and not learning some programming language really, really well. 
RZ And DevOps, in a big way, is equated oftentimes with AWS. 
PF I don’t know, I’m not the expert. But luckily, for once in our life, we went and got an expert for this program.
RZ We got the AWS guy?
PF Corey Quinn is here, perhaps the funniest person to attack AWS billing from every which way, but a world class expert on all things AWS. Corey, hi.
CQ Hi, I’m still waiting for the expert to join, let me know when they do.
PF Ah, don’t even do it. Don’t even do it. Can you explain your very, very specific role in the world to our audience?
CQ Sure, there are two of them really, and they play well together. But it’s hard to discern that from a distance. The first of course, is I help companies fix their horrifying AWS bill, a problem that comes for everyone in the fullness of time. Whereas in the common internet parlance, I spend an awful lot of time doing what the kids call shitposting on Twitter, and other various places, I write the Last Week in AWS newsletter that gathers the news from the AWS ecosystem, curates it, and then gently and lovingly makes fun of it. Because this stuff is pretty boring without a bit of a sense of humor. And if people want to be bored, there are official channels they can be bored through, I try and bring it to life. And then of course, I do a fair bit of podcasting myself to indulge my ongoing love affair with the sound of my own voice. So thank you for indulging me.
PF We’re all three of us indulging ourselves here.
RZ But before we get into it, why this Corey?
CQ It turns out that when you’re setting out independently as I was five years ago, and you’re trying to figure out, okay, how do I parlay my skill set into something that will put food on the table? Because you’re basically unemployable and your primary skill sets getting fired? What can you do that translates itself into something that’s easy to articulate in terms of business value, and this was the first thing I really tried. And it turns out, I sort of tripped over a goldmine in some respects. I came into this with a skill set that looked a lot like oh, I’m really good at cloud architecture. Great. But if I call myself that, I’ll be competing with every consultancy under the sun, and many of them have a larger budget for buying airport ads than I do. Whereas if you go to a bounded problem space, it doesn’t take much to, I guess, rise to the top of the mental SEO stack for whatever it is that you’re targeting. The more narrow the problem description, the likelier it is that people are going to see themselves in your description of what you do and reach out. We’re only now starting to explore with outbound sales work. We’ve been just sitting here waiting for the phone to ring for almost five years and have consistently failed to go out of business every month. So here we are!
RZ Isn’t it terrible that you exist? Not as a human being, but as a consultant.
CQ Well, some would say that especially as a human being. But yeah, in an ideal world, what I do shouldn’t exist. At least that was my theory at the beginning of it. I was worried that in six months, AWS was going to release a service or a feature that put me out of business. And now I’m worried they never will. Because there’s a lot of what I do that is not able to be solved via services. It’s, it comes down to understanding the bill, what drives the impact? Your bill shows up and it shows okay, your spending went up on storage, which team uses storage? Well, basically all of them, how do you attribute that back to teams? How do you predict it? And finally, have a consulting business do is on contract negotiation enterprise scale with AWS. Turns out that whenever you work at Amazon, you’re not going to be able to do a compelling story that even remotely touches oh, you’re spending the right money, because we say you are. Yeah, you’re the person writing the check too. A slight conflict of interest there.
RZ I assume some services that bury their $4.99 in my credit card bill, it’s almost part of their business model. Is it part of Amazon’s business model to put forward just a really shitty complicated invoice that nobody can decipher such that people like you exist now?
CQ From my perspective, it didn’t start off that way. And I don’t believe that was ever their stated intention. Because the idea was, ah, the whole point of utility computing is you only pay for what you use. In practice, you pay for what you forget to turn off. And since every bit of architecture that they put out there from a reference perspective involves basically a bunch of internal kickbacks, as best I can tell from however many service teams you can plug into your solution. There’s a whole bunch of stuff that ties together, the billing is not at all clear. And predicting how much I need these things is going to cost is almost ridiculous. I mean, it’s easy to sit here and say up, so I spend something up, turned it all down again, and now we’re just gonna pay 22 cents in perpetuity. But that doesn’t explain their ridiculous runaway revenue success. The problem really stems down to the idea that on demand billing is somehow transparent, fair and predictable, when in fact, it is in many ways, none of those things.
PF How can I tell if I need you?
CQ Generally speaking, are you spending a million dollars a month in AWS bill? Great, at that point, I promise we can help you. Below that point, it becomes a bit of a question mark. 
PF This is really about scale. Once you get to a certain scale, you’re using more and more the machine.
CQ It never gets smaller on its own, it requires concerted effort to be clear. Also, they show up with a ‘here, sign this contract’ that commits you to use a certain amount over the next one to five years. Yeah, give us a call. We’ve seen this stuff before. But you’re right. It’s hard for us to deliver anything that makes sense economically to folks who are spending $2,000 a month on the bill. But that’s what we’re spending give or take, although we just cut that in half somewhat recently. 
PF Cobblers children are wearing their shoes for once. 
CQ Oh, yeah. It was mostly a question of what are we actually spending money on, we should have a rough idea at our own account, if nothing else. And the problem we keep running into across the board is that there’s also a misunderstanding. If you’re a student, and you’re trying to learn how AWS works, and you expect to be in their free tier, which is basically the biggest lie in computing, then you’re shocked and offended by the fact that you just got a surprise $70 bill, because you forgot that there was one aspect of the billing dimension that wasn’t encapsulated by the free tier. And the entire net tells you oh, you should have known better, which is the most unhelpful statement ever. But yeah, you care about that. From a business perspective, we’re a 10 person company at the moment. And that $2,000 or $1,000 a month is not material. It’s a rounding error. We embezzle more than that an office supplies every month, because that’s the way businesses work. People are always more expensive than the technology that they’re working on. Unless you’re doing something with machine learning and data warehousing.
PF Should I be starting with AWS? Should I be building on AWS? Because at this point, I go and look at that set of logos. I feel like I’m looking at like, Unix in 1983. It’s just infinite and unparsible. 
CQ Good question. I think it comes down to what are your business goals? I know what it takes to want run WordPress super well. 10 years ago, I was a senior engineer at Media Temple and running WordPress at stupendous scale. These days, I have the good sense to pay WP Engine, so I don’t have to think about it. And yeah, could I run it for less infrastructure money? Of course. But I’m the person that would have to maintain this. It is far more cost effective for you to throw the problem to someone else. Now, if you’re going to go with one of the hyperscalers stories, okay, it’s hard to argue against Heroku for folks who were starting out. But if you’re looking at doing it, I want to build it myself from stringing popsicle sticks and lambda functions. And I want to pick a provider to do it. Well, that becomes a strategic decision. Google has a much gentler on ramp for Google Cloud. The Oracle Cloud free tier is actually free. Azure is not like other clouds, and AWS is what people historically have chosen. And some of those companies grew up to the point where they are the 800 pound gorilla in this space. From where I sit, I’m not particularly partisan toward any given cloud, it just comes down to where the expensive business problems are. If Azure somehow starts eating AWS as lunch across the board, yeah, I’m not gonna sit here waving my AWS billing flag. I’ll go with the expensive problems are.
RZ It’s worth noting also that this podcast is brought to you by DigitalOcean.
CQ They have indeed sponsored a bunch of my stuff as well which is always interesting to me when I get angry emails from people on the AWS sales side of ‘why aren’t you having this sponsored by someone who’s aligned with our go to market strategy?’ It’s because I don’t work for you, jackhole! [Paul laughs] It comes down to yeah, anyone who wants to is to sponsor my nonsense. Oracle Cloud sponsors one of my podcasts for which I thank them sincerely.
RZ I’m kidding. By the way. DigitalOcean is not is not sponsoring this podcast. Nobody’s sponsoring this.
CQ Yes. They’re not sponsoring this yet.
RZ This is true. They’re not sponsoring it yet.
PF This is brought to you by PHPwebhosting.com, which I still think of as the greatest host that ever existed. It was $10 a month non stop. That was it.
CQ This is perspective, I’d hate it because that’s the deal with a bunch of $10 a month customers.
RZ Let’s be generous, Corey. I sit somewhere in that, below that threshold to hire you and your expertise. And your your brilliant team. What’s the number one low hanging fruit as a piece of advice you can give to anyone, go look here first?
CQ Turn that shit off. If you’re not using it, turn it off, because you’re getting billed for it anyway. If you have a bunch of pile of data you haven’t looked at since 1998, perhaps it’s okay to either delete that or tear it off to something that’s a lot less expensive on a monthly basis. Figure out what’s in your environment, because often, if you have anything non trivial that’s been there more than a few months. There’s a lot of unknowns there of well, yeah, we have this thing once upon a time, where is it? And this might sound obvious, but it’s not. Start with the big numbers first. The AWS bill likes to come in alphabetically, that’s not the order you should tackle it. 
PF People get tired by that second page, you know, they’re just tired.

RZ Well, you know, here’s the thing–and again, I’m not assuming any hidden agenda other than if you unleash engineers, to handle the invoice processing, for AWS, you’re going to make more money. Because just by definition, things are going to get obfuscated, things are going to get confusing, and people just are, they’re tired. And they’d rather watch TV, or a streaming service of some kind, which should probably also be canceled. But that’s a separate conversation. So it’s not one service, it’s not one thing in the AWS ecosystem that they should go look at, you’re just saying turn it off?
CQ It depends on them, to be clear. Generally speaking, something like 60% of global spend, give or take is EC2 and things driving on top of EC2. Everything else sort of that–you have S3, you have RDS, you have data transfer, which is ever popular, and you have EBS their block storage device. Everything else beyond that is something of a rounding error most of the time, but if you’re trying to fix your own bill, I don’t care what the general statistic is, I care about what’s in yours, take a look at what you’ve got. Something I’d love to do more often with smaller scale bills is tear through them on a Twitch live stream or a Twitter thread and just basically eviscerate them of here’s what I would look at, here’s what I know about your architecture from your bill. That would be fun. But a lot of companies for obvious reasons, tend to view that stuff as being a little more confidential than shown to the entire public in real time. So I get that. But it is fun to play those games every once in a while. 
PF I want to say I would watch that. I would absolutely watch that. 
CQ Oh I’ve done it before. People will in fact watch that. 
PF Oh, yeah, no, no, that’s good content. What constitutes a surprise? Like, are there any sort of platforms out there where you’re like, oh my god, no, not Elasticsearch again, or whatever they’re calling it? Like, are there any where you just see people get snagged over and over?
CQ Data transfer, because it’s not well understood. Sometimes it’s free, people think it’s free in cases where it’s not. Other times, it just winds up adding up with strange data patterns. Where, okay, why are we taking in petabyte a month as a monitoring or telemetry shop, and then passing around 50 petabytes internally of data transfer? That seems odd. If that’s what you’re actually doing, perhaps consider storing multiple copies of the data and accessing it in each availability zone. I promise it’s less money. People don’t quite grasp that. And of course, AWS has on Native services. Oh, yeah. replication traffic between availability zones? Yeah, that’s free. Nothing third party can compete with that.
RZ Corey, what’s the name of your company?

CQ The Duckbill Group. Our mascot is a platypus with a bad attitude, which probably explains a lot.
RZ Boy, it does. 
PF There you have it. That’s the ultimate explanation.
RZ Let me run a fictional scenario by you. And then I want you to take the narrative from where I drop it off. 
CQ Sure…
RZ The Duckbill Group is acquired by the US government, and is now an agency and its job is to go into cloud hosting providers and simplify billing and provide better cost effective opportunities for everyone. What’s the first thing the Federal Duckbill Agency does?
PF Duckbill.gov.
CQ Probably step one, dies of old age waiting for the lawsuits to settle before we’re allowed to do anything. Two, back filling the entire staff who quits because we are not really set on working in very large, very bureaucratic environments because let’s face it, since when is saving money The Federal Government’s goal is almost any project except when it becomes politically feasible to play hot potato with it that month. If you were to somehow, though, asked me to wave the magic wand and fix cloud billing across the board for everyone, I would probably offer a secondary billing model for almost every cloud. And that is great, on a per resource basis, you will be charged a fixed rate per month for that resource, then easy to instance, because those are simple reason to reason about. Up to a relatively high limit, that includes data transfer that includes access to calls that it makes to things like S3 and the rest. And in practice, yes, you’ll wind up spending probably more on a per unit basis that you would otherwise. And I don’t think I know too many companies that wouldn’t jump at the chance just because it makes predictability so much more straightforward. In fact, I wouldn’t even have to guess at this.
RZ Package them up. 
CQ Oh, yeah, with GovCloud and the federal government, they have a billing model there that isn’t offered anywhere else, which is prepay, a fixed rate for a term and anything up to that rate you can use. Now the underlying reason is if you blow the budget, in federal government land, you can go to prison. So let’s make sure we don’t wind up having that particular failure mode happen for a lot of folks. But it lets people at that point waste money, but it also becomes more predictable. And that is what companies care about way more than the actual dollars and cents. Finance gets a surprise, the bill is 40% higher this month. Is that new? Do we have to redo all of our calculations going forward? Alternatively, do we have to sit here and figure out what this means for our unit economic models? Because we’ve started selling things to the market with a certain model in mind, what does this do to our margins? How do we wind up making sure that we’re spending the quote unquote, right amount of money on these things? How is that calculated out?
PF Where does Duckbill begin and end? Obviously, I can send you a PDF and say, here’s my million dollar a month AWS bill. Do you go into the systems and start to figure them out for them?
CQ We do.
PF Oh, okay.
CQ We’re on actually on GitHub, people don’t pay much attention to it, but it’s there. For our entire onboarding process, we start ingesting the cost and usage report, we run it on our site, there are a bunch of internal systems, many of which leading to Tableau. And we take a look during our engagement as far as where the spend is. On workload oby workload basis, we have a series of conversations with the engineering teams and various departments wherever we’re bounded to work within the customer environment. And at the end of it, we come back with what foundationally is a summary report of what we found, do this thing and save this much. And we go fairly into detail on this, depending upon what’s going on with the customer environment. Yeah, if you’ve spent 18 months refactoring this whole thing to a serverless approach, great good for you, we know you’re not going to do it, because no one does that. So all we’re going to gain by putting something like that in the report is make you feel bad. So we don’t suggest that. And we wind up aligning with what is likely to happen, the strategic constraints a company has. And when all is said and done, we come away with a not just a report that people are going to ignore, but something that is meaningful, impactful and likely to be implemented, based upon where companies are as far as their culture of cloud cost management spends.
PF What does AWS think of you?
CQ One of the things that I figured out pretty early on when I was writing a newsletter that made fun of them all the time, was that there is no central person who has AWS’s opinion. Some folks love me, some folks really wish I would go away. But by and large, the most common reaction because it’s a big company is that they haven’t heard of me before, which I think is just unforgivable. But in theory, and in practice both, there’s no one on the other side of what I do, which sounds a little strange. But they don’t want the narrative to be, oh, we’re just going to go ahead and waste all of our money on cloud, because that quickly turns into the clouds expensive. So don’t do it. We’re going back to data centers. They want you to spend more on cloud, but they want you to do it intelligently. So very often, they’re thrilled to have us involved. We’re not partnered with AWS, or any other company in this space, but we still get periodically referred into customers via their account manager.
RZ Before I shift gears for a second, would you take a job at Amazon?
CQ It would have to be the right job. So probably not. Let’s be clear here, at the scale that I’m at now, it’s not just me anymore. That looks like an acquisition. Because we have a bunch of folks here doing interesting things. I don’t think they’d want me to work there. To be perfectly honest with you.
RZ The 78th hyperlink, Paul Ford, there’s 77 hyperlinks for the 77 services and down the road, the 78th hyperlink is just going to be a duck.
CQ There’s over 230 now, to be clear. And it’s Duckbill with the Duckbill platypus.
PF It’s a little tiny platypus, but of course, they wouldn’t do it as platypus. It would just be like for some reason, a green triangle, no reason why. [Rich laughs]
CQ Someone’s paying attention to this current state of AWS graphic design.
PF But then the other options are like Azure’s world of mystery, every time I go in there, I’m like, okay, wow, you own notebooks!
CQ It’s a different customer profile and Azure. They’re inside the ecosystem of MCE types out there who are used to the Windows interface and you can used to contextualizing things through the Windows lens. I’m told that the Azure dashboard is much more resonant with folks who have that background. I’m hoping it’s true. Because if not, their lost. 

PF No, I think that’s right. I mean, you’re in the world there of like, you got Azure certified. And you are this whole sort of like, Microsoft narrative of what it in tech is supposed to be. And then Google like, I’m sort of fond of because it is just that big old menu, and you’re like, well. And they tried to name the things, but they can’t keep up with their own cloud offering. But you know, AWS, you just get that that infinite menu. 
CQ Google is probably most impressive onboarding I’ve seen yet, just because, again, not compared to DigitalOcean. Few things can compare, because they’re terrific at getting you up and running. But it’s also a much more streamlined, straightforward offering.
PF Sell ya a mattress too!
CQ Oh yeah! Google Cloud, on the other hand, they’re a lot more in depth with a lot of their offerings, but they also include batteries a lot of the time where you can start out using it and develop and get something up and running in a reasonable set of defaults. And it’s clear as well that their various service teams in the console are allowed to talk to each other. So you’re not seeing this variety of different eras of design language, like you are with AWS, the company ship their culture. And it’s very clear that AWS has a microservices culture, where they’re not allowed to talk to each other except through API’s. So anytime you see something that requires universal coordination, such as monitoring, tagging, the bill, a design language, it’s one of the seams between their various service teams, and the experience is subpar at best.
RZ Isn’t a big bill, just a byproduct of bad code?
CQ Not always it could be a byproduct of an awful lot of customers. More commonly, it’s a byproduct of an awful lot of engineers. 
RZ Zing.
CQ It’s not entirely a joke, either to be very clear with you.
RZ I mean, a lot of engineers usually leads to inefficiency usually leads to a lack of conformity to an overall approach. People stray.
PF Well, then they solve that with more code.
RZ They solve it with more code. Does your report oftentimes highlight that you’re just doing certain things programmatically in a terribly inefficient way?
CQ It depends, because no one shows up expecting to do a terrible job at work today, unless they work in the Facebook ethics department. So the concern that we have is one of those great, we see something that is inefficient. We don’t surprise people with that in a report. Remember, we have ongoing conversations with service teams as we’re doing this, why is it being done this way? What is the constraint that we are missing? What is the use case here? Is there a reason that doing it differently is non tenable for you. It’s not something that lends itself to a software model, where it just comes back as a report full of shame, it’s figure out why it’s something is the way that it is very often it’s that people didn’t know at the time they built it, there was a different way of doing it. In many cases, there was no other way of doing it at the time when this was built. And then as services become more enhanced, and more featureful, suddenly, other opportunities arise. And no one keeps up with all this stuff. It’s our profound belief that architecture and cost are the same thing just expressed through different terms. 
RZ And let me ask it in a more generous way. Should architects and engineers be thinking about the AWS bill, when they’re thinking through an approach to solving something?
CQ They should be aware of certain contexts, and that’s generally it. They should be aware, generally speaking, that big things cost more than small things, that things will not turn themselves off on their own. Data transfers is nuanced, and anything you store will be charged by the gigabyte per month in perpetuity. And that’s really about it. Because you don’t want engineers thinking about, well, what are the economics of this look like? Instead, you want them focusing on what’s plausible.
PF They want to think about that more than anything in the world, there’s a certain kind of engineer. 
CQ There’s a certain kind of engineer that if you’re not careful, they’ll spend three months of their time knocking 400 bucks off of their monthly AWS developer environment. They cost more than that, you probably don’t want them spending that energy.
RZ CPU, network transfer, and storage, which which should they worry about the least? And should we worry about the CPU?
CQ It depends. It’s always going to be workload dependent. If you’re doing a big data project, then you might want to care about storage. If you’re doing something that involves you taking an awful lot of data from one place to another, data transfer is going to be the big one for you. If we’re talking about an area of compute, well, okay, it’s easy to look at metrics and say, oh, yeah, clearly, these are overpowered idle instances that should be turned off. But if you actually do that, you’re going to cause problems. There are things you can’t see externally such as memory usage. It depends on what the workload is and what the business requirements are around them. Take Lyft, for example. When they went public, it was announced that they were spending over $100 million a year on AWS and the internet lost its collective mind. Because that’s a ridiculous amount of money and you could build a data center for less than that on and on and on and on. First, that is not true. Secondly, let’s back up a second here, they’re losing at the time $900 million a year. If their AWS bill doubles or drops to zero, it does not materially change the trajectory of their business. So why do people care so much? They care because it’s a single big item on a report that doesn’t just go away, you can’t play financial tricks the way you can by spreading across multiple vendors in a data center. It’s pure optics. And it shows up on a consistent basis every month, or every time it’s a contract renewal depending upon who you are. And that becomes something that is very hard to ignore, hard to reason about, in the same way. People were wasting money in data centers long before now, let’s be clear here. But who’s going to be able to tie together all of the staffing costs and the real estate lease and all of the build out costs and what the refresh on the servers is, and the data transfer agreements with your bandwidth providers, and so on, and so on, and so on. This is just one provider, one line, and there’s no hiding,
PF You look inside a lot of big orgs. Right? I mean, to me, the AWS answer to why is this expensive is, well, here, we have all these other wonderful things like lambdas, that will be incredibly efficient, we’ve actually given you a beautiful path towards greater efficiency, fewer instances, so on and so forth. Engineers at Postlight, they love a good lambda. They’re very motivated. But big orgs, I just wonder, do they make any sense of that? Is that even possible?
CQ Some of them do, absolutely. But it’s hard to think of a bigger example of a failed opportunity than lambda. It’s still incredibly complex, it still takes a long time to reason around. And for many things that it’s being used for yet it offers a bunch of options and functionality and the rest. But you need to go to cloud school for a while to figure out how these things work and figure out what failure modes look like and the rest. Whereas for relatively small stuff, you can just have a scheduled job firing off on a server somewhere and call it just as good without having to spend all of that time and energy. It has not gotten substantially easier since it was announced. We have not seen a plethora of architectures out there. And when you do and for example, my newsletter publication pipeline features over 30 lambdas at this point, but no one else can reason about this thing other than me, and it’s there primarily as a technology testbed. If I were having other people who were going to be working on this, I would replace it with something else tomorrow before handing it over.
PF Why do you need 30 lambdas to publish a newsletter? You hit MailChimp, you hit the button? 
RZ Ouch.
CQ Oh, yeah, but how do you curate? Let’s say the average of 400 links a week that I wind up seeing, that wind up getting belted into this system. How do you wind up adding the sarcasm and the snark and the tone? How do you wind up separating out the sponsored links from the rest of them where other people can put those in? So I can review it before it goes out? But what at the time that I’m writing the thing, it doesn’t wind up demonstrating any it doesn’t wind up showing any of this? How can I wind up scheduling something out for a few issues from now with all of those things? How can I make sure that I don’t inadvertently send something out that hasn’t been finalized with a bunch of linter checks tied into it as well? And so on and so forth? How can I categorize all of those links via some of the higher level machine learning powered sentiment analysis so that I can automatically assign tags to specific services? So I can look back later and see what did well? What didn’t? How do you build a custom link tracker that lets you look in the aggregate over a quarter or a year and say, these are the most popular links for roundup issues. As you start diving into this, it gets more and more complex and more and more complicated. And that’s what I built out. I wouldn’t recommend this to other people. But it gives me a deep level of insight.
PF This is your thing.
RZ Can I spin up a conspiracy theory right now?
CQ Please. 
RZ Duckbill is Amazon. And Corey is employed by Amazon.

CQ I smack them off the heart for their labor practices to be an active employee. Let’s be clear on that. 
PF Yeah, that’s true. They take that–you can come after the billing all you want. But if you’re like, hey, what about unionization? Then they’re not going to enjoy that as much.
CQ My point is the same way I talk to my sponsors as well. And the same reason that I don’t have any partnerships at all in the space, you can buy my attention, but not my opinion. And that authenticity with the audience is everything. Because if I were just sitting here slinging mud, then it would be a lot easier for them to ignore me. Whereas I’m right. And when they do something right, I will publicly defend them on it. Whereas when they do something wrong, I will publicly dragging them for it. It’s one of those situations where developing a reputation for being authentic and independent, is everything. I didn’t mean to become almost a media figure inside of the AWS ecosystem that happened almost by chance. I’m gratified that people find what I have to say so compelling that they started paying attention in other respects. But where it all came from is just mostly a willingness to be wrong and a willingness to be dumb of I don’t understand how this thing is supposed to make any sense for customers because it makes no sense to me. Can you help explain it to me? And increasingly I found the answer was no, we’re just sort of marketing hand waving ourselves our hands over it and hoping that customers will explain it to us and oh, so you’re basically advocating your job. Great. Good to know.
PF Are the other platforms, are their billing as bad?
CQ It’s still nuanced. It’s still challenged. To understand where it starts and where it stops, they have less service sprawl. So it’s more intuitive. In some respects, when you’re not the clear number one in the space, it’s a lot easier to get things fixed by calling up your rep and yelling at them. But it also is not something that we are seeing as far the same level of spend. Yes, there are multi 100 million dollar a year spenders out there on other cloud providers, but not nearly with the frequency that we see them on AWS, when I was trying to figure out what to focus on everyone I talked to had an AWS billing problem, I only see it occasionally with other providers. So the problems are still there. Let’s be very clear here. I’ve been saying for a while someone should basically rip off what I’ve done, I’ll tell them how to do it. I have in Twitter threads and podcasts before for other cloud providers, the world needs it. But no one’s taking me up on that yet.
RZ Which leads me to my closing question here. If I am a startup, and I happen to know Corey and run into him at a cocktail party, which cloud hosting provider should I use? 
CQ This is the best part. And this is my actual long term goal for this industry. In five or 10 years, when I start a startup, I don’t want that to be an easy question. I want there to be multiple answers of equal validity and weight. My answer today is AWS for me. And the reason behind that is it’s because it’s the cloud I have the most experience with, it’s the cloud that I know best. I can Google how to do basically anything on AWS and see an entire community out there because their Doc’s are hit or miss at the best of times, and other people will absolutely work with me on it. If I’m working with a different cloud, I don’t have that same level of assurance. I know I can hire people who have expertise within AWS, I know that there is very little new ground I would be having to tread as a startup, tying those things together. Now, let’s be clear, if I wanted to start a startup today, I might completely avoid cloud entirely depending upon what it is. Start off using a bunch of higher level services and just tie them together with someone of the no code, low code solutions. Because I don’t want to run my own database. I just want to give someone or something, a pile of information and have them give it back to me when I asked for it. What is running under the hood? Could not possibly care less. My WP Engine site, the powers last week in aws.com runs on top of Google Cloud. And I have zero problem with that in the slightest, because I don’t care. I just don’t care. It does not help advance my business one iota to have to think about what’s under the hood. That is the future of cloud. And which cloud provider do I pick? It’s how do I build my application? How do I structure something that is completely greenfield in such a way that I can focus on the thing that are likelier to get me to my next milestone? Because I promise you the cloud bill is not that thing.
PF Get ready for these three words together, ruining our lives for the next 10 years enterprise low code. That’s where we’re headed. 
RZ Why would you do that?
PF Because you can’t just have AirTable, because that’s not going to be enterprise enough.
RZ Are we heading to more and more layers of abstraction, such that we’re not complaining about the AWS console anymore? 
CQ Oh, I expect. So for my entire newsletter pipeline system, for example, I am dogshit at the front end. I was about to cut a $20,000 check to have some front end developer build the whole thing for me. And then I stumbled across a low code solution, retool, as it turns out, so suddenly, it’s drag and drop things together for internal tools, just what this is, it doesn’t ever need to see the light of day outside of my nonsense, and ties together a bunch of API’s that I’d already had built, and we’re ready to go. And all I have to do to change the behavior is address what the API does when you click a button, or I can drag a table in or add a text field that then is usable in the space. It’s way more effective for what I do and how. I’m looking forward to the future where an MVP can be constructed like that of almost anything. Because look at the sheer number of apps out there that are doing very well, that are functionally crud apps, were all ours, here’s a database, do different things with the data in your database, spit it back at the right time. There’s nothing wrong with doing that. I don’t think you should have to understand what the hell a dynamodb is for that to make sense. I don’t think you need to understand in any meaningful way what the AWS data transfer billing looks like, I want something that’s going to just package all of that together and make it go. And as far as it being enterprisey, well, great. They do have enterprise customers and that looks like from their perspective is they give you a self hosted version that runs in containers. But the days of enterprise configuration and enterprise user experience stemmed from the time when procurement were the folks deciding what was going to be used and what aren’t. Steve O’Grady 10 years ago wrote about this in his book, The New Kingmakers talking about how developers are in fact, the folks that are driving a lot of the software experience. I look at a lot of the enterprise grade software that companies are adopting now. It doesn’t look that bad compared to consumer grade stuff. Because users are demanding an experience that doesn’t require you send the entire team to training for two weeks to figure it out. The world’s changing and old enterprise software and new enterprise software don’t look the same.
RZ What it really is, is layers that hide complexity, hide how the different pieces glue together and talk to each other. And just make it easy to get going. Is that where the bill is? Because even if I use a tool like retool, I still got to deal with those bills, I still got to go sign up in the four different places. 
CQ Sure. I mean, again, this is what it’s using for my underlying systems. Understand that on a monthly basis, my entire pipeline production system that writes this newsletter, cost me something on the order of $7 a month. Five bucks of that is the AI powered thing that was once upon a time AWS comprehend and increasingly is Algolia, because it’s better. And there’s a 50 cent charge, that’s my route 53 zone for this thing, because I use DNS as a database because of severe a deep seated personality problems. But everything else is pennies a month. And even if it were 10s or hundreds of dollars a month, I still wouldn’t give a shit. Because the value that I’m getting from it as a business far outstrips the cost what it cost me to run the thing. In fact, that’s not strictly true. I am paying $50 per month per user with Retool. And that is, that’s not nothing. But its internal users not external, to be clear.
RZ Still, but that’s incredible, right? I mean, the tool far eclipses the costs of the actual platform. 
CQ Oh, I was also paying a flat rate of $120 a month until last week for Epsagon to monitor all these things and tell me when something was breaking, because God forbid you trust internal AWS tools. I’m now auditioning for replacements having ripped Epsagon out. Great folks, great product acquired by Cisco. And I’m getting out before Cisco pulls a Cisco and absolutely ruins the thing that they’ve just acquired. They have gotten that down to a science. [Rich & Paul laugh]
PF Yeah. Are the clouds a sort of intermediate? To me, I look at the clouds as like, well, they blew up the Linux server, and they glued it back together out of pieces. It looks like this. Great, it’s scalable, it’s good. Are we this sort of caught between the two revolutions? like are we just kind of like, you know, is it headed in that direction? Where it’s like, Look, here it is. It’s blocky, it’s modular, get to market fast. It’s low code, you can go under the hood if you want, but I wouldn’t bother. There’s like 250 services under there. Is that where we’re going? Are we going, am I gonna log in to AWS two years from now and there’s 500 tiny icons?
CQ That’s what’s scary. That is the direction they’re heading in. And there’s no guidance, there’s no best practice. There’s no established golden path of these are the modern services that we think you should build on if you’re starting today. And sure you still have the other stuff that exists. But what do you do with it? How do you wind up making it discoverable? You can still use simple dB, which Andy Jassy is on record to a journalist describing as a failed product, which I’m sure it feels terrific to that service team. But you can still use it in new accounts.
PF I wonder when they shut off AWS, my guess would be like the year 2200.
CQ We’ll be long dead. Because if they shut it off today and said, we’re ending we’re end of life in the entire platform, which would be just fantastically foolish of them. It’s an eight to 10 year project to get some of the larger customers off, because it’s taken longer than that to get on. And that is just one of those stories that I can’t really imagine it what as far likelier and I would bet money on happening is a spin off of AWS separately. 
PF You think it’s coming? 
CQ I wish they would. If it’s going to happen sooner or later. The question is, is it their choice or the governments?

PF Alright, Rich, do you have anything else? I feel that I’ve learned a lot.
RZ No, this was a great conversation actually.
PF Actually? Actually? This was a great conversation!
RZ We meandered out of AWS and into how large orgs try to wrestle with getting things done efficiently, and then deal with the consequences of getting things up and running. It is a classic problem. It’s fascinating to see, it’s been the same problem. I mean, it used to be Rackspace. And it used to be self hosting. And it used to be a lot of different things. Tech gets away from us. We’re ambitious, and we underestimated. And we always do that. And then we always like what the heck just happened. I mean, you exist Corey, Duckbill exists, because tech gets away from us right? And it will continue to get away from us. I can’t wait for the small low code optimization consultancy that will come to life in 10 years, that will bring order out of all the main the madness around all the low code stuff that your–
PF Someone’s gonna fix your AirTable billing within the next five years. Right?
RZ But this was really great, Corey, thank you so much for coming on the Postlight Podcast.
CQ No, thank you for having me. It’s always a pleasure to once again indulge my love affair with the sound of my own voice. 
PF Excellent!
RZ How do people stay connected to you Corey?
CQ The easiest way if they’re not following me on Twitter, which I recommend, no one does. But @QuinnyPig, is to sign up for my ridiculous newsletter lastweekinaws.com. It is exactly what you would expect from someone with my personality defects and ridiculous excuse for a sense of humor.
RZ Excellent. Great.
PF Whenever you receive an email, it’s at the end of like 37 separate lambda functions. That’s just something to know too. [Rich laughs] Rich, tell people how to get in touch.
RZ This is the Postlight Podcast. We are Postlight. We’re a digital product studio, design, engineering, software development. Based in New York, but really just about everywhere now. Check out our case studies at postlight.com and reach out. We’d love to talk [email protected]. Have a wonderful week. Thank you again, Corey.
CQ Thank you for having me, as always.
PF Thank you kindly!
[music ramps up, plays alone, ends]