Fully Automated MySQL slow log analysis on Amazon RDS


At Memonic we rely on MySQL for most of our data storage. In any relational database system the correct creation of indices is important, otherwise queries will be inefficient and slow. The problem with that is, that the indices often are forgotten, especially when updating an existing query. As a tool to detect queries without proper indices MySQL offers the slow query log. All queries that take more than a certain time are logged there.

We host our platform in Amazon’s cloud. For database we rely on their on their Relational Database Service (RDS) service. As we don’t have root access to those machines we can’t just tail the slow log to see what’s up. Instead RDS optionally writes the slow log into a special system table. From there a query can be used to retrieve the data. See the Amazon RDS FAQ about how to configure the slow log on RDS.

For automated analysis of the slow logs we like to use mk-query-digest. This excellent utility groups all logged queries together by their general type and thus allows a big-picture overview. As an example take these three queries that may have been logged:

SELECT * FROM events WHERE user_id = 'foo';
SELECT * FROM data WHERE created_at >= '2011-12-13 10:12:00';
SELECT * FROM events WHERE user_id = 'bar';

These will be grouped together by mk-query-digest as just two queries:

SELECT * FROM events WHERE user_id = '?';
SELECT * FROM data WHERE created_at >= '?';

This is accompanied with how often each query type was executed, how long it took in total, etc. This is a great way to focus any optimization effort first on the queries that are actually used a lot.

Unfortunately mk-query-digest only works with the normal MySQL slow query log format and can’t access the proprietary table that Amazon RDS keeps. To work around this, we wrote the script which we hereby release into the public domain.

#!/usr/bin/env python
Queries the slowlog database table maintained by Amazon RDS and outputs it in
the normal MySQL slow log text format.

import _mysql

db = _mysql.connect(db="mysql", read_default_file="/root/.my.cnf")
db.query("""SELECT * FROM slow_log ORDER BY start_time""")
r = db.use_result()

print """/usr/sbin/mysqld, Version: 5.1.49-3-log ((Debian)). started with:
Tcp port: 3306 Unix socket: /var/run/mysqld/mysqld.sock
Time Id Command Argument

while True:
    results = r.fetch_row(maxrows=100, how=1)
    if not results:

    for row in results:
        row['year'] = row['start_time'][2:4]
        row['month'] = row['start_time'][5:7]
        row['day'] = row['start_time'][8:10]
        row['time'] = row['start_time'][11:]

        hours = int(row['query_time'][0:2])
        minutes = int(row['query_time'][3:5])
        seconds = int(row['query_time'][6:8])
        row['query_time_f'] = hours * 3600 + minutes * 60 + seconds

        hours = int(row['lock_time'][0:2])
        minutes = int(row['lock_time'][3:5])
        seconds = int(row['lock_time'][6:8])
        row['lock_time_f'] = hours * 3600 + minutes * 60 + seconds

        if not row['sql_text'].endswith(';'):
            row['sql_text'] += ';'

        print '# Time: {year}{month}{day} {time}'.format(**row)
        print '# User@Host: {user_host}'.format(**row)
        print '# Query_time: {query_time_f} Lock_time: {lock_time_f} Rows_sent: {rows_sent} Rows_examined: {rows_examined}'.format(**row)
        print 'use {db};'.format(**row)
        print row['sql_text']

view raw This Gist brought to you by GitHub.

It simply dumps the current contents of the RDS slow_log table in the same format that MySQL usually uses for their slow log file. This output can then be piped into mk-query-digest to generate a report.

We ended up doing just that in a daily cron job which sends a mail to our developers.

(/usr/local/bin/db2log | \
    mk-query-digest --fingerprints \
        --filter '$event->{user} !~ m/^(bi|memonic)$/') 2>&1 | \
        mail -s "MySQL slow logs" root

# Rotate slow logs. Will move them into the backup table slow_log_backup. If
# that table exists it's overwritten with the primary slow log.
# So with this strategy we can still access yesterday's slow log by querying
# slow_log_backup.
mysql mysql -e 'CALL rds_rotate_slow_log'
view raw This Gist brought to you by GitHub.

There is one line which needs further explanation: the filter. We filter out any slow log events that were triggered by either the bi or the memonic users. The former is used for asynchronous generation of some statistics and performance isn’t required for that. The latter we use for ad-hoc queries which we don’t need to optimize.

So there you have it: an automated mechanism to analyze slow MySQL queries. From time to time when deploying a new release a new slow query may pop up. But the next day we are informed about it and can fix the issue.

Amazon SQS vs. RabbitMQ » NSONO.NET


Jul 012011

We’ve been using Amazon SQS where I work for awhile. We have a fairly heavy (though, that’s relative: we’re a small company) cloud application that makes use of a bunch of the Amazon services (SimpleDB, S3, EC2) and, when we needed a message queue, SQS was just there for us. It was convenient, simple, and reasonably quick to code at.

Our application has now grown to a point where ‘convenient’ doesn’t quite cut it anymore. Performance and cost are starting to seriously matter – and SQS is a pretty serious point of pain. So, after some searching for alternatives: RabbitMQ to the rescue.

Here’s the core problem: with Amazon SQS, the message consumer is forced to poll the queue in order to determine whether or not a message is available. If a message is available, that’s great, the consumer is happy and receives the message. If not, the consumer must idle for awhile, and then poll again.

In our use case, we face relatively long periods in which queues are idle (no messages) followed by bursts of activity that are time-critical (well, ok, not critical, but time-sensitive). Worse, we have a chain of services with queues sitting in between them: one service forwards a message to another, which produces new work for other services, and on… Obviously, the amount of time that a message spends on the queue waiting for a consumer to wake up and poll matters. The latencies multiply out when you have a chain of services, like we do. So, fine: poll like crazy, right? Constantly, and with a very low delay?

Well, no: SQS charges for every request made. 1 million requests = $1. Sounds OK, but multiply out the number of queues you’re polling and the number of consumers doing the polling, and you can get some serious $$$ by polling like crazy on empty queues. 10 consumers polling once every 250ms for 1 month (31 days) = 107 million requests = $107. The solution that we had been relying on was to use a simple back-off algorithm: if a consumer is receiving messages, continue polling like crazy, but every time we don’t receive a message from the queue, back off a bit – slow the polling down, up to a certain threshold (10 seconds for us). That’s fine and all, but then you’re introducing up to 10 seconds of latency at each point in the chain.

So, fine then, we need something new: RabbitMQ.

RabbitMQ is a message queue system based on Erlang and conforming to AMQP (a standard and heavily used message queue protocol).  I’ve really only scratched the surface with it (we’re still in the process of running tests to make sure that RabbitMQ really does what we need), but I like what I’ve seen so far.


  1. It’s fast
    • It’s built for telcos and SMS switching and all kinds of serious heavy-load craziness that there’s just no way SQS could handle. It supports clustering. It supports subscriptions (RabbitMQ will deliver the message to your consumer immediately instead of forcing consumers to poll).  The underlying resources are under your control (no competing with noisy neighbours, as with Amazon).
  2. It’s free
    • RabbitMQ is open source. All you’re paying for is the instance that runs it. There’s no request charge, so you can poll as frequently as you want. If your clients subscribe to queues, then there’s not even a need to poll. RabbitMQ will just tell the client when a message is available!
  3. It supports all kinds of crazy message patterns
    • For right now, this isn’t particularly important to us, but RabbitMQ supports a bunch of different message passing patterns: one-to-one, one-to-many, many-to-many, RPC, complex routing, topics, etc..
  4. It supports ‘durable’ messaging
    • It was particularly important to us that a crash of RabbitMQ (or the EC2 instance) not lose the contents of the message queues. RabbitMQ supports both durable and transient messaging. If all you care about is lightning fast performance, set durable=false on the queue. If you care about crash recovery, set durable=true. There’s a performance penalty there, though, because each queued message needs to be written to disk.
  5. It supports true FIFO message ordering
    • I wrote a ton of code to deal with SQS’s ‘more or less in order, sort of’ message ordering. With RabbitMQ, the only time a message will be received out-of-order is if the message is requeued for one reason or another (of course, it depends on your configuration).
  6. It is consistent
    • Eventual consistency can be a real problem with the Amazon services.  It is not infrequent to have the same message delivered multiple times to the same client (or multiple times to multiple clients).  With RabbitMQ, most of that pain is gone.  RabbitMQ does not guarantee exactly-once delivery, but more-than-once delivery will really only occur if the first message delivery fails (is not acknowledged by the client), or if the message is resubmitted.
  7. Easy to set up, easy to swap in
    • It’s taken me about a day to set up RabbitMQ, swap out our old SQS code and replace it with RabbitMQ code. It’s a hack job for now, but it’s 100% functional and gives us an environment with which to test out RabbitMQ.


  1. High-availability.
    • We haven’t quite gotten there yet, but: making RabbitMQ highly available is up to you. With SQS, high availability comes along for free.
    • Additionally, making RabbitMQ highly available seems a bit tricky. RabbitMQ’s clustering is built for performance, not availability. There is, from what I’ve seen, no built-in support for redundancy and failover. You kind of have to glue it all together yourself.
  2. Worryingly large number of cases in which a message can disappear into vapor.
    • Watch out: there seem to be a number of ways to configure RabbitMQ where, if a consumer is not present or a queue is not present or an exchange is not present or whatever, your message will just vanish. Vaporize. I’m just in the process of testing out our migration, and this is one thing that I still need to spend some time on: it’s not acceptable for a message to vaporize, ever, at all, for us. With SQS, when a message is published, Amazon guarantees that it will be delivered at least once (provided a client eventually polls the queue).

Rip Rowan - Google+ - Stevey's Google Platforms Rant I was at Amazon for about…

Stevey's Google Platforms Rant

I was at Amazon for about six and a half years, and now I've been at Google for that long. One thing that struck me immediately about the two companies -- an impression that has been reinforced almost daily -- is that Amazon does everything wrong, and Google does everything right. Sure, it's a sweeping generalization, but a surprisingly accurate one. It's pretty crazy. There are probably a hundred or even two hundred different ways you can compare the two companies, and Google is superior in all but three of them, if I recall correctly. I actually did a spreadsheet at one point but Legal wouldn't let me show it to anyone, even though recruiting loved it.

I mean, just to give you a very brief taste: Amazon's recruiting process is fundamentally flawed by having teams hire for themselves, so their hiring bar is incredibly inconsistent across teams, despite various efforts they've made to level it out. And their operations are a mess; they don't really have SREs and they make engineers pretty much do everything, which leaves almost no time for coding - though again this varies by group, so it's luck of the draw. They don't give a single shit about charity or helping the needy or community contributions or anything like that. Never comes up there, except maybe to laugh about it. Their facilities are dirt-smeared cube farms without a dime spent on decor or common meeting areas. Their pay and benefits suck, although much less so lately due to local competition from Google and Facebook. But they don't have any of our perks or extras -- they just try to match the offer-letter numbers, and that's the end of it. Their code base is a disaster, with no engineering standards whatsoever except what individual teams choose to put in place.

To be fair, they do have a nice versioned-library system that we really ought to emulate, and a nice publish-subscribe system that we also have no equivalent for. But for the most part they just have a bunch of crappy tools that read and write state machine information into relational databases. We wouldn't take most of it even if it were free.

I think the pubsub system and their library-shelf system were two out of the grand total of three things Amazon does better than google.

I guess you could make an argument that their bias for launching early and iterating like mad is also something they do well, but you can argue it either way. They prioritize launching early over everything else, including retention and engineering discipline and a bunch of other stuff that turns out to matter in the long run. So even though it's given them some competitive advantages in the marketplace, it's created enough other problems to make it something less than a slam-dunk.

But there's one thing they do really really well that pretty much makes up for ALL of their political, philosophical and technical screw-ups.

Jeff Bezos is an infamous micro-manager. He micro-manages every single pixel of Amazon's retail site. He hired Larry Tesler, Apple's Chief Scientist and probably the very most famous and respected human-computer interaction expert in the entire world, and then ignored every goddamn thing Larry said for three years until Larry finally -- wisely -- left the company. Larry would do these big usability studies and demonstrate beyond any shred of doubt that nobody can understand that frigging website, but Bezos just couldn't let go of those pixels, all those millions of semantics-packed pixels on the landing page. They were like millions of his own precious children. So they're all still there, and Larry is not.

Micro-managing isn't that third thing that Amazon does better than us, by the way. I mean, yeah, they micro-manage really well, but I wouldn't list it as a strength or anything. I'm just trying to set the context here, to help you understand what happened. We're talking about a guy who in all seriousness has said on many public occasions that people should be paying him to work at Amazon. He hands out little yellow stickies with his name on them, reminding people "who runs the company" when they disagree with him. The guy is a regular... well, Steve Jobs, I guess. Except without the fashion or design sense. Bezos is super smart; don't get me wrong. He just makes ordinary control freaks look like stoned hippies.

So one day Jeff Bezos issued a mandate. He's doing that all the time, of course, and people scramble like ants being pounded with a rubber mallet whenever it happens. But on one occasion -- back around 2002 I think, plus or minus a year -- he issued a mandate that was so out there, so huge and eye-bulgingly ponderous, that it made all of his other mandates look like unsolicited peer bonuses.

His Big Mandate went something along these lines:

1) All teams will henceforth expose their data and functionality through service interfaces.

2) Teams must communicate with each other through these interfaces.

3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team's data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.

4) It doesn't matter what technology they use. HTTP, Corba, Pubsub, custom protocols -- doesn't matter. Bezos doesn't care.

5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

6) Anyone who doesn't do this will be fired.

7) Thank you; have a nice day!

Ha, ha! You 150-odd ex-Amazon folks here will of course realize immediately that #7 was a little joke I threw in, because Bezos most definitely does not give a shit about your day.

#6, however, was quite real, so people went to work. Bezos assigned a couple of Chief Bulldogs to oversee the effort and ensure forward progress, headed up by Uber-Chief Bear Bulldog Rick Dalzell. Rick is an ex-Armgy Ranger, West Point Academy graduate, ex-boxer, ex-Chief Torturer slash CIO at Wal*Mart, and is a big genial scary man who used the word "hardened interface" a lot. Rick was a walking, talking hardened interface himself, so needless to say, everyone made LOTS of forward progress and made sure Rick knew about it.

Over the next couple of years, Amazon transformed internally into a service-oriented architecture. They learned a tremendous amount while effecting this transformation. There was lots of existing documentation and lore about SOAs, but at Amazon's vast scale it was about as useful as telling Indiana Jones to look both ways before crossing the street. Amazon's dev staff made a lot of discoveries along the way. A teeny tiny sampling of these discoveries included:

- pager escalation gets way harder, because a ticket might bounce through 20 service calls before the real owner is identified. If each bounce goes through a team with a 15-minute response time, it can be hours before the right team finally finds out, unless you build a lot of scaffolding and metrics and reporting.

- every single one of your peer teams suddenly becomes a potential DOS attacker. Nobody can make any real forward progress until very serious quotas and throttling are put in place in every single service.

- monitoring and QA are the same thing. You'd never think so until you try doing a big SOA. But when your service says "oh yes, I'm fine", it may well be the case that the only thing still functioning in the server is the little component that knows how to say "I'm fine, roger roger, over and out" in a cheery droid voice. In order to tell whether the service is actually responding, you have to make individual calls. The problem continues recursively until your monitoring is doing comprehensive semantics checking of your entire range of services and data, at which point it's indistinguishable from automated QA. So they're a continuum.

- if you have hundreds of services, and your code MUST communicate with other groups' code via these services, then you won't be able to find any of them without a service-discovery mechanism. And you can't have that without a service registration mechanism, which itself is another service. So Amazon has a universal service registry where you can find out reflectively (programmatically) about every service, what its APIs are, and also whether it is currently up, and where.

- debugging problems with someone else's code gets a LOT harder, and is basically impossible unless there is a universal standard way to run every service in a debuggable sandbox.

That's just a very small sample. There are dozens, maybe hundreds of individual learnings like these that Amazon had to discover organically. There were a lot of wacky ones around externalizing services, but not as many as you might think. Organizing into services taught teams not to trust each other in most of the same ways they're not supposed to trust external developers.

This effort was still underway when I left to join Google in mid-2005, but it was pretty far advanced. From the time Bezos issued his edict through the time I left, Amazon had transformed culturally into a company that thinks about everything in a services-first fashion. It is now fundamental to how they approach all designs, including internal designs for stuff that might never see the light of day externally.

At this point they don't even do it out of fear of being fired. I mean, they're still afraid of that; it's pretty much part of daily life there, working for the Dread Pirate Bezos and all. But they do services because they've come to understand that it's the Right Thing. There are without question pros and cons to the SOA approach, and some of the cons are pretty long. But overall it's the right thing because SOA-driven design enables Platforms.

That's what Bezos was up to with his edict, of course. He didn't (and doesn't) care even a tiny bit about the well-being of the teams, nor about what technologies they use, nor in fact any detail whatsoever about how they go about their business unless they happen to be screwing up. But Bezos realized long before the vast majority of Amazonians that Amazon needs to be a platform.

You wouldn't really think that an online bookstore needs to be an extensible, programmable platform. Would you?

Well, the first big thing Bezos realized is that the infrastructure they'd built for selling and shipping books and sundry could be transformed an excellent repurposable computing platform. So now they have the Amazon Elastic Compute Cloud, and the Amazon Elastic MapReduce, and the Amazon Relational Database Service, and a whole passel' o' other services browsable at These services host the backends for some pretty successful companies, reddit being my personal favorite of the bunch.

The other big realization he had was that he can't always build the right thing. I think Larry Tesler might have struck some kind of chord in Bezos when he said his mom couldn't use the goddamn website. It's not even super clear whose mom he was talking about, and doesn't really matter, because nobody's mom can use the goddamn website. In fact I myself find the website disturbingly daunting, and I worked there for over half a decade. I've just learned to kinda defocus my eyes and concentrate on the million or so pixels near the center of the page above the fold.

I'm not really sure how Bezos came to this realization -- the insight that he can't build one product and have it be right for everyone. But it doesn't matter, because he gets it. There's actually a formal name for this phenomenon. It's called Accessibility, and it's the most important thing in the computing world.

The. Most. Important. Thing.

If you're sorta thinking, "huh? You mean like, blind and deaf people Accessibility?" then you're not alone, because I've come to understand that there are lots and LOTS of people just like you: people for whom this idea does not have the right Accessibility, so it hasn't been able to get through to you yet. It's not your fault for not understanding, any more than it would be your fault for being blind or deaf or motion-restricted or living with any other disability. When software -- or idea-ware for that matter -- fails to be accessible to anyone for any reason, it is the fault of the software or of the messaging of the idea. It is an Accessibility failure.

Like anything else big and important in life, Accessibility has an evil twin who, jilted by the unbalanced affection displayed by their parents in their youth, has grown into an equally powerful Arch-Nemesis (yes, there's more than one nemesis to accessibility) named Security. And boy howdy are the two ever at odds.

But I'll argue that Accessibility is actually more important than Security because dialing Accessibility to zero means you have no product at all, whereas dialing Security to zero can still get you a reasonably successful product such as the Playstation Network.

So yeah. In case you hadn't noticed, I could actually write a book on this topic. A fat one, filled with amusing anecdotes about ants and rubber mallets at companies I've worked at. But I will never get this little rant published, and you'll never get it read, unless I start to wrap up.

That one last thing that Google doesn't do well is Platforms. We don't understand platforms. We don't "get" platforms. Some of you do, but you are the minority. This has become painfully clear to me over the past six years. I was kind of hoping that competitive pressure from Microsoft and Amazon and more recently Facebook would make us wake up collectively and start doing universal services. Not in some sort of ad-hoc, half-assed way, but in more or less the same way Amazon did it: all at once, for real, no cheating, and treating it as our top priority from now on.

But no. No, it's like our tenth or eleventh priority. Or fifteenth, I don't know. It's pretty low. There are a few teams who treat the idea very seriously, but most teams either don't think about it all, ever, or only a small percentage of them think about it in a very small way.

It's a big stretch even to get most teams to offer a stubby service to get programmatic access to their data and computations. Most of them think they're building products. And a stubby service is a pretty pathetic service. Go back and look at that partial list of learnings from Amazon, and tell me which ones Stubby gives you out of the box. As far as I'm concerned, it's none of them. Stubby's great, but it's like parts when you need a car.

A product is useless without a platform, or more precisely and accurately, a platform-less product will always be replaced by an equivalent platform-ized product.

Google+ is a prime example of our complete failure to understand platforms from the very highest levels of executive leadership (hi Larry, Sergey, Eric, Vic, howdy howdy) down to the very lowest leaf workers (hey yo). We all don't get it. The Golden Rule of platforms is that you Eat Your Own Dogfood. The Google+ platform is a pathetic afterthought. We had no API at all at launch, and last I checked, we had one measly API call. One of the team members marched in and told me about it when they launched, and I asked: "So is it the Stalker API?" She got all glum and said "Yeah." I mean, I was joking, but no... the only API call we offer is to get someone's stream. So I guess the joke was on me.

Microsoft has known about the Dogfood rule for at least twenty years. It's been part of their culture for a whole generation now. You don't eat People Food and give your developers Dog Food. Doing that is simply robbing your long-term platform value for short-term successes. Platforms are all about long-term thinking.

Google+ is a knee-jerk reaction, a study in short-term thinking, predicated on the incorrect notion that Facebook is successful because they built a great product. But that's not why they are successful. Facebook is successful because they built an entire constellation of products by allowing other people to do the work. So Facebook is different for everyone. Some people spend all their time on Mafia Wars. Some spend all their time on Farmville. There are hundreds or maybe thousands of different high-quality time sinks available, so there's something there for everyone.

Our Google+ team took a look at the aftermarket and said: "Gosh, it looks like we need some games. Let's go contract someone to, um, write some games for us." Do you begin to see how incredibly wrong that thinking is now? The problem is that we are trying to predict what people want and deliver it for them.

You can't do that. Not really. Not reliably. There have been precious few people in the world, over the entire history of computing, who have been able to do it reliably. Steve Jobs was one of them. We don't have a Steve Jobs here. I'm sorry, but we don't.

Larry Tesler may have convinced Bezos that he was no Steve Jobs, but Bezos realized that he didn't need to be a Steve Jobs in order to provide everyone with the right products: interfaces and workflows that they liked and felt at ease with. He just needed to enable third-party developers to do it, and it would happen automatically.

I apologize to those (many) of you for whom all this stuff I'm saying is incredibly obvious, because yeah. It's incredibly frigging obvious. Except we're not doing it. We don't get Platforms, and we don't get Accessibility. The two are basically the same thing, because platforms solve accessibility. A platform is accessibility.

So yeah, Microsoft gets it. And you know as well as I do how surprising that is, because they don't "get" much of anything, really. But they understand platforms as a purely accidental outgrowth of having started life in the business of providing platforms. So they have thirty-plus years of learning in this space. And if you go to, and spend some time browsing, and you've never seen it before, prepare to be amazed. Because it's staggeringly huge. They have thousands, and thousands, and THOUSANDS of API calls. They have a HUGE platform. Too big in fact, because they can't design for squat, but at least they're doing it.

Amazon gets it. Amazon's AWS ( is incredible. Just go look at it. Click around. It's embarrassing. We don't have any of that stuff.

Apple gets it, obviously. They've made some fundamentally non-open choices, particularly around their mobile platform. But they understand accessibility and they understand the power of third-party development and they eat their dogfood. And you know what? They make pretty good dogfood. Their APIs are a hell of a lot cleaner than Microsoft's, and have been since time immemorial.

Facebook gets it. That's what really worries me. That's what got me off my lazy butt to write this thing. I hate blogging. I hate... plussing, or whatever it's called when you do a massive rant in Google+ even though it's a terrible venue for it but you do it anyway because in the end you really do want Google to be successful. And I do! I mean, Facebook wants me there, and it'd be pretty easy to just go. But Google is home, so I'm insisting that we have this little family intervention, uncomfortable as it might be.

After you've marveled at the platform offerings of Microsoft and Amazon, and Facebook I guess (I didn't look because I didn't want to get too depressed), head over to and browse a little. Pretty big difference, eh? It's like what your fifth-grade nephew might mock up if he were doing an assignment to demonstrate what a big powerful platform company might be building if all they had, resource-wise, was one fifth grader.

Please don't get me wrong here -- I know for a fact that the dev-rel team has had to FIGHT to get even this much available externally. They're kicking ass as far as I'm concerned, because they DO get platforms, and they are struggling heroically to try to create one in an environment that is at best platform-apathetic, and at worst often openly hostile to the idea.

I'm just frankly describing what looks like to an outsider. It looks childish. Where's the Maps APIs in there for Christ's sake? Some of the things in there are labs projects. And the APIs for everything I clicked were... they were paltry. They were obviously dog food. Not even good organic stuff. Compared to our internal APIs it's all snouts and horse hooves.

And also don't get me wrong about Google+. They're far from the only offenders. This is a cultural thing. What we have going on internally is basically a war, with the underdog minority Platformers fighting a more or less losing battle against the Mighty Funded Confident Producters.

Any teams that have successfully internalized the notion that they should be externally programmable platforms from the ground up are underdogs -- Maps and Docs come to mind, and I know GMail is making overtures in that direction. But it's hard for them to get funding for it because it's not part of our culture. Maestro's funding is a feeble thing compared to the gargantuan Microsoft Office programming platform: it's a fluffy rabbit versus a T-Rex. The Docs team knows they'll never be competitive with Office until they can match its scripting facilities, but they're not getting any resource love. I mean, I assume they're not, given that Apps Script only works in Spreadsheet right now, and it doesn't even have keyboard shortcuts as part of its API. That team looks pretty unloved to me.

Ironically enough, Wave was a great platform, may they rest in peace. But making something a platform is not going to make you an instant success. A platform needs a killer app. Facebook -- that is, the stock service they offer with walls and friends and such -- is the killer app for the Facebook Platform. And it is a very serious mistake to conclude that the Facebook App could have been anywhere near as successful without the Facebook Platform.

You know how people are always saying Google is arrogant? I'm a Googler, so I get as irritated as you do when people say that. We're not arrogant, by and large. We're, like, 99% Arrogance-Free. I did start this post -- if you'll reach back into distant memory -- by describing Google as "doing everything right". We do mean well, and for the most part when people say we're arrogant it's because we didn't hire them, or they're unhappy with our policies, or something along those lines. They're inferring arrogance because it makes them feel better.

But when we take the stance that we know how to design the perfect product for everyone, and believe you me, I hear that a lot, then we're being fools. You can attribute it to arrogance, or naivete, or whatever -- it doesn't matter in the end, because it's foolishness. There IS no perfect product for everyone.

And so we wind up with a browser that doesn't let you set the default font size. Talk about an affront to Accessibility. I mean, as I get older I'm actually going blind. For real. I've been nearsighted all my life, and once you hit 40 years old you stop being able to see things up close. So font selection becomes this life-or-death thing: it can lock you out of the product completely. But the Chrome team is flat-out arrogant here: they want to build a zero-configuration product, and they're quite brazen about it, and Fuck You if you're blind or deaf or whatever. Hit Ctrl-+ on every single page visit for the rest of your life.

It's not just them. It's everyone. The problem is that we're a Product Company through and through. We built a successful product with broad appeal -- our search, that is -- and that wild success has biased us.

Amazon was a product company too, so it took an out-of-band force to make Bezos understand the need for a platform. That force was their evaporating margins; he was cornered and had to think of a way out. But all he had was a bunch of engineers and all these computers... if only they could be monetized somehow... you can see how he arrived at AWS, in hindsight.

Microsoft started out as a platform, so they've just had lots of practice at it.

Facebook, though: they worry me. I'm no expert, but I'm pretty sure they started off as a Product and they rode that success pretty far. So I'm not sure exactly how they made the transition to a platform. It was a relatively long time ago, since they had to be a platform before (now very old) things like Mafia Wars could come along.

Maybe they just looked at us and asked: "How can we beat Google? What are they missing?"

The problem we face is pretty huge, because it will take a dramatic cultural change in order for us to start catching up. We don't do internal service-oriented platforms, and we just as equally don't do external ones. This means that the "not getting it" is endemic across the company: the PMs don't get it, the engineers don't get it, the product teams don't get it, nobody gets it. Even if individuals do, even if YOU do, it doesn't matter one bit unless we're treating it as an all-hands-on-deck emergency. We can't keep launching products and pretending we'll turn them into magical beautiful extensible platforms later. We've tried that and it's not working.

The Golden Rule of Platforms, "Eat Your Own Dogfood", can be rephrased as "Start with a Platform, and Then Use it for Everything." You can't just bolt it on later. Certainly not easily at any rate -- ask anyone who worked on platformizing MS Office. Or anyone who worked on platformizing Amazon. If you delay it, it'll be ten times as much work as just doing it correctly up front. You can't cheat. You can't have secret back doors for internal apps to get special priority access, not for ANY reason. You need to solve the hard problems up front.

I'm not saying it's too late for us, but the longer we wait, the closer we get to being Too Late.

I honestly don't know how to wrap this up. I've said pretty much everything I came here to say today. This post has been six years in the making. I'm sorry if I wasn't gentle enough, or if I misrepresented some product or team or person, or if we're actually doing LOTS of platform stuff and it just so happens that I and everyone I ever talk to has just never heard about it. I'm sorry.

But we've gotta start doing this right.

SOA done right: the Amazon strategy


June 26th, 2007

SOA done right: the Amazon strategy

Posted by Robin Harris @ 11:04 pm

I survived the Google Seattle scalability conference

And Seattle’s “summer” weather! The conference was titled “The Seattle Conference on Scalability”. The Google hosts were wonderful. I met some fellow bloggers and a lot of smart folks from massive scale websites, and some that hope to become massive scale websites.

What is the leading edge of scalability? 1,000,000 node clusters in five years, up from 25,000 node clusters today. Another spoke of 100x current scalability in the next few years. New plumbing to enable 10 GigE to handle massive clusters. Programming tools that enable a few dozen lines of code to put thousands of nodes to work.

Rather than summarize the entire conference in a few paragraphs or pages, which would hardly do it justice, I’m going to hit on some topics of particular interest over the next few days. Today’s topic is the Amazon approach to a service oriented architecture (SOA). Werner Vogels and Swami Sivasubramanian presented at length on Amazon’s experience.

SOA done right
I’ve been critical of SOA hype because - in a nutshell - SOA isn’t based on business metrics. Architects and devs can gush all they want, but until you can show hard business benefits you’ll get a wave-off from the bean counters. As you should.

I’ve also been impressed by Amazon’s path to highly scalable SOA because they’ve made every mistake in the book, starting with a hairball app, moving to a mainframe and finally a fully distributed SOA model.

How distributed?
Every Amazon web page calls on at least 150 services. IMHO, the most important statement of the day was when Werner noted that “resource sharing without contracts kills scaling.” If your services can’t scale, your SOA dreams are DOA.

Bi-lateral talks
The contract covers what the service promises against certain demand levels. The key is that both the demand side and the supply side are covered by the contract. This is the simplest level of business metric.

If the service doesn’t meet the SLA when the demand exceeds the contract, too bad. As he noted several times, Amazon is all about getting the customer’s money. It is a business, and developers are expected to know business requirements.

Decompose the problem
So what is the key to scalable services? State management. Saving state to a database guarantees scaling failure. So how do you keep some having to save state? By decomposing the problem so it can be solved with smaller, simpler tools. Simple tools are more scalable.

Databases are dinosaurs
I already think that RAID arrays are dinosaurs - a kludge that has outlived its usefulness. Werner made a related point: databases are dinosaurs too. They are difficult to manage, don’t scale well and architecturally indistinguishable from their forebears 15 years ago. Also like RAID arrays.

Do you want availability or consistency?
SOA has also required new thinking about consistency and availability. It turns out that in Amazon’s experience you can either have high availability or strong consistency. You just can’t have both at the same time.

My take on why is this: strong consistency means that there is lots of communication among the services, and if the communication doesn’t complete, the system has to stop until it does. If consistency is relaxed (or eventual) things keep moving even when services get hinky. Of course there has to be a final review process to remove possible inconsistencies, but in the meantime you’ve moved the customer to the money moment.

The Storage Bits take
Amazon has amassed a wealth of experience in SOA. If you think SOA is worth investing in, you owe it to yourself to listen to what Amazon has to say. At some point the talk will be up on YouTube and you can watch it yourself. I’ll post the URL here when I get it.

Amazon Simple Storage Service (Amazon S3)


Amazon Simple Storage Service (Amazon S3)

Amazon S3 is storage for the Internet. It is designed to make web-scale computing easier for developers.

Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.


Pay only for what you use. There is no minimum fee. Estimate your monthly bill using the AWS Simple Monthly Calculator.

We charge less where our costs are less, and prices are based on the location of your Amazon S3 bucket.

Data transfer “in” and “out” refers to transfer into and out of an Amazon S3 Region. There is no Data Transfer charge for data transferred within an Amazon S3 Region via a COPY request. Data transferred via a COPY request between Regions is charged at regular rates. There is no Data Transfer charge for data transferred between Amazon EC2 and Amazon S3 within the same Region or for data transferred between the Amazon EC2 Northern Virginia Region and the Amazon S3 US Standard Region. Data transferred between Amazon EC2 and Amazon S3 across all other Regions (i.e. between the Amazon EC2 Northern California and Amazon S3 US Standard Regions) will be charged at Internet Data Transfer rates on both sides of the transfer.

Storage and bandwidth size includes all file overhead.

(Amazon S3 is sold by Amazon Web Services LLC.)

Amazon CloudFront (beta)


Amazon CloudFront (beta)

Amazon CloudFront is a web service for content delivery. It integrates with other Amazon Web Services to give developers and businesses an easy way to distribute content to end users with low latency, high data transfer speeds, and no commitments.

Amazon CloudFront delivers your static and streaming content using a global network of edge locations. Requests for your objects are automatically routed to the nearest edge location, so content is delivered with the best possible performance. Amazon CloudFront works seamlessly with Amazon Simple Storage Service (Amazon S3) which durably stores the original, definitive versions of your files. Like other Amazon Web Services, there are no contracts or monthly commitments for using Amazon CloudFront – you pay only for as much or as little content as you actually deliver through the service.


Pay only for what you use. There is no minimum fee. Estimate your monthly bill using the AWS Simple Monthly Calculator.

United States Edge Locations

Data Transfer

$0.150 per GB – first 10 TB / month data transfer out
$0.100 per GB – next 40 TB / month data transfer out
$0.080 per GB – next 100 TB / month data transfer out
$0.070 per GB – next 100 TB / month data transfer out
$0.060 per GB – next 250 TB / month data transfer out
$0.050 per GB – next 250 TB / month data transfer out
$0.040 per GB – next 250 TB / month data transfer out
$0.030 per GB – data transfer out / month over 1,000 TB


$0.010 per 10,000 GET requests

European Edge Locations

Data Transfer

$0.150 per GB – first 10TB / month data transfer out
$0.100 per GB – next 40 TB / month data transfer out
$0.080 per GB – next 100 TB / month data transfer out
$0.070 per GB – next 100 TB / month data transfer out
$0.060 per GB – next 250 TB / month data transfer out
$0.050 per GB – next 250 TB / month data transfer out
$0.040 per GB – next 250 TB / month data transfer out
$0.030 per GB – data transfer out / month over 1,000 TB


$0.012 per 10,000 GET requests

Hong Kong Edge Locations

Data Transfer

$0.190 per GB – first 10 TB / month data transfer out
$0.140 per GB – next 40 TB / month data transfer out
$0.120 per GB – next 100 TB / month data transfer out
$0.110 per GB – next 100 TB / month data transfer out
$0.100 per GB – next 250 TB / month data transfer out
$0.090 per GB – next 250 TB / month data transfer out
$0.080 per GB – next 250 TB / month data transfer out
$0.070 per GB – data transfer out / month over 1,000 TB


$0.012 per 10,000 GET requests

Japan Edge Locations

Data Transfer

$0.201 per GB – first 10 TB / month data transfer out
$0.148 per GB – next 40 TB / month data transfer out
$0.127 per GB – next 100 TB / month data transfer out
$0.117 per GB – next 100 TB / month data transfer out
$0.106 per GB – next 250 TB / month data transfer out
$0.096 per GB – next 250 TB / month data transfer out
$0.085 per GB – next 250 TB / month data transfer out
$0.075 per GB – data transfer out / month over 1,000 TB


$0.013 per 10,000 GET requests

We charge less where our costs are less, thus some prices vary across geographic regions and are based on the edge location through which your content is served. Usage tiers for data transfer are measured separately for each geographic region. The prices above are exclusive of applicable taxes, fees, or similar governmental changes, if any exist, except as otherwise noted. Effective January 1, 2010, the prices for usage out of Japan edge locations are inclusive of Japan consumption tax.

Origin Server

Amazon CloudFront uses Amazon S3 as the origin server to store the original, definitive versions of your files. Normal fees will apply for Amazon S3 usage, including “origin fetches” – data transferred from Amazon S3 to edge locations.

Amazon Elastic Compute Cloud (Amazon EC2)


Amazon Elastic Compute Cloud (Amazon EC2)

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers.

Amazon EC2’s simple web service interface allows you to obtain and configure capacity with minimal friction. It provides you with complete control of your computing resources and lets you run on Amazon’s proven computing environment. Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change. Amazon EC2 changes the economics of computing by allowing you to pay only for capacity that you actually use. Amazon EC2 provides developers the tools to build failure resilient applications and isolate themselves from common failure scenarios.


Pay only for what you use. There is no minimum fee. Estimate your monthly bill using AWS Simple Monthly Calculator. The prices listed are based on the Region in which your instance is running. For a detailed comparison between On-Demand Instances, Reserved Instances and Spot Instances, see Amazon EC2 Instance Purchasing Options.

On-Demand Instances

On-Demand Instances let you pay for compute capacity by the hour with no long-term commitments. This frees you from the costs and complexities of planning, purchasing, and maintaining hardware and transforms what are commonly large fixed costs into much smaller variable costs.

The pricing below includes the cost to run private and public AMIs on the specified operating system (“Windows Usage” prices apply to both Windows Server® 2003 and 2008). Amazon also provides you with additional instances with other option for Amazon EC2 running Microsoft and Amazon EC2 running IBM that are priced differently.

Pricing is per instance-hour consumed for each instance type, from the time an instance is launched until it is terminated. Each partial instance-hour consumed will be billed as a full hour.

Reserved Instances

Reserved Instances give you the option to make a low, one-time payment for each instance you want to reserve and in turn receive a significant discount on the hourly usage charge for that instance. After the one-time payment for an instance, that instance is reserved for you, and you have no further obligation; you may choose to run that instance for the discounted usage rate for the duration of your term, or when you do not use the instance, you will not pay usage charges on it.

Reserved Instances can be purchased for 1 or 3 year terms, and the one-time fee per instance is non-refundable. Usage pricing is per instance-hour consumed. Instance-hours are billed for the time that instances are in a running state; if you do not run the instance in an hour, there is zero usage charge. Partial instance-hours consumed are billed as full hours.

If Microsoft chooses to increase the license fees that it charges for Windows, we may correspondingly increase the per-hour usage rate for previously purchased Reserved Instances with Windows. The initial one-time payment for a Reserved Instance will be unaffected in this situation. Any such changes would be made between Dec 1 – Jan 31, and with at least 30 days’ notice. If the per-hour usage rate does increase, you may continue to use your Reserved Instance with Windows with the new per-hour usage rate, convert your Reserved Instance with Windows to a Reserved Instance with Linux, or request a pro rata refund of the upfront fee you paid for the Reserved Instance with Windows.

Reserved Instances are currently available for Linux/UNIX and Windows operating systems. Click here to learn more about Reserved Instances.

Spot Instances

Spot Instances enable you to bid for unused Amazon EC2 capacity. Instances are charged the Spot Price, which is set by Amazon EC2 and fluctuates periodically depending on the supply of and demand for Spot Instance capacity. To use Spot Instances, you place a Spot Instance request, specifying the instance type, the Region desired, the number of Spot Instances you want to run, and the maximum price you are willing to pay per instance hour. To determine how that maximum price compares to past Spot Prices, the Spot Price history is available via the Amazon EC2 API and the AWS Management Console. If your maximum price bid exceeds the current Spot Price, your request is fulfilled and your instances will run until either you choose to terminate them or the Spot Price increases above your maximum price (whichever is sooner).

Click here to learn more about Spot Instances. For information on how to get started, click here.

The following table displays the Spot Price per Region and instance type (updated every 30 minutes).

If you would like to go straight to a view of the latest Spot Instance pricing:
  1. Log in to the AWS Management Console, then click the “Amazon EC2” tab.
  2. Click on “Spot Requests” in the navigation pane on the left.
  3. Click on “Pricing History” to open a view of pricing selectable by instance type.

Data Transfer

Internet Data Transfer

The pricing below is based on data transferred "in" and "out" of Amazon EC2.

Data Transfer In  
All Data Transfer Free through June 30, 2010*

Data Transfer Out  
First 10 TB per Month $0.15 per GB
Next 40 TB per Month $0.11 per GB
Next 100TB per Month $0.09 per GB
Over 150 TB per Month $0.08 per GB

There is no Data Transfer charge between Amazon EC2 and other Amazon Web Services within the same region (i.e. between Amazon EC2 US West and Amazon S3 in US West). Data transferred between Amazon EC2 instances located in different Availability Zones in the same Region will be charged Regional Data Transfer. Data transferred between AWS services in different regions will be charged as Internet Data Transfer on both sides of the transfer.

Usage for other Amazon Web Services is billed separately from Amazon EC2.

* Data Transfer In will be $0.10 per GB after June 30, 2010.

Availability Zone Data Transfer

  • $0.00 per GB – all data transferred between instances in the same Availability Zone using private IP addresses.

Regional Data Transfer

  • $0.01 per GB in/out – all data transferred between instances in different Availability Zones in the same region.

Public and Elastic IP and Elastic Load Balancing Data Transfer

  • $0.01 per GB in/out – If you choose to communicate using your Public or Elastic IP address or Elastic Load Balancer inside of the Amazon EC2 network, you’ll pay Regional Data Transfer rates even if the instances are in the same Availability Zone. For data transfer within the same Availability Zone, you can easily avoid this charge (and get better network performance) by using your private IP whenever possible.

See Availability Zones for tools to describe instance location.

Amazon Elastic Block Store

Elastic IP Addresses

No cost for Elastic IP addresses while in use

  • $0.01 per non-attached Elastic IP address per complete hour
  • $0.00 per Elastic IP address remap – first 100 remaps / month
  • $0.10 per Elastic IP address remap – additional remap / month over 100

Amazon CloudWatch

Auto Scaling

Auto Scaling is enabled by Amazon CloudWatch and carries no additional fees. Each instance launched by Auto Scaling is automatically enabled for monitoring and the Amazon CloudWatch monitoring charge will be applied.

Elastic Load Balancing

(Amazon EC2 is sold by Amazon Web Services LLC.)

Elastic Load Balancing


Elastic Load Balancing

Elastic Load Balancing automatically distributes incoming application traffic across multiple Amazon EC2 instances. It enables you to achieve even greater fault tolerance in your applications, seamlessly providing the amount of load balancing capacity needed in response to incoming application traffic. Elastic Load Balancing detects unhealthy instances within a pool and automatically reroutes traffic to healthy instances until the unhealthy instances have been restored. Customers can enable Elastic Load Balancing within a single Availability Zone or across multiple zones for even more consistent application performance.

Amazon Elastic MapReduce


Amazon Elastic MapReduce

Amazon Elastic MapReduce is a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3).

Using Amazon Elastic MapReduce, you can instantly provision as much or as little capacity as you like to perform data-intensive tasks for applications such as web indexing, data mining, log file analysis, machine learning, financial analysis, scientific simulation, and bioinformatics research. Amazon Elastic MapReduce lets you focus on crunching or analyzing your data without having to worry about time-consuming set-up, management or tuning of Hadoop clusters or the compute capacity upon which they sit.


Amazon Elastic MapReduce currently is available in the US and EU Regions. Pay only for what you use – there is no minimum fee. Amazon Elastic MapReduce pricing is in addition to normal Amazon EC2 and Amazon S3 pricing.

Amazon EC2, Amazon S3 and Amazon SimpleDB charges are billed separately. Pricing for Amazon Elastic MapReduce is per instance-hour consumed for each instance type, from the time job flow began processing until it is terminated. Each partial instance-hour consumed will be billed as a full hour. For additional details on Amazon EC2 Instance Types, Amazon EC2 Reserved Instances Pricing, Amazon S3 Pricing, or Amazon SimpleDB Pricing, follow the links below:

Amazon EC2 Instance Types

Amazon EC2 Reserved Instances Pricing

Amazon S3 Pricing

Amazon SimpleDB Pricing

Amazon SimpleDB TM (beta)


Amazon SimpleDB TM (beta)

Amazon SimpleDB is a highly available, scalable, and flexible non-relational data store that offloads the work of database administration. Developers simply store and query data items via web services requests, and Amazon SimpleDB does the rest.

Unbound by the strict requirements of a relational database, Amazon SimpleDB is optimized to provide high availability, ease of scalability, and flexibility with little or no administrative burden. Behind the scenes, Amazon SimpleDB creates and manages multiple geographically distributed replicas of your data automatically to enable high availability and data durability. The service responds to changes in traffic by charging you only for the compute and storage resources actually consumed in serving your requests. You can change your data model on the fly, and data is automatically indexed for you. With Amazon SimpleDB, you can focus on application development without worrying about infrastructure provisioning, high availability, software maintenance, schema and index management, or performance tuning.

Featured Use Cases
Learn about data sets and use cases well-suited to take advantage of Amazon SimpleDB’s scalability, availability, flexibility, and zero-administration service model. Featured applications include:


Free Tier*
You can get started with SimpleDB for free. Amazon SimpleDB users pay no charges on the first 25 Machine Hours, 1 GB of Storage, and 1 GB of Data Transfer Out consumed every month. Under the new free inbound data transfer promotion, all data transfer into Amazon SimpleDB is free of charge until June 30, 2010. In most use cases, the free tier enables approximately 2,000,000 GET or SELECT API requests to be completed per month before incurring any usage charges. Many applications should be able to operate perpetually within this free tier, such as a daily website analysis and traffic reporting tool, a web indexing service, or an analytics utility for online marketing programs.

As your demand grows, you still pay only for what you use. As with other AWS services, there is no minimum fee and no long-term commitment. Also, note that we charge less where our costs are less, thus some prices vary across Geographic Regions. The prices listed are based on the Region in which you establish your Amazon SimpleDB domain(s). Amazon SimpleDB may be used from most countries, so long as payment is made in US Dollars.

Machine Utilization

Amazon SimpleDB measures the machine utilization of each request and charges based on the amount of machine capacity used to complete the particular request (SELECT, GET, PUT, etc.), normalized to the hourly capacity of a circa 2007 1.7 GHz Xeon processor. See below for a more detailed description of how machine utilization charges are calculated.See below for a more detailed description of how machine utilization charges are calculated.

Data Transfer

  • Data Transfer In is free until June 30, 2010*

  • First 1 GB of data transferred out per month is free; thereafter:
  • $0.15 per GB – first 10 TB / month data transfer out
  • $0.11 per GB – next 40 TB / month data transfer out
  • $0.09 per GB – next 100 TB / month data transfer out
  • $0.08 per GB – data transfer out / month over 150 TB
* Data Transfer In will be $.10 per GB after June 30, 2010

Data transfer “in” and “out” refers to transfer into and out of Amazon SimpleDB. There is no additional charge for data transferred between Amazon SimpleDB and other Amazon Web Services within the same Region (i.e., $0.00 per GB). Data transferred across Regions (e.g., between Amazon SimpleDB in the EU (Ireland) Region and Amazon EC2 in the US-East (Northern Virginia) Region, will be charged at Internet Data Transfer rates on both sides of the transfer.

Structured Data Storage

Amazon SimpleDB measures the size of your billable data by adding the raw byte size of the data you upload + 45 bytes of overhead for each item, attribute name and attribute-value pair.

Amazon SimpleDB is designed to store relatively small amounts of data and is optimized for fast data access and flexibility in how that data is expressed. In order to minimize your costs across AWS services, large objects or files should be stored in Amazon S3, while the pointers and the meta-data associated with those files can be stored in Amazon SimpleDB. This will allow you to quickly search for and access your files, while minimizing overall storage costs. See below for a detailed explanation of how storage in Amazon SimpleDB and storage in Amazon S3 differ and a more detailed description on calculating your Storage Costs.

* The free tier is a monthly offer. Free usage does not accumulate.

** Any data stored as part of the free tier program must be actively used. If a domain is not accessed for a period of 6 months, it will be subject to removal at the discretion of Amazon Web Services.

(Amazon SimpleDB is licensed by Amazon Web Services LLC.)

(1 - 10 of 15)