79226895

Date: 2024-11-26 13:30:11
Score: 1
Natty:
Report link

I feel like you have a good general idea of the main messaging services that AWS provides, but I think a couple details are missing:

SQS

For this use case it might be a good option to consider an SQS FIFO queue which means the messages will be handled in order.

Otherwise one bid might reach the backend before another, even though it was published at a later timestamp.

Furthermore, make sure that the visibility timeout is larger than the time you require to process the message. Otherwise you could accidentally pull the same SQS message twice. This is a general best practice though, so nothing particular with your use case.

Kinesis

Kinesis will not force each consumer to read all messages.

With kinesis you have a dynamodb lease table where each consumer registers which shards it will be pulling information from.

In this setup with 2 instances it could be a good idea to have 2 shards (or more if needed) and each instance will only pull from the shards it has resisted in the lease table.

Furthermore, it would probably be a good idea to use the bidding item id as a hash key for the kinesis messages, assuring that each bid for a certain item always goes to the same shard, since kinesis only promises ordering on a shard level.

Backend Application

Depending on how your application runs it might still be a good idea to have some sort of locking in place in the application itself.

If you are sure it will only ever pull and process one message at a time though, I guess this might be overly cautious.

As to the "I also assume that every service instance maintains an in-memory mapping..." part:

I'm not quite sure if you're talking about the "leasing" part or the data storage of the bids. For both cases though I would ask why you think it should be in memory?

A centralised location both for the "leasing" and storage of the bids seems more appropriate, since it's then decoupled from the instances.

With kinesis there is the dynamodb lease table, and with Kafka I guesss it's something different, but still something central that all instances communicate with. Otherwise there would be no way of detection collisions where multiple consumers try and process the same shards etc.

As for the storage of the bids themselves, that should also generally be in a separate storage (e.g. RDS / DynamoDB) since otherwise they would of course disappear if you had an issue with your application or performed a deployment etc.

Reasons:
  • Long answer (-1):
  • No code block (0.5):
  • Contains question mark (0.5):
  • Low reputation (1):
Posted by: Adam Olsson