Event-Driven CI at Enterprise Scale: Decoupling Webhook Ingestion with Kafka

Key Takeaways

• Synchronous webhook to runner systems do present some issues but we see that they are at risk for certain problems. The solution isn't to add more runners, instead what we do is we separate input functions from processing via the partitioned event bus.
• Partitioning effected on the basis of organization, type of repository, type and name of reference preserves arrangement of references while avoiding the creation of hot partitions for big repositories. Because there is a lack of needs of the execution process, the stateless receiver only needs to verify and publish.
• When queuing arises, the protocol takes a new form. As it happens, the messages can be delayed, restored, and reverted which implies that some verification must happen before the process of publishing, deduplications on the basis of delivery ID must occur at the moment of delivery and processing should also be checked once again.
• One must...

Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE