Episode 2: CQRS Building Blocks
In this episode we discuss the various concepts and patterns that surround CQRS as an architecture and how each of these patterns can elegantly solve a number of business and technical problems.
Please post your comments below. All feedback would be appreciated!
MP3 Download: Distributed Podcast, Episode 2
Episode references
- CQRS Info and CQRS Starting Point
- What is Remote Procedure Call (RPC)
- Messaging
- Retry Policy and Circuit Breaker
- Domain Driven Design by Eric Evans (DDD)
- Event Driven Architecture (EDA)
- Rules Engine
- Task-Based UI: Microsoft Inductive UI
- Event Sourcing defined by Martin Fowler and explained by Greg Young
- Event Storage Architecture Overview by Jonathan Oliver
- NServiceBus
- NCQRS
- Git
- CQRS Google Group
Our blogs
I think you guys are providing a much needed service to the dev community. Keep up the good work.
A suggestion for the john and john problem, call j. Matheus Mat or John John.
One last thing I was reading Rinat’s blog about his ideas on usin the RX framework with cq kaybe you could do a round table discussion about this subect.
Sorry about the gramar I am entering this from my nook.
Periop, thank you for the feedback!
Indeed, we’ll need to figure out the naming problem, to reduce confusion.
Re Rx framework: this could provide an interesting topic for discussion in a few months from now. At the moment there isn’t enough production experience on this topic, though.
Love the content. I have no knowledge of CQRS and DDDD but am learning some through the 2 podcasts so far, but i’m still a complete n00b.
Couple things that come to mind.
You need to sell me on Event Sourcing instead of the normal persistance store. Yes I know that all subscribers will come to eventual consistency and scalability. What I’m thinking about is more in terms of support/maintaince issues/fixes/tools (yes I know submit an event to make the fix, but it can’t be that simple; what is a hard issue you’ve dealt with?). How to interact with it, what tools to use or do we build our own? What is the persistence mechanism or product choices out there for event sourcing (i.e. NoSQL, MSMQ, etc.???)? How do we do reporting on Event Sourcing? (i.e. Sums, Avg, Counts, Factors etc)
Other questions:
How do we deal with a Task Based UI but needs a value submited by the user client and fed back to the user client (i.e. web page/control)? Such as a calculation? What if the calculation needs to be displayed on the same UI screen/page instead of moving to the next step?
Chris, thank you for the interest in the podcast!
Re Event Sourcing: We’ll probably do a podcast on it’s own on this topic. In short, in the CQRS/DDD world, event sourcing is just another way to persist aggregates. Even if you decide to use relational DB for persistence, you can still have eventual consistency in form of events that get published by command handler and consumed by various event handlers to populate views.
Re Task based UIs: That’s a common question (we’d probably talk about that later as well). In short: you can do either request/responce style or use some sort of service with RPC. It all depends on the case.
One other question. What happens if an event is delayed for whatever reason, but the next step/task requires that event to already be in the event store but isn’t. What happens then? What can we do to mitigate that or handle a series of tasks that require that value as that value is used for a calculation later on in the steps/tasks ahead?
One of the possible answers is batching (multiple commands are sent as one message). We mention that in episode 3 on messaging (to be published later)
Lets say a website has millions of users, i.e. Amazon.com. We need to bring up another event source node to handle an influx of users (mainly reads). So we bring up another node, does that new node require all events from day zero to come up or can there be from a point of time (like a snap shot of a period of events) in order to speed up the start up? Or maybe this isn’t needed????? Or is this product specific? roll your own?
Short answer: depends on the case.
There is no such thing as “event source node” (at least I’m not aware of). There are command handlers (server side) which can be partitioned by aggregate ID, there could be view handlers, which can be partitioned by view type and ID. For the reads we usually need to scale the views. The simplest way to do that is to use the cloud/CDN for storing denormalized views (it will take care of scalability).
Another option is indeed, to spawn new view handler’s, populating them from the event stream, if needed (or you can copy view completely from another instance and just subscribe it to the new events).
We’ll probably go into deeper details in later podcasts.
Does this help?
I just went throught the your blog post on Retry Policy and Circuit Breaker listed in the show notes, Recently I have been testing and researching AOP. One of the AOP examples I came across was using the Circuit Breaker as describe in your blog. Do you use AOP in production environments? If so would you consider doing a show on the subject and how it relates to builidng distributed systems? I could see your Retry Policy as something that would be perfect for AOP. Then that got me thinking about your whole policy framework and how it would appear to be a good fit with AOP as well. What do you think? Is this something you already do? Again keep up the good work. And I will try to make my comments from a PC instead of my nook in the future.
Periop,
I don’t use AOP at all. It complicates everything too much in my taste (high complexity raises dev and maintenance costs). Simple manually coded interface wrapper (that just executes every method call of an inner interface inside an action policy passed in the ctor) is a simpler solution.
Great podcast guys. The subject is very interesting and I love the roundtable format. Looking forward to future podcasts!
Martin, thank you for the kind words. We’ll try to keep it up!
Love the podcast. Would really like to hear about troubleshooting, especially debugging and monitoring, as this is an important part of getting a stable distributed system that is overlooked and often difficult.
Also, I’d like to hear more about implementations (ideally .net for me) including any gotcha’s (things to look out for). My main concerns here are about performance, including loading of aggregates, replaying of events and batch processing.
Keep up the good work!
Jon, thanks for the compliment!
We’ll be talking about the operations and implementations along the podcasts (aside from potential podcast focusing on the subject). Meanwhile you can check xLim 4 articles (esp “Lifehacks from Lokad Production” section) for some additional ideas in the field.
As for the performance – aggregate loading performance from event sourcing loading can be solved mechanically by using the concept of rolling snapshots. Another approach is focusing on the cloud platforms, where performance problems can be solved by elastic scaling out.
Great podcast. Do you guys have some schedule of when the next podcast(s) will be available? Did not see anything on this site.
There are some nice questions and answers in the comments: My request it to put some FAQ or just plain a questions section.
Thanks for the show so far!
Derick, thank you for the kind words!
We will be trying to publish a podcast once every two weeks. 3rd episode on Messaging is in the post production.
Great idea about the FAQ! I’ll try to assemble it later today.
Hi,
nice work, I’m trying to get into CQRS/EventSourcing, I can understand that if I have a start new project it is nice to try DDD, CQRS…. But we now have huge ERP CRUD system. Is there some patterns how to start implement DDD/CQRS into existing project.
I think that it can be done by use commands and so in ItemChanged in UI, but it is so much work and without some nice efect.
In another way I think that our customers are fine with CRUD, because of that if they typing invoice, they type it like writing it on paper they do not need to specify task, only type input and then they du some actions, like change state of invoice, and so.
I know that my English is not good, so sry for really n00b questions and if there some point where start with moving existing CRUD large project into DDD/CQRS I will be really happy to read/see/hear more about it.
A.
Ales,
The recommended approach, that seems to work for gradual migration of existing business application:
1. Get used to DDD (read the book and start practicing) and leverage it to understand your business and break it into smaller domains.
2. Pick domains, where introduction of CQRS/ES elements will bring the most benefit (performance, reporting, scalability etc) and start by capturing user’s intent at the UI level. Then, work your way slowly down within this domain along the lines outlined on cqrsinfo.com site.
3. Repeat for the other domains, that are worth migrating to CQRS/DDD
@Ales,
Great question. Unfortunately I don’t have a lot of experience migrating existing systems to CQRS. One thing that has worked in my limited experience is to carve off selected pieces of a system and to re-implement each piece.
Hi thanks for yours replies, but even if I think that I’m good for that bussines domain of our system, it is hard to imagine some task based UI instead of UI which is used now. May be that change UI into task based can be compared to change UI from old dos application to Win GUI aplication or Web app. Some users are really scared on any change.
May be that it will be better to try do some small app (as homework) from some domain which I do not know too much, my mind is not so tied up with that domain.
Anyway it is really interesting and I will give some time to find more about CQRS…..
Again ty for your work and ty all other who trying propagate CQRS and other new views on developing apps. It is a lot of fun learning new stuff.
A.
On a mental level, from the perspective of a user, task-based UIs are easier to handle and used, compared to usual forms and data-entry windows. They allow to explicitly capture business processes, instead of keeping them in mind. This actually reduces learning curve and potential errors, leading directly to the financial benefits.
An old paper on Microsoft Inductive UI talks about this approach in much better detail. It might require a bit of rethinking, though (something which took me more than 4-5 years).
Is there some OpenSource TaskBased UI application may be that simply I cannot imagine what can be done using TaskBasedUI.
Users of our CRUD are fast-typing accountants which are using keystokes and I cannot Imagine that all atomic change on Invoice they have to type and confirm by some buttons.
If there is some app which is rich of typing information into and has TaskBased UI it will be really good for me.
Thank you A.
I just looked on link to MS Inductive UI, it is now more clear. Task based UI does not mean that every Value typed into system has to be a command. I will look at that and sry for noob question I just try to do some small app and then I hopefully will have better type of questions.
What is better way to learn more about CQRS try to create CQRS app from scratch or use some of framework (NCQRS,..) and after that go more into deep ?
A.
Ales,
Resharper (and other JetBrains apps) are a great example of TaskBased UIs that are keystroke-friendly.
Re learning CQRS – trying to build your own framework is much better for learning, in my personal opinion, However, for the actual study process, I would recommend instead to start with DDD, which is *really* important (if you haven’t already), then try to pick a legacy application, gradually separating one domain in it (which has the most business value), and migrate it this portion to CQRS.
There are much less greenfield projects, compared to the brownfield projects.
In this podcast you guys talked about integrating with legacy systems. Specifically, you talked about a couple of solutions.
1) sync all data
2) perform data diffs (read through data and write out changes to another table)
3) introduce triggers that write changes to another table (in a better format for messaging)
4) introduce triggers that submit messages to Sql Server Service Broker
I need to integrate with a legacy system that is based on a file system database, which means I cannot modify it and it has no triggers. Unfortunately, its query provider can be quite slow and I have no way to modify indexes to make it more performant. It seems I’m stuck with #1 or #2 above, but does anyone have any further suggestions for how to make this work efficiently and with minimal complexity?
Thanks,
–Jeff
Are you comfortable writing code that queries directly into the files themselves? If so, you could write some simple code that completely bypasses any/all query providers. Then you could dispatch messages based upon any new information added. Working with something like that can be very nasty and there aren’t a lot of great ways to work around this type of problem.