JDD 2012 and local space distractions
I am back. With tons of ideas to try, new memories and people I will always remember.
I am posting a link to my presentation, which turned into “abstract musings and my own trauma of frameworks, languages and paradigms” according to one of attendees. Thanks all who have come to my talk, gave me feedback afterwards and sipped “cherry” with me till late night at after party.
See you soon.
“Do you really hate frameworks?” – this question won a contest for the most popular question after my talk at JDD this year. I don’t remember exactly how many discussions I had after my presentation, how many questions about my personal trauma caused by programming languages, paradigms and frameworks. I have to say that this makes me feel good about my presentation. I don’t know if you liked it, if you liked my style, slides and my English was good enough. One thing I know for sure is that my message was heard and was not lost in translation. I tend to have people confused after my presentations, they don’t know what I wanted to say.I usually have too many thoughts at once, too many views and never ever show last slide 
.
Do I really hate frameworks?. Yes I do. Do I? No, I don’t hate them. It is really complex and dynamic relation. It is hard to say if I love or hate, something that is not alive. I can love or hate people, but I cannot say the same about frameworks. At the end of the day it is all about people, people who create and use frameworks, paradigms and programming langues. It is all about us,
lost in a universe of unknown , which expands with the speed of light. It is all about us, who master our skills to communicate better and better with machines, and lost our ability to express our thoughts in a “human readable” form.
It is all about expressiveness, about telling stories (thanks @pbadenski for your
inspiring talk “Catcher in the code”). Frameworks, paradigms and programming languages are not about solving problems. As I said during presentation “frameworks solve author’s problems. Not yours”. You don’t need zillions of dependencies in your POM xml to solve your problems. And let’s be honest. It is all not about solving problems, it is all about describing domain. If your description of domain is complete and expressed in a clear way, there will be no problems to solve, almost .
Many, if not all problems in our daily life come from wrong, incomplete and ambiguous description of the domain. In short words your model is wrong.
Model is not a description of a problem, of a domain.It is one of the means of communication. Model exists to communicate your understanding of the domain, and by definition every model is wrong. So what is a “good model”? It is the model which communicates well your understanding of the world, your view of a reality.
This is what my personal problem with frameworks is all about. We tend to use them to solve problems, at the same time loosing “model” from our sight, the need to express domain in a clear way.
Let me prove it to you. Let’s imagine that we just got a new task in our TODO list.
Let’s imagine that we have a map of string keys associated with integer values. This map has to be transported over network as list of strings, where key and value are separated by ” = “. Simple model, simple enough so we don’t have to focus on understanding, but rather we are going to discuss implementation.
Let’s first start with Java.
Map<String, Integer> map = new HashMap<String, Integer>();
map.put("one", 1);
map.put("two", 2);
map.put("three", 3);
List<String> list = new ArrayList<String>();
for (Map.Entry<String, Integer> entry : map.entrySet()) {
list.add(entry.getKey() + " = " + entry.getValue().toString());
}
So what’s the story? First we create new hash map, then we put first key value pair, then second and third and create new array list. What happens next?
We get set of map entries and iterate over it. For every entry we get key and value, and join them with ” = ” sign in the middle. Once it is done we add newly created string into the list.
The truth is that in fact only line 10 expresses what we wanted to do, converts map into list of strings.
It is like play in a theater, when for first two acts actors introduce characters they play, repeating their last and first name, on and on.
And in the third act which lasts 5 minutes, they tell you that this play is about “meaning of life”. Interesting? Isn’t it?
Of course since we all are “framework seekers and followers” let’s use one. Lambdaj, please come and hear my prayers.
Map<String, Integer> map = new HashMap<String, Integer>();
map.put("one", 1);
map.put("two", 2);
map.put("three", 3);
List<String> list = convert(map.entrySet()),
new Converter<Map.Entry<String, Integer>, String>() {
public String convert(Entry<String, Integer> entry) {
return entry.getKey() + " = "
+ entry.getValue().toString();
}
});
Is it better? It is even worse. It all looks better until monster called “anonymous class” steps out of its cave.
We even have some nice words, like “convert” which express our needs far much better, but concatenation of strings is devoured by Anonymous The Dragon. Do you really think you need framework?
Or maybe you chose the wrong one? Or maybe you just skimmed through docs and examples, it worked and you moved to next task?
Yeah, that’s the thing I hate too, really hate. All this copy paste from examples jumping out right on my screen from framework’s docs and example pages, and recently from StackOverflow.
The thing is that such “smelly code” will sneak through key strokes, and will infect your codebase faster than you expect.
Its readability is even worse then plain old Java code. But “It solved my problem” you may say. The same as first snippet, right?
So what’s the deal? Yeah, I know, your CV. The longer, the more cryptic the better.
During JDD with group of people I have come up with an idea to write Maven mojo, with goal “cv:generate”, which will create your CV based on project dependencies in 3 most popular formats. Wouldn’t it be nice?
You know what’s the another problem with frameworks, paradigms and programming languages?
You can hide time pressure or your ignorance behind it. Yeap, you really can.
“Architects forced us to use it”.
“This is a bug in a framework we will have to live with it until someone fixes it, in the meantime we have prepared a quick workaround (which will stay there forever)”
“There was time pressure to deliver this feature on time, we didn’t have time to get familiar with it, we will leave it in projects dependencies and will use it in next version (read it – never)”
“This framework is soo great, but it is complex, we will use it and learn it later”
Is Lambdaj so awful, no it is really nice but you have to take time and unlearn your old thinking and behaviors and learn new.
So let’s give it one more chance.
public void convertMap() {
Map<String, Integer> map = new HashMap<String, Integer>();
map.put("one", 1);
map.put("two", 2);
map.put("three", 3);
Closure closure = closure();
{
of(this).toString(var(Map.Entry.class));
}
List<?> list = closure.each(collect(map.entrySet()));
System.out.println(list);
}
public String toString(Entry<?, ?> var) {
return var.getKey() + " = " + var.getValue();
}
How do you like this approach? Personally I like it more then the previous one. But you know what? This is not obvious for novice lambdaj user, it takes time to get how closures work in this framework, and expressiveness comes with the price of performance penalty, you don’t get it for free.
Is this all we can get from Java? I think so. It is still a whole lot of code for such simple case, so let’s make another step forward, and write same example in Fantom.
map := ["one" : 1 , "two" : 2, "three" : 3]
list := Str[,]
map.each |v, k| { list.add(k + " = " + v.toStr) }
Isn’t it lovely? Does it mean Fantom is cure for all our headaches? Are you sure we will be able to express other models in Fantom in the same, really expressive way? I will leave this question unanswered, to feed your thoughts. It is better to answer such questions to yourself.
Warning!!!
This post is not about greatness of Lambdaj, functional programming or Fantom language product placement.
If you think it is, start to read this post once again.
And I know you will try to flood me with my example written in you favorite framework or language. But something that is more readable for you is not necessarily readable for others.
Frameworks are great, as long as they help you express your thoughts, as long as they help you tackle accidental complexity of the code. Yeah, complexity, an elephant in the room. You cannot hide from it, you have to face it and embrace it, a topic for long discussion.
I hope I have helped you understand my frustration, and I have explained that my presentation was about a need to better express our thoughts, and a need to focus more on a systems structures and models. Once we picture these often hidden structures and models, we will be able to better solve our problems without need to juggle with frameworks. Wise people before us understood that need, and developed a tool for us, for our brains to help us express our model, our understanding of reality.
It is called systems thinking.
This time I will conduct training, “Practical Spring Data” at 33rd Degree conference. So if you want to become master in Spring Data, different NoSQL models and get to know how to write less to express more, come and join me on 11th of March in Warsaw.
Software architecture is not about technology, and what is even more important it is not about business.
It is not a full time job,
it is not position in CV,
it is not set of rules and principles,
and for god’s sake it is not about best practices.Software architecture is about change. About being leaders of change.
Another quote from my famous stoned friend.
- “What is software complexity, my friend?”
- “It is number of possible end states, man”
- “If this is true, isn’t like software’s complexity is a “new entropy”?”
- “You’re right”
- “So what can we do about it?”
- “Looking back at the history of science, not much my friend”
I am back home after architectural kata, where I played role of a “customer”, for the first time.I am tired, but smiling. Team was unbelievable, problem we were trying to solve quirky enough, discussions during presentations of architectures were great, tons of good “critical thinking”. Everyone was treated equally and had to survive under constant fire of questions from the audience.
On my way home I realized that I have learned something really important, even more important then discussed designs and technology stacks.
These things are so important that I have to share it with you, my dear reader. It is nothing that would change your life, rather things we tend to forget about.
Architecture is a collaborative effort.
We had three teams, 3 to 4 members in each. Every member of the team had its own view on what “customer” said, even when I thought there was no place for misunderstanding. This collaborative view on problem ended up with more open, flexible design. When team members were not able to agree on what was expected from the system they were creating subsystems and boundaries, to postpone final decision until “customer” provided more details.I have also found that if you create your architecture in a collaborative way, seniority of people involved matters less. Why? We had really unbalanced teams, with different experience and seniority levels, and at the end of the day they created pretty similar designs, they cut the problem domain in same places.I think it is because common sense and “gut feeling” is a thing that matters more in software architecture then we want to admit. So, if seniority and experience doesn’t matter so much, why all architects are people with long gray hair and beards? Why does it take so much time to become one of them, Gandalfs or Sarumans of software development lands? Thanks to architectural kata I think I know the answer.
Architecture needs to be validated (through constant communication).
If designs were pretty similar, at least at the components and collaborations level, does it really matter who creates architecture in your project?
Yes, it does matter. I was amazed how different our communication styles are, and when it came to share with others effort of 45 minutes design session in my personal opinion there was only one team during kata which outperformed. Shined like a crazy diamond (if you know what I mean).
I was just standing with my eyes wide opened, staring at the whiteboard, and I was hearing voice in my head “you dumpass, you too often forget how important is to communicate architecture”. This is what makes software developers great architects.
Ability to communicate architecture.
Second thing that is connected with that, a need for more communication, is something that eats my CPU cycles for years. There are hundreds of ways to validate software, some are good, other are even better and even more expensive , some work, some doesn’t. But how to validate architecture? Find design flaws, possible bottlenecks and “hidden traps” in technology stacks? Again, through communication, but make sure that your audience was not involved in a design process, because they will protect their “kid”, new shiny beautiful design.
Once we started to share results of our work during kata, I suddenly started to hear many “oops we didn’t think about it, oops does it really work this way?, oops we made wrong assumptions about technology”. Count every “ooops” as a failing test, failing unit test and you got ODA, Oooops Driven Architecture
So next thing to remember, if you have an architect who doesn’t collaborate and communicate his work, find him another place, it will just simply not work, period.
And make sure your architect doesn’t fall too often in love in particular technology, he should not be constrained by technology,and when you are in love you are biased. On the other side, he just needs to know this stuff, that’s it, there is no place for feelings
And don’t worry, we are all in love, everyone has their technology stack of choice, but you need to be critical about it, honest to yourself.
Problem domain can and will impact architecture, the trick is that you never know when and how.
Yeah, this one is a nasty thing. It looks to me that some designs where immune to particular changes of requirements,
while other simply just exploded. As a “customer”, I have hidden few domain details, simply skipped it . During presentations,I got my memory back, and communicated that I forgot about few details, here and there. Some teams reacted to the change saying,
“yeah, we have plugins so what you said can be implemented as a one more plugin”, others just had to rethink big chunks of the system.
What can we do about it? Nothing? Communicate with customer? I was available all the time, something that doesn’t happen in a real world, and still I was able to surprise teams.
These things just happen, my only advice is simple. When change comes, throw what you have and start from scratch. It is cheap as long it is still a design phase. You just simply don’t know how many places in the system this change will affect, don’t take this risk. I recommend you to build new design and validate it once again. In an ideal world….
You can also build open, flexible architecture from the beginning, of course when you can afford it. During kata I have also learned that there are few things you need to do to open yourself for endless possibilities of design space in front of your eyes. ( whooohoo I used “design space” for the first time in my post, and it looks so cool )
Do you want to know what you need to do?
That’s all folks. My personal take away from this architectural kata. Believe me, once you try it you will want more and more, it so rewarding.
If anyone is interested to try it at one of upcoming conferences in Poland, let me know, I will be more than happy to help.
WARNING
This post is not for people who take life too seriously, and not for people who know the answer to the ultimate question about meaning of life.
This post is not for people who don’t get Monty Python jokes, and not for people who do not get sarcasm, take everything personally, and who don’t share this vision,that life is just a ride.
I promised myself, that my next post will be about some nice framework, or some important architecture pattern or challenging technical problem,like reliable multicast or exclusive consumers, my little two daemons, my Moirai, my own personal road to hell.And again I have failed, failed to not get distracted by Twitter and blogs
So this post is my own take on “megasoftwarecraftsmanshipper”, strategic refactorization and somehow lously coupled with latest discussion between Uncle Bob and Ted Neward.
Reading through last tweets and blogs I realized that I have just one advice for us.
“Stop refactoring!”
Do you think that I have some secret trick which will change your refactoring “kung fu” forever? And I am going to share it with you? No way. There is no magic, unless you call “critical thinking” magic. It is a really hard job to do, it requires special skills (like “Extract Method” spell, or magical items like “Wand of God Object” which shines every time you get close to “God Object”). It is a job which requires mixture of knowledge, experience and courage.
It is job which will give you headaches.You will loose faith in human race. You will cry, shout and laugh at the same time and you will earn respect in da hood.
That’s all and nothing more, because it is garbage, really. Only you care about it, and you know what? You throw company money with every run of your useless tests suite. Did somebody ask you to do it? Really? Don’t lie yourself. You could develop next shiny feature with the time you wasted for refactoring.
Yeah, I know somebody will have to maintain it, but lets be honest, it will not be you, so why bother? (you get the sarcasm, I hope )
Is this some kind of “refactoring” bashing? No, it is not. I really think refactoring is a waste, this is just a painkiller, not a cure.
You feel better because you are a “craftsman”, you care about your habitat. In your mind filled with “craftsmanship manifesto”, not doing refactoring, is like throwing pieces of paper on a sidewalk, like not helping animals in a cold winter night.
Your management is proud, because they understand “technical debt”, they can fill reports with numbers about money saved, thanks to their strategic skills and investments.
And your customer doesn’t give a …. about it.
And you know what? You behave like compulsive, addicted jerk. Yeah, me too, no worries it is my fault too. Because you keep coming for more, every line of “wasted years” you see, you feel this addictive shiver of temptation, to make world a better place, to bring professionalism back to life,”to shine like a crazy diamond”. You are so blinded by your filthy thoughts about getting rid of “shared global state” and replacing endless “if” spaghetti structures with Strategy pattern, that you don’t see that this is the same code you worked with 3 months back, when you were working on previous release. You are addicted, and refactoring is your drug, and you keep coming for more.
Take a look back at your SCM logs, haven’t you worked on the same files few months back? You keep coming to the same places in the system, over and over again.You fix and improve, but system erodes so fast, in so many phantasmagorical shapes, that you don’t even recognize that you have been through this nightmare so many times. Stop refactoring and step back, and realize that what you see here is fix that fails, another lovely “system trap”.
Let me explain you why your lovely fix fails, and I am not going to bore you with systems theory and their explanation, no worries.
You make corrective actions in the system (you call it refactoring), and you see immediate effects, you feel better, complexity metric is at right level,and you know it was good. You improved it and you moved on. You forgot about side effects of your changes, you forgot about cause and effect.Somebody got your changes, junior person or “Sunday” coder. It really doesn’t matter. What matters is that they had to add some nice customization for important client and they became mentally damaged in a moment when they opened your changes. Your “clean code” was sooo clean, that they were afraid to touch it, so they “copy pasted” old version, added customization for customer, added one small ‘if’ in the beginning of your freshly refactored code and moved on.
You know what will happen the day you will open this “masterpiece”? You will sing along with Brittney, “Ooops I did it again”, admit it. You will refactor it, using more and more “spells” and magical items. It will be even worse with every cycle.
You have to break this addiction.Stop refactoring! and ask yourself, why do I have to refactor? Code is old, project to complex, lack of design, deadlines or technology stack… it is what it is. The important thing is that you can improve code without touching it, almost.
Organization is what you need to refactor in first place. It doesn’t mean you have to fire management, and become “usurper CEO”. You can start from your team.
Architect you organization in a way you will not have to refactor in a endless loop. Share knowledge, respect others and improve your own habitat, which is not your code but your team mates, your manager and architect. Does it sound like a utopia? Maybe, but isn’t it nice?
Or maybe if you are on a senior, lead, architect or principal position, this is what you should do? Not just only bash people for their misunderstanding of your great ideas? For their misunderstanding of your addiction to refactoring, SOLID principles, low coupling and so on? Easier said then done.
What should I do? Listen to the wise man, look outside of your comfort zone and understand that this is not problem with people and their skills or attitude. Maybe this is just manifestation of your organization structure. Give this thought a chance, let it grow. Maybe you have to constantly refactor, because people know that you will fix it anyway? Because they don’t have time? Because they don’t care? Because we don’t do thing like this here?
Don’t get me wrong. Refactoring can be a cure for lot of your problems, but only when you have organization which needs cure not painkillers.
Refactoring as a short-term solution, quick fix, will not help you. It will make things even worse, craftsman will become more skilled, and junior people more confused.
It can even lead to a silent war between “craftsman” and “workers” camps, and both will try harder and harder.
“Craftsman” will use more and more sophisticated refactorings, structures and patterns, and “workers” will play “ignorance game”.Believe me I was in a middle of such silent war. Refactorings were blessed only in a situations of critical bugs and customer escalations, and this organization strategy didn’t work.It was not a good place to be, and believe me I don’t care if you are a “craftsman” or “worker”, and I don’t respect others more or less. This is who we are, this kind of diversity is good, but organizations should learn how to use it.
Refactoring as a daily habit, between sips of coffee and planning and demo sessions will improve you code, your habitat. Refactoring in a small, energizing “shoots” will make you a drug addict. So, stop refactoring of code and refactor your organization.
Good night!
What is Software Architecture?
Are you brave enough to stand up in front of the crowd and answer this question? What is Software Architecture? Are you brave enough to go away from frameworks and programming languages and talk about design, paradigms and systems’ scalability, performance and resiliency? Call for papers is still open and we still accept topic’s submissions. Let’s meet on 12 April,2013 in Warsaw at 4Developers conference.
This is my first every “quicky”.
Recently I didn’t have time to write posts as I was travelling back and forth to conferences, code katas and so on. I have a long list of things to write about, and not enough time to give it attention they need. In the meantime I get distracted by my “pet” projects, articles and posts. Usual thing. Ok, enough excuses
Whenever I got a chance to squeeze time and space, I am working in a spare time on my projects… , actually haven’t committed to my “pet” projects a single line of code for a month or so. This is sad story about every single of my “pet” projects. It is not that I am lazy , it is more because of a broken connection in my brain. Because of this broken connection when I build concept or an idea for a “pet” project my brain “gets high” pretty fast, I work on it day and night, commits flow into repository like waters of holy river Ganges, until the “aha” moment.
Aha, I know how to implement it.
Aha, I am able to validate and justify the concept and design in my head.
Aha, I proved that it can be done.
Broken connection in my brain causes short-circuit between my synapses, which then sends “it is boring” message to the center of “pleasure department”
Somebody can say, what a waste of time and energy. Actually it is not. With every of my “pet” projects I have learned something, about programming languages, testing and frameworks. As “pet” project doesn’t give any value to the community I started to think that it would be at least worth to share what I have learned.
So this is idea of “quickies”. Short posts, about a small little things I have learned. Some of them will be obvious, some of them will give you “aha” moment.
In my most recent project I have lots of fun with threads, concurrency and testing. I am again amazed how asynchronous communication and concurrency can hurt your brain during testing. Sometimes I want to give up, and narrow boundaries of testable parts of the system. It is tempting, so tempting. But one the other side, it is much more tempting to solve this problem.
In one of cases, I want to test if message send asynchronously is being delivered with proper message content. I know that this can be quite easily done with a little bit of coding here and there.But when I looked at my code after few days I didn’t understand what was going on in it. I needed something which expresses my intent in a more descriptive way, and I found it. It is called SettableFuture, from Google Guava library.
Here is short snippet how it works.
”
import com.google.common.util.concurrent.SettableFuture;
final SettableFuture<String> future = SettableFuture.<String> create();
new Thread(new Runnable(){
public void run(){
future.set("Hello world!!!");
}
}).start();
String result = future.get(4, TimeUnit.SECONDS);
assertThat(result).isNotEmpty();
How it works?
SettableFuture implements java.util.concurrent.Future interface, which in short represents result of asynchronous operation. Aha
According to contract of method Future.get(long timeout,TimeUnit unit) it will block until you get result of operation, or throw java.util.concurrent.TimeoutException when operation timeout occurs.
What SettableFuture gives you? SettableFuture is the implementation of java.util.concurrent.Future, which gives you ability to set operation result. Nothing more, nothing less. Thanks to this you can pass SettableFuture to asynchronous operation and notify test code whenever results are available.
Because it implements java.util.concurrent.Future contract your tests will not run forever and in case of time out you can test timeout with expected exceptions. Of course if asynchronous call executes faster then you reach future.get(4,TimeUnit.SECONDS) it will return immediately returning results of asynchronous operation.
That’s all for today, in my first “quicky” . I hope you liked it.
This time something different, a framework
From time to time, even I, professional framework “hater”, need to write or contribute to some framework. Just to prove how “bad, bad, bad” frameworks are Just to feel “filthy” and “dirty”. Just to feel this chill down my spine, when I have to frame my thoughts within boundaries set by somebody else:)
This time I was playing with Spring Integration, an interesting approach to implementation of patterns described in a book
“Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions” by Greg Hohpe and Bobby Woolf, a real classic.
As I was playing with different adapters and channels I have noticed that there is missing implementation of JGroups transport.
So here is my short story how I have built channel adapters and XML namespace for JGroups in Spring Integration.
This small chunk of code is of course available for your eyes only and convenience at Github, https://github.com/kcrimson/spring-integration-jgroups.
If you are into Enterprise Integration Patterns, you probably know that Channel Adapter pattern allows you to access particular API or data using Message Channel semantics.Just like well known Adapter pattern from “GoF” book, it wraps one interface into another one, so you can access it without changing existing code.
Channel Adapter is one of key patterns, which allows you to “hide” complexity and different architectural styles or APIs, behind simple uniform interface of Message Channel.Thanks to this pattern you can easily access FTP sites, invoke HTTP based services and send JMS messages without diving too deep into complexities of each protocol or API, and at the same time you can easily replace FTP with email or HTTP transport, thanks to this pattern.
This is also simplest, first step, you can take when you want to implement your own transport/endpoint in Spring Integration. Of course in majority of cases, you will have to implement two Channel Adapters, one for inbound and one for outbound communication.
So let’s start with inbound channel adapter, which receives messages send to group of nodes (JGroups cluster). In the case of JGroups we deal with “push” communication.Out of many abstract endpoint implementations and styles available in Spring Integration, org.springframework.integration.endpoint.MessageProducerSupport looked like a most obvious choice.
It has some basic code for setting Message Channel reference, convenient lifecycle methods and as well handling of messaging errors. So what’s left to do? Put some glue code which registers receiver on JGroup’s JChannel object.
@Override
protected void doStart() {
jgroupsChannel.setReceiver(new ReceiverAdapter() {
@Override
public void receive(Message msg) {
Object object = msg.getObject();
Map<String, Object> headers = headerMapper.toHeaders(msg);
sendMessage(MessageBuilder.withPayload(object).copyHeaders(headers).build());
}
});
}
Few commits later (including “crash course” on XML Schema), I was able to connect to JGroups cluster and send received JGroups messages to any channel, with a little bit of XML which looks like this.
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jgroups="http://www.springframework.org/schema/integration/jgroups" xmlns:int="http://www.springframework.org/schema/integration" xsi:schemaLocation="http://www.springframework.org/schema/integration/jgroups http://www.springframework.org/schema/integration/jgroups/spring-intergration-jgroups.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration-2.2.xsd http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd"> <jgroups:cluster name="mygroup"> <jgroups:xml-configurator resource="classpath:udp.xml" /> </jgroups:cluster> <jgroups:inbound-channel-adapter id="cluster-adapter" cluster="mygroup" channel="inbound"/> <int:channel id="inbound"> <int:queue/> </int:channel> </beans>
I hope you noticed, JGroups cluster name is a also name of a bean which we inject into inbound channel adapter, using attribute cluster
Outbound channel is also trivial piece of code, which extends org.springframework.integration.handler.AbstractMessageHandler. So if you want to send message from Spring Integration to JGroups cluster, you can simply
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jgroups="http://www.springframework.org/schema/integration/jgroups" xmlns:int="http://www.springframework.org/schema/integration" xsi:schemaLocation="http://www.springframework.org/schema/integration/jgroups http://www.springframework.org/schema/integration/jgroups/spring-intergration-jgroups.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration-2.2.xsd http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd"> <jgroups:cluster name="mygroup"> <jgroups:xml-configurator resource="classpath:udp.xml" /> </jgroups:cluster> <int:poller fixed-rate="100" default="true"/> <int:channel id="inbound"> <int:queue/> </int:channel> <jgroups:outbound-channel-adapter id="cluster-adapter" cluster="mygroup" channel="inbound"/> </beans>
This code is not complete, works only in really basic cases, but it already has basic header mapper, and as you saw some support code to start JGroups in Spring’s application context.
I welcome any thoughts, comments and suggestions and I am happy to maintain this code if you need it , good night.
Tym razem w ojczystym języku, czyli moja prezentacja o cache w aplikacjach, dzięki uprzejmości i gościnności Polish JUG.
Stokrotne dzięki za możliwość prezentacji premierowego materiału, ciągle na etapie eksperymentów z formą i treścią
“Enjoy the ride!”
My “Practical Spring Data” accepted JavaDay conf in Riga
This is going to be a great time for me and I hope for attendees as well . I must admit I have never ever been in Riga and I have never ever been beyond east side of Polish border. I need to start looking for accommodation and flight. Any suggestions are more than welcomed
See you there in November.
Real World Enterprise Integration Patterns workshop at JDD conference in Krakow, 16-17 October 2013
Pipes and filters, content routers, messaging endpoints and gateways, aggregators and splitters. Want to hear what is all about? Want to hear some real world stories about building application integrations with Spring Integration. Come and join me at Java Developers Day conference in Krakow, 16-18 October 2013.
For last couple of months I have been thinking, researching, trying and failing to build different approach to software architecture.As I am getting closer to my 40ties it is high time to summarize what I have learned so far. All the past I spent following others, learning from people, adapting paradigms, testing them in real life, and throwing majority of them to a trashcan, as they were idealistic, utopian visions of reality.
Does it mean I want to concur the world with yet another manifesto? Give consultants a chance to write one more book? Does it mean I want to start a revolution? Ask you to forget all you have learned? For God’s sake, no. If I every do this, it means that I was drunk, or somebody forced me to do it.
The things I think about is rather a librarian like work, a village’s shaman who decided to put all the collective knowledge and wisdom of the tribe, written in the stone. It is rather collection of articles,blog posts, discussions, things so obvious they even don’t have name, so it is hard to talk about them.
So I named this thing, I called it Patterns of Organic Architecture. Nice name, would look nice at the cover book I will never have time to write.
What’s all about? Over the years I tried many approaches to software architecture. We, as an industry have written tons of worthless material around it, and we still struggle what does it really mean “to do software architecture”, “to be software architect”. We tried big design upfront, we tried bottom-up and top-down, we tried “don’t give a f..k about it” (so called Agile, or rather how we interpreted it). And yes this post is NOT against Agile, so stop whining about how bad I am, that I don’t understand it, I don’t believe in values. Again f..k it.
I am talking about reality here, not about Agile as an idea. I am talking about us, about us who had to implement what was said by Mr. Senior Principal Enterprise Architect, about us asked to implement one of these “world class” monolithic architecture frameworks, about us being asked if we can improve overall architecture, in the meantime adding some sexy features here and there. About us hearing everyday that there is no time for architecture, because we need to deliver some mythical business value, which usually turns into customers leaving product, because they had to wait to long to log in, because they had to wait to long to get new feature. Whatever you do to build your architecture, or even if you don’t care, you still can use what I call Patterns of Organic Architecture. A nature’s way of building simple and beautiful stuff.
Let it grow, go with the flow, but don’t leave it alone. Watch it. Measure it. Dig in, and when necessary take action. Understand forces that drive your system, and use them for your own good.
One of the patterns I have identified, “grow and harvest”, assumes that over time there will be pieces of your system which will become stable, by stable I mean ratio of changes in last months is close to zero.
One of the problems is how to identify these pieces, so we can “harvest” them from the code base, seal in separate repository, release binary artifact, or even remove them because they are not used, and enjoy smaller code base, faster build times and so on.
The idea is not new, this is something Michael Feathers is talking about quite often in his posts. Your SCM has all the information you need, you just need to dig in, reach out. The problem I have is that I tend to find myself in situations where I need to deal with gargantuan code bases, years of history, tons of files and technology and architecture changes, and if in majority of cases simple Perl script will do the job, it is hard, really hard to reason about such beasts without better tools.
Recently a friend of mine (to some known as @LAFK_pl) asked me for help with some interesting problem he was trying to solve with graph database, this way I found out that Neo4j was just updated to version 2.0. I was working with Neo4j couple of years back, trying to replace some legacy system for airlines with lovely graph model of airports and flight connections. Since then I didn’t had time to really track what is happening in this space, until last week. And suddenly I realized that graph is all I was looking for. What if I push all the information I have about files, Maven modules, packages and such plus SCM change sets in one graph, I can ask for anything that comes to my mind? Couple minutes later I had came up with this dirty code snippet which reads data from Mercurial log and puts content of this log into Neo4j database. Beware!!! This is not OOP, DDD, TDD code, this is just few minutes hack.
import static com.google.common.collect.FluentIterable.from;
import java.io.FileReader;
import java.util.List;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathFactory;
import net.sf.saxon.om.NodeInfo;
import org.neo4j.graphdb.Direction;
import org.neo4j.graphdb.DynamicLabel;
import org.neo4j.graphdb.DynamicRelationshipType;
import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.Node;
import org.neo4j.graphdb.Relationship;
import org.neo4j.graphdb.Transaction;
import org.neo4j.graphdb.factory.GraphDatabaseFactory;
import org.neo4j.graphdb.index.Index;
import org.neo4j.graphdb.index.IndexHits;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.uncommons.maths.combinatorics.CombinationGenerator;
import org.xml.sax.InputSource;
import com.google.common.base.Optional;
import com.google.common.base.Predicate;
public class App {
private static final Logger LOGGER = LoggerFactory.getLogger(App.class);
public static void main(String[] args) throws Exception {
GraphDatabaseService graphDatabase = new GraphDatabaseFactory().newEmbeddedDatabaseBuilder("db")
.newGraphDatabase();
XPathFactory factory = XPathFactory.newInstance();
XPath xpath = factory.newXPath();
@SuppressWarnings("unchecked")
List<NodeInfo> changesets = (List<NodeInfo>) xpath.evaluate("/changes/changeset", new InputSource(
new FileReader("out.xml")), XPathConstants.NODESET);
for (NodeInfo node : changesets) {
@SuppressWarnings("unchecked")
List<NodeInfo> files = (List<NodeInfo>) xpath.evaluate("file/text()", node, XPathConstants.NODESET);
if (files.size() >= 2) {
CombinationGenerator<NodeInfo> generator = new CombinationGenerator<NodeInfo>(files, 2);
Transaction tx = graphDatabase.beginTx();
Index<Node> nodes = graphDatabase.index().forNodes("files");
for (List<NodeInfo> pair : generator) {
NodeInfo first = pair.get(0);
NodeInfo second = pair.get(1);
LOGGER.info("changeset pair {}<->{}", first.getStringValue(), second.getStringValue());
Node firstNode = addNode(graphDatabase, first, nodes);
final Node secondNode = addNode(graphDatabase, second, nodes);
createChangeset(firstNode, secondNode);
createChangeset(secondNode, firstNode);
}
tx.success();
tx.close();
}
}
}
private static void createChangeset(Node startNode, final Node endNode) {
Iterable<Relationship> relationships = startNode.getRelationships(
DynamicRelationshipType.withName("changeset"), Direction.OUTGOING);
Optional<Relationship> firstMatch = from(relationships).firstMatch(new Predicate<Relationship>() {
public boolean apply(Relationship r) {
return r.getEndNode().getId() == endNode.getId();
}
});
Relationship relationship = firstMatch.orNull();
if (relationship == null) {
Relationship relationshipTo = startNode.createRelationshipTo(endNode,
DynamicRelationshipType.withName("changeset"));
relationshipTo.setProperty("times", 1L);
} else {
Long property = (Long) relationship.getProperty("times");
relationship.setProperty("times", ++property);
}
}
private static Node addNode(GraphDatabaseService graphDatabase, NodeInfo df, Index<Node> nodes) {
IndexHits<Node> indexHits = nodes.get("filename", df.getStringValue());
Node single = indexHits.getSingle();
if (single == null) {
single = graphDatabase.createNode();
single.addLabel(DynamicLabel.label(df.getStringValue()));
nodes.putIfAbsent(single, "filename", df.getStringValue());
}
return single;
}
}
This piece of code reads all change sets from the XML, and for each file, which in this case is a node in a graph, creates relation to another file in the same change set. If such relation already exists it just simply increases counter which is stored on the edges (relation) between nodes. Of course I am lazy enough to not write my own implementation of combinations from combinatorics, so I use Uncommons Math, and its CombinationGenerator, which generates pairs of all of the combinations of files within one change set. One thing I have spotted working later on with this graph is that, because Neo4j stores directed graphs, I had to generate two relations per each pair, incoming and outgoing. Which in fact is true, file A was changed together with file B, and file B was changed together with file A. Thanks to this I could simplify my Cypher queries. For those of you who are new to Neo4j, Cypher is a language which allows you to work with graphs, including queries and modifications. So what kind of information I can get from graph?
neo4j-sh (?)$ MATCH (a)-[c:`changeset`]->(b) RETURN labels(a),c.times,labels(b) order by c.times desc limit 5; +-----------------------------------------------------------------------------------------------------------+ | labels(a) | c.times | labels(b) | +-----------------------------------------------------------------------------------------------------------+ | [".../listeners/SummaryScenarioListener.java"] | 13 | [".../listeners/LoggingScenarioListener.java"] | | [".../listeners/LoggingScenarioListener.java"] | 13 | [".../listeners/SummaryScenarioListener.java"] | | ["pom.xml"] | 12 | ["roadrunner-core/pom.xml] | | [".../cli/BenchTest.java"] | 12 | [".../cli/RunTest.java"] | | [".../cli/RunTest.java"] | 12 | [".../cli/BenchTest.java"] | +-----------------------------------------------------------------------------------------------------------+
This is the simplest query, which shows all pairs of files which where modified together, ordered by number of times this pair occurred in any change set. What is important to understand is that this kind of analysis not only shows code level dependencies, but as well feature/function level dependencies, it can also show cross technology dependencies, between your JavaScript, CSS and Java files, which is pretty hard to get even with all modern IDEs we have at our disposal.Of course this type of SCM “big data” analysis can be sometime misleading, there can and will be a lot of “falsy truths” about your code base. Especially when your teams favor large commits at the end of each sprint (which I hope doesn’t happen in any organization taking code quality and continuous integration seriously). But compare this kind of information with no information at all. At least you have places in the system from which you can start you travel back in time. Of course you use other tools to visualize your code, like Gephi or Graphviz. In my past I did many such “back in time” travels, and I was always coming back with interesting and precious information. In many cases as a result we were reorganizing code base, cutting of new libraries, Maven modules and so on. Better built times, less complex code. It is worth to look back. It is worth to “grow and harvest” you code. Enjoy!
So, end of call for papers for 4Developers’2015 is approaching fast, it is actually end of January. I will be honest with you, I don’t have a clue what I will be talking about at “the only”, “the biggest” software architecture track in Poland, this year :). Yeah, I little bit of megalomania is never enough.
But actually it is true, in 2014 we had two full rooms of people, in parallel, listening to real world cases studies from battlefields.
Why I am writing this post at my almost dead blog? Simply, because a single tweet is not enough to tell you about my plans for this year. My dream is that every year, software architecture track at 4Developers will have different leitmotif. I also hope it will help you to find a topic you can talk about.
The word for this year is:
Failure in many different shapes. Tell us how and why you failed implementing newest, coolest architecture paradigm, how you design your systems, so they can fail in a controlled way. Tell us how you design for failure and how you fail fast.
Of course if you want to talk about something else, that’s cool, but I will prefer talks about failures, in any technology and framework, over other talks. So don’t wait to fail, submit your talk to 4Developers at CfP page, or write to me directly. In case you are not convinced you are a right person to stand in front of crowd to talk about failures, believe you are.
WARNING
This post is full of emotions, strong words, my own opinions. It is far from being politically correct in any dimension. If you know me, you know what I mean. You read this post on your own responsibility.
I struggled since Friday two weeks ago to actually get it in shape and post it. I have compressed a year long of frustrations and anger.
Enjoy!
Friday, finally. When I was leaving office I was filled with anger, disappointment and this weird feeling of lost hope and faith in humanity. I am coder, by heart, and by accident I work with people.
On the way back home I was thinking about this post as a way for me to deal with all my frustration, being mainly driven by my interactions with other people. As I was passing tram stops I finally realized that there is something different which is a reason of my bad mood and what a surprise, the reason is different then I thought.
I think the way I was feeling that day is not only my problem, my dear angry, frustrated fellow coders, bytecode lovers, monad travellers, big data miners. As I travel through space and time, at conferences, code retreats, beer talks, JUGs and many other places I meet tired and frustrated coders. Frustrated for different reasons, constantly talking how codebases we work with suck, how stupid architects are (and yet majority of us wants to become one, to join this army of “codeless” monkeys, remember I work as architect at the moment :) ), how we hate trendsetters at conferences, how technology X sucks big time, and Y is new cool, and will suck in a year from now, for sure. Man, I have never met so many frustrated people as I meet at software conferences, we are so sad drinking energy drinks, looking for next sticker to put on our laptop and talking in corridors that last year this conference was much better. Still looking for Big Next Thing(TM).
Fuuuck, stop it. Right now.
Yeah, actually I used F word, and you know what? You can stop reading, I will not force you to continue. Use your free will or relax and fuck it (oops again).
Conference sucks? Don’t go.
Technology sucks? Don’t use.
Codebase sucks? Change it or leave it.
Your manager is an asshole, who doesn’t understand software? Become one.
I was thinking about all the bad emotions in my life as code mangler and system’s destroyer and maybe, just maybe I have an idea why it is so. Look, we are creators, we turn null terminated arrays of characters into electrical impulses, which create motion in resistors, capacitor and transistors, this motion of electrons changes people lives, wakes up good and bad emotions, helps people meet, get married and divorce, buy new shit, sell old shit, travel, get lost and be found.
Yes, it is you my friend, who created this mess. As I said at JDD this year, We are Gods!!! We own this planet!!!
… and yet we are crowd, we are anonymous.
We are like a wannabe rockstars, still rehearsing at parents garage with this notion, that we will never actualy record an album and play gig for a crowded house.
I think this is hidden root cause of all evil.
How many times you have heard from your customer, “I saw your recent change to registration form, from the first look I knew it was you who coded it. That’s so much better then previous one” or “I think this pagination on orders page was probably coded by Tomasz, it must have been him”. Yes we are anonymous, we create stuff, but we don’t get direct recognition, our code doesn’t hang in art galleries so everyone can see it and be amazed (or maybe it just sooo bad). I miss this feeling of direct connection between my creation and people who see it. I am a man behind wall of text, and you know what? Somebody could write different sequence of characters and nobody would notice. Especially true for all back-end people, for whom character, black and white terminal is an ultimate form of human thoughts manifestation.
And it will be always this way. Nobody will notice if your code is good or bad, if its reactive or declarative or imperative. Nobody will notice your fancy lambdas, sexy tricks with communicating sequential processes, and how many microseconds you saved due to use of sun.misc.Unsafe. Nobody.
Really?
That’s not true. Fortunately or not there are other software developers in the world, who will understand you techno mumble bumble, your 3 letter acronyms and beautiful details of endofunctors, monoids, actors and such. It is just simple as that, stand up and go to meet up or conference. Don’t tell me you need approval or training budget, this is just your stupid excuse for not moving your ass of the chair. Meetups are for free. Conferences?
Become a speaker! Give something to get something back. Stop complaining. Do something. Are you afraid to speak in public? Seriously? Do you know that all people who do it are afraid every time they go on stage. I am freaking scared every time, doesn’t matter if this new or old material, I am scared.
“People will eat me alive, if I say something stupid”. Seriously? If they will be right, because you actually said something stupid, that’s good for ya. You learn something. If they were trolls or morons, ignore them. And yes I am talking to you mister, in stupid baseball hat with more stickers on your laptop then successful pull requests in your whole life. Don’t wander around in corridors during conferences with this Mr. Knowitall face expression. Do you think you could do better on stage? Talking to crowd about your passion? Feel warmly welcomed. But just stop complaining and shut up. Majority of us speaking at conferences don’t get paid for it, we do it because we love it, show some respect.
“I don’t have anything interesting to say, I am doing JEE code maintanance”. Seriously? That’s awesome. Tell me about it, share your tricks and techniques. Every project is different and unique in a way it was fucked during design or development stage (and yes we do design and development phase, shoot me).
Don’t want to talk about it? It is even better. Find something. Try it. Fail. And share.
We are crowd, we are anonymous.
That’s the essence of why there is so much tension between us and the world. I am not complaining, really. I am just trying to point out how hard it is to take pride of our work. Surrounded by deadlines, messy code, procedures, processes and approvals, “agile”, “lean”, “risk assessment”, “business value”, end year reviews, improvement programs. Oh yes, do you love these moments when somebody non technical tells you that you need to improve your coding skills. It is like eunuch telling you how to make love with your wife.
All we create is “just a code”. It couldn’t be so hard? How many times you have heard this?
Or my other all time favourite “I used be a coder, too”. Wow, how fucking scientific, can you tell me why you don’t code any more? Is there any particular reason your rights to code repository were revoked?
I choose “a way of code”. Long time ago, I had my ups and downs. But this is my choice. And if you made same decision you need to accept that you need to find other ways to be proud of your work. And your day job is not necessary the only place. Conferences, meetups, hackathon, finding other freaks like you.
Just do me one favour, stop complaining, criticizing and whining. If you choose “a way of code”, do something about it. Be full of respect for past generations, because the way you talk about old code today, will be the same way people will be talking about your code years from now. Be full of care about future generations, they will have to live with your legacy code. Be full of childish curiosity, embrace failures, be open, experiment and share. Because as I heard recently “sharing is caring”, not only about others, but also caring about yourself. Being proud of what you are doing is most rewarding thing in our life. Be proud of who you are, don’t accept mediocrity but don’t punish people for it. Maybe, just maybe they didn’t choose “a way of code” and that’s fine. That’s good. Don’t make people feel bad about (I know it is hard, trust me).
So if code is bad, change it, if your not happy with a conference talk, submit your own, if you think your manager is an ass hole, become one, for a moment, to see it , to feel it. Do anything, to be proud of who you are, to be proud of your work.
I love when my brain has some spare cycles. I love this feeling of being bored. And I love this moment when one of my brain’s cells wake up and shouts out loud, “hey, let’s do something”. In these moments everything looks like a fun thing to do, the next most important project that is going to change the world. I have so many such projects, not finished, never actually fully implemented, the next big thing. Because life is too short to finish projects
.
This time, one of my projects turned into something that works, and I have been using it for the last two years more than 20 times, a Neo4j workshop.
It was a dark, snowy night in Cracow, Poland when said to myself, “hey, what if, I put all my emails into Neo4j”. Doesn’t this sound like a fun project? Of course, it does. You may ask, so what’s the business value of such project. Let me explain you, my business value minded people, fun is the ultimate value, together with exploration and finding people I can go for a drink. That’s what I call business value.
So let me welcome you to a journey on how to find your “true” friends by analysing your inbox in a graph.
This post assumes, at least, basic knowledge about Neo4j and Cypher. It is for people who already started their exploration of this amazing land. This is just one of many ideas you can play with, to get better feel of graphs.
In case you are not familiar with Neo4j, a my favourite graph database, it should be fine as well, as long as you follow my instructions, have downloaded neo4j copy from http://neo4j.com/download/ and read tutorial how to run neo4j shell :).
Enjoy the ride!
First of all we will need a way to import all of your emails into graph. This is kind of boring task. So I will not go into much details about implementation, as you can find it at http://bitbucket.org/kcrimson/recograph.
hg clone http://bitbucket.org/kcrimson/recograph cd recograph mvn package cd target/appassembler/bin ./import-gmail -u <your@user.name> -password <your.password> -f Inbox
`-f` option allows you to set folders you want to fetch. It also supports sub folders, so you can pass for example `-f “[Gmail]/AnotherFolder”`. In order to import all of your emails you can just fetch single folder, called “[Gmail]/Wszystkie”, if you use GMail with Polish language.
WARNING
GMail users need to enable IMAP protocol.
It will take a while. It will probably fail few times. But don’t worry you can re-run it as many times as you wish, as it will import every email just once, based of message identifier. You can also check file `import-gmail.log` to check for status of the import and lookup for errors. If you have enough patience, after an hour or so (depending on the size of your inbox) you will end-up with `mails` directory with Neo4j graph in it. With a structure which looks more or less like this.
So what’s in it? All your mails as nodes with label Message and properties `Subject`, `Message-Id` and `In-Reply-To` and all of your recipients as nodes with label InternetAddress and properties `address` and `personal`. You also find relationships of types FROM,TO,CC and BCC, between your Messages and InternetAddreesses.
This is a simple graph, which doesn’t tell you much about your social life, let’s tune it a little bit.
First thing that came to my mind was, emails threads, how can I find all emails threads in my inbox. After few “I am feeling lucky” searches I found that there are actually two headers in emails, which I can use to explore threads. They are called `Message-Id` and `In-Reply-To`. So it was a time to materialize few relationships.
CREATE INDEX ON :Message(`Message-Id`); CREATE INDEX ON :Message(`In-Reply-To`);
MATCH (n:Message) MATCH (m:Message) WHERE n.`Message-Id`=m.`In-Reply-To` CREATE n<-[:IN_REPLY_TO]-m;
What happened here is, first I created two indexes, just to spice up things, sorry, speed up, when I will be marching through the whole graph, to find nodes (mails) which are related. Next is a lovely Cypher query which matches nodes where Message-Id equals In-Reply-To and creates a new relationship. This is the thing I love in graphs, exploring, building layers of knowledge from simple facts, layer after layer, till the moment self-consciousness of graph awakens
So as you can see it is no longer a simple graph. We have enriched it with `IN_REPLY_TO` relationship, which points to mail which was a response to another mail.
Now, let’s look for a longest email thread in our mailbox.
MATCH thread=((firstMail:Message)<-[:IN_REPLY_TO*]-(lastMail:Message)) WHERE NOT ()<-[:IN_REPLY_TO]-firstMail AND NOT (lastMail)<-[:IN_REPLY_TO]-() WITH firstMail, length(nodes(thread)) AS threadLength RETURN firstMail.Subject, threadLength ORDER BY threadLength DESC LIMIT 5;
Isn’t it lovely? I will print this query, someday, on my t-shirt. It is so beautiful.
To be honest it took me some time to make this query use the power of Cyhper, in its pure form. The trick is in the question I asked myself. What is email thread? For the sake of simplicity, something that has beginning and end. I know it is wrong, because email threads are actually trees, not lists, but this is something for another post, to explore. I had to find first and last email in the thread. Once you draw it on a piece of paper it is easy. First email in thread is the mail which doesn’t have outgoing `IN_REPLY_TO` relationships, NOT ()<-[:IN_REPLY_TO]-firstMail, and last email is the email which doesn’t have incoming `IN_REPLY_TO` relationship, NOT (lastMail)<-[:IN_REPLY_TO]-(). Just few ASCII arrows away.
This is a place when things get even more interesting. Once I have email threads I can try to explore clusters of friends. But let’s come up with definition of friend. If I have your email address, it means I am your friend (it is not so straightforward in Poland :)). It leads to interesting observation, and another layer of facts. If I send email to Tomasz and Kuba, it means they are my friends. But when Kuba responds to all, adding Wiktor, it means he knows Wiktor, but Wiktor not necessarily knows Tomasz and me. And here is another crazy query, with even more arrows.
OPTIONAL MATCH (m:Message)<-[:IN_REPLY_TO]-(n:Message)-[:TO|CC|BCC]->r WHERE NOT m-[:TO|CC|BCC]->r WITH n,r MATCH n-[:FROM]->f MERGE r-[:KNOWS]-f;
So I am looking for all recipients of the email, and check if there are relationships to these recipients in a previous email in the thread. For these recipients I create `KNOWS` relationship.
And last but not least. Clusters of friends. This time we will use well-known algorithm, called local clustering coefficient. I am not going to bore you with details, but in short, it counts number of relationships between your direct friends versus number of potential relationships between them. Which in short, looks like this
MATCH (a)-[:KNOWS]-(b) WITH a, count(distinct b) as neighbours MATCH (a)-[:KNOWS]-()-[r:KNOWS]-()-[:KNOWS]-(a) WHERE exists(a.name) WITH a, neighbours, count(distinct r) AS connected_neighbours WHERE neighbours>1 RETURN a.name, (2*toFloat(connected_neighbours))/(neighbours*(neighbours-1))
In short clustering coefficient equal 1, means that friends of this person know each other very well, really close circle of friends. We could also explore complex world of cliques, for example Bron–Kerbosch algorithm. One interesting fact, finding maximal clique in a graph is on the list of the most complex computational problems, as listed on Wikipedia.
And that’s all folks, I hope you enjoyed it, especially the moments when out of simple facts, we were able to build knowledge.
There is of course a lots of things we can, like adding time dimension to our graph. We can use `Date` header and spread time tree and look at who responded to our emails within minutes or days.
I hope also I have shown you that Cypher is a great language, but as with every abstraction you have to unlearn old tricks.
Happy traversing.
Actually, when I was writing this article I was on vacation, relaxed, far away from work, with limited access to Internet, reading books, enjoying time with kids. It is just an example of what happens when I spend too much time away from coding. I start to think. Seriously. I recommend this exercise to everybody.
Oh, and one more note, this article contains strong language. Some statistics, word bullshit is used nine times (including this sentence) and word fuck is used two times (including this sentence). You are likely to disagree with my points, and that’s ok. It is ok to disagree and have different point of view, and even call me dump. I would love to hear some arguments with this article, because it would mean I am not surrounded by zombies. This would bring some hope into my life.
I’m tired. This is the feeling I’m hiding deep inside of me. It is not a burnout, because I have it under control.
But I am tired. After 20 years (or more, depends how you count), I can say it out loud. I am tired.
There have recently been a couple of blog posts, articles on happiness level in our overpaid, teased by recruiters with shining like a diamond benefits, eco offices with best of breed fair trade coffee, thin and lightweight Macs, scrum owners (actually psychologists in disguise, and you know what I think about psychology) who are here for you to listen and tokenize and parse your feedback, with tons of empathy for your everyday struggle with The System.
The System, the Monster your created, so later on you can refactor it, using new shining Sword Of New Technology Blessed On Another Cool Conference By Some Unknown Priest of Whatever. It reminds me about the “DragonHeart” movie, where the last living dragon, played by Sean Connery, makes a deal with dragonslayer. The dragon rides villages, burns and destroys everything, then comes the dragonslayer, he “slays” the dragon in well played “heroic fight”, gets the prize and they move to the next village. The System is the dragon and we are the dragonslayers. We get prizes, and move on. The Dragon (khmmm… The System), stays the same. Complex, with dependencies descriptors, getting bigger and bigger, and soon, all our microservices will turn into The System. Where information density is equal on all levels of abstractions, where partial decomposition is impossible and database (this time NoSQL database) is central design lock.
Bullshit.
Don’t get me wrong. I am part of all of this. It is not like I am complaining because I am not part of this craze. I got angry when somebody wrote an article in Polish blogosphere about getting government to regulate this market as it creates huge gap between people and it’s just unfair, destabilizing economy and other interesting observations.
People are asking me what I do for a living. They are like, really? Can I be a programmer too? Can you teach me this stuff? They give up quickly, because what motivates them is money.
What motivates us? Mastery (and money, you hypocritical bastards). Getting better at things. We have even bring back to life a centuries old idea of craftsmanship.
You say, it is so cool we can learn, improve, gain experience, continuously. Some say, it is part of our work ethic (and even write books about it). And I say bullshit, bingo. Double soy latte, bullshit. Who gave me the right to say so? Let me tell you my young friend. I gave myself such right, right to be honest with all of you, and yes I was not touching lemon vodka since last conference after party
. I am relaxed as hell.
I am feeding my brain with whitepapers and paperswelove since 1998. I consume weekly, on average one PDF, sometimes I find monsters like “Software and Hardware Techniques for Efficient Polymorphic Calls”, which took me a month to digest. I still don’t understand last two paragraphs, which makes me feel guilty. Yes, I am talking about this feeling, when you go to bed too early, with a PDF half printed, with links on Twitter you put on your TODO list, with all tweets from your wolfpack, who tweet on lovely, just found paper from 1976 on virtual machines construction, “just in time compilation” and other stuff. You feel guilty for sleeping, while others are expanding their boundaries of understanding. And what do you do? On a next meetup, conference you meet a group of young adepts of dark arcane magic, and you pass the guilt. You tell them about message selectors, single dispatch and overhead of dispatch table row compaction algorithm. They feel guilty, you earn +10 points in necromancy, and you’re free from guilt. Nice trick. This is how it works. Our industry is built on vicious circle of guilt.
Where is the trick? Why can’t we break this cycle? Because our brain is limited, our time is limited and yet it’s evolution, baby. One paper leads to another. One research didn’t prove anything but built foundations for another research. Because we made all so complex, adding layers of abstraction, to simplify things, that we failed (or forgot) to make things simple. Thus, one paper is never enough. Every paper has references section, which is like an address to a next drug dealer. And we need drugs to silence this feeling, this scratching of the soul, that you don’t know enough. It is so hard to admit: oops I didn’t read the original Smalltalk paper; I don’t know that Strongtalk made foundations for inlining in JVM (or it was Self? fuck it). You see. Don’t you feel like shit? I feel like a king. Have you ever thought, if a person with 20 years in business feels like this, what people, new to this whole circus, feel? I am talking about empathy. Yes, your are reading the blog post of a person, who starts his presentations with the sentence “in general, I don’t like people”.
We are in an unpleasant place to be, despite salary, job opportunities, free gym, free beer and whatever free HR department comes with. We are under pressure from our peers to understand, not to stop in our efforts. In the meantime, our lives become empty as we lose track of our ultimate goal: to conquer The System, to work less, and have more fun. Life is not a game. It is only a chance you have, so don’t waste it on feeling guilty.
I can rant for chapters about other tricks we are playing with ourselves. Like hiring for culture fit, 20% of time for doing cool stuff at work (if your are not doing cool stuff at work and company needs to give you 20% special time slot, something is wrong). But it is waste of your and my time. And let’s not cheat ourselves. Somebody needs to maintain the shit I created. Life is not all about roses and love, sex and rock’n’roll. Sometimes you need to dive in shit until you smell it.
There are two (or more, I will not count) things which amplify this behaviour, so let’s find the leverage point (oh, lovely systems thinking).
First thing is hard, like really, really hard. It is so hard, I will not even try to tackle it. I gave up a long time ago. But I am talking about it when my neurons are floating in toxic liquids, with cherry flavour. People who party with me are so tired of this topic, that they usually release another bottle of poison, just to make me stop talking about it. We don’t have foundations, like 10 papers which define software engineering, which define our discipline, which, if we follow them, The System, the dragon will be under control. It will not be killing random people with fire. What’s funny, some of us say, we have. We have our 10 commandments. But it is like with the religion. Every camp has different rules and as usually with us, humans, pitiful, war loving animals, we fight each other. High Priests of Higher Kinded Types camp fight with knights from Order of Temple Of Object Oriented, wearing steel and composite helmets Paladins Of Strongly Typed Church Of Truly One Type To Rule Them All are filling battle fields with bodies of Dynamically Typed Ninjas of Duck Typing. Somewhere behind all of this, Monks Of Curly Braces train new armies to, someday, confront Reactive Functional Hippies Of Evergreens. And, each camp has its prophet, and priests, and followers. They invest their time in finding new souls to join their ranks. This is how it is. One giant battlefield, and every project is a battle and every line of code is a prove. That this is bullshit. And yet, don’t forget about machine loving Transformers Of Mechanical Sympathy, kicking asses of Viking Warriors Of Immutable Containers. Everybody fights for your soul, yours, you young padawan. Will we find peace? Maybe, someday, but I am not brave enough to follow this path. Because, I am tired.
So what’s the second thing? This is something we can try to change. Together, create an utopia, where not knowing is cool, where new people will feel welcomed. But there always will be people who know and want to share their path, their experience and time.
Hey, wait, aren’t we doing it already? Like sharing on conferences and meetups?
Yes, but I think, it is just a play. Conferences have their goals and targets. They usually are part of one or another camp. We have conferences for A and B, and talking about C ,and where D is the new cool. That’s the first challenge.
The second challenge is: “celebrities”, “hot names”, “he always has room full of people”. Stars. I am not saying these people are doing a collateral damage. No, not in general (but some of them should get life time ban of public speaking, it’s just my personal opinion). Yeah, I am like shitting in my own nest (old polish idiom). But, you know what? It’s all bullshit. Being nice for the sake of being nice, isn’t part part of my work (or life) ethic. Deal with it and I will deal with consequences.
We are like masonry. It’s hard to join our secret club, it is hard to get to a new level. Because, we work together, review our presentations together, travel, and party together. I hear it more often these days. That we even create our own ghettos during parties. There are special speakers dinners, and speakers rooms, and speakers pub crawling, and special insight code, language we use.
Some people even make a living out of this, not having written production line of code in ages. What a bullshit.
I have recently met two of high rank speakers and tried to talk with them. They were so focused on defending their way of seeing things, that I felt like shit after the discussion. Then it struck me. They were protecting their job security. It was more important for them to protect their dragon, then to be open, honest, ready for discussion about framework, technology or language they promote. If you don’t have enough patience, if you think speaking at main stage makes you a superior human, you are in a wrong business.You write shitty code too (or don’t write it at all, bingo), you feed the dragon like we do. What makes you special?
Don’t let me continue. Yes, I know it is extra work, extra effort. But we are in this together. Together, we shape this industry. Without the audience, we are nothing. Our ideas are dead, without people who actually implement them. They have important feedback for us, and this feedback is: stop creating ghettos. This is my another observation. People are either afraid to come and ask, or we are building walls around us. I thought, hey, I am not nice, it is not welcoming when you say “in general, I don’t like people”.
Recently, on one of the conferences, I understood, it is not a problem. It is just a filter for people who don’t get sarcasm. The problem is, even when people get sarcasm, they are trapped in this guilty hell, because they follow us on Twitter, they see my sick tweets at 2:05 about another PDF I found, and they didn’t read it.
Hey, I say it is ok. It is cool not to know. I have read it. Stay cool, give me a shot of something strong and cold, and I will tell you what I understood, why I think it is important.
Somebody asked me, why I do it for free. I don’t know. Maybe, I like when I turn my sleepless nights into something worth it. And, of course, I love to be the wisest man on the block:), c’mon, deep inside it is all about this. We are like a 10 year old boys playing football in the backyard, and somebody just showed a new trick.
I’m tired of chasing my dream alone. Having to read all these papers. Not knowing what’s next.
Tired of our wars, because they are pointless, no matter FP, OOP, AOP, FRP, at the end there is CPU and OS. And they don’t care. And we created abstractions which are here rather to make people feel stupid, than to make things simpler. We feed new dragons, when the old one is still alive and doing well.
I am dreaming about a place where there is no audience and speakers, where there is one afterparty, where lack of knowledge is privilege, and if you know and understand, it is your duty to share it. Where there are no camps, no knights, no dragons. A sacred place where, before you enter, you have to leave all of your weapons, langs, frameworks and libs, at the front door. Where the only stars are inside us, lonely stars looking for understanding to become one.To explore space. To enjoy the things we don’t know we don’t know.
I am naive.
I know, this is my utopia. I need to free myself from being tired talking about technology, about frameworks, and libs, and containers.
I am not saying they are not important. I see them like a forest full of trees. They hide the truth, that the king is naked and the princess is ugly, and there is no handsome prince on a white horse.
The truth is, deep inside, behind all of the abstraction layers, there is OS and CPU, deep inside, behind all of ORM frameworks there is SQL and deep inside, behind all of MVC frameworks and JS libs dependency hell there is DOM and HTTP, for God’s sake.
I would love to see more and more talks about foundations, about things that build frameworks, languages and libraries, layers of abstraction below. These are stable layers of the system to which you should design dependencies of your knowledge and experience. Build on things that will not change in a year, oops, month.
Would you do me a favour? Next time you are going to talk about a new framework, rather talk about the layer this framework tries to hide and “simplify”.
… yeah, this article was not reviewed, accepted or read by any of my friends and peers, because I don’t strive for perfection, I love chaos and improvisations. So sorry for all mistakes
Some time ago I found my old articles, from previous blog. Some of them are 5 or more years old, written in Polish, from times where I was spending more time far away from coding. In those times, I was focused on managing and “architecting”. It gave me a strange perspective on things. Few months back, I woke up with this idea to comment and criticize my own thoughts. I don’t know if you are going to like it, but at least for one person, it was fun
And it was me.
Before we start, lots of love and thanks to my wife, who took this burden on her shoulders, and translated these articles.
I include the whole article with my comments and thoughts inlined. Enjoy, or not
The 1980’s. Simple Minds and their classic Alive and Kicking (especially in
live version).
I have not been here for a long time. I neglected this small group of
readers, the blogosphere, and Googlebot. In the meantime, I changed my job, explored Northern Europe, survived the deadly “afterparty” after the first edition of the refreshed Confitura, and started a PhD in JBoss 4.2.3 GA.
I shut myself in my small space, at the new desk with a view of Krakow and repeat like mantra the mvn clean package, run and subsequent curse through gritted teeth and git reset. I turned off Google Reader, InfoQ and a few other “disturbers”. The only entertainment is to keep alive the noble idea of “zero inbox”.
I think I owe you a little bit of explanation. Back in days, when I was writing the original version, I worked as a “chief architect” (I still don’t know what that means). I worked with a system with ~500 kloc of Java + ~500 kloc of Coldfusion and around ~400 kloc of JavaScript (jQuery) code. It was like a baby born after a heavy drinking party with T Rex, xenomporh and Princess Xena, but you would not know who had sex with who.
Everything was dipped like nachos in spicy salsa in JBoss 4.2.3.custom_fixes_and_extensions, something I consider last stable version of JBoss. This application was a pure proof that it is not a challenge to make simple things simple, but it is a true mastery to turn CRUD application into “feature set” orgy, customizations “gang bang” and “we have wizards everywhere” extravaganza. Deal with it.
For clarity, the code I am working on, is not bad, terrible, unreadable or fatal. It is also not that it underwhelmed me, disappointed or plunged into the black as espresso despair. The man already knows what to expect, and accepts the fact that the client is the Lord and basta. And a racial purity of the code, its kosher, remains in a beer reverie in a cigarette smoke.
Actually, it was. Bad, terrible, unreadable and fatal code. This is, my dear friend, the first stage of working with legacy systems. Just admit it and say it out loud: “this code is garbage”. But do it one time. I think we spent too much time on whingeing: “who wrote this code?”, “this is shit”, “I am not going to work with it”, “this is pure spaghetti with hand grenade in it”. It is, and so what?
Keep in mind, you Mr. Perfect High And Mighty, that probably somebody, in 10 years from now, will tell all of these things about your “shiny code”.
Yes, somebody had a bad day 20 years ago. Somebody was forced to follow “architecture guidelines”. Somebody didn’t care at all, because he knew he would be moved to another project in a week. I can think about hundreds of reasons why this code is bad. There are days in my life where I shouldn’t be allowed to code or even get close to an IDE. Because I am fed up, because I am tired or with a mountain high hangover. Or my kids are sick, or my parents are not well.
First of all, you and I, we are human beings and work is not life. (Maybe it is when you are 20 or 30 but not when you are 40
)
So just don’t try to improve your self-esteem by saying: “I would never ever write this code this way, these people were stupid”. I bet you would have written even worse code in the same context this person was in.
So, “yes, this code is bad”, end of story. Let’s do something about it. The first step in a whole organization is to admit how bad it is and embrace the fact that it actually makes money, because you still have your job. Period.
So what pushes me to these late evening thoughts? For some time now, I live outside the mainstream of a TDD, BDD, DDD and Software Craftsmanship (now these toys do not quite fit into my sandbox). I realized how hard it is to work without these tools. How hard, at times, it was when I could use them on a daily basis but it is even harder to survive without them (maybe it’s just a matter of habit). Currently, with bare hands, I decompose the code structure, make destruction of local dependencies, discourage next application layers, and remove/disassemble piles of false assumptions and recognized truths. My human nature does not allow me, however, to accept the existing state of things and trying to understand the motivations and context.
What did go wrong? Because it went wrong all the way down to the gates of hell. It exploded in our hands right into faces of surprised CEO and other C-level executives. Oh God, this was so much fun talking with these people during these days.
I thought that quality was important. I was so fooled by all this software quality talks that it narrowed my view on the system. Don’t get me wrong, it is important but not when you have to dive into savage. When you are right before jump into shit and you don’t know how deep it is, last thing you need is quality. You need a world-class life support system.
You need a working continuous integration. You need tests which are not “false positive” all the time. You need bloody fast build times so you can experiment with “workarounds” all the day and all of the night. You need static code analysis quality gates which will make sure you will not get a system into worse state with your refactorings. You need access to production, without stupid procedures which give you systems logs a week after crash. As you are diving into shit, so how will these procedures make your journey more enjoyable?
And last but not least. You have to remove all signs of bullshit in management of your organization, at all levels. All this “our best people are working on it” (of course they are), “we need to break silos” (of course we need to, first start with all these management levels), “we have certain values” (no shit, really? I am proud of you. Do your values reflect quality of code you have?), “business value is priority” (how about all the money flushed down the pipes with all this technical debt you have?).
Back to the topic.
Fast turnaround and feedback is key. Especially in first days, weeks and months. It means it is more important to have “one click deployment” and working development environment on people’s laptops than “clean code”. Remember, the deeper you dive, your focus will need to change. But in first moments, everything you do, should be driven by this question: “Will I break the system? Yes? How fast will I recover?”
And this is something I have ignored. I thought that if I had a better coverage, response for class or cohesion metrics, everything would be fine.
It wasn’t.
I try not to search the guilty (git log will not change anything). I assume the good will of creators, unclear requirements and pressure of time.
So where does the reason lie? In JBoss AS. Do not take it too personally. It might as well be the Spring Framework, OSGi, Grails or homemade, tailor-made platform. Each platform/framework, or other kind of animal, carries the initial technical debt. You think that JBoss does not have a technical debt, and its code is clean as the proverbial glass of “wyborowa”. Stop delude yourselves. Every piece of code that was created is burdened with lesser or greater debt.
There is also another category of debt that comes along with a choice of “this only solution”.
Our lack of complete knowledge of the selected technology, gaps in documentation, outdated tutorials, things that developers do not want to speak loud and “miraculous” solutions available in the network. So this is what Dan North presented in the “Deliberate Discovery”. Our debt is what we do not know that we do not know :). And in this way, the simple things, as provided in platform are entwined by “hacks”, aspects, modified libraries just to make ends meet and release it.
I , kind of, still agree with these statements. With a small addition. Very often, we invest in certain technology, make it a backbone of the whole system. We love it so much that we see every problem through it. Even when deep in the heart we know it doesn’t fit our stack, our needs or our infrastructure. But because we have invested in it so much, so many hours, we think it is “the solution” to all of our problems. Sooner or later we are going to create “extensions”, “patches” and “proxies” to justify its existence in our environment. The truth is, when it doesn’t fit, it doesn’t fit. I see this kind of behaviour as one of key technical debt sources. Together with “I know shit about this NoSQL database, but will use it everywhere” and “cool kids use it these days, it has to be good”.
One thing that can be helpful in these moments is architecture decisions log. A place where you store all your stupid decisions with a context. Why have you decided to use X? What kind of assumptions did you make about X? When these assumptions appear to be false, you will have a clear evidence why it is a stupid choice.
And I didn’t have architecture decisions log.
Of course, you can still throw a pebble of our ignorance in the area of the domain business but at this stage I’m not going to wear a noose around my neck.
Now we get to the root of the problem. How important is the choice of our target, unique and beloved platform? In my opinion, it does not quite matter, and regardless of the choice, we incur a technical debt prior to writing the first line of code. And we will amaze future generations by our ignorance.
Oh, how wrong I was.
Yes, it does matter. The technology you choose, will drive your success or failure, will be there with you till the end of your life, will be your “hand of doom” in these lovely moments where whole your SRE team will sit by the camp fire and watch your DC burn in flames.
So why is the technology important? That’s tricky. In my old brain there is this hidden thought, hidden deep in darkest corners of my being.
“It is important to know how easy it will be to remove this technology from the stack, when the time comes”
Really.
There are frameworks, libraries and other stuff, which require you to bond your code with it really, really strong. You have to put import, annotation, extend, here and there. And slowly, your code is strangled into the framework, you cannot move, you cannot breathe, you cannot do anything which is not permitted by “uber framework”.
What if we could build our applications without single import from a framework whose name I am not going to reveal? What if we could build whole systems just using language features and its standard library?
That’s nothing new. Ports & adapters or onion architecture.
But hey, sooner or later, you will have to change your technology stack. Why do you need to change your so-called “business logic” code? It’s orthogonal concern for me.
Don’t let your framework strangle your “business logic” code.
Our debt will grow with each new version of JBoss or Spring to which we will not have time to move our application. Theoretically, the ignorance of the creators of those platforms will diminish with each new version. So lagging is not in our interest. But who can afford it? It is better to run up a debt.
It will not be shown in the annual financial statements anyway.
What’s interesting, we have not yet worked out a clear and effective approach to such problems (talking about “legacy code, architecture and design”). I know you will bombard me now with comments about my reading ignorance. But of course, I did read the “Working with legacy code” and “Refactoring to patterns”. However, each such application with which you will work, is unique in the set “brainfucks” which surrendered its creator. And that is why I believe, that if you seek real challenges, intellectual puzzles, and work that will bring you waves of elation and the avalanche of falls, look for some “legacy” system. Personally, I believe that none of the new, written from scratch applications, in the latest “frameworks”, extremely dynamic languages, giving you sharp tools into your hands, does not make so much pleasure and pride in a job well done as solid “legacy” put in your hands for straightening. Why, you ask?
Welcome to the moment when you start to accuse about lack of diversity and empathy in my way of thinking. But I have stopped worrying about what people thought ages ago (except carefully chosen a small circle of people who I respect).
Ready?
“Everyone can code”. That’s a terrifying idea. We should, of course, create an open, welcoming place where everyone would feel safe, regardless of our background, skills and things that make as different (in the first blink of an eye). The thing that we have to even discuss it, it’s a pity and shame for us as human beings. That we are so miserable, that we put people in boxes, and create boundaries. As we now talk about it, it creates the other extreme of this problem. However, let’s leave it for another discussion.
Do we really believe that the only way to solve shortage of talent in IT is to have more people coding. Really? If it is true, it means that we have also failed as an industry in building right tools, paradigms and practices. Do you remember this joke about pregnant woman and project management? This is what I am talking about.
I have recently read a nice write up about Alan’s Kay idea of Moore’s law in software development. That’s sad, that it didn’t happen. That hardware outperforms software at so many levels. That we are where we are mainly because of advances in hardware. Maybe I am wrong. I hope so.
Good app is built around hacks. This is not TDD, BDD, DDD and Software Craftsmanship but HFDD (Hack it and Fuck it Driven Development). The new better paradigm
The future of IT, to the fear and general scandal, is programming archeology. And yet it is because of archaeology we learn about the past generations and their customs, we reconstruct course of events, we expand our experience and thus we become better. Sometimes also, we discover an extinct technology, and when we see that it is good, we put the new sticker on it and sell as a breakthrough discovery.
That’s the end. No final word.No closing comment. Just many thanks to Wiktor, Krzysiek, Kuba, Paweł and Jarek. You didn’t know, but you were inspiration too many thoughts in this post. I have a feeling that lots of these thoughts deserve their own post but I am too lazy to continue;)
