Moose 5.0 roadmap (video)

Stephan Eggermont kindly made available the video of the presentation I gave at the Moose Day about the Moose 5.0 roadmap.

More details about the talk can be found in the previous blog post.

Moose 5.0 Roadmap

PharoConf | MooseDay | 2013 is now over. It was an exciting event. The mixture of roadmap talks, tutorials and demos kept us all engaged. I for one learnt quite a bit.

While PharoConf was focused on various Pharo technologies (from the low-level vectorial canvas to the neat Mongo support, and to the funky debugger infrastructure), MooseDay had a narrower focus. To set the context, the MooseDay kicked off with me laying out what Moose is, and where we want to get Moose in the following year. The roadmap talk is summarized below.

Over the past two years we have released four versions of Moose:

There are several things to notice. First, every major release of Moose came with significant new features. Second, the releases of Moose go almost hand in hand with the releases of Pharo. This is because the two projects both complement and overlap each other both in terms of functionality and in terms of the people involved.

In the near future we plan the following versions:

The current plan for 5.0 go as follows:

Consolidating the visualization engines

Roassal will become the core visualization engine, and on top of it we will implement both Mondrian for easy graph visualizations (already supported to a large extent), and EyeSee for charts.

Roassal + vectorial support

A major improvement is expected in the rendering area. Roassal will move from a bitmap backend to a vector-based one. This will be achieved by adopting the Athens canvas.

PetitParser + Islands

PetitParser already offers an impressive infrastructure for building parsers. We want to add One direction of improvement comes from improving support for island parsing.

One infrastructure for hierarchical graphs

At the moment, we have three implementations that use the concept of hierarchical graphs (DSM, Carrack, and Quicksilver). We want to move towards one central support. Most likely this will come from the Quicksilver project.

Fame/FAMIX + Traits

An important feature of Moose comes from the ease of building new models. This is achieved through Fame meta-descriptions, and when it comes to software models, the FAMIX family of meta-models offers already a good start that can be reused in different contexts. However, until now, the reuse mainly depends on inheritance, and as a consequence, in less classic cases it is difficult to get a small meta-model that exactly fits the purpose without any superfluous details. To fix this, we will work on introducing a new mechanism of reuse based on Traits.

GToolkit + Glamour

Last but least, both Glamour and the Glamorous Toolkit will continue to evolve. In particular, the goal of the toolkit is to offer an extensible and assessment-enabled development environment for Pharo.

Stronger web presence

Moose' focus on making crafting analysis easy is rather unique among similar projects. The webpage will have to change to make this mission more explicit.

The slides I used for the presentation can be seen below.

Demystifying cycle detection (with Moose)

Cyclic dependencies is one of those concerns that occupy a top position in developer's consciousness. Usually, it comes right after the size of the code, and next to dead code.

Cycles are bad, goes the urban legend. But, why are they bad? And what cycles are we talking about? Are all cycles bad? Are cycles within packages bad? Are cycles within projects bad? Are cycles between classes bad? Are they all bad in the same way?

Generalization usually leads to easy solutions, but at the same time, every generalization comes with a price. However, in any serious engineering discipline, details are important, and we should be able to deal with them.

Let's consider some Java-based examples and see how details can play an important role. The examples are accompanied by detection approaches based on Moose.

Detecting all cycles between packages

The simplest form of cycle detection is to simply retrieve all cyclic dependencies between all namespaces (in our meta-model, Java packages are called namespaces) in the system. Moose comes with a built-in cycle detection algorithm that can be used like:

mooseModel allNamespaces cyclesToAllProviderNamespaces.

This expression is applied on a model of a Java system, and it returns groups of namespaces that form cycles within the given system. These groups can later be manipulated in various ways. For example, when applied on a Java system I dealt with recently, I obtained some 33 cycles. One of these cycles involved 38 namespaces. The pictures below show the namespaces involved in the cycle and their interdependencies (as a Dependency Structure Matrix, and as a graph).

Simple. But, is it efficient? Do you truly want to deal with all dependencies throughout your entire system at once? Often this is not desired, especially when you start to look into cycles late in the project.

Cycles inside components only

When working on a large project you might want to restrict the detection to only a part of it based on some criteria. A common approach is to identify sub-projects or components and look at cycles only inside of these.

(mooseModel allNamespaces select: #isPartOfMyComponent)
  cyclesToAllProviderNamespaces

In this expression, we first select the namespaces from the desired component via the isPartOfMyComponent custom method, and then simply call the detection of cycles on the result. What is isPartOfMyComponent? It is an extension that you have defined. For example, if you can identify a component based on a naming convention, a possible implementation would look like:

FAMIXNamespace>>isPartOfMyComponent
  ^ self mooseName beginsWith: 'com::example::mycomponent'

Cycles between components

Looking inside a component reveals detailed cycles. A similarly legitimate idea would be to look for cycles between components to identify high-level architectural violations.

This is a slightly more complicated issue. For this, we need two things:

1. A way to identify all components, and
2. A way to identify all their inter-dependencies while taking into account that a component can be formed by many sub parts.

Let's start with identifying the components, and let us suppose that you have a way to identify the components based on names. In this case, we can associate the root namespace with the component and make all sub-namespaces be able to say what is its parent component.

FAMIXNamespace>>component
  self mooseName = 'com::example::mycomponent' ifTrue: [ ^ self ].
  self mooseName = 'com::example::yourcomponent' ifTrue: [ ^ self ].
  "..."
  ^ self belongsTo isNil
      fTrue: [ self ]
      fFalse: [ self belongsTo component ]
FAMIXNamespace>>isComponent
  ^ self component = self

By extending the default meta-model with the notion of your components, you can ask the namespace com::example::mycomponent::implementation for its component and it will return the namespace called com::example::mycomponent.

Once this point achieved, we can tackle the second point:

FAMIXNamespace>>allProviderComponents
  ^ (self allChildScopes flatCollect: #providerNamespaces )
      collectAsSet: #component

The trick here is to collect all dependencies (this is what we essentially get when invoking providerNamespaces) from all the nested namespaces, and then to collect the result as components. Armed with these helper extensions, we can tackle the original problem of identifying cycles between components:

(mooseModel allNamespaces select: #isComponent)
  cyclesToAll: #allProviderComponents

Through this expression, we essentially aggregate the cycles at component level by: (1) applying the detection only for components, and (2) considering dependencies only to components.

Cycles induced by a subset of code entities

But, what happens if your components include tests that introduce strange dependencies that you actually do not want to be bother with at the beginning? Well, you should be able to ignore those dependencies.

FAMIXNamespace>>allOutgoingAssociationsWithoutTests
  ^ self queryAllOutgoingAssociations select: [ :assoc |
      assoc from classScope isJUnit4TestCase not and: [
        assoc to asOrderedCollection anyOne classScope isJUnit4TestCase not ]]
FAMIXNamespace>>providerNamespacesWithoutTests
  ^ self allOutgoingAssociationsWithoutTests
      atNamespaceScope withoutSelfLoops asSet

With these extensions, we define a new way of obtaining provider namespaces by ignoring all associations that involve a test class. Afterwards, we can simply redefine the way we compute allProviderComponents by using the new dependency semantics:

FAMIXNamespace>>allProviderComponents
  ^ (self allChildScopes flatCollect: #providerNamespacesWithoutTests)
      collectAsSet: #component

Cycles between Stateless beans that cause injection issues

While the previous examples showed more classic use cases, sometimes, cycle detection can be used as a tool to identify specific problems. For example, in JBoss 5, there exist a bug in the implementation of the memory pools used for the allocation of beans. However, the bug is only apparent in special circumstances when (1) there exist cyclic dependencies between beans, and (2) the pool is smaller than the maximum amount of beans wanted at a time.

Long story short, if you have cyclic dependencies between beans you are susceptible to experience memory leaks. To guard against the bug, we need to set the pools to have a larger size than the amount of beans used at a time. However, to find out what this right size is, we need to form an idea of the runtime scenarios, and the first step in this direction is to know what cycles between beans are there in the system.

Such a cycle could look like this:

public class BeanA implements IBeanA {
  @EJB
  private IBeanB beanB;
  ...}

public class BeanB implements IBeanB {
  @EJB
  private IBeanA beanA;
  ...}

This is a kind of a cycle, and it should be be detectable like any other cycle. The only problem is that there is no direct cycle in the code because both references point to the interfaces, and not to the implementation. The cycle only happens at runtime. Thus, to detect our problem we have to manufacture the dependency:

(mooseModel allModelClasses select: #isBean)
          cyclesToAll: [ :class |
            class attributes flatCollectAsSet: [ :attribute |
              attribute declaredType
                ifNil: [#()]
                ifNotNil: [:type | type withSubclassHierarchy ] ] ]

The expression gets all the beans from the system, and for each of these will look at the possible cycles induced by the types of the attributes to all the sub types of the declared type.

Summary

The topic of cyclic dependencies occupies a top position in developer's consciousness for good reasons, but it is often surrounded by a mystic aura. The above examples show five different ways in which cycles can be approached within the context of a software system. Of these, the generic detection is the least useful one. Yet, this is what most tools offer. To extract real value, you need to customize the detection for the context of your problem. In order to do that, you need a platform that allows you to customize cheaply. Moose shows how this customization can be both cheap and powerful, in particular when combined with other querying mechanisms.

The bottom line is that cycle detection is a tool that should make it in the toolkit of any software engineer.

Architecture is a group artifact

Architecture is not to be found in the drawing from the architect's office. Architecture is to be located in the source code repository. And in the build system. And in the configuration management repository. And in the deployed artifacts. And in the running system.

All these artifacts determine, in one way or another, the architecture of the system. They all have an impact on it.

These artifacts are created by a group of people. This implies that whoever is involved in placing a piece in the system is also contributing to the architecture of that system. Architecture is a group artifact. As such, it should be regulated by the group, too. Including developers.

One argument that goes against developers having a say in the architecture is that their context is too narrow and therefore, they do not have enough distance to make strategic decisions. This is wrong.

Indeed, their context is typically narrow. And indeed, due to the nature of their work, especially in a world made of short sprints, developers tend to focus on the immediate tasks. This makes them less keen on making decisions that might have far fetching impact, but it does not mean that they are not capable of making long term decisions.

They can make them, and should be at least involved in those decisions. In the end, it is developers that have to deal with the consequences of those decisions. Developers do live in narrow contexts, but this also means that they know those contexts. A broad architecture that does not fit the narrow context is worse than having no architecture at all.

Context matters and any process of decision making benefits from knowing it. The trick is to project a long term goal onto immediate actions. Like anything in software development, it requires a bit of trial and error to find a productive rhythm.

For example, one daily assessment pattern is to have all developers participate in an assessment stand-up in which everyone can raise a technical concern, can debate a concern, and can volunteer to solve it once the consensus have been reached.

The main benefit of the daily assessment exercise does not come from the specific decisions. The critical benefit comes from making architectural decisions explicit, from involving the entire team, and from the daily system cleaning. This works particularly well in agile settings where the team understands the importance of feedback and explicitly invests in obtaining it.

During the daily assessment, discussions only take place if (1) there exist one stakeholder, and (2) the issue has been made apparent through some sort of detection. This has two interesting aspects. First, the stakeholder has a direct interest in the problem and can defend the concerns he raised, thus providing good basis for a healthy debate. Second, the discussion is carried around hands-on examples obtained through checkers, thus ensuring that the debate is concrete. The combination of the two consistently leads to distillation of immediate actions.

Architecture is a group artifact, and it can and should be steered by the group, too.

Moose 4.7

The Moose Suite version 4.7 is out: http://moosetechnology.org/download

What is new:

• Integration of the Roassal visualization engine
• New PetitParser browser with integrated refactorings
• Improved Glamorous Toolkit for Pharo including the Glamorous Inspector and Debugger
• Several features in Glamour including Watcher support (similar to OS X preview)
• Significant improvements in FAMIX
• Improved external VerveineJ parser for Java
• Based on Pharo 1.4

A list of issues addressed in this release can be found at: http://code.google.com/p/moose-technology/issues/list?can=1&q=status=Fixed%20milestone=4.7