Join the UML Forum Discussion Group: 900+ members, 600+ topics and growing! Click here to join…

How to Define UML Tool Evaluation Criteria for Your Team & Project

The UML modeling tools reviewed on this site are evaluated using weighted evaluation criteria. This article describes the UML modeling tool evaluation criteria applied, and shows how they can be weighted to customize tool evaluations to meet team and project specific needs.

Evaluation Criteria

The following evaluation criteria separate functional and non-functional tool features. For functional tool features they distinguish between diagram drawing functions , which are can be supported by templates for popular drawing tools (e.g., Visio, OpenOffice Draw, iDraw), from modeling tool specific functions, such as Round-Trip Engineering functions (code generation, reverse engineering).

Usability

We evaluate modeling tool Usability while exercising the tool Functional Features described below. Usability factors that we evaluate include, but are not limited to: User Interface (UI) design, UI learnability, UI efficiency (as measured by #UI gestures/function), UI response times, context-sensitive help, and technical documentation.

Functional Features

Functional features that we evaluate are categorized into Drawing features and Round-Trip Engineering features:

Drawing Features

We evaluate basic, intermediate, and advanced drawing features of modeling tools. Basic drawing features that we assess include, but are not limited to: opening/saving models, opening/saving diagrams, drawing model elements and relationships, editing properties of elements and relationships, and multiple undo/redo capabilities. Intermediate drawing features that we assess include, but are not limited to: support for recursive design techniques, requirements traceability across diagram types, generating reporttables from diagrams, and automated document generation. Advanced drawing features that we assess include, but are not limited to: model management and tool customizability via profiles, options, and scripts.

Round-Trip Engineering Features

In the context of visual modeling with UML we define Round-Trip Engineering (RTE) as the seamless integration of forward-engineering (generating software code from models) and reverse-engineering (generating models from software code) functions while preserving both model and software integrity. Round-Trip Engineering features that we evaluate for UML modeling tools include, but are not limited to: forward engineering for various object-oriented programming languages (Java, C#, C++), reverse engineering for the same object-oriented programming languages, and change management processes that preserve the integrity of both models and software code.

Standards Compliance & Interoperability

We evaluate tool compliance to UML notation (syntax) and semantics as defined by the current UML 2.x specification. We also evaluate the interoperability of XMI files generated for model import/export, and the interoperability of UML diagrams with SysML and BPMN diagrams for modeling projects that require mixed language usages.

Technical & Team Modeling Support

We evaluate general tool support to include installation, set up, update mechanisms, and technical support.  We also evaluate team modeling support for project sharing, versioning, and user permissions.

Value

We calculate Value as a function of Feature (Completeness,Quality) / Price in order to rank tools in relative order.

Weighting Criteria

After you have specified your selection criteria you should assign numerical weights to them to quantify their relative importance. The assignment of relative numerical weights to your criteria allows you to tailor your tool selection to specific team and project needs, and it can also reduce evaluator bias. For example, consider weighting the evaluation criteria described above as follows:

\(Score_ {tool} = \overline{x} = \sum_{i=1}^{n} w_i * c_i\)

where Tool Evaluation Score (\(Score_ {tool}\)) is the weighted mean (\(\overline{x}\)) of the following six (6) evaluation criteria (c1 = Usability, c2 = Functionality: Drawing, c3 = Functionality: Round-Trip Engineering, c4 = Standards & Interoperability, c5 = Technical & Team Modeling Support, c6 =Value) with corresponding weights (w1 = 15%, w2 = 15%, w3 = 15%, w4 = 15%, w5 = 10%, w6 = 15%).

The weights that you apply to your own UML tool evaluations should be adjusted for your team and project modeling needs.

Comments on this entry are closed.