jCleanCim readme

jCleanCim is an open source tool for validation and documentation generation from Enterprise Architect UML models of IEC TC57 CIM and IEC61850 UML models.

Up until end of 2015 it has been hosted by the CIM Methods & Tools for Enterprise Integration group on the CIM Users Group web site, with access limited to the CIM and IEC61850 users community members only.

To make it accessible also to non-CIMug members, since 2016, I decided to share it as a full open source tool and host it at my web space.

This is a non-GUI Java application and the Java code is fully platform independent. However, it unfortunately must be run on MS Windows machine due to the usage of Enterprise Architect and MS Word automation libraries (.dlls).

• Documentation
• Which distribution should I download?
• Dependencies
• Performance indicators
• Hints - please read carefully before reporting problems!
• Disclaimer
• What's new in this release

(Old release notes)

Documentation

Once you unzip a jCleanCim distribution, doc directory contains the full documentation. The important parts of the source code are documented and that documentation is generated as so-called javadoc - namely, a set of web pages that allow for easy navigation.

• Each distribution contains an up to date set of slides doc/jCleanCimIntro.pptx - you may want to start from there.
• Each distribution contains the javadoc in doc/api/index.html. The documentation of the root package org.tanjakostic.jcleancim (Description section) is a good starting point.
• Binary distribution contains also the PDF version of that same javadoc, in a file doc/jCleanCim-[version].pdf (source distribution contains the Apache ant build script to produce this document, so it is not packaged in source distributions).
• Source distribution contains the PDF version of the depency graph of the Apache ant build targets (this is useful for a developer only).
• Finally, each distribution contains the test report, run during the build of the distribution, under doc/testReport/index.html. For most users this is not really of interest (but keeps our developer spirit in peace :-).

Which distribution should I download?

jCleanCim is available in two distributions, depending on how you want to use it.

Even if you have a 64-bit Windows OS, ensure you install a 32-bit Java (JRE: runtime) or SDK (software development kit) and have it appear on your PATH before the potentially already installed 64-bit Java, because Enterprise Architect is still a 32-bit application and requires a 32-bit Java. See the commented text in the run.bat script.

Note for source distribution (and if you need to create the distribution yourself): Ant build file contains targets that invoke GraphViz application for creation of graphical elements for the documentation (UML-enhanced javadoc, and javadoc in pdf format). If you do not have GraphViz installed on your local system, these targets will be just skipped during the build (even if you get exceptions from the doclet, the javadoc gets generated, just without the UML diagrams). However, if you want to produce jCleanCim distributions, you should install GraphViz in order to have nicer documentation.

Dependencies

To be self-contained, jCleanCim distributions bundle relevant third party open source/distributable libraries. Java jars are in the project's lib directory, and MS Windows dlls are in the dlls directory.

The following libraries are packaged with all the distributions of jCleanCim:

• For access to EA model file:
  • When we need to export images or XMI, we use EA Java API, which in turn depends on a MS Windows dll. By working directly with EA model file, jCleanCim tries to identify problems at the source (.eap file), before any XMI or other artefact generation takes place.
  • When we don't need to export images or XMI, we use Jackcess library that enables light-fast access from Java to MS Access database (EA model file is exactly that), so this is totally OS independent.
• For document generation functionality, we use Jacob library that enables access from Java to MS Word automation interface, wrapped into a dll for 32bit MS Windows. The MS Word/Office used is the one on your local machine, and not distributed here.
• For logging, we use Apache's log4j.
• For command line processing, we use Apache's Commons CLI.
• For string markup processing and some stopwatch functionality, we use Apache's Commons Lang.

The following libraries are packaged only with source distribution of jCleanCim:

• For unit testing, we use JUnit 4. You will need this only if you run or develop tests, or if you are producing distributions (that include running tests and producing rest reports).
• For generation of ant target dependencies graph, we use Grand library; it will be ignored if you do not have GraphViz installed. These are not used from the jCleanCim source code.
• For enhancement of regular javadoc with the UML class diagrams, we use UmlGraph library; it will be ignored if you do not have GraphViz installed. These are not used from the jCleanCim source code.
• For pdf generation from the javadoc, we use PDFDoclet application. You will need this only if you generate the pdf documentation for binary distribution (from an ant target). The application is otherwise not used from the jCleanCim source code.

Performance indicators

Since we talk to EA and to MS Word through their automation APIs, the model building (as a first step in the application) and the MS Word document generation (if enabled) take time:

• EA automation API implementation unfortunately does not know of bulk CRUD operation, so for every single item to be returned through the API, they perform an SQL query on the underlying Access RDBMS (even for items in a collection!). Determinant factor for performance here are: number of elements (classes, attributes, ...) and the number of diagrams that need to be saved to file. In release 01v07, we had provided a fully new implementation for reading the UML model from EA (with option model.useSql = true, in that release only). Since release 01v08, we added one more implementation (see Fast loading of .eap file) and replaced the model.useSql boolean option with the one taking one of three pre-defined string values. In short, if you need to export XMI for a model release, or diagrams for document generation, ensure you use model.builder=sqlxml, otherwise leave the option empty or set it to model.builder=db. See also hint on fixing ordering errors .
• MS Word is extremely slow at inserting captions for figures, and in particular for tables, as well as in populating and formatting tables. As the number of figures/tables grows, MS Word takes more and more time to insert their captions - similar would happen if you insert captions by hand in an open Word document: higher the number of captions in the document, more time MS Word takes to calculate the number for the caption (and it is impossible to disable this automatic calculation if we want to create tables of figures/tables). Since release 01v05, we provide a configuration option docgen.saveReopenEvery that you should definitely use to speed up MS Word document generation . Default value is 12, but you should play with your document to find out whether higher value would make it faster. See also discussion on this option and its resulting performance improvement.

Java processing - for validation, statistics calculation and documentation collection from in-memory model to pass to the actual writer (s), as well as XML document generation for web-access - takes a couple of seconds for all the models and scenarios tested.

Hints

Here items thay may be considered as issues (but will likely not be addressed soon) and performance-related advice, so please take them into account when running jCleanCim:

• If you run only to validate the model or generated XML or Word doc without diagrams, you should use model.builder=db as the fastest way; notice that no digrams can ever get exported with this builder. With any other builder option, and when docgen.on=true, diagrams get exported from the EA model (in order to be used in the generated doc), while if docgen.on=false (or empty, or absent), document generation is disabled and we know that we don't need diagrams, so they don't get exported at all. Not exporting diagrams saves a lot of time: between 300-500 ms per normative diagram - for 100 normative diagrams, you save at least half a minute to read the model.
• EA ordering errors. If you use model.builder=db or model.builder=sqlxml in 01v08, or model.useSql=true in 01v07 (which you should, because they are orders of magnitude faster), you may see logged ERROR (+++ EA ordering error) for several UML elements (diagrams, packages, classes, etc.). This comes from the fact that EA internal storage for some reason does not always keep up to date the order of items in a container (we have seen this already with attributes order before). When using API, we just follow the order of elements as returned by the EA API. However, when processing SQL query results or reading the tables directly, we may find those position indexes totally uninitialised (that's how they are stored in the repository). To ensure the order gets preserved, just open the model and manually move an item from the indicated list up and down - this should trigger the EA internal update mechanism - and then you're set as the error should disappear on the next run.
• If for some reason you're still using model.builder=japi for reading from EA file:
  • Remove any baselines. They grow the size of data in the underlying RDBMS and every SQL query that EA does (unfortunately: 1 for each single attribute/association/class/package/tagged value/constraint/dependency) will take longer on a larger dataset.
  • Compact EAP file. From within EA, run regularly Tools - Manage .EAP file - Compact .EAP file. There is no effect if this is executed from EA that you've opened with your own model (that is why this is not feasible programmatically either), so you must open EA with an arbitrary, another model (or just launch EA alone), then select your file to compact.
• When running MS Word document generation:
  • (since 01v08) If you can, save your template as native Office 2007/2010/2013 document (.docx), without compatibility enabled. This will run much faster, because in this case only, it is possible to programmatically disable field updates.
  • This will run correctly only on an English version of MS Office (or on a local language installation plus the language support package for English). Reason: We create MS Word documents primarily for IEC, and these need to be in English; supporting various languages in MS Word (for e.g. style names) would be too much work at this moment. If you have a patch that can generically support MS Word installation in any local language, please submit it.
  • Ensure to update all fields in the MS Word template before starting document generation with change tracking turned OFF , in particular if you have added/removed tables/figures and their captions. MS Word is known to screw up caption numbers (read: identifiers) when you edit them with change tracking enabled.
  • Disable automatic spell checking in the styles 'PARAGRAPH', 'TABLE-cell' and 'Normal'.
  • On our developer Windows 7 machine, we have noticed that for some special formatting of diagrams (with resizing or such), the process PrintSpooler (the Windows printing service) kicks in immediately after EA repository gets opened. This manifests at building Domain package from the sample EA repository, when it seems like never-ending. When stopping the PrintSpoolerService (Computer/Manage/Services and Applications/Services/PrintSpooler/Stop) at that point, there are about 20 popups from EA process reporting on some font problem (that is how we concluded it is about some interaction of EA with the OS). If this manifests on your machine, the easiest is to temporarily stop PrintSpooler service while running jCleanCim. Note that this has never happened on Windows XP, only on Windows 7.

Disclaimer

Starting with release 02v00, jCleanCim has been licensed under the terms of GNU LGPLv3 license and includes a modified copyright. The copyright as well as a reference to the license for this software is available at the download site, and is included in every distribution and in every java source file.

Have a look here for a relatively accessible comparison of licenses.

This software has been developed in my free time. The contained IP is not related in any way to my previous or current employer.

Release notes for jCleanCim-02v01, 2016-07-23

New home

Since 2016, jCleanCim has new home at my web space - which is accessible also to non-CIMug members.

New features

Hyperlinks in the Word document

Credit: Prototype by Laurent Guise (thanks!) has inspired the implementation in this release.

This (by IEC 61850 community) long awaited feature is in and it is controlled by a new configuration option, docgen.word.useHyperlinks . To not incure performance penalty, the default is 'disabled' (i.e., you need to set it to true explicitly .

The hyperlinks are introduced for those classes and enumeration literals that have been printed in the same document. For example:

• For CIM editors, hyperlinks to IdentifiedObject will be printed only in IEC 61970-301, all the other CIM canonical model documents will not have hyperlinks to IdentifiedObject.
• For IEC 61850 editors, hyperlinks to e.g., INT32 will be printed only in IEC 61850-7-2 (where INT32 is defined), while in IEC 61850-7-3 there will be no hyperlinks to INT32. Another example, because we print in several annexes presence conditions, DA, CDC and LN tables will all have presence conditions (except for M and O) hyperlinked.

New MS Word document placeholders

In this release, we have added three new placeholders for MS Word document generation:

• startUmlClass.{packageName.className}.endUml: This was on request by IEC 61850 community, but is generally useful if you want to quickly print a couple of classes only. You don't need to modify the UML model (to put them into a package) only to be able to print them. However, please do not misuse the feature to end up with a proliferation of individual class placeholders (it is always wiser and better maintainable to mainly include full packages).
• startUmlDiagNote{packageName}.{diagramName}.endUml: We added this for a custom document generation, where we want to have a table with two columns: diagram in one column, its description in the other. Feature is now available, so you can experiment with your own layouts if you need.
• See below IEC 61850-specific features for the third new placeholder.

We have also added examples in the base-small-template.docx to show how to use the new placeholders.

Support for EA Enumeration type (that discards class with "enumeration" stereotype)

In more recent versions of EA, on XMI import, EA refuses to import the original class with stereotype "enumeration", but transforms it automatically to the new Enumeration type. We found no way to force EA to keep enumeration stereotype imported through XMI. The result was that the enumeration classes were then not recognised in jCleanCim as enumerated types and one had to manually add enumeration stereotype.

This release provides proper handling for this case, as well - i.e., the EA Enumeration type is now recognised as enumerated type, the same way as a class with "enumeration" stereotype.

Performance changes in release 02v01

None. Note however that newer versions of Enterprise Architect seem to always be somewhat slower than the older ones (we noticed the slow down in diagram generation).

Potential backwards compatibility breaking changes for the application user

• Before this release, we have been outputing Web Access XML documentation for elements and diagrams by default and only as HTML (text with mark-up), while the default for MS Word was raw text, without markup. To consistently support choice for printing markup or not in both MS Word and Web Access XML, the configuration option docgen.word.printHtml has been renamed to docgen.printHtml, incidating that this choice is not anymore limited to MS Word. By default, the value remains false, which means no change for MS Word output, but change for Web Access XML output (now printing raw text instead of markup). Note: Enabling markup for MS Word still does NOT work properly (and probably never will), so it is wise to stay with the default.

Potential forwards incompatibility for the application user: old jCleanCim with new EA

• -

Other functional changes

• builder.ea: Improved identification and signaling of the problematic entries in messages "+++ EA problem: attr count on ..." (when reading EA model); e.g. "+++ attr Adp: pos = 14 DUPLICATE". Moving any one of the entries ending with DUPLICATE should initiate EA's renumbering and solve the problem for the class. Note that this has been applied in the previous release for everything but class attributes and operations (now we fixed those as well).
• docgen.collector: Improved generation of locally and globally unique IDs.
• model: Support for mutual dependencies without going into infinite recursion.
• model: Added EA "Process" as recognised skipped element, for some custom extensions.
• CIM-specific:
  • -
• IEC61850-specific:
  • model: Per IEC 61850-7-3 editor request, replaced "ARRAY of min...max of XYZ" with "ARRAY min...max OF XYZ".
  • stats: Improved logging of Abbreviated terms usage section, now alphabetically ordered.
  • collector: Enabled support for CDC ERY of IEC 61850-7-420 (it was implemented since a while but for some reason commented out... Thanks to L. Guise for investigating the issue!)
  • collector: Added a dedicated Word placeholder startUmlIec61850NsName.{className}.endUml to allow correct printing of derived IEC 61850 name space name. Word API didn't behave as expected. If there are no separator between placeholders, it doesn’t recognise a placeholder, so we either miss a piece of information or need to introduce the space character between e.g. '2007' and 'B'. This is not to be acceptable for namespace.name, so this new placeholder is the clean resolution for the issue.
  • model: To derive Namespace "Needs" for Web Access XML, we now collect all the containing package dependencies, in addition to any native one. This allows us to have, e.g. subpackages of WG17 (7-420 and 90-8) get derived the hand-drawn dependencies of the containing package (WG17) to 7-2, 7-3, 7-4.

Validation rules:

• util (validation): -
• New common / general validation rules:
  • -
• New CIM-specific rules:
  • -
• New IEC61850-specific rules:
  • Iec61850LNClassesWithSuperfluousConstraints (to catch constraint for inexisting DO; typical copy/paste error, or consequence of DO renaming)

Bug fixes:

• collector (61850): Brought back printing default value for an FCDA (attribute in CDC).
• main: Fixed NPE when launching with arguments -help or -version (thanks to Jacob Dall for reporting).
• builder.ea: Now catching exceptions during EA processing and properly closing .eap file.
• builder.ea: While building dependencies from EA, now retaining only those linking directly package-2-package or class-2-class (class, interface, enum); all other dependencies are created as a skipped connector.
• collector (xml): In XML output, now printing properly 'this' side of association end for inherited association (thanks to Laurent Guise for reporting).

Known issues / limitations:

• nothing new.

Implementation and packaging:

• collector, writer: split placeholder functionality to two classes.
• model, collector: added some more unit tests.
• writer: started adapting Caption class for non-English MS Word, not finished yet...
• upgraded libraries: n/a.
• all sources: updated copyright year.
• moved code to a new svn, which resulted into rejuvenated svn IDs (had to discard the history due to space limitation)
• added Apache POI jar; implementation far away from finished...

Documentation:

• updated the presentation.
• added description on how to use a custom config.properties file.

Built on 2016-07-23T22:39:05

Copyright License

Feedback