ASA statement on p-values

There has been controversy around p-values in recent years, often linked to issues with reproducibility in psychology.  p-values are also often reported in empirical software engineering papers. We haven’t yet seen widespread public controversy about software engineering studies, but that’s not because there aren’t problems!

The American Statistical Association has just released a clarifying statement about p-values. (pdf)

p-values are not inherently broken. The problems are about mis-interpreting them, about poor study design and practice, and about poor reporting. The ASA statement seems like a useful contribution to the debate.

Back at NICTA, and a few Reflections on Medical Device Engineering

In February I moved back to NICTA after what proved to be an interesting sabbatical year working at Saluda Medical.  Saluda is going strong – they have great technology, and very recently closed a $10Mill VC round which will lead towards major clinical trials of their implanted spinal cord stimulator in the US.

I learned a lot working at Saluda, which is always fun – I hadn’t previously worked alongside mechanical engineers on product design, nor alongside electrical engineers doing signal processing, nor thought much about manufacturing process and product validation for manufactured devices. My role included work defining user requirements, system specifications, system architecture, system validation, and system verification. But perhaps the most interesting thing was risk management, which is central to systems engineering and is highly interdisciplinary. The system I was involved with is now undergoing clinical trial in the US. I also contributed to my first (very cool) patent application, and helped co-author a (conference) publication on placement of paddle leads for spinal cord stimulation. There is perhaps another (journal) publication in the works on adverse events for spinal cord stimulation. And I had the opportunity to learn Python, which was fun, and to learn more about Microsoft Word Interop scripting than I ever wanted to know.

The medical device industry has an interesting regulatory environment.  Of course it’s very conscious about risks and ethics. However, there is a surprising amount of flexibility about how companies can choose to engineer medical devices. Nonetheless, when a company has said how they’ll demonstrate safety and/or effectiveness (and having had that plan approved), regulatory monitoring and review is a powerful way of making sure that happens. That’s especially pointed when companies are selling medical devices (which Saluda hasn’t yet started to do).

More Philosophy of Engineering

In January, the the journal Synthese accepted and published the first of two papers of mine on the philosophy of engineering.  The second installment is now also accepted and published: “Critical rationalism and engineering: methodology” (author’s preprint here). Woot!  In the new paper I use the three worlds schema from the first paper to look at possible sources and responses to falsification of engineering theories.  I also discuss the growth of knowledge in engineering.  Finally, I talk about assurance in engineering.  There are perhaps more open questions than answers, but the questions are important and interesting.

Assurance is key for engineering.  Engineers design and create artefacts that other people use.  But engineers don’t just throw artefacts “over the wall” (or into the market) – they also warrant that those artefacts can be used to meet people’s needs. Those assurances don’t just get made up.  They are backed by explicit justifications – arguments using empirically-validated engineering theories.  For safety-critical systems, if those arguments are invalid or those theories are false, people will die or get hurt.  That’s why it’s worth understanding engineering epistemology.

 

Philosophy of Engineering

What is engineering? Sometimes people think engineering is just the same as science, but in a new paper on the philosophy of engineering (preprint here), I argue why that’s not the case. Engineering is similar, but different to Science, and its epistemological issues are also similar but different.

I got into this question because of problems in assurance for software engineering and formal methods that are essentially philosophical problems. But having work available on the philosophy of engineering available should also help with perennial questions like “Is Software Engineering a field of engineering?” and “Is Computer Science a science?”.

What Software Engineers Should Know

Software Engineering would be a more mature discipline if we had spent more time reading What Engineers Know and How They Know It: Analytical Studies from Aeronautical History rather than A Pattern Language: Towns, Buildings, Construction.

I was an Apple Mac kid

Don was an Apple II kid, and he credits Apple with helping him dive so deep and so early into writing software.  I never had an Apple II, but I got a taste of that kind of experience with the Vic-20 at home, and the BBC computer room at school.

But then, we upgraded our home computer to the Apple Mac.  My experience on the Apple Mac was exactly opposite to Don’s on the Apple II.

The Mac was the start (well, after Lisa) of Apple’s focus on the creativity of the users of computers, rather than on the creativity of software developers.  The Mac had amazing useability and rich interactive applications, but there was no out-of-the-box development environment.  Even when years later I did get the MPW, there was a killer learning curve to create simple apps that conformed to Apple’s strict UI guidelines.  Hypercard (especially Hypertalk) was ahead of its time and did encourage bespoke coding creativity, but then Apple ditched it.

Apple’s success is due to their user and customer focus, but ever since the Apple Mac they’ve been mostly hostile to developers.

 

Congratulations Marek and Innoboard Team!

Marek Kowalkiewicz from the SAP Research in Brisbane just last week won the international “Demo Jam” competition in the SAP TechEd event in LA, for the “Innoboard” software.  Innoboard is an augmented reality technology, which lets distributed teams interactively share whiteboards that mix projected images and physical sticky post-it notes.  All using the low-cost iphone camera and an ordinary projector.  Cool demo!  The idea at the end of taking streamed information out of the interactive session and using that to drive other workflow software (Jira in this case) is also cool, and just hints at the huge potential of ideas like this.

The Innoboard team found its first industry trial partner through the Future Logistics Living Lab, which is run by NICTA, SAP, and Fraunhofer IESE, and has around twenty (and growing) industry & research participants.  (Fraunhofer’s involvement is through the Fraunhofer Project Centre in Transport and Logistics at NICTA).  Industry trials for Innoboard are continuing, in a use-case for distributed logistics operations planning.  The Future Logistics Living Lab is also hosting a demo instance of Innoboard, and setting it up in the lab has helped contribute to ironing out some of the use & set-up issues in the early prototypes.

 

Is Informatics a Science?

Robin Milner gave a presentation “Is Informatics a Science?” at a conference at ENS, 10 December 2007, where he discussed the challenge of better understanding relationships between models in computer science – how they “explain” (specify, refine, implement, abstract, realise) each other. I don’t believe he captured these thoughts in a journal or conference paper, but the ENS presentation follows an earlier similar 2006 presentation (for which there is a transcript) on “Scientific Foundation for Global Computing” .

An audio recording of the ENS presentation exists.  I’ve created a PDF transcript of that recording.  However, I don’t have the slides that Robin presented – I’d be interested to have a copy if anyone could send me one.

Windows 7 Deletes all my Shortcuts

Stuff has started mysteriously disappearing from my desktop.  I had installed Fences, so thought that was the culprit, and uninstalled it.  But the problem keeps happening, and it turns out that Windows 7 is causing my pain.

Windows runs a weekly maintenance task that deletes all your shortcuts if it thinks you have more than four “broken” ones.  Bad!  The problem is exacerbated my Windows being mistaken about what’s a “broken” file.  It’s counting all my shortcuts coming from an “always online” shared network drive, even though none of them are really broken.  System maintenance task can clean stuff out of temp all it likes, maybe even delete old log files, but stay away from messing with my desktop!

MSFT offers two solutions: don’t have more than four shortcuts on your desktop (ummm, no), or turn off all system mainteance (ummm, no).  The best solution I’ve found is to hack the Windows 7 maintenance scripts to stop them mis-counting files.  Doing this felt like being a Linux user used to feel like.  I guess I should be grateful that I can hack these files – I was a little surprised to be able to.

 

What is Software Architecture?

What is software architecture? There have been many definitions. Here’s mine.

First let’s consider some of the earlier definitions. SEI has a huge collection of definitions on its website, including “classic” definitions, bibliographic definitions (stops in 1996?), “modern” definitions, and definitions submitted from the communityPerry and Wolf (1992) have perhaps the most classic definition, though it’s a little sketchy:

Architecture = {elements, form, rationale}

where elements are Processing, Data, or Connecting elements. Taylor et al. (2010) note that when people talk about software architecture in terms of Components and Connectors, that’s an over-simplification of Perry and Wolf’s definition – over-simplified because it doesn’t always work. For example in REST, Data elements are pre-eminent.

The ANSI/IEEE 1471-2000 definition expands on Perry and Wolf’s definition, and also slips environment into the scope of form (relationships).

Architecture is the fundamental organisation of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution.

Elements (components) and their form (relationships) are clearly key to understanding what software architecture is, but many people think that’s all it is! Instead, software architecture researchers have understood for a long time that rationale (design/evolution principles) is also a key part of what software architecture is.

But contrari-wise, software architecture is not just rationale. So although Taylor et al. (2010) is a great architecture textbook, their definition of software architecture isn’t so great:

A software system’s architecture is the set of principal design decisions made about the system.

I would instead say that design decisions are the means by which elements, form, and rationale are created. The design decisions are not the architecture per se.

Software architecture is commonly misunderstood to be an exclusively structural model. Perhaps that’s because UML class diagrams and deployment diagrams are often presented as iconic for software architecture. The definition from the Bass et al. (2008) classic textbook also encourages this view:

The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them.

But there has been a shift in the software architecture research community to think about architecture more abstractly: in terms of rules or constraints or styles. So instead of “structures”, I prefer using the word “abstractions”, to more easily accommodate a rule-based architectural perspective. The abstractions here are not just of software, but also non-software elements in the system, including the environment. Fielding (2000) also talks about abstractions this way (though for some reason he limits software architecture to run-time, which I don’t agree with):

A software architecture is an abstraction of the run-time elements of a software system during some phase of its operation. A system may be composed of many levels of abstraction and many phases of operation, each with its own software architecture.

Apparently Clements et al. (2010) has another new definition:

The software architecture of a system is the set of structures needed to reason about the system, which comprise software elements, relations among them, and properties of both.

Here we have (ignoring “structures” for the moment) Perry and Wolf’s elements and form (relations) again, but now also with properties for each. Perry and Wolf’s rationale has not disappeared, but here appears as a qualifier (“needed to reason about”). I like this. I think the whole point of architecture is abstraction and analysis of a system for particular purposes. Here the “reasoning” objective implicitly encompasses those purposes and analyses.

I’d slightly prefer to say “communicate and reason” instead of “reason”, though perhaps you could say that understanding a communication is-or-requires reasoning. I’d also prefer to talk about “a” (not “the”) software architecture of a system (and similarly “a set” not “the set”), to more readily accommodate multiple views/perspectives of a single system.

So, in summary, my definition would be something like:

A software architecture of a system is a set of abstractions needed to communicate and reason about software in the system. A software architecture models elements of the system and its environment, relations among those elements, and properties of those elements and relations.