[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

"a skilled backdoor-writer can defeat skilled auditors"?

On Wed, Jun 04, 2014 at 08:50:14AM -0400, Tom Ritter wrote:
> On 4 June 2014 01:54, Stephan Neuhaus <[email protected]>
> wrote:
> > If you fail the audit, it's your duty as a professional auditor to
> >  provide evidence that there is something actually wrong with the
> > software.  It's OK to single out some pieces of code for closer
> > inspection because of code smells, but if you try your darnedest to find
> > something wrong with it and can't, then either the code is OK or you're
> > not good enough an auditor.  In either case, you can flag the code, you
> > can recommend rewriting it according to what you think is better style,
> > but you can't in good conscience fail the audit.


I strongly disagree.  There are implementations that are Just Too
Complicated and are Impossible To Audit.  Such implementation choices
*do*, empirically, provide cover for bugs; and as we as a society build
more and more software into the fabric of our life-critical systems it's
imperative that "the implementor liked this complexity and refuses to
change it" gives way to the larger goals at stake.  The auditor
absolutely must have leeway to say "no you don't get to write your own
string processing, you are going to use the standard ones."

This kind of feedback is precisely what happens in the higher quality
audits that are becoming standard practice for security-critical

> Perhaps this is getting too far into nits and wording, but I audit software
> for my day job (iSEC Partners).  I'm not speaking for my employer. But,
> with very few exceptions (we have a compliance arm for example), one does
> not 'Pass' or 'Fail' one of our audits.  (Perhaps they might be better
> termed as 'security assessments' then, like we call them internally, but
> we're speaking in common english, and people tend to use them synonymously.)

As a satisifed iSec customer (at a previous job), I have a bit of
insight here.  iSec is a leader in this space and definitely leads by
example.  Across the industry, the average quality of discourse in the
source auditing business is pretty good in my experience; only the
bottom-skimming truly awful auditors reduce their customer-facing
feedback to just a binary pass/fail.

However, inevitably, in the societal analysis of software quality for
practical purposes, reductive reasoning happens.  (This is not a bad
thing, it's absolutely necessary -- we humans don't have the cognitive
capacity to hold a complete decision tree in our head while doing this
reasoning.)  Thus statements like "you should use $OSS_CRYPTO_PACKAGE,
it has passed its audits" end up playing a role in the discourse.

We as domain experts have an obligation to ensure that our contribution
is given appropriate weight in the debate and decisions -- in both
directions.  For example if an auditor sees their results being
mis-interpreted in customer marketing material or media coverage, the
auditor has a moral obligation to correct that and insist that the
mischaracterization stop.  (And yes, I believe that this moral
obligation would override an NDA between the customer and the auditor;
the contract should be structured to recognize this fact.)