Yesterday, I volunteered a couple of hours to help review the translation of an ISTQB paper into Hebrew. Specifically, the "CT-AI" syllabus. I did it mostly because someone asked and I thought it won't hurt to give it a little time a small part out of wanting to see how bad they managed to get it, and a tiny fraction of hope that maybe this time it won't be as terrible (spoiler - it was).
So I got an excuse to read parts of the document (I didn't bother with the entire thing, just a random chapter or two and the example exam), and it's the same problem all over again - a silly multiple choice questionnaire that you can pass with some common sense and guessing. One bonus point is that the syllabus mentions some practical hands on exercises that can be done. No, of course there's no enforcing of this, but a serious training vendor could, in theory, cram some useful learning that will not be needed for the certification exam but is, officially, expected. Self learners and less reputable training providers? they will probably skip those exercises.
Also, very much like CTFL, CT-AI also misses the mark completely, but it does so on a more profound level. To be more specific - It aims for the wrong goal. Most of the syllabus is aimed at "how to test a neural network" (it does recognize that there are other types of "AI" in the form Bayesian algorithms or oven simple decision trees, but almost all properties they attribute to "AI" are more in place with neural networks than with other sorts of machine learning algorithms - which is ok, most people saying "AI" today do mean a neural network of a sort). Now, neural network development (or, "training" is the parlance goes) is testing. The data people let the computer run until it passes their defined test cases and then they declare it done. They might tweak with the network's geometry, weighting or calculations, but most of your work will be collecting the right data, and evaluating the model's performance. This sort of testing usually involves some heavy math and big data techniques - both are not really something mandatory in a tester's background. For testers, such as me, who don't have those math skills sharpened, their contribution to testing the model itself will be fairly limited and will be inferior to that of the data people.Testers who do have shaken the dust of the relevant math (or learned it from scratch) - don't need this basic mess of a certification.
A better goal for this certification would have been to accept the fact that most testers won't spend their time testing models but rather test systems with embedded "AI" in them and deal with the complexity of a nondeterministic element in the system. Only caveat with that? Testers have been doing that for years, and it's not really different than simply testing, so you can't scam more money out of people.
So, sadly, no surprises here - somehow the ISTQB organization has managed yet again to get a bunch of really smart people to produce a piece of work that has a negative value. It's a waste of time for everyone involved in writing, teaching, or learning it. It has unrealistic goals that are unmatched by the weak certification conditions.
And one last thing - The first (and currently latest) version of the syllabus was out in October 2021, riding the wave of "deep neural network" that seemed to be in every start-up name. ChatGPT was out a mere year later, and today most of the material in this syllabus might not be outright obsolete, but it's heavily dated, as the capabilities of "AI" have leaped and our understanding of how we might use such systems has evolved quite a bit.
As for why do I keep doing this - mainly, it comes to a strange dissonance - while I do think that ISTQB is harming our profession, in a strange turn of events, the local chapter, ITCB, is doing some good by being the center of the local testing community - organizing meetups, releasing a quarterly magazine and maintaining a podcast - so I keep around those people and when they ask for help, I'm sometimes there, even if it means paying the devil his due. I should probably make better judgement in the future.
No comments:
Post a Comment