Saturday, February 12, 2022

Troubleshooting


 גרסה עברית

Last week I faced a moment of frustration. And not the good "why isn't it working" kind of frustration. That kind I'm used to, and I normally end up learning something. This time I was frustrated with the people I work with. The story is quite simple - we had (yet another) problem in our systems, and they set out to investigate it. I was busy with a super-urgent-clock-is-ticking kind of task, so I wasn't available to help in this investigation. I did try to ask some guiding question, such as "what error do you see in the log", or "can you reproduce it on your system?" but other than that I was doing my best not to get involved. 

After a while they have been struggling with the problem, my manager asked me to time-box 20 minutes for this, as it was blocking most of the team. After checking that the urgent task can wait this long, I took a look. Then I got upset. Two people have been looking on this, and the best they could come up with was to quote a partial error and the step which was failing. No guess of what could have happened, no narrowing down of the problem, a simple "it fails with this message". Yet, when I took a look, it was less than 30 seconds to figure out what was wrong, then perhaps 15 more minutes to find a workaround that is actually working. 

I reflected a bit on this negative feeling - somewhere between disappointment and annoyance - and figured out why I was so upset, and this helped me notice something I didn't see before.

I was upset because I always assume that the people I'm working with are smart and capable people who are doing the best they can., and any contrary example is touching a raw nerve for me and makes me wonder why I'm bothering investing so much time and effort trying to collaborate with them instead of working individually. Then, after processing it a bit more and recalling the fundamental attribution error I could say that it's probably not that the people who failed in a task I found trivial are not smart or that they don't try their best, it's more likely that there are some other factors I'm not aware of that make this behavior reasonable. Both of them had other tasks putting pressure on them, and both are fairly inexperienced - between them they have less than 18 months of experience. In addition, it reminded me that troubleshooting is a skill that needs practice and learning, which prompted this post - I want to share the way I approach troubleshooting, hoping it might help people. 

The first thing worth noticing about troubleshooting is that almost anyone related to software development need to do this quite a lot - programmers, testers, CS, Operations, and however you might call the team managing your pipelines. The second thing worth noticing is that  it looks a lot like bug investigation, so being a better troubleshooter will make you a better tester as well. In fact, the main difference between troubleshooting and bug investigation is the goal we have: troubleshooting is about making a problem go away, or at least find a way to do our task around it, where bug investigation is more about understanding the cause and potential impact of such problem, so if a bug just flickers away we'll hunt it down.

So, how do I troubleshoot? Here's a short guide:

  1. Is it obvious? Sometimes the errors I get or the symptoms I experience are detailed enough that no investigation is actually needed. I can easily tell what has happened and skip directly to fixing or finding a workaround.
  2. Can I reproduce it? Does it happen again? if not - great, problem gone. It might come back later, in which case I might try a bit harder to reproduce it or trace its cause, but usually, a first time problem that isn't easily reproducible doesn't really need shooting. I skip to "done for now" and continue with whatever it is that needs doing.
  3. Create a mental model - what was supposed to happen? Which components are talking with which systems? What relevant environmental factors should be considered? What has recently changed?
  4. Investigation loop:
    1. gather information. Google the error message or symptom, gain visibility on the relevant flow, ask around if anyone have seen such a problem, etc.
    2. Hypothesize. Guess what might be causing the problem. 
    3. Try to disprove the hypothesis: 
      1. Create a minimal reproduction of the problem
      2. Find contrary evidence in log file, side effects, etc.
    4. Tweak. Based on my current guesses, try working around or mitigate the cause of failure. I suspect a code change? I'll revert to a previous state. Server can't be reached? I'll tweak in order to gain more information. I might check for ping, or DNS resolution. 
    5. Check to see if problem has gone away. If so - update the theory of what happened and finish.
    6. Update and narrow the model. Using the information I gained, zoom in on the relevant part of the model and elaborate it. For example, a model starting with  "I can't install the product", might narrow to "I have a remnants from a faulty uninstall that are preventing a critical operation" or to "the installation requires active internet connection and the computer has been plugged out", it can be more complicated than that. 
    7. If  I can't narrow down the model, or can't come up with a next theory of what might be wrong, I still have two options:
      1. Hail Mary - I'll change a random thing that I don't expect to help but is related in some way. For instance, I might follow instructions on the internet to find a relevant configuration change, or reboot the system. Who knows? I might be lucky and gain more information, or even make the problem go away for a while. 
      2. Ask for help. Find someone who might have more knowledge than me, or just a fresh perspective, and share my model, failed attempts and guesses I couldn't or didn't act upon, and we'll look at the problem with that person's knowledge and tools. 
  5.  Now we know what's wrong, or at least we're confident enough that we know, time to shoot down the problem. Find a way to configure a tool we were using and was causing problems, change the way we operate, sanitize our environment, or whatever will work to our satisfaction. 

That's it. I hope this flow will be helpful, at least to some extent. If you have additional tips on troubleshooting - I'd be happy to hear about them. 

תפעול תקלות


English version

בשבוע שעבר היה לי רגע של תסכול. לא תסכול טוב של "אבל למה זה לא עובד??" אלא תסכול שקשור לאנשים שאני עובד איתם. הסיפור, בתכל'ס די פשוט - הייתה לנו (עוד) תקלה עם סביבת ההרצה שלנו, ושני אנשים מהצוות ניסו לתקן אותה. אני, מצידי, הייתי עסוק במשימה סופר-דחופה-הדדליין-זה-אתמול, אז לא הייתי זמין לעזור גם כשהם ביקשו עזרה. כן ניסיתי לזרוק איזו שאלה מנחה או שתיים - דברים כמו האם הסתכלתם בלוג? מה הוא מלמד אתכם? האם אתם יכולים לשחזר את הבעיה בסביבה שלכם? האם אתם יכולים להגדיר את הבעיה בצורה יותר מדוייקת מאשר "זה לא עובד" ? אבל חוץ מזה דחיתי בתקיפות את בקשות העזרה שלהם. 

אחרי איזשהו זמן המנהלת שלי ביקשה שאשים את המשימה הדחופה בצד לעשרים דקות כדי לראות אם אני יכול לקדם אותם קצת, כי הבעיה שהם מנסים להתמודד איתה תוקעת את כל הצוות. אז ביררתי שאפשר להתעכב בשעה (כי עד שחוזרים בחזרה לריכוז גם לוקח זמן) ואז הסתכלתי לראות מה קורה שם. ואז התרגזתי. לקח לי שלושים שניות כדי להבין מה הבעיה, ולראות בלוג שיש הפנייה לטיקט ספציפי בגיטהאב (שנכנס לפני שבוע) של אחד הכלים שאנחנו משתמשים בהם והצעה למעקף. אז אוקי, המעקף לא עובד, אבל ברגע שזיהיתי את זה לקח לי עוד רבע שעה למצוא איך להוריד גרסה של הכלי הנ"ל ולפתור את הבעיה. ועדיין, הכי טוב שקיבלתי משני אנשים שישבו לחקור את הבעיה היה "זה נופל, הנה הודעת השגיאה" (מנותקת מכל ההקשר). לא הבנה טובה יותר של מה קורה שם, לא דברים שהם כבר ניסו. כלום. 

אני לא אוהב לכעוס על שותפים לעבודה, אז המחשבה הזו נשארה איתי קצת, והקדשתי לעניין הזה תשומת לב. הבנתי למה זה כל כך הציק לי, וקיבלתי תזכורת למשהו שחמק ממני.הסיבה שמשהו כל כך טריוויאלי (כולה רבע שעה מהחיים שלי, ועוד בזמן עבודה) היא שאני פועל תמיד מתוך הנחה שכל מי שאני עובד איתו הוא גם חכם וגם עושה את המקסימום שהוא יכול. זו הנחה חזקה יחסית, אבל היא חלק חשוב ממה שגורם לי ליהנות בעבודה. כשדוגמאות בסגנון הזה מערערות אותה, אני תוהה בשביל מה אני משקיע כל כך הרבה מאמץ בניסיון לעבוד יחד איתם במקום פשוט להתקדם לבד עם משימות, ושיסתדרו. התבשלתי קצת במיץ של עצמי, ואחרי שעיבדתי את הנושא עוד קצת והזכרתי לעצמי את טעות הייחוס הבסיסית (ההסבר בויקיפדיה קצת מעורפל, כאן יש הסבר טוב יותר), הבנתי שנכון יותר לומר ששני האנשים שלא הצליחו להתמודד עם משימה שאני מקטלג כטריוויאלית לא חיפפו, אלא עשו כמיטב יכולתם בנסיבות בהן מצאו את עצמם. יש לפחות שני גורמים שאני יכול לחשוב עליהם שתורמים לתוצאה הזו - לשניהם יש מלא משימות אחרות דחופות לא פחות מאשר המכשול הזה שנפל עליהם, ולשניהם ביחד יש רק טיפה יותר משנה של ניסיון, כך שהם עדיין לא ראו ארבע מאות דברים דומים. חוץ מזה, זה הזכיר לי שפתרון תקלות הוא כישור שצריך לפתח, לא משהו שקורה באופן אוטומטי, מה שמביא אותי למטרת הפוסט - אני רוצה לחלוק עם כל מי שיגיע לכאן (כנראה בטעות) את הדרך בה אני מתמודד עם תקלות טכניות בתקווה שזה יעזור למישהו.

הדבר הראשון שכדאי לשים לב אליו כשמדברים על תפעול תקלות הוא שכמעט כל מי שמתעסק עם תוכנה יצטרך לעשות את זה, ולא מעט. מתכנתים, בודקי תוכנה אנשי DevOps (או איך שלא תקראו לצוות הזה שמתחזק את ג'נקינס בשביל כל הארגון) וכמובן שגם אנשי תמיכה. כולם עושים את זה - כל הזמן. הדבר השני הוא שיש לא מעט קווי דמיון בין תפעול תקלות לבין חקירת באגים, כך ששיפור ביכולת תפעול התקלות תהפוך אתכם גם לבודקי תוכנה טובים יותר. למעשה, ההבדל המרכזי בין שתי הפעילויות הוא המטרה: כשאנחנו מתפעלים תקלה יש משהו שלא עובד לנו, הוא חוסם לנו את הדרך למשהו שאנחנו צריכים לעשות, ואנחנו רוצים לגרום לו לא להיות כאן יותר, או לפחות למצוא דרך לעקוף את המכשול המציק הזה. בחקירת באגים, מצד שני, נרצה להבין מה הם הגורמים לתקלה, מה הנזק הפוטנציאלי וכמה הוא חשוב. כך שאם באג נעלם ככה סתם, אנחנו נצוד אותו, אבל אם תקלה נעלמת בלי שאנחנו יודעים למה, נגיד תודה רבה ונמשיך בחיינו. 

 אז, איך אני ניגש לתקלה? הנה מדריך קצר:

  1. האם זה ברור מאליו? לפעמים הודעת השגיאה טובה מספיק, או שראינו כבר משהו ממש דומה, או שהתסמינים פשוט צועקים "זו הבעיה". במקרים כאלה אפשר לדלג על שלב החקירה ולהתחיל לחפש פתרונות. 
  2. האם אני מצליח לשחזר את הבעיה? עדיף - בסביבת העבודה שלי. לא? מצויין, הבעיה הלכה למקום אחר ולא צריך לטפל בה. יכול להיות שהבעיה תחזור אחר כך, ואז אולי אשקיע קצת יותר זמן בניסיונות לשחזר אותה - כי בעיה שחוזרת פעם בחודש ושורפת לי זמן תעלה לי יותר מאשר להשקיע בה יומיים ולפתור אותה אחת ולתמיד. 
  3. בניית מודל.
    אני מתחיל לבנות לעצמי תמונה בראש - אילו חלקים נעים במערכת? מי מדבר עם מי? מה היה צריך לקרות ובאילו שלבים? אילו משתני סביבה משפיעים על מה שקורה? מה השתנה לאחרונה ויכול להיות קשור?
  4. לולאת החקירה:
    1. איסוף מידע. אפשר לחפש את הודעת השגיאה בגוגל, לצלול לתוך הלוגים בחיפושים אחרי מידע נוסף, לנסות ולעקוב אחרי המידע (למשל, שימוש בפרוקסי כדי לראות תעבורת רשת, או הדפסות מסויימות שאמורות לקרות לפני נקודת הכשל)
    2. ניחוש - על בסיס המודל שיש לנו בידיים, מה יכול לגרום לבעיה? 
    3. ניסיון להפריך את הניחוש:
      1. שחזור מינימלי שיאפשר לי לבודד את גורם התקלה ולראות שבלעדיו זה לא קורה (אם כן - זה לא זה)
      2. חיפוש ראיות סותרות במידע שיש לי. 
    4. שפצורים - אני מתחיל לשחק עם כל מיני פרמטרים קשורים לניחוש בתקווה שאחד מהם יעזור לפתור את הבעיה. 
    5. האם זה עבד? האם השינוי שעשיתי גרם לבעיה להיעלם? אם כן, סבבה. הבעיה נפתרה. 
    6. עדכון המודל עם המידע החדש שנאסף בשלבים הקודמים. 
    7. לפעמים נגמרים הרעיונות - חקרתי את כל הכיוונים שהצלחתי לחשוב עליהם, בחנתי את הכל הראיות שמצאתי, וזה לא מתקדם לשום מקום. עדיין יש לי שני כלים בארגז שאפשר להפעיל:
      1. אללה-הוא-אכבר: אני אעשה משהו שאין לו באמת סיכוי לעבוד, או אשנה איזה פרמטר אקראי  שמצאתי בגוגל ולא נראה קשור.מי יודע, אולי יהיה לי מזל וזה יוסיף לי מידע, או אפילו, חלילה, יפתור את הבעיה? 
      2. לבקש עזרה - אין לי את כל הידע בעולם, ולא כל הרעיונות נובעים מהראש שלי. אולי למישהו אחר יהיו רעיונות חדשים שיקדמו אותי? אולי הוא יודע להפעיל כלי רלוונטי שאני לא מכיר? אני אתפוס מישהו, אציג לו בקצרה את ההתקדמות שעשיתי עד עכשיו (את המודל הנוכחי, וכמה צעדים שעשיתי כדי לבסס אותו) ונשב לעבוד ביחד על הבעיה. 
  5. זהו, בשלב הזה אנחנו צריכים כבר לדעת מה הבעיה - גם אם לא את הפיתרון עבורה. עכשיו זה רק החלק של למצוא איך לעקוף את גורם הבעיה ולגרום למה שאנחנו צריכים לעבוד. זה יכול להיות משהו קצר ופשוט כמו "לוודא שתיקייה מסויימת קיימת לפני שמריצים פקודה" או משהו קצת יותר כבד כמו לקמפל מחדש ספריית קוד-פתוח שאנחנו משתמשים בה כי יש שם באג. בדרך כלל, אם יודעים מה הבעיה, לא מאוד מורכב לפתור אותה. 

זהו, פלוס מינוס. אני מקווה שזה יהיה שימושי למישהו. וכמובן - אם יש לכם דרכים נוספות להתמודד עם תקלות, אשמח לשמוע עליהן. 

Sunday, January 23, 2022

A world class test team?

Source: XKCD

 

I've read Simon Prior's blog post Building a world class test team, and I didn't like what I've seen. 

There's the overuse of my least favorite 4 letter word, but putting that aside, I found that I'm disagreeing with the definitions he's using.  His idea of a "good test team" is a safety net role that enables others to disregard their responsibilities and directly contribute to worse software. The "great" test team is the first thing that starts to get close to an almost decent team, as he's added "preventing defects" which I chose to interpret generously as being involved in crafting the requirements and planning the work with the other members of the software engineering group.
Then, the "World class test team" is complete contradiction. It's described as "all of the above" AND some properties of what I would call pretty good testing - not being the scapegoat, coaching, improving other functions work - a lot of things that are not very compatible with owning the large part of the testing effort. Sure, I can imagine this working in some places, but it would be like hammering a bolt that doesn't fit well into place - it will work in the end, but you've invested more effort than you should have, and it will be a mess to deal with it every time you need to change something. 

Instead, I want to suggest another definition - in almost all cases, a world-class testing team is one that has disbanded (or is actively working towards that goal). As much as I like being a tester by role as well as by identification, my main aspiration is to bring the place I work in to a state where they don't need dedicated testers (note - I said "don't need",I didn't mean "just get rid of") - the way to achieve this is to embed the testing mindset and activities as part of every step in the process, from feature definition to deployment. Something very much like Dan Ashby's model of where testing fit in DevOps. It will take a while to help build the necessary infrastructure needed for self-testing and even longer to instill a culture where testing is part of being a professional, but if we want "world class" testing, one that people can learn from, it's this property we are looking for. 


In addition to this, I found two points of disagreement with the practical advice of building a good test team. The first one is using "9 to 5 testers" as a derogatory term. It's covered beneath some layers of "there's nothing wrong with it", but still confounds working reasonable hours with box-ticking and lack of development. When building a team, respecting their private time is super important, and accommodating improvement within regular working hours is key to building  the team. want to fend off stagnation and rigidness? find a term that will not push towards expecting people to put their free time for the business. 

The second issue is the advice to diversify with hiring. It's not a bad idea, but in many cases it's not a viable option - by my experience, finding senior testers is way more difficult than finding experienced coders, and it seems that it is so in other countries as well

 

Monday, January 17, 2022

Deep work - Book review


 


 

Following a recommendation from Tomer Cohen (He wrote in Hebrew), I finally got to listen to Cal Newport's "Deep Work: Rules for Focused Success in a Distracted World", and my reaction to it is  quite ambivalent.

The book starts with a very tempting, yet quite weak proposition of value: Deep work is valuable, and its value increases with the proliferation of knowledge work in today's world. On the other hand, increasing connectivity and interruptions makes this particular skill rarer, so it can make you stand out in your profession and earn more. It then moves on to claim that deep work will also make you happier. Basically - it's a panacea to all of the modern world's ailments. And, like the lovely book "Calling Bullshit:The art of skepticism in a data driven world" has reminded us - great claims requires great deal of evidence. Not only that this book does not provide that sort of evidence, it goes on to actually show a counter-example, acknowledging that deep work is not the only way to wealth, and dismissing it with a statement along the lines of "even though there are other ways to wealth, it does not mean that deep work isn't a way to put you ahead of everyone else (perhaps except the CEO that provided the counter-example, but for most of us it's not an option)". Convinced? Neither was I.

But, before I go on rambling about deep work, it might be worth it to define the term. The definition used in the book is: "Professional activities performed in a state of distraction-free
concentration that push your cognitive capabilities to their limit. These efforts create new value, improve your skill, and are hard to replicate". long story short - deep work is hard, valuable mental work.

Anyway, while the evidence fall short of the grand claims, the book actually tells a compelling story, and if we accept some of the assumptions made hastily - its conclusions are inevitable. For instance, one reason that deep work should make one happier goes as follows: Quote the work in Winifred Gallagher's RAPT to base that people happiness has more to do with what they focus on than with what objectively happens to them, state (assume) that deep work is focusing on more meaningful activity than the shallow trudge of interrupts, conclude that since one spends more time focused on meaningful events, they will feel a greater sense of meaning in life, or quote Mihaly Csikszentmihalyi's "Flow", assume that deep work is more prone to lead a person to the sweet spot between challenge and success to greatly increase one's chances of experiencing flow, and since being in a state of flow is known to increase happiness, so does deep work.
As you might have noticed, the assumptions are quite plausible. good enough to be motivating. One issue it sidesteps completely is the question "are there paths other than deep work that are as likely to generate comparable levels of happiness or value? How do we identify we are on such a path or that we might benefit more from one?"

The rest of the book is about tips to incorporate deep work in our day to day. Most advice here is at least convincing, and the author takes sufficient time to base some of the claims - shutting down interruptions and educating your environment can work in most contexts, focus is a skill that needs constant training and is prone to deterioration, actually resting and not doing any work after the work day is a way to increase productivity, and so on. My unease on this part has more to do with the totality of the approaches described in the book. Commitment to deep work is subjugating one's entire being: The default is being in a state of deep work, pausing only for as brief periods as necessary. A lot of it feels like what you might expect in a time-management book, only that the focus is not about "not wasting time", but rather about constantly training your deep working skill. I got tired only from trying to grok the message.

This fatalistic approach is a bit much, but it helps to emphasize that deep work is demanding, even if I believe that it can be done in a less extreme way it is still useful to learn the purists approach, where it is the clearest. It also helps to see various examples of building a schedule for deep work. It can be getting up early to spend a couple of hours in full focus before the day starts for other people who will create interference, it can be planning work so that we divide our calendar time between weeks of solitary focus and those of interruption heavy busy-work. Then there's the part where I felt I was being cheated - after explaining at length how deep work is something that takes time to start (research I think I recall from other places states that it takes about 15 minutes to enter "the zone") and that 4 half-hours of deep work are not as effective as 2 straight hours of it, the author goes to describe the "journalistic" approach which is based on the assumption that people trained in deep work can skip this time and just dive into deep work immediately whenever they have a spare 10 minutes or more. It really felt as a way to say "I'm doing deep work" while ditching aside all of the principles mentioned in the book before. It was also an unpleasant surprise to hear that this is the author's main mode of work - Preaching for deep work and claiming to be quite adept at it, then redefining it as "I do deep work when I decide to do it and have free time from my other distractions" sure does feel like hypocrisy.


That being said, I still took some insights with me.

First of all, I'm still debating with myself about the actual value of deep work in my context - much of my day is about helping others, jumping for a short period and dropping a question that might set things on a better path, mentoring others to do the actual work themselves. Sure, some of the work is done by me, and I find great joy in being able to still contribute directly, but my added value there is not always significant - I might do things faster or a bit better than the less experienced people in my team, but in the time it takes me to do one unit of work myself, I can probably help three team mates to do better work and complete 3 units of work that will be 80% as good as if I would have done it. So, where do I see more value? If I'd try to frame this in deep-work lingo, I'd borrow some of Kahneman's Thinking fast and slow and claim that by practicing, I managed to move some of the skills that require deep work and concentration to the more automatic parts of my mind and now I provide value by doing deep work in a shallow fashion - using those automatic skills to improve my environment. This means that my skill is based on deep work, and even if I believe that most of the value I provide is collaborative and interruption heavy (collaboration can, in rare cases, be deep work in itself, but the book usually treats it as solitary work), I should still invest some time in deep work to expand the base I'm building upon and to make sure I'm still connected to the work.

It also serves as reminder - not everything I do is equally important, and there are some things such as e-mail or instant messaging that can be pushed aside instead of sapping my attention.
Another thing that intrigues me is the claim that we are addicted to interruptions - I'm definitely going to try some of the tactics to train my mind to be more focused, such as defining breaks from concentration (perhaps using pomodoro) and learn to stick to them - no breaks until the buzzer. 

I think that the most fitting analogy to this book would be an olympic runner sharing tips on how to become a better runner. This athlete will surely recognize that the amount of effort needed to get to their level is ridiculous, but even when they will try to dial down the effort, they cannot unsee what they know - What you eat, how you sleep, how balanced are your core muscles, all of this has an impact on your running. Getting advice from someone like that will create the impression that there is nothing more important in life, which is true for very specific cases.
This is why I probably won't invest the effort to become a professional deep-worker who's managed by the hunt for deep work, it might be relevant for highly competitive fields such as academia or writing, but I suspect the reward is much smaller in software development where good employees are rather hard to find. A focused deep worker might be a lot better than me, but the difficulty of measuring productivity and the fact that most places require the sort of work that my skills are more than enough for - it will be like purchasing a Ferrari to stand in traffic. It *can* get to silly speed, but you would do just the same with a car tenth of that price. I need and want to be good at my work, trying to be the best of the best is not worth the effort. Instead, I'll treat it the same way I treat my bicycle. It's fun, it builds some sort of muscle, and I gain a lot from this as a hobby.

Thursday, June 17, 2021

Is all testing context driven?


So, AST is crowdsourcing a book on testing, and I was reminded of the current question by James Thomas's response (more details about it in his blog). So, right before reading his answer, I went to the AST slack channel and posted my own response.

As this blog has been quite dormant for a while, despite my best intentions, I thought it would be a good start to post my answer here as well. 
Who knows, I might even get to translate it to Hebrew later. 

After reading James's answer, it seems that we say roughly the same thing, only he's wording it better and more precisely. We both make a point about being context driven.


So, without any further hassle, here's my response:

All testing is driven by context, some of the testing is doing this explicitly.

If we face the truth of it, context provides a set of limitations and pressures that affect the way our testing is done and the extent of thoroughness one feels they can and need to examine an application. In that sense, context is driving everything.
A more precise look on the state of the industry would make the distinction between context driven and context dragged, the main difference would be the place that context plays in the way we talk about our testing. For example, in a context driven situation, one might say "We're in a fast-paced team, if something goes horribly wrong the worse thing that can happen is that the company will lose some money, which is gained multiple times over by releasing quickly, so we devise a strategy that will lay out decent automated checks, but do deep exploration after the product is deployed. We still need to improve on exploring edge cases while defining features to reduce the noise coming back from such exploration"
In a context-dragged situation we might say "Testing in production is the way to go forward! We deploy faster, getting important things fixed and increased collaboration within our team." and would strive to do the same in a more risk averse environment (say - testing a self driving car algorithm).
There's also a third, very common type of testing, which we might call "poor testing", those are normally context-oblivious, and uses the process they know and the test level they are familiar with without considering the business needs. Naturally, it's not a very interesting type of testing to discuss

Wednesday, February 17, 2021

The age of Surveillance capitalism - book review


 

It's the second time I listen to an audio-book and then go and purchase a hard copy one (the previous one was Data and Goliath), and much like it, I feel that it would be beneficial for just about anyone to read, and just like Weapons of Math Destruction should be read by anyone who works as a data scientist (or frankly, developing an "AI" solution of any sort), so does this book should be read by any software engineer, especially if they are working in a data hogging company (and yes, most people might not notice how much data their product is collecting before reading this book, so just read it).

One thing to remember before we dive into the book - The first and by far the most important takeaway is that breaching our privacy is not about knowing our secrets, it's about manipulation, coercion and control.

The book itself is divided into three parts depicting the advancement of surveillance capitalism - from its foundation, mostly attributed to Google, to the ways it has moved to influence the physical world and then to the social world, each step providing it with more instrumentation capabilities. The main challenge the book raises is that surveillance capitalism is "unprecedented", that is - we've yet to see something like it and therefore the discourse about it is unfit and lacking, hindering us from grasping the potential cost our society is paying, from understanding the mechanisms behind its operation and from drawing boundaries in appropriate places. The fourth part (as the author counts four) is a conclusion

The term "surveillance capitalism" is not coincidental, nor is it neutral - the basic claim in this book is that like the early capitalist "robber barons" who, in the name of making profit have committed atrocities and invoked their "right to private property" to defend their ability to extract  wealth out of unclaimed lands or "freedom of contract" as a way to defend child labor or minimal wages with horrid conditions, so does the surveillance capitalism harm society in their actions while hiding behind verbal jiu-jitsu and "freedom of speech\press" 

But what is surveillance capitalism anyway? While the book is a bit more detailed, the short answer is that it's an industry that harvests personal information and produces predictions of human behavior (namely - on which ads we'll click, what products we'll buy, and what message will convince us to vote). In order to improve those predictions - it moves to shaping said behaviors. No, no need to reach for your tin foiled hats, but there is  reason to be worried.   

Here's the part that could be a summary of each chapter, one that will be much shorter and won't do it any justice, so instead I'll share some of the key points that stuck with me so far. 

  • The first one is that the entire business is shrouded in euphemisms and intentional misguiding - from terms such as "data exhaust" or "digital bread crumbs" hinting that the data about our online activity is useless and that making it into something useful is not only harmless - it's benign and similar to recycling, to "personalization" which is an intentional mask to the fact that the profiling is not done to the benefit of the user. 
  • We are not the product, we are the resource being mined and cultivated for the raw materials that compose the product. We are not Ford's T-model, or even the metal used to build it, we are the ore being mined for our data. While the book does not go there, it's quite clear to me that this setup is a strong incentive to squeeze us for as much as possible - if the raw materials are second grade, the quality of the product is hindered, but the source of those materials is far enough the chain so that it's cheaper to dig deeper and mine more than it is to maintain and cultivate it. 
  • Science finds - Industry applies - Man conforms: This motto from Chicago's Fair of 1933 provides a succinct reminder to how much technology changes the way we behave: we use calculators, we drive with a navigation app, and we prioritize communication with people far away over those we're sitting with right now, simply because they ping us through our phone. In similar manner we've all been accustomed to various services provided by the surveillance capitalist firms, and we learned to depend on them. 
  • Breaking the law is a business strategy. The playbook goes roughly that way: Invent something never before seen, or that is not explicitly and clearly defined by law, start doing it. When being sued or when attempts to enforce existing regulatory mechanisms are starting - delay. Finally, claim that people are relying on the service you provide (and have been providing during the entire delay period). It can be Google Street view that drags court orders for years and in the meanwhile continue to operate and update, infringing on people's privacy in the meanwhile, and we can see this being done with Uber & Lyft trying to fend off their drivers employee rights by delaying and extending the litigation process as much as they can. 
  • Surveillance capitalism is pushing towards omnipresence - thanks to IOT and "smart" everything, there are multiple sources of information for just about anything - "smart" cities sharing our location data with Uber &co under the excuse of optimized traffic routing, our sports heart-rate meter sharing health information with a running app, Roomba-like devices mapping our apartments and the Nest thermostat communicating with our door lock and keeping track on our daily routine. 
  • It's not about knowing, it's about nudging. Facebook did an experiment of increasing voter turnout, Pokemon go was herding people to stores that paid to have that privilege. While it isn't mind control, imagine this scenario - Next elections, Facebook uses its own data to determine the users expected vote and showing the effective promotions only to those voting "right", tipping the scale just a bit. Too far fetched? What if it was about "encouraging" people to vote against a law that would limit Facebook's ability to make money? 
  • Industrial capitalism, while devastative to nature and natural resources, was rather beneficial to people since it was relying on ever increasing scale of production, which means affordable products (also, if people can't buy, it's worth paying them a salary so that they'll have money to spend on luxuries). Surveillance capitalism, on the other hand, is parasitic by nature, it depends on scale of extraction. Surveillance capitalism companies make their profit out of the few rather than out of the many, and while today most of the revenue comes from product advertising, even in the remote case no products will advertise anything, the market for behavioral predictions and influence will always have buyers - Every political actor - Be it formal candidates, NGOs or even dissidents and malicious actors.
  • Not that we didn't know that beforehand, but users are forced to sign unfair and unreasonable "contracts" and submit to a misleadingly named "privacy policy" (or, as it being called in the book - "surveillance policy") which might change at the company will at any time.Refuse to submit? you are hereby denied service in an disproportional scale, sometimes forcing you to stop using a product you purchased or keep it non-functional and even dangerous.
  • Last, and probably the most important point: It does not have to be this way. Search queries are not stored because "that's how search engines work", they are stored because the company makes profit from storing and using them, mining our email to enrich our hidden user profile is not just the price of getting a "free" inbox that syncs to our "free" calendar, it's a dangling bait to make sure we continue to provide the company with more data - both about ourselves and realistic samples for making datasets for various ML training that would in turn enhance its ability to access more data. Since the way things currently are is a choice, it can be changed to a more balanced and fair system. 

 

The book, while remarkable, is not free of weak spots - People who take this book at face value can get the impression that surveillance capitalism controls every aspect of our lives, to the point where we lose the will to will (there is, in fact, a chapter that has this very claim). While I don't think it's wise to dismiss this risk, we are not there yet. Most manipulation tactics used to nudge people actions are not super-effective, and people devise defense mechanisms against them, so that effect is erodes over time (think on how accustomed you've become to glance over web-ads without really noticing them and how quick you are to categorize them as such) - in this case, I believe it's best to look at intentions. Even if the power is limited, nudging-power is what those companies sell, so it's fair to assume they'll get better at it over time but even if they never make it to a really scary level, the attempt of perfecting coercive power is harmful in itself. 

Another thing I found less effective in the book has a somewhat Marxist feeling to it. The terminology it tries to coin, being put out to oppose surveillance capitalism, is very similar to the terminology used to oppose industrial capitalism - "means of production" is scattered in the book, and a good place is dedicated to the "division of learning", and to what the author tries to convince the readers (not sure about how successful this attempt is) that the essential questions we should be asking all the times are "who knows? who decides? who decides who decides?" While reading I found myself agreeing with the message while at the same time finding my defense mechanisms raised high by this choice of words. Perhaps it's by design, perhaps it's just an effect of having a different milieu which makes me resist those idea because of this wording. 

 

So, to sum everything up - I found this book important mostly because it is a very good first step. It's a first step towards creating a critical language when evaluating data collection and usage, and a first step towards reclaiming our right to privacy and control. I hope it will foster deeper, clearer discussion about how are we moving forward to create the right sort of technology, supported by an appropriate regulation to make a better society.

Friday, July 17, 2020

Being right about the wrong things


So, following a link I encountered on Josh Grant's blog I came across a post named "against testing" where the author (someone named Ted, I think) has some very strong claims about tests and why most people shouldn't write them. On the face of it, most of those arguments are sound and represent a well thought of observations. It is true that most automated tests, especially unit tests, rarely find bugs, and that they are tightly coupled to the existing implementation in a way that means that whenever you come to refactor a module you'll find yourself in need of refactoring the tests as well.It is also true that there are a lot of projects out there where testing something in isolation is simply not that easy to do (There's a reason why Michael Feathers defines legacy code as "code without tests" ) and investigating a test fail requires putting a lot of time into understanding code that won't help you when you come to develop the next feature. Furthermore, having test code does introduce the risk of delivering it to production, where all sorts of nasty things might happen. No test code? This risk does not exist. 
All of those seem to follow a sound logic chain resulting in one conclusion - don't write tests. Not surprisingly, I disagree.
The easiest way to dismiss most of those claims is to respond with "most of those problems indicate that your tests are not good", and while I do hold true that a well written test should be clear to understand, at just about the right level of abstraction to minimize refactoring pain and targeting the tested unit commitments rather than implementation, I also know that those are hard to write, and that most people writing test code are not doing such a pristine job most of the time. In fact, for my response I'm going to assume that people write mediocre to trivial tests, simply because that's the most likely scenario. Most people indeed don't learn to write proper tests, and don't practice it. They get "write tests" as a task they must do to complete their "real" work and thus do the bare minimum they must. 

From my perspective, the post is wrong right at the beginning, stating that "In order to be effective, a test needs to exist for some condition not handled by the code", that is - a test is meant to "find bugs". For me, that's a secondary goal at best. I see tests as scaffolding - making the promises of a piece of code explicit, and here to help people refactoring or using that piece of code. If someone is working in a TDDish manner (no need to be strict about it) they can use this scaffolding to figure out earlier how their code should look like - when an internal logic that totally makes sense when implementing is just too cumbersome to use or when we need some extra dependencies. It is also a nice way to put things I don't want to forget in a place where I'll be reminded on. 
But, that's assuming TDD, and not enough people are using this method to justify writing tests, or to not delete them once I'm done, and that's when I get to two of the most common tasks a developer faces: refactoring code and investigating bugs. Starting with the fun part - refactoring. When refactoring a piece of code, there's one single worry - did I break something? Testing alone does not answer this question, but it does help in reducing it, especially in a language without a strict compiler. Imagine a simple python project, where there is a utility module that is being called extensively. I go and change the type of one of the parameters from string-duck to Object-duck (let's imagine I'm assuming a .foo() method to be available). It is already the case in 99% of the project, but not necessarily in all. If I wasn't using proper type hinting (as is sadly way too common), the only way I'll find this will be if I'll run the specific piece of code with the faulty code. a 100% line coverage increases my chances of finding it. Too far fetched? ok. What about a piece of code that is not straightforward? one that has what the linters like to call "high complexity". Just keeping those intricate conditions in mind is heavy lifting, so why not put those in a place where I'll get feedback if my refactor broke something?
Those types of functions are also a nightmare to debug or fix, and here I want to share an experience I had. In my previous workplace we had a table that was aggregating purchases - if they matched on certain fields we called them equal and would merge them. Choosing what to display had a rather complex decision tree due to what matched our business. I got a task of fixing something minor in it (I don't recall exactly what, I think it was a wrong value in a single column or something like that). Frankly? The code was complicated. complicated enough so that I wasn't sure I could find the right place. So I added a condition to an existing test. It wasn't a good test to begin with, and in fact, when I first approached it, it couldn't fail because it asserted that "expected" was equal to "expected" (it was a bit difficult to see that, though), Once I added the expected result to the test I could just run it, fix the problematic scenario, and move on to the next one. The existing tests did remind me of a flow I completely forgot about (did  I mention it was a very complicated decision tree?). 

Another useful way to use tests is as an investigative tool. In my current workplace we are using python (Short advice - don't do it without a good reason). Moreover, we are using python 3.6. we do a lot of work with JSON messages, and as such it's nice to be able to deserialize a message into a proper object, such as can be done with Jackson or Gson in Java. However, since python has a "native" support for json, I didn't manage to find such a tool (not saying there isn't), so in order to avoid string literals, we defined a new type of class that takes a dictionary and translate it to an object. Combined with type hints - and we have an easy to use auto-complete friendly object (In python 3.8 they introduced a "data class" that might do what we need,  but that's less relevant here). To do that we've overrode some of the "magic" methods ( __getattr__, for instance), which means we don't really know what we did here and what side effects there are. What we did know were our intended uses - we wanted to serialize and deserialize some objects, with nesting objects of various types. So, after the first bug manifested, we added some tests - we found out that our solution could cause an endless call-loop and that we don't really need to support deserializing a tuple, since a json string can only be a simple value, a list or a dictionary (not something we thought about when we started implementing the serialization part, so we saved some time by not writing this useless piece of code). Whenever we were unsure about how would our code behave - we added a test about it. Also, since we didn't really understand what we were doing, we managed to fix one thing while breaking another several times. Each time our existing tests showed us we had a problem. 

There is, however, one point on which I agree with the author - writing unit tests does change the way your code is written. It might add some abstraction (though that's not necessarily the case with the existing mocking tools) and it does push towards a specific type of design. In fact, when I wrote tests for a side project I'm working on, I broke a web controller to 5 different classes just because I noticed that I had to instantiate a lot of uninteresting dependencies for each test. I'm happy with that change since I see it as something that showed me that my single class was actually doing 5 different things, albeit quite similar ones. As a result of this change, I can be more confident about the possible impact a specific change can have - it won't affect all 5 classes, but only one of them, and this class has a very specific task I know how to access and which flows it's involved in. Changing an existing code this way does introduce risk, so everyone need to decide whether the rewards are worth taking those risks. If all you expect from your tests is to find bugs or defend against regression - the reward is indeed small. I believe that if you consider the other benefits I mentioned - having an investigative tool that will work for others touching the code, being explicit about what a piece of code promises and in the long run having smaller components with defined (supported) interactions between them - it starts to make much more sense. 


So, to sum up what I wrote above in a short paragraph - the author is completely right in claiming that if tests are meant to find bugs and defend against regression, they don't do a very good job. But, treating tests in such a way is to claim that a hammer is not very effective because it does poor job in pinning screws to a wall. Tests can sometimes find bugs, and they can defend against some of the bugs introduced by refactoring. but they don't do these tasks very well. What tests do best is mostly about people. They communicate (and enforce) explicit commitments, they help investigate and remember tasks and they save a ton of time wasted on stupid mistakes and leaves more time to deal with the real logic difficulties a code presents. I think that by looking at those properties of having tests, their value is represented better, and it also becomes easier to write better tests.