We’re Living in a World of Black Box Systems

Sounds like another episode of the dystopian tv-series Black Mirror.
Reading Wired this morning, journalist Siva Vaidhyanathan thinks Members of U.S. Congress don’t fully understand the tech companies they’re supposed to regulate and goes on to suggest that none of us really do either.

So as we look at the myriad ways Google and Facebook have let us down and led us astray, let’s remember that no one has the manual. No one fully understands these systems, even the people who designed them at their birth. The once impressive, now basic, algorithms that made Google and Facebook distinct and useful have long been eclipsed by even more sophisticated and opaque data sets and machine learning. They are not just black boxes to regulators, journalists, and scholars. They are black boxes to the very engineers who work there.

Siva Vaidhyanathan, wired.com

Use any of the popular web apps and services offered by FAANG and you might be wondering what’s actually going on behind the scenes. What data points are collected? How is the price I’m seeing on that Amazon purchase ultimately determined — by my location/IP address? — by a purchase history algorithm? — by my recent social media activities?

Many of these systems collect obscene amounts of data. While our inputs and outputs can be viewed, the inner workings — the bits of personal data scraped up behind the slick front-end UI — are largely hidden to us. Smart phones and smart home hubs are good examples of devices that run a plethora of background ‘services’ that are always connected to the net, constantly pinging information back to the mothership. These devices are perhaps also listening to our private conversations despite assurances from manufacturers and software developers otherwise. Buried somewhere in the 10-page jargon-laden terms of service agreement, you might have knowingly (or unknowingly) granted access to your device’s text messaging, microphone and/or photos. Unscrupulous data brokers will gladly harvest this information for a variety of purposes, the least sinister of which is probably to just serve you targeted advertisments.

Tech giant Google is currently facing a class-action lawsuit that alleges they track users on hundreds of thousands of apps even when they opt out of “Web & App Activity” in the settings. This yet again raises the spectre of privacy erosion in the digital age with many of us now realizing the convenience of using “free” apps and services typically means we’re paying with our personal information which may be sold (and re-sold) to 3rd parties. We also pay through diminished battery life on our mobile devices (and the added requirement for higher capacity data plans) because of all the data pinging back and forth. Is it any wonder 5G networks are touted as the next big thing. Our data, and more of it, will just get pushed around a helluva lot quicker!

Maybe I’m a bit old-fashioned, but I’m not keen of the idea my headphones are collecting personal data every time I listen to something. I question the necessity of bluetooth lightbulbs controlled via smartphone or my refrigerator collecting data on what our family eats each week and able to tell us when we’re out of milk and eggs. There’s an endless array of IoT products and services that are unnecessarily complex and over-engineered with user data collection perhaps foremost in mind.
I’m old enough to remember a time when household appliances were designed and engineered to last more than 5 years and most automobile engine issues could be fixed without special computer diagnostic equipment. Instead, we have a trend towards OEM products that can’t be readily serviced by 3rd parties or mechanically-inclined owners. Core product functions are increasingly controlled by proprietary software that will lock out, or worse ‘brick’ a device or system from users who attempt any tinkering or repair. This has bolstered the right to repair movement in several countries among consumers seeking greater control over the products they own.

All of this reminds me of Nicholas Carr’s 2015 ideacity talk where he asks the audience to “go out onto the sidewalk and smash your smart phones“. Don’t you sometimes feel like doing that?