M ARCO A URÉLIO F ERNANDES G ARCIA (018099174B) I NFERENCE - BASED ALGORITHMIC DISCRIMINATION : LEGAL REMEDIES UNDER THE G ENERAL D ATA P ROTECTION R EGULATION LL.M in Space, Communication and Media Law University of Luxembourg Faculty of Law, Economics and Finance Thesis Supervisor: Professor Dr. Hielke Hijmans Luxembourg, June 2020. 2 Acknowledgments To Letícia Francisco Alves Ribeiro Dias, for being an unsurpassable source of inspiration. I address my sincere gratitude to my parents, José Leonardo Garcia and Suzete Fernandes Garcia, to my brother, Marcos Vinicius Fernandes Garcia, for the unconditional support throughout all my life, and to my Law firm partner, Luís Fernando Costa Oliveira, for always being there when needed. I would like to acknowledge the University of Luxembourg and the assistance and support of my supervisor, Professor Hielke Hiejmans, for being responsible for my instruction in data protection law, and to the Professors of the University of Luxembourg that have contributed to my formation, particularly Professor Mark Cole, for his never-ending enthusiasm with classes, and Professor Mahulena Hoffman, for her passion on the themes lectured. I would like to express my thankfulness for each colleague that took part of this four months, then ten months, and finally two years journey in Luxembourg. Particular thanks are directed, in no particular order, to my friends of the University of Luxembourg, Delphine Ernst, Sigrid Heirbrandt, Veerle Bregman, Yi Gong, Nicolas Wurth, Irian Pinillos, Kevin Calzada, Athanasios Mavroulis, Corentin dal Pra, Júlia Drummond, Guilherme Belotto, Natalia Olechnowicz, Chloé Catallo, Eriona Prençi, and Mohammed Amr, as well as to my colleagues at the Luxembourg Space Agency, especially Dovilé Matuleviciuté, Sarah Fleischer, Joseph Mousel, and Gary Martin. I am gratefully indebted to the lessons learned in Luxembourg and I hope they are the start of a journey. This work is just the start; we shall not cease from exploration, and, in the end of all our exploring, we shall arrive where we started and know the place for the very first time. “ It seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers... They would be able to converse with each other to sharpen their wits. At some stage therefore, we should have to expect the machines to take control. ” (Alan Turing, 1947) 3 Abstract The emergence of sophisticated machine-learning systems led to new challenges that traverse the fields of artificial intelligence and data protection, requiring equally new legal responses. As algorithms may either be a force to foment or to curtail discrimination, discussing their improvement is of the utmost importance to the society. This work addresses the applicability of legal remedies under the General Data Protection Regulation to detect and deter inference-based algorithmic discrimination. Firstly, we explore the contours of the basic concepts, and underlying epistemic and normative concerns, of algorithms and discrimination. Secondly, we discuss under what circumstances should inferred data be considered “ personal data ” under the General Data Protection Regulation. In the sequence, we analyze the overall availability of GDPR remedies to address inferred personal data, as well as the application of each one of the “ candidate ” GDPR rights , i. e ., the right of access, rectification, erasure, object, data portability, and to oppose automated decision-making, to address inferred personal data. Finally, we discuss the combination of GDPR legal remedies with the current framework of anti-discrimination law in the European level, presenting brief remarks as how to improve protection against inference-based discrimination. Although absent a general provision on anti-discrimination in the GDPR, the combination of different GDPR legal remedies may assist the data subject to detect and challenge discrimination on the individual level. The possibility and effectiveness of collective anti-discrimination action under the GDPR remains unclear and, without collective action, discriminatory (either facially or apparently neutral) algorithms are bound to continue discriminating against new data subjects. Due to GDPR lackluster anti-discrimination framework, the combination of GDPR remedies with the current acquis of European anti- discrimination Law may stand out as a more suitable option for the data subject to collectively challenge inference-based discrimination on protected grounds. Finally, we propose the implementation of a right not to be discriminated under the GDPR, in the same line of anti- discrimination EU Law, in order to guarantee procedural and substantive advantages for the data subject who is a victim of discrimination, including, inter alia , shifting the burden of proof, eligible representation, and imposing the obligation to take affirmative actions. Keywords: data protection; general data protection regulation; inferences; inferred data; discrimination. 4 INTRODUCTION ................................................................................................................................................................... 7 CHAPTER I – THE CONCEPT OF ALGORITHM ......................................................................................................... 10 1. D EFINITION OF “ ALGORITHM ” ..................................................................................................................................... 10 2. E PISTEMIC AND NORMATIVE CONCERNS RELATED TO ALGORITHMS ............................................................................ 11 3. A LGORITHMIC PROFILING ............................................................................................................................................ 16 CHAPTER II – ALGORITHMIC-BASED DISCRIMINATION ..................................................................................... 20 1. C ONCEPT OF DISCRIMINATION .......................................................................................................................................... 20 2. T HEORIES ON DISCRIMINATION ........................................................................................................................................ 23 3. A DDRESSING DISCRIMINATION UNDER THE L AW .............................................................................................................. 24 4. A LGORITHMS AS A SOURCE OF DISCRIMINATION AND BIAS .............................................................................................. 27 5. D ISCRIMINATION UNDER THE GDPR ................................................................................................................................ 30 CHAPTER III – INFERRED DATA A S “PERSONAL DATA” UN DER THE GDPR ................................................. 32 1. I NFERENCES AND INFERENCE - BASED DISCRIMINATION ..................................................................................................... 32 2. A FIRST APPROXIMATION TO THE CONCEPT OF INFERRED PERSONAL DATA ...................................................................... 34 3. T HE ECJ DECISIONS ON THE DEFINITION OF “ PERSONAL DATA ” ....................................................................................... 37 4. D RAWING THE LINE : INFERRED DATA THAT ARE PERSONAL DATA , AND THOSE THAT ARE NOT ........................................ 41 CHAPTER IV – ANALYSIS OF LEGAL REMEDIES FOR INFERENCE-BASED ALGORITHMIC DISCRIMINATION UNDER THE GDPR ......................................................................................................................... 45 1. A VAILABILITY OF THE LEGAL REMEDIES TO INFERRED PERSONAL DATA : PERSONAL DATA “ PROVIDED ”, “ COLLECTED ” FROM , OR “ CONCERNING ” THE DATA SUBJECT ...................................................................................................................... 45 2. I MPLEMENTATION OF GDPR REMEDIES IN RELATION INFERRED DATA AND INFERENCE - BASED DISCRIMINATION ............ 51 A. Right of access to inferred personal data ................................................................................................................... 51 B. Right to rectify inferred personal data ....................................................................................................................... 54 C. Right to erase inferred personal data ......................................................................................................................... 57 D. Right to object processing of inferred personal data ................................................................................................. 60 E. Right to data portability of inferred personal data ..................................................................................................... 62 F. Right to oppose automated decision-making and a right to contest automated decisions related to inferred personal data ................................................................................................................................................................................. 63 3. C OMBINING GDPR REMEDIES WITH THE ANTI - DISCRIMINATION FRAMEWORK TO DETER INFERENCE - BASED DISPARATE TREATMENT AND DISPARATE IMPACT ................................................................................................................................... 66 CHAPTER V – IMPROVING THE PROTECTION AGAINST INFERENCE-BASED DISCRIMINATION ........... 75 1. C HALLENGES ON THE IMPROVEMENT OF THE PROTECTION AGAINST INFERENCE - BASED DISCRIMINATION ....................... 75 2. A RIGHT TO REASONABLE INFERENCES ? ........................................................................................................................... 77 3. A RIGHT NOT TO BE DISCRIMINATED ? ............................................................................................................................... 78 CONCLUDING REMARKS ................................................................................................................................................ 81 BIBLIOGRAPHY .................................................................................................................................................................. 82 5 List of Abbreviations CEDAW Convention on the Elimination of All Forms of Discriminati on Against Women [1979] 1249 United Nati ons, Treaty Series 13 Data Protection Directive Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995 on the protection of individuals with regard to the processing of personal data an d on the free movement of such data [1995] OJ L 281 ECJ European Court of Justice Equal Treatment in Goods and Services Directive Council Directive 2004/113/EC of 13 December 2004 implementing the principle of equal treatment between men and women in t he access to and supply of goods and services [2004] OJ L 373 Equal Treatment in Employment and Occupation Directive Directive 2006/54/EC of the European Parliament and of the Council of 5 July 2006 on the implementation of the principle of equal opportu nities and equal treatment of men and women in matters of employment and occupation (recast) [2006] OJ L 204 Equality Framework Directive Council Directive 2000/78/EC of 27 November 2000 establishing a general framework for equal treatment in employment and occupation [2000] OJ L 303 EU Charter Charter of Fundamental Rights of the European Union [2000] 2000/C 364/01 6 European Convention on Human Rights European Convention for the Protection of Human Rights and Fundamental Freedoms, as amended by Protoc ols Nos. 11 and 14 [1950] ETS 5 GDPR Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and re pealing Directive 95/46/EC (General Data Protection Regulation) [2016] OJ L 119 ICCPR International Covenant on Civil and Political Rights [1966] 999 United Nations Treaty Series 171 ICESCR International Covenant on Economic, Social and Cultural Rights [1966] 993 United Nations Treaty Series 3 ICERD International Convention on the Elimination of All Forms of Racial Discrimination [1965] 660 United Nations Treaty Series 19 Race Equality Directive Council Directive 2000/43/EC of 29 June 2000 implement ing the principle of equal treatment between persons irrespective of racial or ethnic origin [2000] OJ L 180 WP29 Article 29 Data Protection Working Party 7 INTRODUCTION Discrimination is ubiquitous, every so often clear, occasionally disguised. It derives both from the manner a person acts and the way a person does not act. It stems from unchangeable characteristics, such as “ race ” 1 , as well as from the littlest details of everyday life. Although profoundly unjust, one cannot properly imagine a society in which discrimination is non-existent, not even including M ORE ’ s Utopia 2, in which every household was allowed two slaves. Recognizing the detrimental effects of discrimination in several spheres of human life, the Law establishes mechanisms to detect and deter discrimination. Nevertheless, discrimination is not always easy to uncover, as it usually requires, among many others factors, an understanding of context, the access to different sets of information, a recognition of covertly detrimental effects to groups, an external benchmark to compare, and, ultimately, a value judgment as to the justice of the situation. In this regard, computer algorithms are neither the most context-sensitive entities nor particularly good at gathering insights on justice. Algorithms are present in every aspect of modern life in society, including via the substitution of decisions historically made by humans 3. However, the black box nature of computer algorithms, i. e. , the inscrutability of these systems, represents a natural obstacle to combat discrimination, which is particularly enhanced by the development of recent machine-learning techniques. During the preceding years, programmers have believed that there is a trade-off between accuracy and interpretability, ultimately foregoing the development of interpretable models 4. Such mystification of technology ultimately leads to society calls for more accountability concerning these programs. Algorithms may either be a force to promote or curtail discrimination. As they are developed by humans, they usually reflect the values, desired outcomes, interests, and biases of their creators and 1 Some terms used in this work may be considered derogatory by individuals or groups. However, it is indeed necessary to refer to them when discussing discrimination. Although we are discussing with such a sensitive topic – discrimination – we should not refrain from addressing the usual categories of discrimination, including extremely sensitive categories, such as “blacks”, “LGBT”, among others. The use of these words should be fully understood as part of an academic word in which the author intends to produce no harm. 2 Written in 1516, Utopia is a satire novel written by Thomas More describing a fictional island ruled by a perfect government, under which More is able to discuss and expose contemporaneous European politics through illustrations of (supposedly) perfectly balanced institutions and lives. Thomas More, Utopia (first published 1516, Penguin 2012) 3 Joshua Kroll, Solon Barocas, Edward Felten, Joel Reidenberg, David Robinson, Harlan Yu, ‘Accountable Algorithms’ [2017] 165 University of Pennsylvania Law Review 633, 633-634 4 Cynthia Rudin, ‘Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead’ (2019) 1 Nature Machine Intelligence < https://arxiv.org/abs/1811.10154> accessed 18 May 2020, 206 8 users. Operation within pre-defined accepted parameters of static algorithms may lead to ethically unacceptable behaviors either intentionally or due to the difficulty to predict outcomes from the outset of the system ’ s development. The graveness of the situation rose with the design of “ learning ” algorithms, which are able to alter operation parameter and decision-making rules during the execution of the system ’ s intended functions 5. In both cases, but especially in the latter, the use of non-initially- intended data by the algorithm, such as the use of special categories of data by inference, may be occurring, with detrimental consequences to the rights and freedoms of individuals. The Gener al Data Protection Regulation (“GDPR” ) 6 addresses algorithm-based processing under the heading “ automated decision-making ” , including “ profiling ” . Surprisingly, “ discrimination ” is only mentioned at the Recitals 7 . No particular reference to “ inferential analytics ” is made, although one Recital reflects concerns on discriminatory effects of profiling 8. The scope of GDPR mechanisms to protect data subjects against malpractices related to automated decision-making is unclear and subject to ongoing debate 9. The first cases decided under the GDPR provided no further guidance on the matter 10. Available and effective legal mechanisms to deter discrimination arising out of automated decision-making related to inferential analytics under the GDPR are certainly scarce. Hence, we intend to study the legal remedies currently available under the GDPR against algorithm discrimination by inference in order to verify whether the data subject is properly safeguarded against these novel practices and systems. This work addresses this question exclusively on the EU-level. In that sense, we first describe the basic idea of algorithm, algorithmic profiling and basic underlying epistemic and normative concerns related to algorithms ( CHAPTER I ), and the intersection 5 Brent Mittelstadt, Patrick Allo, Mariar osaria Toledo, Sandra Wachter, Luciano Floridi, ‘The Ethics of Algorithms: Mapping the Debate’ (2016) Big Data & Society < https://journals.sagepub.com/doi/full/10.1177/2053951716679679> accessed on 13 May 2020, 2 6. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) [2016] OJ L 119 7 Recitals 71, 75 and 85 of the GDPR 8 Recital 71, second part 9 Sandra Wachter, Brent Mittelstadt, Luciano Floridi, ‘Why a Right to Explanation of Automated Decision -Making Does Not Exist in the General Data Protection Regulation’ [2017] 7(2) Internatio nal Data Privacy Law 76, 76-77 10 Joined Cases C-141/12 and 372/12 YS v Minister voor Immigratie, Integratie en Asiel, Minister voor Immigratie, Integratie en Asiel v M, S [2014]. Case 436/16 Peter Nowak v Data Protection Commissioner [2017] 9 between algorithms and discrimination ( CHAPTER II ). Then, we consider the relationship between inferred data and the GDPR, defining under what circumstances inferred data should be considered “personal data” ( CHAPTER III). In the sequence, we proceed to an overview of proposed legal remedies to improve the data protection system in relation to algorithmic discrimination ( CHAPTER IV ). Finally, we present our observations on how to improve the protection against inference-based algorithmic discrimination ( CHAPTER V) 10 CHAPTER I – THE CONCEPT OF ALGORITHM This introductory part presents the underlying basic concepts of algorithm, bias and discrimination that are object of further discussion in the following chapters. In order to introduce the intricacies of the relationship of the GDPR and algorithmic discrimination, we shall divide this chapter in three sections. The first section concerns the basic definition of algorithms ( Section 1 ), the second address the underlying epistemic and normative concerns related to algorithms ( Section 2 ), and the third discusses the practice of algorithmic profiling ( Section 3 ). 1. Definition of “ algorithm ” Algorithms are responsible for every computation operation in computers; in essence, they are a “ method for solving problems ” in a field concerned with “ systematic problem solving ” 11 . S KIENA defines algorithm as a procedure to solve a well-specific problem, which is specified by a certain set of instances to work and the outputs derived from these instances, transforming the possible input instances in a desired output 12. H ILL presents a similar definition, considering algorithms as mathe matical constructs composed by “ a finite, abstract, effective, compound control structure, imperatively given, accomplishing a given purpose under given provisions ” 13 For a more precise and mathematical representation of algorithm, we may use the following scheme presented by R APAPORT 14 : An algorithm (for executor E) [to accomplish goal G] is: 1. a procedure P, i.e., a finite set (or sequence) of statements (or rules, or instructions), such that each statement S is: (a) composed of a finite number of symbols (better: uninterpreted marks) from a finite alphabet (b) and unambiguous (for E — i.e., (i) E “knows how” to do S, (ii) E can do S, (iii) S can be done in a finite amount of time 11 Martin Erwig, Once upon an Algorithm: How Stories Explain Computing (MIT Press, 2017) 3-4 12 Steven Skiena, The Algorithm Design Manual (2 nd edn, Springer, 2008) 3 13 Robin Hill, ‘What an Algorithm Is’ (2015) 29(1) Philosophy & Technology 35, 47 14 William Rapaport, ‘On the Relation of Computing to the World’ in Thomas Powers (ed.), Philosophy and Computing – Essays in Epistemology, Philosophy of Mind, Logic, and Ethics (Springer 2017) 42 11 (iv) and, after doing S, E “knows” what to do next— ), 2. P takes a finite amount of time, i.e., halts, 3. [and P ends with G accomplished]. Although computerized algorithms are certainly a new phenomenon in history, K ITCHIN dates back the term “ algorism ” to the 12 th century region of modern Spain, where the transcripts of M ūsāal - Khwārizmī containing methods of addition, subtraction, multiplication and division were translated to Latin. These systematic methods for performing elementary arithmetic were translated as “ algorisms ” This root would have been revisited in the creation of the programming language ALGOL 58, in 1958, meaning algorithmic language in German ( algorithmische Sprache ) 15. Applications using algorithms are a commonplace in society, ranging from the most basic calculations to online service providers, advanced facial recognition techniques, internet of things applications, among several others 16. Algorithms are not programmed in English (or any other natural languages) because natural languages have too many ambiguities 17. Since algorithms are ever-present, this situation begs the question on why these computational creatures would pose any threat of discrimination. The answer and the concerns underlying algorithms reside on the particular characteristic of these processes, which we address in the following sections. 2. Epistemic and normative concerns related to algorithms According to M ITTELSTADT , concerns related to algorithms may be classified under two dimensions: epistemic and normative. The former refers essentially to the manner humans strive to understand algorithms results, whereas the latter involves the effects caused by algorithms. On the first category, we have issues arising out of uncertainty of knowledge produced by algorithm (inconclusiveness), unintelligibility of the processes that lead to the verified outputs (inscrutability), and unreliability of outputs due to questionable inputs (misguidance). On the second category, we observe questions concerning unfair outcomes of algorithmic processes (unfairness), the transformation of our behavior and perspectives due to algorithmic-based analysis, such as profiling (reontologization), and difficulties 15 Rob Kitchin, ‘Thinking Critically about and Researching Algorithms’ [2017] 20(1) In formation, Communication & Society 14, 16 16 Brent Mittelstadt, Patrick Allo, Mariarosaria Toledo, Sandra Wachter, Luciano Floridi, ‘The Ethics of Algorithms: Mapping the Debate’ (2016) Big Data & Society < https://journals.sagepub.com/doi/full/10.1177/2053951716679679> accessed on 13 May 2020, 1 17 Martin Erwig, Once upon an Algorithm: How Stories Explain Computing (MIT Press, 2017) 5 12 on holding responsibility for harm caused by algorithms (traceability) 18. As a thesis on Law, we are generally more preoccupied with the normative dimension of algorithms. In particular, discrimination and bias are themes closely connected to the fairness of algorithmic processing. However, we cannot ignore the epistemic challenges of algorithms and their importance in the underlying normative responses. In order to balance this issue, we shall briefly explore the epistemic dimension of algorithms in this section, and then proceed to address mostly normative concerns throughout the remained of the work. A point that combines the inconclusiveness, inscrutability and misguidance of algorithms is that a lgorithms are “ blind ” ; they might be excellent calculators without even knowing what algebra or arithmetic are. One simple example of such blindness is posed by E RWIG concerning the calculation of the square root of two, since the computer need not to know what a square root is in order to perform this operations. For instance, it may draw a horizontal line one centimeter long and a vertical line of the same length, perpendicular to the other and starting from its end. Then, by drawing a diagonal connecting the two open ends the computer has finished the calculation of the square root of two 19. This initial perspective on artificial intelligence is usually attributed to T URING , since pre-T URING the common knowledge was that computers were to learn new operations as people and thus would need to actually know these fields in order to reproduce mathematical operations. To the question on whether machines can think, T URING ’ s reply is that this is “ too meaningless to deserve discussion ” , since the computers are intrinsically involved in an “ imitation game ” that even the greatest storage capacity would not alter, although it would be expected that computers would be so profoundly proficient in the imitation game such as to induce humans to confuse machines with humans more often than not 20. Such reasoning is known to be premised in D ESCARTES consideration that even if humans were able to construct machines that resembled men, they would still not be men because their actions and words would infallibly lack reason and meaning 21. 18 Brent Mittelstadt, Patrick Allo, Mariarosaria Toledo, Sandra Wachter, Luciano Floridi, ‘The Ethics of Algorithms: Mapping t he Debate’ (2016) Big Data & Society < https://journals.sagepub.com/doi/full/10.1177/2053951716679679> accessed on 13 May 2020, 4-5 19 Martin Erwig, Once upon an Algorithm: How Stories Explain Computing (MIT Press, 2017) 1 20 Alan Turing, ‘Computing Machinery and Intelligence’ [1950] 59(236) Mind 433, 442 21 René Descartes, Discours de la méthode (first published 1637, Édition Electronique Les Échos du Maquis 2011) 35. The consideration that T URING ’ s reasoning is premised on D ESCARTES ’ is found on: Stanford Encyclopedia of Philosophy , ‘The Turing Test’ (2016) < https://plato.stanford.edu/entries/turing -test/> accessed on 16 April 2020 13 T URING ’ s considerations produced an inversion of such reasoning, causing computers to be understood as imitators from whom you may “ squeeze out the last tiny smidgens of understanding leaving nothing but brute, mechanical actions ” 22 . According to D ENNET ’ s argument reverberating T URING ’ S thinking, just as Darwinian evolution need not (and it is not) be rational in order to produce a living organism, computers are not under any particular duty to understand the operations they perform 23. The machines prevailing in T URING ’ s era were certainly not cognitive agents. However, one may wonder whether the newest developments do not effectively reject this characteristic. Many arguments have been raised for both sides of the dispute. We need not to provide a definite answer, but a few remarks on this point are useful for the themes developed in the next chapters. Certain points raised by R APAPORT stand out in favor of the T URING ’ s approach. Computers are not syntactic entities, as they instead work with number-theoretic functions, meaning they do not compute symbols with meaning, but simply symbols. This implies that no matter the increase in computer capacity, the development of so-called “ deep learning ” algorithms, among others, computers would continue to be context-blind entities and it is hardly possible to call “ intelligent ” an entity that is systematically unable to understand context 24. Just as a loom is programmed to weave a specific pattern regardless of the thread, the computational processes are insensitive to the properties of their inputs 25. Hence, if we consider computers not particularly “ intelligent ” , but incredibly fast and efficient mechanical engines, then we could reiterate the question posed in the last section: how could they discriminate? Part of the answer definitely relies on this very aspect, since the high speed of processing allows for further detrimental and discriminatory effects. However, brute force and speed are not relevant without considering that algorithms are inscrutable and value-laden. It is the combination of efficient computers, that have their rules dictated by humans and that are hardly interpretable that ultimately causes discriminatory effects. We shall address the two latter points in the next section. 22 Daniel Dennet, ‘Turing’s “Strange Inversion of Reasoning” in S. Barry Cooper, Jan van Leewuwen (eds), Alan Turing: His Work and Impact (Elsevier, 2009), 571 23 Daniel Dennett, ‘Darwin’s “Strange Inversion of Reasoning” 106(1) Proceedings of the National Academy of Sciences of the United States of America <https://www.pnas.org/content/106/Supplement_1/10061> 10062. 24 William Rapapo rt, ‘On the Relation of Computing to the World’ in Thomas Powers (ed.), Philosophy and Computing – Essays in Epistemology, Philosophy of Mind, Logic, and Ethics (Springer 2017) 48 25 Example found in Gualtiero Piccini, ‘Computers’ [2008] 89 Pacific Philosop hical Quarterly 32, 39 14 Algorithms are written by humans and reflect their values; they are “ inescapably value-laden ” and reflect the subjectivity of their programmers 26. Algorithms implicitly or explicitly present “ essential value-judgments ” that compromise the status of their operations as “ value-free ” 27 . Not taking any particular ethical stance may be an ethical stance per se or may lead to detrimental results, such as Amazon ’ s sexist recruitment software that was eventually scrapped because women were penalized, due to the fact that Amazon historically hired more men than women for IT positions 28. This applies even to more technical discussions, such as mapping diseases within a cell, in which the definition of “disease” is discussed and where researchers usually have to input their own preferences on whether more false-positives or false-negatives are desirable within the system 29. In this regard, H IJMANS and R AAB propose that both deliberate targeting and the failure to anticipate differentials may be questioned on legal and ethical grounds 30. The inner working of an algorithm, i. e. , its source code, is illegible for nonexperts; in reality, even experts usually need further assistance, such as having access to the computer program, since inspecting a source code can is not completely helpful to determine how the algorithm will behave. This situation is slightly worse when dealing with machine-learning algorithms, in which the decision rule may itself emerge automatically from the data under analysis and that humans may not be able to explain. The important distinction here is that while programmers may show the text (the source code), the data-driven decisions instilled in the algorithm are the most relevant tool to assess potential outputs generated by the program and these are not always readily available 31. 26 Brent Mittelstadt, Patrick Allo, Mariarosaria Toledo, Sandra Wachter, Luciano Floridi, ‘The Ethics of Algorithms: Mapping the Debate’ (2016) Big Data & Society < https://journals.sagepub.com/doi/full/10.1177/2053951716679679> accessed on 13 May 2020, 1 27 Felicitas Kraemer, Kees van Overveld, Martin Peterson, ‘Is There an Ethics of Algorithms?’ [2011]13 Ethics and Information Technology 251, 251-252 28 ‘Amazon Scraped “Sexist AI” Tool’ BBC (10 October 2018) <https://www.bbc.com/news/technology-45809919> accessed on 19 April 2020 29 Felicitas Kraemer, Kees van Overveld, Martin Peterson, ‘Is There an Ethics of Algorithms?’ [2011]13 Ethics and Information Technology 251, 255 30 Hielke Hiejmans, Charles Raab, ‘Ethical Dimensions of the GDPR’ [2018] in Mark Cole, Franziska Boehm (eds), Commentary on the General Data Protection Regulation (Edward Elgar Publishing 2018, forthcoming) <https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3222677> accessed on 18 April 2020, 6 31 Joshua Kroll, Solon B arocas, Edward Felten, Joel Reidenberg, David Robinson, Harlan Yu, ‘Accountable Algorithms’ [2017] 165 University of Pennsylvania Law Review 633, 638 15 To illustrate this difficulty of interpretability, we may refer to K ITCHIN , who proposes six different approaches to gain access to the insights of the programmers who built the algorithms. These methods are: examining the source code 32 (i); reflexively producing code (ii); reverse engineering (iii); interviewing the coding team (iv); investigating the socio-technical context of the elaboration of the algorithm (v); and investigating how people engage with the algorithm through ethnographic research (vi) 33. Moreover, algorithms can be “ easily, instantly, radically, and invisibly changed ” 34 This characteristic of opacity and inscrutability, combined with the increased capacity of data processing, naturally lead to demands of more accountability. The underlying ethical concern is quite clear: society strives to have meaningful control over machines’ decisions, in particular artificial intelligence. Humans are legally and morally responsible for their actions and the effects they cause are of an increasing moral gravity 35. Control comes along with responsibility, to the extent that one should not place products or applications in the open market without proper (and human) oversight. Calls for more accountability of algorithms have been voiced for over twenty years, but the suggestions on how to implement accountability mechanisms vary 36. H ILL argues that algorithms shoul d surpass the mere instruction “ Do P ” to include the form “ To G, Do P ” , under the premise that the user should be aware of certain aspects underlying the mathematical processing made by the algorithm. This would imply that the algorithm is interpretable, i. e. , an external observer could reasonably draw meaningful conclusions other than the mere observance of the inputs and outputs 32 The author points out three ways to perform such examination: deconstructing the source code and separating the rule set to determine how the algorithm translates inputs, analysing the genealogy and evolution across different versions of code, and testing the translation of the task of the algorithm across different platforms. 33 Rob Kitchin, ‘Thinking Critically about and Researching Algorithms’ [2017] 20(1) Information, Communication & Society 14, 22-26 34 Tarleton Gillespie, ‘The Relevance of Algorithms’ in Tarleton Gillespie, Pablo Bockzowski, Kirsten Foot (eds), Media Technologies: Essays on Communication, Materiality, and Society (MIT Press 2014) 178 35 Shannon Vallor, George Bekey, ‘Artificial Intelligence and the Ethics of Self - Learning Robots’ in Patrick Lin, Ryan Jenkins, Keith Abney, Robot Ethics 2.0 from Autonomous Cars to Artificial Intelligence (Oxford University Press 2017) 344 36 Robert Sloan, Richard Warner, ‘When Is an Algorithm Transparent? Predictive Analytics, Privacy, and Public Policy’ [2018] 16(3) IEEE Security & Privacy 18, 18-19 16 would allow 37. S LOAN and W ARNER have defended that the solutions relies on creating informational coordination norms for predictive systems in order to ensure transparency to the user 38. K ROLL ET A L indicate that reinforcing certain types of blindness in the algorithmic decision-making process may be more productive, such as design an algorithm that is “ race-blind ” , “ gender-blind ” or “ income-blind ” 39 S CHWARTZ advocates for the structuring of transparent data processing systems, providing procedural and substantive rights to data subjects and establishing independent governmental institutions tasked with monitoring data processing systems 40. More recently, W ACHTER and M ITTELSTADT propose a direct normative solution and developed in further detail a possible “ right to reasonable inferences ” appl icable to “ high risk inferences ” , implying an ex ante justification given by the data controller establishing whether an inference is reasonable 41. Our analysis of improving accountability of algorithms is detailed in the last chapter of this work ( infra , Chapter V). For the moment, our intention is to highlight the most essential concerns regarding algorithms. In the next section, we briefly address a particularly important topic related to these concerns: the practices of profiling, clustering and labeling. 3. Algorithmic profiling According to the famous construction of W ARREN and B RANDEIS , the right to privacy has been understood “ as part of a more general right to the immunity of the pe rson” , meaning that it is “the right to one ’ s personality ” , or the right “ to be let alone ” 42 . Profiling is a particularly concerning algorithmic process, since it not only constitutes an important source of privacy-invasive, discriminatory practices, but it also refers expressly to the identity of the individual. In this sense, 37 Robin Hill, ‘What an Algorithm Is’ [2015] 29(1) P hilosophy and Technology 35, 36 38 Robert Sloan, Richard Warner, ‘When Is an Algorithm Transparent? Predictive Analytics, Privacy, and Public Policy’ [2018] 16(3) IEEE Security & Privacy 18, 22 39 Joshua Kroll, Solon Barocas, Edward Felten, Joel Reidenberg, David Robi nson, Harlan Yu, ‘Accountable Algorithms’ [2017] 165 University of Pennsylvania Law Review 633, 685 40 Paul Schwartz, ‘Data Processing and Government Administration: The Failure of the American Legal Response to the Computer’ [1992] 43 Hasting Law Journal 1 321, 1323 41 Sandra Wachter, Brent Mittelstadt, ‘A Right to Reasonable Inferences: Re -Thinking Data Protection Law in the Age of Big Data and AI’ [2018] 2019(2) Columbia Business Law Review 1, 2 42 Samuel Warren, Louis Brandeis, ‘The Right to Privacy’ [1890] 4(5) Harvard Law Review 193, 207. The latter construction (“a right to be let alone”) is attributed by W ARREN and B RANDEIS to Judge C OOLEY 17 identity is not merely a philosophical topic, but “ a device used to decide who is in and who is out ” 43 The profiling processes may be understood as comprising three different stages: collection and storage of digitized obs ervations regarding individuals’ behavior or characteristics (data warehousing); analysis of the correlations between different behavior and characteristics (data mining); and production of inferences (inference stage) based on certain observable behavioral variables or characteristics specific to a generally identified individual 44. Profiling examples abound, ranging from the determination of gender, sexual orientation (and other prot