The collections of ancestral remains of indigenous peoples from southern Africa which are kept in museums and research institutes or university centers seem to be the prime example of “problematic” anthropological collections, for which “undoing empire” [Rassool, 2015b] implies restitution to contemporary descendants. However, this restitution does not involve only a “simple” reburial of the remains, but also a dialogue with the descendants around the status of the raw ethnographic material gathered by anthropologists who studied the Khoekhoe and San peoples—field notebooks, catalog records, photographs, plaster casts, sound recordings—as well as around the research conducted currently (or in the future) on these collections. Focusing on several South African case studies, I show how consultations around collections and the knowledge derived from the analysis of these collections bring into opposition multiple local, national, and transnational players and rationales, which go well beyond the simplistic dichotomy “academics” versus “descendant communities”. The various players and rationales are often contradictory, but they need to be taken into account when creating shared digital infrastructures and databases in order to be able to question (rather than reinforce) “the colonial archive and its modes of evidence” [Lalu, 2009].