Trust & Safety teams are making difficult decisions every day. The content, data, and context involved in these choices varies widely, but so do the nature and structure of the decisions themselves.
While this profoundly impacts the way Trust & Safety teams operate, it rarely gets the attention it deserves. That matters because the range of Trust & Safety decisions require different tools, workflows, and processes, and failure to systematically analyze the entire spectrum of these choices limits preparation for effective decisions in specific verticals, let alone drive efficiencies across all of them.
At Cinder, we think about Trust & Safety decisions on a spectrum defined by complexity. It drives the work we do internally, along with our partners and customers, and we wanted to outline it here so other organizations can take advantage of this framework to increase operational efficiencies and overall make faster and more accurate decisions where, especially in Trust & Safety, every decision counts.
Our decision spectrum is a spectrum of complexity, which itself is a function of the content, context, impact, and process required for those decisions. This is most apparent in large Trust & Safety enterprises, where disparate teams specialize in decisions of varying complexity. But the conceptual framework applies even if there is just one person in the entire Trust & Safety organization.
.png)
On one side of the spectrum are the more straightforward decisions. Although these choices may be vexing and difficult (managing CSAM, identifying satire, understanding the context of a slur, etc), the basic decision process is relatively simple. Basic data labeling falls into this category. For example, you might have an image or single block of text to be assessed without additional context and to be labeled with a single policy. Such reviews often require little context and are very common. They are often conducted by reviewers individually or via automated systems.
A more complex entity requires a more sophisticated decision process. For example, you might review a user account, which has a variety of associated attributes and is directly connected to other entities - like images, purchases, posts, etc. It might also be important to understand the history of the account and its relationship with other accounts. The additional context requires more data and perhaps more training to make effective decisions.
An even more complex decision requires multiple, complex entities, such as a dispute among multiple parties. This could include review of existing communication, understanding context around each user, and even real-time communication to gather additional information. These decisions may be resolved by individual reviewers, but may require a longer time period and repeated visits to the decision. Such decisions are far more likely to require escalation and complex triage than the most simple reviews.
The most complex decisions are essentially unstructured. These are open-ended investigations that may require months to adjudicate, generally involve adversarial networks of accounts, and demand deep subject matter experts with extensive knowledge and training. The decision process is often collaborative, involves multiple business units, and often can implicate political, business, and legal judgments generally unheard of with individual data labeling and content moderation choices. It is tempting to refer to such decisions as unusual, and they are certainly less common than less complex decisions, but platforms of all sizes face these decisions constantly.
Identifying the type of decision to be made does not tell you what the substance of that decision should be, but it does provide guidance on which processes, people and tools are optimal. On one end of the spectrum are decisions essentially made in a factory that is highly optimized for scale and repeatability; on the other end are decisions that more closely resemble legal investigations capped by judicial outcomes freighted with geopolitical and business consequences.
These decisions may require different organizational structures as well, but those entities should not be built in a vacuum nor siloed from each other. Trust & Safety leaders must understand the full scope of the decisions that occur in their enterprises, conceptualize the complexity of those decisions in reference to each other, and ensure that the outcome of those decisions is synergistic even when the processes driving them are largely distinct.
At the crux of leveraging the full spectrum of potential decisions is data. Centralized, normalized, and accessible data that can be maintained as a single source of truth. Cinder was built to facilitate decisions across the entire spectrum, maintain effective knowledge management across those organizational and decision structures, and thereby drive those positive interactions. Once your data is in a single place, new workflows can be spun up and decisions can be tackled. While the output - logging, metrics, and transparency data - is normalized and ready to go.
Failing to prepare for and proactively manage the entire decision spectrum will undercut platform efforts to build a future-facing team, prepare for the Digital Services Act, harness the full potential of AI, and build a safe and transparent platform for users. That’s true for both large and small platforms. And while we think that Cinder is a critical tool for platform leaders looking to manage and direct the full spectrum of decisions within their Trust & Safety enterprise, our hope is that the spectrum is a useful framework for every Trust & Safety decision maker.