Sometimes, we tell people not to buy Cinder. It’s usually start-up teams with small, straightforward Trust & Safety problems. Maybe they can get by with an off-the-shelf classifier and can manage the human elements at low scale by cobbling together a customer service ticketing system with a messaging tool. For some companies, this approach is good enough. It will leave all sorts of gaps and will not scale, but it can be the right option at a particular moment in time.

The problem, of course, is that Trust & Safety enterprises often limp along with this type of inefficient infrastructure far too long. In the past, Trust & Safety teams knew they would have to cobble together internal tooling themselves, usually with a mix of internal builds and commoditized software built for some other purpose. Now, however, Trust & Safety teams have a robust set of vendor options and so face a question long-pondered by leaders in other areas of the business: whether to build or to buy.

Building & Buying

Trust & Safety is deeply complex. The policy choices are vexing; operations across the entire Decision Spectrum require specialized processes and precise execution on short timelines; and decision makers often must make choices with imperfect information, both because of the short timeframes and because of inadequate tools.

But turning to vendor solutions is also disconcerting for many companies: Trust & Safety issues are highly sensitive and the vendor space is relatively new, which means that vendors must establish bona fides. Moreover, Trust & Safety teams often do not have experience buying enterprise software, which can increase reticence and slow passage through corporate purchasing processes. Compare these purchases to a CISO’s budget, with expected line items for VirusTotal, or a SIEM, as well as for FTE headcount and contractors or consultants.  

At some basic level, teams considering buying enterprise software of any kind are asking whether it will be worth it. Do the benefits outweigh the costs? In the Trust & Safety context, this requires considering a company’s overall exposure to Trust & Safety risks and comparing the total costs and benefits of using an internally-developed solution to the costs and benefits of buying tools.

This process alone can be daunting for Trust & Safety teams that haven’t had the opportunity to evaluate more than one choice (build internally), until recently. Our blog on measuring Trust & Safety may be a starting point for teams thinking through what matters and how they may, or may not, be able to make progress against business goals.

At Cinder, we can draw some lessons about how companies approach these questions - and central to that is thinking about three concepts.

• What are the pros and cons of the current tools?
• What are the switching costs associated with improving the current tools?
• And what are the relative costs and benefits of building versus buying in the future?

Benefits: What's the Purpose of Trust & Safety Tools?

Trust & Safety tools have many benefits, but at the core they aim to provide efficiency, accuracy, completeness, and confidence.

Efficiency

We like to think about two categories of efficiency; operational and innovative. Operational efficiency is the measure of how quickly decision makers can execute a decision. In traditional scaled content moderation, this is usually measured by Active Handle Time (AHT), throughput, and cost. Many companies struggle with operational efficiency because their operations teams require multiple tools and, literally, dozens of clicks to make a single decision. Innovative efficiency reflects how quickly tools can be updated to support a changing policy environment, respond to crises, or enable Trust & Safety support for new products. Efficiency also promises real cost savings in terms of the number of reviewers necessary to review the same amount of material and in the ability to push reviews to outsourced specialists.

Accuracy

Accuracy reflects the ability of a Trust & Safety team to enact its policies as intended. Tooling impacts this in multiple ways, for example by displaying relevant information, routing jobs to appropriate reviewers, and ensuring that communications remain associated with decision tasks.

Completeness

We think about completeness in multiple ways. On the one-hand, this refers to detection and the ability to surface potential violations. But this also means giving investigators the ability to access all appropriate data and ensuring that every action that should result from a decision actually gets made, including appropriate communications, recording data, taking steps to limit an account or piece of content, etc.

Confidence

We divide confidence into security, operational, and strategic sub-categories. Security confidence speaks to the knowledge that information is safe, which includes storage guidelines, whether it is sent to APIs hosted in the cloud, and in-app permission systems. Operational confidence is the ability of managers to allocate resources, use metrics to assess whether teams are executing appropriately, and determine whether their capabilities match needs. Strategic confidence is reflected in high-level metrics but also in the basic ability to trust that data returned by tools accurately reflects reality - and can therefore safely inform regulatory filings and statements to the media

Of course, these elements often interact. For example, tooling that provides complete information to moderators is likely to produce more accurate decisions with improved efficiency. Integrated tools are likely to be more efficient and enable operational confidence because metrics can be tracked more simply.

Costs: Measuring Soft and Hard Metrics

The costs associated with building and buying Trust & Safety tools vary significantly - and are sometimes hidden. It’s useful to think about them in four buckets: software vendor costs, engineering costs, operational costs, and risk. Importantly, these categories usually contain some real cost regardless of whether a company chooses to build or buy Trust & Safety tools, though obviously the amount and type of cost sub-category will change.

Software Vendor Costs

There are valuable freely-available Trust & Safety tools, but they are limited in scope. Buying a Trust & Safety platform will cost money. But many home-brewed Trust & Safety systems also depend on vendor-based tools as well. Teams pay for ticketing systems like Zendesk and Salesforce, training tools like Guru, QA tools like Maestro, and dashboard tools like Retool or Splunk.

Engineering Costs

Building in-house tools or connecting various off-the-shelf tools requires engineering investments. Data pipelines must be connected and updated, tools configured, and APIs maintained. A great challenge for operational Trust & Safety teams is that engineering teams rarely have the capacity to invest in these efforts, leaving tools partially built or unreliable. At the same time, the more an engineering team builds, the more maintenance it requires, which manifests as an increased engineering burden as a Trust & Safety program becomes more complex. Buying Trust & Safety tools should significantly reduce the engineering load on your teams, but it will not eliminate it. Engineers will still need to facilitate data pipelines and deploy the tool if it is hosted locally. But well-built vendor tools should allow internal engineers to mostly focus on other problems.

Operational Costs

Inefficient or ineffective tools produce real costs associated with Trust & Safety operations. As companies scale, the direct dollar cost of these inefficiencies can surpass many millions of dollars annually. Many companies pay very large fees to outsource reviewers, in large measure because of the inefficiency of their review processes. Other companies are forced to keep significant portions of Trust & Safety operations work in-house, and at a high cost, to mitigate the data privacy risks and operational complexities embedded in their tools. Every tool has operational costs: it demands maintenance and people to operate it. Efficient tools save costs not just on tooling but on those related line items in the budget.

Risk

Trust & Safety risks manifest in various ways: the general risk of harmful material on your platform, the acute risk of real-world harm being facilitated on your platform, and the business risk to your platform of substandard Trust & Safety practices. All of these risks constitute a cost, even if they are hard to measure and represent in dollar figures. But it is important to note that settling for ineffective Trust & Safety infrastructure carries costs, though estimating honestly the costs associated with rectifying those inadequacies by building internally or buying tools will often manifest with other assessments of cost.

Importantly, these costs persist through time. We find companies concerned about their present-day costs, thinking about switching costs (whether that means internal upgrades or purchasing software), and trying to estimate the future cost of maintaining whatever system they choose to build or buy. Companies may anticipate increased engineering and operational costs during the switching period but often find that buying tools will significantly reduce expensive engineering and operational costs over time.

Facilitating Innovation

Successful digital platforms innovate. Trust & Safety professionals often find that they are the last to hear when a new product is ready for release and then must struggle to build systems to manage that new product. The market increasingly demands that every product comes with Trust & safety protections, but many legacy Trust & Safety tools were not designed for such flexibility. We had long assumed customers would be interested in Operational Efficiency as a mechanism to reduce costs while maintaining high Trust & Safety standards. Since launching Cinder, we have been surprised by how often companies purchase Cinder because they prioritize Innovative Efficiency - because it powers the core business goal of safely enabling experimentation and the development of new products.

Buying Trust & Safety Tools

We are Trust & Safety professionals. None of us went to business school. But we have learned a lot about how various companies evaluate the build vs. buy decision since founding Cinder nearly two years ago. Trust & Safety teams are often unaccustomed to buying software. This is changing now that options like Cinder exist, but we are still relatively early in the process of Trust & Safety vendors being totally normalized across the Trust & Safety ecosystem.

This often manifests as an inability to compare the full-cost of building and maintaining an internally built system against the cost of purchasing software. That is particularly true because the personnel and opportunity costs associated with spending engineering time to build are managed and controlled by different people with separate accounting buckets than those that manage spend on vendor software. Still, companies compare these costs in other arenas, including cybersecurity and customer support. As companies navigate this shift for Trust & Safety, they should consider the following to determine whether buying software may be a better option than building it:

• Consider buying: This may seem obvious, but many Trust & Safety teams still default to build. If an operational or engineering challenge arises, do not just assume that the only solution is to build. Ask whether a vendor might be a useful option.
• Identify target business outcomes: Teams often focus narrowly on tactical problems, like enabling a particular type of escalation or tracking a specific regulatory demand. It is worth stepping back to make sure your investments align with business outcomes. Why is escalation important? Are there better ways to produce the necessary outcomes?
• Identify must-have requirements: Make sure you identify critical requirements and that you are really clear on what is mandatory and what is not. The best vendors will be able to help you think through this consultatively - and even if you ultimately build a solution in-house that process will help ensure you invest strategically.
• Speak to more than one vendor: Talking to vendors will help you imagine different solutions and is valuable for benchmarking the pros and cons of various companies and the building internally. Focus on how well the vendors understand the problem you are trying to solve. Companies face similar Trust & Safety challenges; strong vendors will bring experience to bear from other companies that may illuminate your challenges.
• Bring in multiple stakeholders: Trust & Safety is a team sport. Engage your colleagues from operations, engineering, policy, product, legal, and potentially even finance or business operations as you consider whether to build or buy. An effective Trust & Safety investment should bring ancillary benefits across teams - and those benefits may be critical to understanding the ultimate pros and cons to your company of the build vs. buy decision.
• Align with executives: Digital platforms, even those that famously default toward building, manage innumerable external vendors. Executives may not be used to Trust & Safety vendors, but they are likely very familiar with vendors for other critical business tasks, like HR, customer support, or internal communications. They will likely have preferences for how a build vs. buy decision should be made, including with key stakeholders like information security and legal. Use your leadership as a resource to plan this process accordingly.
• Compare apples to apples: If you identify an appealing vendor, compare it directly against building internally - but make sure you assess all facets of that decision. Which will deliver more complete outcomes faster? Which will require more engineering support? Which will be more sustainable as your operation scales? Which will cost more today and in two years? Which will save more time and money today and in two years? Which will do the best job mitigating data privacy, reputational, and other risks? Which will facilitate the dynamic innovation your company likely seeks?
• Prepare to onboard: If you do choose to buy work with the vendor to create a smooth onboarding process. Identify the key stakeholders, assign responsibility, and set clear expectations with the vendor about deliverables and timelines. A strong vendor will be able to help with this process.
• Measure outcomes: Establish your core business KPIs and measure them prior to transitioning to a new solution, whether you build it or buy it. Assess whether that solution has produced the shifts you wanted to achieve. If you aim to build internally, do not sidestep this process because you will need to know whether that approach works. If you go with a vendor, hold them accountable for helping you measure these shifts.
  ‍

It comes as no surprise that a founder of a Trust & Safety vendor sees real opportunity in the Trust & Safety vendor ecosystem. We think Cinder and others in the space will continue to improve offerings and make buying tools and capabilities ever more viable. Importantly, though, the responsibility and decision making authority for Trust & Safety cannot and should not be outsourced. Companies can and should purchase tools and information, but they bear ultimate responsibility for scoping Trust & Safety policies and enforcing them. The reason to purchase tools is not to avoid that responsibility, it is because those tools allow companies to live up to those obligations more effectively.

‍