OpenAI has come up with a new content moderation tool

To ensure the safe and productive use of their API and AI models, OpenAI has developed a new Moderation API that interprets user input and flags whether it is considered harmful or not. This new Moderation API is available to all developers using the OpenAI API, and while it's currently intended for use with other OpenAI API endpoints, it is also available for use with non-API traffic in beta with some restrictions. This release is a little more involved than just a simple policy update; Since this is a brand-new endpoint in the API intended for direct use by developers, there's a guide and documentation to go along with its use, as well as a paper published on the creation of the new model which drives it and the dataset used to evaluate it. The new API is free to use for developers working within the OpenAI API.

Behind the scenes, the Moderation endpoint has access to a GPT-based model which was trained to assess input text for potentially harmful content. When used with other parts of the OpenAI API, the user-provided input text is first passed through a Moderation API call to block misuse of OpenAI's models. In many applications of the OpenAI API, the use of the Moderation endpoint is required. Use of the Moderation endpoint in applications within the OpenAI API is at no cost to the developer, and use for external applications such as content moderation on another platform is in beta and does come with fees.

The Moderation API is useful in identifying problematic language created by both humans and AI. It can be used for blocking harmful inputs provided by malicious human interactors and at the same time block any harmful text content that a model might generate on its own.

A paper describing the creation of the GPT-based model was released, as well as the dataset which was used to evaluate it, if you're interested in the fine details of the model driving the Moderation API.

More Trending Stories