judges · PyPI

judges is a small library to use and create LLM-as-a-Judge evaluators. The purpose of judges is to have a curated set of LLM evaluators in a low-friction format across a variety of use cases that are backed by research, and can be used off-the-shelf or serve as inspiration for building your own LLM evaluators. The library provides two types of judges: Classifiers: Return boolean values. Graders: Return scores on a numerical or Likert scale. All judges can be used by calling the .judge() method. This method accepts the following parameters: The .judge() method returns a Judgment object with the following attributes: If the underlying prompt for a classifier judge outputs a Judgment similar to True or False (e.g., good or bad, yes or no, 0 or 1), the judges library automatically resolves the outputs so that a Judgment only has a boolean label. The library also provides an interface to combine multiple judges through the Jury object. The Jury object has a .vote() method that produces a Ve

saved by

• Leonardo