1. Define Evaluation Criteria

• Why: Establishing clear criteria ensures your evals are aligned with the goals and are actionable.
• How:
  • Draft initial criteria based on inputs, outputs, and protocols.
  • Use the model to generate suggestions for evaluation criteria. For example:
    • "Here’s my task and goals. What are some measurable criteria I should consider for evaluating success?"
  • Review and refine these suggestions as a human in the loop.

2. Leverage the Model to Generate High-Quality Inputs and Labels

• Generate Inputs:
  • Feed the model a few representative examples and a task definition.
  • Ask it to produce variations or new examples based on these.
• Draft Labels:
  • Task the model with labeling the generated data.
  • Example: "Here’s an input and the corresponding task. Draft possible output labels."
• Review Only: Humans should primarily focus on reviewing and refining these, rather than creating them from scratch. This reduces cognitive and time overhead.

3. Optimize Performance via Prompt Iteration

• Use the Model as a Grader:
  • Test your outputs against criteria and have the model grade the results.
• Iterative Prompting:
  • Identify failure modes by analyzing cases where the model performs poorly.
  • Use meta-prompting: Provide the model with failure modes and ask it to suggest prompt improvements.
  • Example meta-prompt: "Given this failure case and prompt, what changes could improve performance?"

4. Reviewing Outputs is Faster Than Drafting

• Reviewing AI-generated outputs (inputs, labels, or criteria) is much quicker than creating them manually.
• Focus human time on high-value tasks like confirming accuracy, refining definitions, and addressing edge cases.

5. Experiment with Advanced Flows

• Meta-Prompting: Iteratively refine the prompts using feedback loops based on observed model behavior.
• Failure Analysis: Document failure cases systematically to identify patterns.
• Use O1 Preview (or similarly powerful models) for complex grading and iterative testing to enhance workflows.

Benefits of These Practices:

• Efficiency: Reduces manual work.
• Quality: Improves evaluation depth and consistency.