Criterion Evaluations

A Criterion Evaluation that represents a determination of a candidate on whether they meet a certain criterion against a tag, job, etc.

🛠️ This page is a work in progress. Please refer to the in-depth guides and API reference in the meantime.

Relationships:

• Nomination to Criterion Evaluation is 1-Many

• Candidate to Criterion Evaluation is 1-Many

Criterion Evaluation Result Type:

Successful criterion evaluations will be returned in this format.

{
  criterion: {
    id: UUID,
    prompt_text: string
    display_name: string | null,
    scope: {
      scope_type: 'organization' | 'job' | 'tag',
      tag: {
        tag_type: string,
        id: uuid,
        is_standardized: boolean,
        name: string,
        description: string,
        created_at: datetime string
      } | null,
      job_id: uuid | undefined
    }
  },
  evaluation: {
    state: "EVALUATED" | "FAILED" | "IN_PROGRESS",
    result: {
      is_criterion_met: boolean,
      confidence_score: number (0-1),
      explanation: string | null
      evaluated_at: datetime string
    } | null,
    error: {
      type: "EVALUATION_GENERATION_FAILED"
    } | null
  }
}

Coming soon: Evaluation accuracy assessments

To help build trust and transparency with users, Endorsed will soon begin rolling out the Assessments feature, in which the system will automatically evaluate its evaluations across several accuracy axes, and provide that result in the /criterion_evaluations endpoints to Platform customers.

Please reach out to us if you would like early access to this feature.

Assessments will look like this, as a nested object within the evaluation result:

"assessments": {
  "errors": [],
  "assessments": [
    {
      "isGap": false,
      "assessmentKey": "false_positive",
      "confidenceScore": 0.9,
      "assessmentVersion": 1
    },
    {
      "isGap": false,
      "explanation": "The claim that the candidate meets the criterion of having great communication skills is supported by the source material, which explicitly lists 'Good communication skills, great with quick responses to questions' under qualifications.",
      "assessmentKey": "unfaithful",
      "confidenceScore": 1,
      "assessmentVersion": 1
    }
  ]
}

Get the criterion evaluations for a nomination

GET https://endorsed.ai/api/v1/nominations/{id}/criterion_evaluations/

Gets all the evaluations for each criterion for a nomination based upon the nomination's job's criteria

Path Parameters

Name	Type	Description
id*	string	UUID id of the nomination

Headers

Name	Type	Description
Authorization*	string	Authorization header containing the Bearer Access Token from the /auth/tokens endpoint. Example: Bearer <ACCESS_TOKEN>

200 Successfully fetched

{
  "is_error": false,
  "results": [
    {
      "criterion": {
        "id": "5f00531e-43c9-4bcc-a904-e5c3c2ac0481",
        "prompt_text": "Has 5 years of experience in Javascript development",
        "display_name": "Javascript development",
        "scope": {
          "scope_type": "job",
          "tag": null,  // Optional | tag may not exist if the criterion is not a tag based criterion
          "job_id": "af837f55-5d2d-4746-b26b-8ddb4c19aa5f" // Optional | job_id / job may not exist if the criterion is not a job-scoped criterion
        }
      },
      "evaluation": {
        "state": "EVALUATED",
        "result": {
          "is_criterion_met": true,
          "confidence_score": 0.9,
          "explanation": "The candidate has worked two unique software engineering jobs developing Javascript web applications totaling 7 years",
          "evaluated_at": "2023-09-25T22:06:01.000Z"
        },
        "error": null
      }
    },
    {
      "criterion": {
        "id": "69632995-93e2-4a78-aaed-b5779d879b54",
        "prompt_text": "Has previously worked in the food service industry",
        "display_name": null,
        "scope": {
          "scope_type": "job",
          "tag": null,
          "job_id": "af837f55-5d2d-4746-b26b-8ddb4c19aa5f" // Optional | job_id / job may not exist if the criterion is not a job-scoped criterion
        }
      },
      "evaluation": {
        "state": "FAILED",
        "result": null,
        "error": {
          "type": "EVALUATION_GENERATION_FAILED"
        }
      }
    },
    {
      "criterion": {
        "id": "176b6e49-0649-4509-be0d-f77a3bf8a7f3",
        "prompt_text": "Has previously worked as an Account Executive",
        "display_name": "Account Executive experience",
        "scope": {
          "scope_type": "job",
          "tag": null,
          "job_id": "af837f55-5d2d-4746-b26b-8ddb4c19aa5f" // Optional | job_id / job may not exist if the criterion is not a job-scoped criterion
        }
      },
      "evaluation": {
        "state": "IN_PROGRESS",
        "result": null,
        "error": null
      }
    }
  ]
}

202: Accepted

Indicates that all of the criterion evaluations for the nomination have not completed yet. Another request should be made to get the resource(s) once the processing is complete.

🛠️ This API response is not yet stable and may be subject to change. Dependent code or logic on the status code is okay, but please avoid writing any dependent code or logic based upon these fields as we iterate on its structure.

{
  "is_error": false,
  "results": null,
  "message": "Your request has been accepted and is still being processed. Please execute your request again",
  "estimatedCompletionTime": "2024-02-05T23:31:00.000Z"
}

401 Unauthorized

{
  "is_error": true,
  "error": {
    "code": "UNAUTHENTICATED",
    "issues": [
      {
        "message": "Resource requires authentication. Please provide a valid access token."
      }
    ]
  }
}

403: Forbidden Permission denied

{
  "is_error": true,
  "error": {
    "code": "FORBIDDEN",
    "issues": [
      {
        "message": "Access to this resource is not authorized."
      }
    ]
  }
}

404: Not Found

{
  "is_error": true,
  "error": {
    "code": "NOT_FOUND",
    "issues": [
      {
        "message": "This resource entity does not exist."
      }
    ]
  }
}

500: Internal Server Error

{
  "is_error": true,
  "error": {
    "code": "INTERNAL_SERVER_ERROR",
    "issues": [
      {
        "message": "An unexpected error occurred on our end. Please try again later, and if the problem persists, contact our support team"
      }
    ]
  }
}

Examples:

cURL

curl -X GET "${BASE_URL}/api/v1/nominations/6427f970-70c0-4b42-8fae-5609cefbb1bc/criterion_evaluations" \
-H "Authorization: Bearer ${ACCESS_TOKEN}"

Get the criterion evaluations for a candidate

GET https://endorsed.ai/api/v1/candidates/{id}/criterion_evaluations/

Gets all the evaluations for each criterion for a candidate based upon the organization's criteria

Path Parameters

Name	Type	Description
id*	string	UUID id of the candidate

Query Parameters

Name	Type	Description
criterion_type*	string	Type of criterion to filter criterion evaluation results upon. Currently only supports tag and org_wide. job specific criterion evaluations should be fetched from `/nominations/[id]/criterion_evaluations.

Headers

Name	Type	Description
Authorization*	string	Authorization header containing the Bearer Access Token from the /auth/tokens endpoint. Example: Bearer <ACCESS_TOKEN>

200 Successfully fetched

{
  "is_error": false,
  "results": [
    {
      "criterion": {
        "id": "5d2374b9-f637-46aa-a131-f5a4aa409414",
        "prompt_text": "Has qualifying experience as a Videographer",
        "display_name": null,
        "scope": {
          "scope_type": "tag",
          "tag": {
            "tag_type": "experience_based_role_title",
            "id": "2fd70a5a-1e03-412f-b990-f4248e18af07",
            "is_standardized": true,
            "name": "Videographer",
            "description": "Adept in filming and operating camera equipment",
            "created_at": "2023-09-25T22:06:01.000Z"
          }
        }
      },
      "evaluation": {
        "state": "EVALUATED",
        "result": {
          "is_criterion_met": true,
          "confidence_score": 0.9,
          "explanation": "The candidate's role in producing video content for recognized brands and his involvement in video content creation for over 10 years demonstrate a strong fit for the Videographer tag.",
          "evaluated_at": "2023-09-25T22:06:01.000Z"
        },
        "error": null
      }
    },
    {
      "criterion": {
        "id": "2fc0f456-6c20-405e-b8a2-eec5059c5e6c",
        "prompt_text": "Has previously worked as a Voiceover Artist",
        "display_name": null,
        "scope": {
          "scope_type": "tag",
          "tag": {
            "tag_type": "experience_based_role_title",
            "id": "48ed311b-6fa2-4d71-85a7-91fbe7e8e946",
            "is_standardized": true,
            "name": "Videographer",
            "description": "Adept in filming and operating camera equipment",
            "created_at": "2023-09-25T22:06:01.000Z"
          }
        }
      },
      "evaluation": {
        "state": "FAILED",
        "result": null,
        "error": {
          "type": "EVALUATION_GENERATION_FAILED"
        }
      }
    },
    {
      "criterion": {
        "id": "3476fb35-9725-4d5c-b1d9-81d64eee69d9",
        "prompt_text": "Has previously worked as a Voiceover Artist",
        "display_name": null,
        "scope": {
          "scope_type": "tag",
          "tag": {
            "tag_type": "experience_based_role_title",
            "id": "c3d44241-ab36-42c8-b8ff-7bac8fbdc86f",
            "is_standardized": true,
            "name": "Videographer",
            "description": "Adept in filming and operating camera equipment",
            "created_at": "2023-09-25T22:06:01.000Z"
          }
        }
      },
      "evaluation": {
        "state": "IN_PROGRESS",
        "result": null,
        "error": null
      }
    }
  ]
}

202: Accepted

Indicates that all of the criterion evaluations of the criterion_type have not completed yet. Another request should be made to get the resource(s) once the processing is complete.

🛠️ This API response is not yet stable and may be subject to change. Dependent code or logic on the status code is okay, but please avoid writing any dependent code or logic based upon these fields as we iterate on its structure.

{
  "is_error": false,
  "results": null,
  "message": "Your request has been accepted and is still being processed. Please execute your request again",
  "estimatedCompletionTime": "2024-02-05T23:31:00.000Z"
}

401 Unauthorized

{
  "is_error": true,
  "error": {
    "code": "UNAUTHENTICATED",
    "issues": [
      {
        "message": "Resource requires authentication. Please provide a valid access token."
      }
    ]
  }
}

403: Forbidden Permission denied

{
  "is_error": true,
  "error": {
    "code": "FORBIDDEN",
    "issues": [
      {
        "message": "Access to this resource is not authorized."
      }
    ]
  }
}

404: Not Found

{
  "is_error": true,
  "error": {
    "code": "NOT_FOUND",
    "issues": [
      {
        "message": "This resource entity does not exist."
      }
    ]
  }
}

500: Internal Server Error

Overarching Evaluation Type Error

{
  "is_error": true,
  "error": {
    "code": "INTERNAL_SERVER_ERROR",
    "issues": [
      {
        "message": "The criterion evaluations for evaluation_type [tag] had an overar ching failure. Please try again later, and if the problem persists contact our support team"
      }
    ]
  }
}

Other Unexpected Error

{
  "is_error": true,
  "error": {
    "code": "INTERNAL_SERVER_ERROR",
    "issues": [
      {
        "message": "An unexpected error occurred on our end. Please try again later, and if the problem persists, contact our support team"
      }
    ]
  }
}

Examples:

cURL

curl -X GET "${BASE_URL}/api/v1/candidates/b8c76214-d518-4d69-9aab-77b0166696d2/criterion_evaluations?criterion_type=tag" \
-H "Authorization: Bearer ${ACCESS_TOKEN}"