Criterion Evaluations

A Criterion Evaluation that represents a determination of a candidate on whether they meet a certain criterion against a tag, job, etc.

🛠️ This page is a work in progress. Please refer to the in-depth guides and API reference in the meantime.

Relationships:

Nomination to Criterion Evaluation is 1-Many
Candidate to Criterion Evaluation is 1-Many

Criterion Evaluation Result Type:

Successful criterion evaluations will be returned in this format.

{
  criterion: {
    id: UUID,
    prompt_text: string
    display_name: string | null,
    scope: {
      scope_type: 'organization' | 'job' | 'tag',
      tag: {
        tag_type: string,
        id: uuid,
        is_standardized: boolean,
        name: string,
        description: string,
        created_at: datetime string
      } | null,
      job_id: uuid | undefined
    }
  },
  evaluation: {
    state: "EVALUATED" | "FAILED" | "IN_PROGRESS",
    result: {
      is_criterion_met: boolean,
      confidence_score: number (0-1),
      explanation: string | null
      evaluated_at: datetime string
    } | null,
    error: {
      type: "EVALUATION_GENERATION_FAILED"
    } | null
  }
}

Coming soon: Evaluation accuracy assessments

To help build trust and transparency with users, Endorsed will soon begin rolling out the Assessments feature, in which the system will automatically evaluate its evaluations across several accuracy axes, and provide that result in the /criterion_evaluations endpoints to Platform customers.

Please reach out to us if you would like early access to this feature.

Assessments will look like this, as a nested object within the evaluation result:

"assessments": {
  "errors": [],
  "assessments": [
    {
      "isGap": false,
      "assessmentKey": "false_positive",
      "confidenceScore": 0.9,
      "assessmentVersion": 1
    },
    {
      "isGap": false,
      "explanation": "The claim that the candidate meets the criterion of having great communication skills is supported by the source material, which explicitly lists 'Good communication skills, great with quick responses to questions' under qualifications.",
      "assessmentKey": "unfaithful",
      "confidenceScore": 1,
      "assessmentVersion": 1
    }
  ]
}

Get the criterion evaluations for a nomination

GET https://endorsed.ai/api/v1/nominations/{id}/criterion_evaluations/

Gets all the evaluations for each criterion for a nomination based upon the nomination's job's criteria

Path Parameters

Name

Type

Description

id*

string

UUID id of the nomination

Headers

Name

Type

Description

Authorization*

string

Authorization header containing the Bearer Access Token from the /auth/tokens endpoint. Example: Bearer <ACCESS_TOKEN>

{
  "is_error": false,
  "results": [
    {
      "criterion": {
        "id": "5f00531e-43c9-4bcc-a904-e5c3c2ac0481",
        "prompt_text": "Has 5 years of experience in Javascript development",
        "display_name": "Javascript development",
        "scope": {
          "scope_type": "job",
          "tag": null,  // Optional | tag may not exist if the criterion is not a tag based criterion
          "job_id": "af837f55-5d2d-4746-b26b-8ddb4c19aa5f" // Optional | job_id / job may not exist if the criterion is not a job-scoped criterion
        }
      },
      "evaluation": {
        "state": "EVALUATED",
        "result": {
          "is_criterion_met": true,
          "confidence_score": 0.9,
          "explanation": "The candidate has worked two unique software engineering jobs developing Javascript web applications totaling 7 years",
          "evaluated_at": "2023-09-25T22:06:01.000Z"
        },
        "error": null
      }
    },
    {
      "criterion": {
        "id": "69632995-93e2-4a78-aaed-b5779d879b54",
        "prompt_text": "Has previously worked in the food service industry",
        "display_name": null,
        "scope": {
          "scope_type": "job",
          "tag": null,
          "job_id": "af837f55-5d2d-4746-b26b-8ddb4c19aa5f" // Optional | job_id / job may not exist if the criterion is not a job-scoped criterion
        }
      },
      "evaluation": {
        "state": "FAILED",
        "result": null,
        "error": {
          "type": "EVALUATION_GENERATION_FAILED"
        }
      }
    },
    {
      "criterion": {
        "id": "176b6e49-0649-4509-be0d-f77a3bf8a7f3",
        "prompt_text": "Has previously worked as an Account Executive",
        "display_name": "Account Executive experience",
        "scope": {
          "scope_type": "job",
          "tag": null,
          "job_id": "af837f55-5d2d-4746-b26b-8ddb4c19aa5f" // Optional | job_id / job may not exist if the criterion is not a job-scoped criterion
        }
      },
      "evaluation": {
        "state": "IN_PROGRESS",
        "result": null,
        "error": null
      }
    }
  ]
}

Indicates that all of the criterion evaluations for the nomination have not completed yet. Another request should be made to get the resource(s) once the processing is complete.

🛠️ This API response is not yet stable and may be subject to change. Dependent code or logic on the status code is okay, but please avoid writing any dependent code or logic based upon these fields as we iterate on its structure.

{
  "is_error": false,
  "results": null,
  "message": "Your request has been accepted and is still being processed. Please execute your request again",
  "estimatedCompletionTime": "2024-02-05T23:31:00.000Z"
}

{
  "is_error": true,
  "error": {
    "code": "UNAUTHENTICATED",
    "issues": [
      {
        "message": "Resource requires authentication. Please provide a valid access token."
      }
    ]
  }
}

{
  "is_error": true,
  "error": {
    "code": "FORBIDDEN",
    "issues": [
      {
        "message": "Access to this resource is not authorized."
      }
    ]
  }
}

{
  "is_error": true,
  "error": {
    "code": "NOT_FOUND",
    "issues": [
      {
        "message": "This resource entity does not exist."
      }
    ]
  }
}

{
  "is_error": true,
  "error": {
    "code": "INTERNAL_SERVER_ERROR",
    "issues": [
      {
        "message": "An unexpected error occurred on our end. Please try again later, and if the problem persists, contact our support team"
      }
    ]
  }
}

Examples:

curl -X GET "${BASE_URL}/api/v1/nominations/6427f970-70c0-4b42-8fae-5609cefbb1bc/criterion_evaluations" \
-H "Authorization: Bearer ${ACCESS_TOKEN}"

Get the criterion evaluations for a candidate

GET https://endorsed.ai/api/v1/candidates/{id}/criterion_evaluations/

Gets all the evaluations for each criterion for a candidate based upon the organization's criteria

Path Parameters

Name

Type

Description

id*

string

UUID id of the candidate

Query Parameters

Name

Type

Description

criterion_type*

string

Type of criterion to filter criterion evaluation results upon. Currently only supports tag and org_wide. job specific criterion evaluations should be fetched from `/nominations/[id]/criterion_evaluations.

Headers

Name

Type

Description

Authorization*

string

Authorization header containing the Bearer Access Token from the /auth/tokens endpoint. Example: Bearer <ACCESS_TOKEN>

{
  "is_error": false,
  "results": [
    {
      "criterion": {
        "id": "5d2374b9-f637-46aa-a131-f5a4aa409414",
        "prompt_text": "Has qualifying experience as a Videographer",
        "display_name": null,
        "scope": {
          "scope_type": "tag",
          "tag": {
            "tag_type": "experience_based_role_title",
            "id": "2fd70a5a-1e03-412f-b990-f4248e18af07",
            "is_standardized": true,
            "name": "Videographer",
            "description": "Adept in filming and operating camera equipment",
            "created_at": "2023-09-25T22:06:01.000Z"
          }
        }
      },
      "evaluation": {
        "state": "EVALUATED",
        "result": {
          "is_criterion_met": true,
          "confidence_score": 0.9,
          "explanation": "The candidate's role in producing video content for recognized brands and his involvement in video content creation for over 10 years demonstrate a strong fit for the Videographer tag.",
          "evaluated_at": "2023-09-25T22:06:01.000Z"
        },
        "error": null
      }
    },
    {
      "criterion": {
        "id": "2fc0f456-6c20-405e-b8a2-eec5059c5e6c",
        "prompt_text": "Has previously worked as a Voiceover Artist",
        "display_name": null,
        "scope": {
          "scope_type": "tag",
          "tag": {
            "tag_type": "experience_based_role_title",
            "id": "48ed311b-6fa2-4d71-85a7-91fbe7e8e946",
            "is_standardized": true,
            "name": "Videographer",
            "description": "Adept in filming and operating camera equipment",
            "created_at": "2023-09-25T22:06:01.000Z"
          }
        }
      },
      "evaluation": {
        "state": "FAILED",
        "result": null,
        "error": {
          "type": "EVALUATION_GENERATION_FAILED"
        }
      }
    },
    {
      "criterion": {
        "id": "3476fb35-9725-4d5c-b1d9-81d64eee69d9",
        "prompt_text": "Has previously worked as a Voiceover Artist",
        "display_name": null,
        "scope": {
          "scope_type": "tag",
          "tag": {
            "tag_type": "experience_based_role_title",
            "id": "c3d44241-ab36-42c8-b8ff-7bac8fbdc86f",
            "is_standardized": true,
            "name": "Videographer",
            "description": "Adept in filming and operating camera equipment",
            "created_at": "2023-09-25T22:06:01.000Z"
          }
        }
      },
      "evaluation": {
        "state": "IN_PROGRESS",
        "result": null,
        "error": null
      }
    }
  ]
}

Indicates that all of the criterion evaluations of the criterion_type have not completed yet. Another request should be made to get the resource(s) once the processing is complete.

{
  "is_error": false,
  "results": null,
  "message": "Your request has been accepted and is still being processed. Please execute your request again",
  "estimatedCompletionTime": "2024-02-05T23:31:00.000Z"
}

{
  "is_error": true,
  "error": {
    "code": "UNAUTHENTICATED",
    "issues": [
      {
        "message": "Resource requires authentication. Please provide a valid access token."
      }
    ]
  }
}

{
  "is_error": true,
  "error": {
    "code": "FORBIDDEN",
    "issues": [
      {
        "message": "Access to this resource is not authorized."
      }
    ]
  }
}

{
  "is_error": true,
  "error": {
    "code": "NOT_FOUND",
    "issues": [
      {
        "message": "This resource entity does not exist."
      }
    ]
  }
}

{
  "is_error": true,
  "error": {
    "code": "INTERNAL_SERVER_ERROR",
    "issues": [
      {
        "message": "The criterion evaluations for evaluation_type [tag] had an overar ching failure. Please try again later, and if the problem persists contact our support team"
      }
    ]
  }
}

{
  "is_error": true,
  "error": {
    "code": "INTERNAL_SERVER_ERROR",
    "issues": [
      {
        "message": "An unexpected error occurred on our end. Please try again later, and if the problem persists, contact our support team"
      }
    ]
  }
}

Examples:

curl -X GET "${BASE_URL}/api/v1/candidates/b8c76214-d518-4d69-9aab-77b0166696d2/criterion_evaluations?criterion_type=tag" \
-H "Authorization: Bearer ${ACCESS_TOKEN}"

Last updated 5 months ago