model-evaluation
Skill from awslabs/agent-plugins
Install
npx skills add awslabs/agent-plugins --skill model-evaluation
Runs the open skills CLI and installs the skill for your agent (Claude Code, Codex, Cursor …). Alternatively copy the skill folder into .claude/skills/ or upload it in Claude.ai under Settings → Skills.
As a Claude Code plugin
/plugin marketplace add anthropics/claude-plugins-official
/plugin install sagemaker-ai@agent-plugins-for-aws
This skill is part of the plugin “sagemaker-ai”. In Claude Code the plugin installs it together with the plugin’s other skills, commands and hooks.
What does the model-evaluation skill do?
Description from the SKILL.md in the source repository.
Questions about model-evaluation
What does the model-evaluation skill do?
Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.
How do I install model-evaluation?
In a terminal: npx skills add awslabs/agent-plugins --skill model-evaluation. The skills CLI asks which agents to install it for. In Claude.ai you can upload the skill folder as a ZIP under Settings → Skills.
Which agents can use model-evaluation?
Skills use the open SKILL.md format, which Claude Code, Claude.ai, the Claude API, OpenAI Codex, Cursor, Gemini CLI and other agents support.
Is model-evaluation safe to use?
Our automatic check found no risky patterns (piped shell scripts, hidden characters, secret access, prompt overrides). It is not a full audit – read the SKILL.md and any scripts before installing.
More skills from awslabs/agent-plugins
Diagnose and remediate cluster-wide HyperPod (EKS or Slurm) problems — creation / deployment failures (CloudFormation, EFA health check, lifecycle scripts, capacity), EKS access, node replacement, CloudFormation nested-stack errors, post-maintenance rollback state, dangling …
Diagnose performance issues on Amazon SageMaker HyperPod clusters — uneven NCCL bandwidth across nodes and poor filesystem throughput. Read-only.
Diagnose and remediate per-node issues on a HyperPod cluster (EKS or Slurm) — a specific node is unhealthy, unresponsive, stuck, or needs replacing.
Diagnose NCCL failures and adjacent training-pod failures on HyperPod GPU clusters (EKS or Slurm) — training hangs, AllReduce / collective-op timeouts, EFA or libfabric errors, rendezvous failures, EFA TCP fallback, /dev/shm or …
Generate comprehensive issue reports from HyperPod clusters (EKS and Slurm) by collecting diagnostic logs and configurations for troubleshooting and AWS Support cases.
Selects a fine-tuning technique (SFT, DPO, RLVR, or RLAIF) for the user's use case and validates it against the selected model's available recipes.
Manages project directory setup and artifact organization. Use when starting a new project, resuming an existing one, or when a PLAN.md needs to be associated with a project directory.
Diagnostic-only skill for Slurm scheduler and node-daemon issues on Amazon SageMaker HyperPod Slurm clusters. Scope mirrors the HyperPod troubleshooting guide.
Related skills in Development
Scan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.
Diagnosis loop for hard bugs and performance regressions. Use when the user says "diagnose"/"debug this", or reports something broken/throwing/failing/slow.
Review the changes since a fixed point (commit, branch, tag, or merge-base) along two axes: Standards (does the code follow this repo's documented coding standards?) and Spec (does the code match what …
Investigate a question against high-trust primary sources and capture the findings as a Markdown file in the repo.
Guidance for instrumenting webapps with Azure Application Insights. Provides telemetry patterns, SDK setup, and configuration references.
Analyze Azure resource groups and generate detailed Mermaid architecture diagrams showing the relationships between individual resources. WHEN: create architecture diagram, visualize Azure resources, show resource relationships, generate Mermaid diagram, analyze resource group …
Configure Azure API Management as an AI Gateway for AI models, MCP tools, and agents. WHEN: semantic caching, token limit, content safety, load balancing, AI model governance, MCP rate limiting, jailbreak detection …
Troubleshoot and resolve issues with Azure Messaging SDKs for Event Hubs and Service Bus. Covers connection failures, authentication errors, message processing issues, and SDK configuration problems.