Skip to main content
To evaluate AI models effectively, we use a multifactor scoring system that assesses popularity, activity, operational integrity, and security. Each model is assigned a composite score based on the following criteria. Expand each category to see its contributing factors.
The popularity score reflects the model’s adoption and recognition within the AI community. Higher scores indicate greater usage and community engagement.
  • Number of downloads: More downloads indicate widespread adoption.
  • Number of likes: More likes suggest a positive reception from users.
  • Published papers: Models with linked academic papers receive higher credibility.
  • GitHub repository: Models with an associated GitHub repository score higher.
  • Number of spaces using the model: More integrations suggest broader utility.
Models with many downloads, likes, citations, and integrations score higher. Models with fewer engagements score lower.
The activity score measures how actively a model is discussed and maintained.
  • Discussion posts: Active community discussions contribute positively.
  • Pull requests: Indicates ongoing maintenance and improvements.
Models with frequent discussions and active pull requests score higher. Models with limited activity receive lower scores.
The operational score assesses the model’s reliability, transparency, and usability.
  • Reputable provider: Models from well-known sources score higher.
  • Model age: Older, well-maintained models may score higher, but outdated models may receive penalties.
  • Authorization requirements: Restricted-access models score lower for accessibility but may gain points for security.
  • Gated models: If a model requires special access, it may impact usability.
  • License information: Models with clear licensing receive higher scores.
  • License type: Open licenses (permissive, unencumbered) generally score higher than restrictive ones.
The following metadata factors are also considered.
  • Metric information: Essential for model evaluation.
  • Dataset information: Transparency about training data boosts the score.
  • Base model information: Important for derivative works.
  • Training data, fine-tuning, and alignment training information: Increases credibility.
  • Evaluation results: Demonstrates model performance.
Models with comprehensive metadata, reputable providers, and clear licensing score higher. Models with unclear ownership, restrictive access, or missing details score lower.
The security score evaluates potential risks associated with a model’s implementation and distribution.
  • Use of safe tensors: Secure tensor formats boost the safety score.
  • Use of potentially unsafe files: Formats such as pickle, PyTorch, and Python code files pose security risks.
  • Typosquatting risks: Models that could be impersonating popular models receive lower scores.
  • Example code availability: Models that contain example code or code snippets can introduce potential issues and hence receive lower scores.
Models that follow best security practices such as safe tensors, clear documentation, or vetted repositories score higher. Models receive lower scores if they use potentially unsafe formats such as pickle (.pkl) and unverified PyTorch (.pth) or show signs of typosquatting.

​
Final score calculation

Each category contributes to the overall model score. The final score is a weighted sum of these factors, with weights adjusted based on real-world relevance and risk impact. Higher scores indicate well-documented, popular, actively maintained, and secure models, while lower scores highlight potential risks or lack of transparency. This scoring system enables users to make informed decisions when selecting AI models for their projects. Endor Labs continuously refines and expands its evaluation criteria; this document represents the current methodology snapshot.