You are viewing a preview of this job. Log in or register to view more details about this job.

LLM Software Evaluation Engineer

Compass Tech International
Email

Advanced AI Evaluation Software Engineer (Part-Time)

Location: Fully Remote
Commitment: 10 hours per week, 6-month contract

Company Overview:

We are at the forefront of AI innovation, specializing in the development and refinement of large language models (LLMs) designed to enhance software development. Our goal is to create AI systems that understand and generate high-quality code, transforming the way developers work.

Job Description:

As an LLM Evaluation Expert specializing in Coding, you will be responsible for assessing and improving the coding capabilities of AI models. Your expertise will guide the evaluation of AI-generated code responses, ensuring that the output aligns with best practices, industry standards, and user needs.

Key Responsibilities:

  • Analyze and evaluate LLM-generated code across multiple programming languages and development paradigms.
  • Use expert judgment to identify the most efficient and appropriate code solutions from AI-generated outputs.
  • Make informed decisions to ensure selected code aligns with industry best practices and client-specific requirements.
  • Develop coding examples that serve as benchmarks for high-quality AI-generated code.
  • Provide detailed feedback to improve the AI model’s coding capabilities.
  • Collaborate with research teams to enhance AI-generated code quality.
  • Stay up to date with software engineering best practices, coding standards, and AI advancements to refine evaluation processes.

Required Qualifications:

  • Advanced degree in Computer Science, Software Engineering, or a related field.
  • 5+ years of experience in software development across multiple programming languages.
  • Strong ability to evaluate code quality, efficiency, and adherence to best practices.
  • Excellent analytical and decision-making skills, particularly in ambiguous situations.
  • Strong written and verbal communication skills for explaining technical concepts clearly.
  • Experience in technical writing, including creating coding examples or tutorials.

Preferred Qualifications:

  • Experience working with or evaluating AI systems, particularly in AI-driven code generation.
  • Familiarity with a range of software development methodologies and architectural patterns.
  • Understanding of machine learning concepts, particularly related to natural language processing and code generation.
  • Experience contributing to coding standards or style guides.