Trilon is building a supercharged, technology-enabled future for our people and partners. The QA Engineer plays a critical role in that mission by ensuring every AI-powered tool shipped by their product pod meets a high bar for quality, reliability, and real-world usability.
This is an embedded role within a product pod, acting as the end-to-end owner of quality across everything the team delivers. You are responsible for validating both traditional software behavior and the more complex, non-deterministic outputs of AI systems, ensuring that every solution performs as expected in real-world engineering workflows.
You work closely with the Lead Engineer and Applied AI Engineer to define acceptance criteria, design test strategies, and build automated test frameworks that catch issues early and prevent regressions over time. You are accountable for production readiness and serve as the final quality gate before tools are released to operating company engineers.
This role requires strong test automation skills, hands-on experience testing AI-enabled applications, and the judgment to define what “good” looks like in an AEC context. You are comfortable writing production-quality test code, evaluating AI output quality, and clearly communicating risks and tradeoffs to both technical and non-technical stakeholders.
Quality Ownership and Test Strategy
- Own the full quality lifecycle for your product pod, from test planning through production sign-off
- Define acceptance criteria in partnership with engineers and product managers
- Design comprehensive test strategies that cover functionality, performance, and reliability
- Establish and maintain quality standards for all deliverables
Test Automation and Frameworks
- Design, build, and maintain automated test suites for APIs, services, and user interfaces
- Develop regression tests to detect issues as systems evolve
- Integrate automated testing into CI/CD pipelines to ensure continuous validation
- Write clean, maintainable test code that scales with the product
AI Output Evaluation
- Design and implement testing approaches for AI-generated outputs, including prompt evaluation and RAG validation
- Define evaluation criteria and benchmarks for accuracy, consistency, and usefulness
- Monitor for quality drift as models, prompts, or data sources change
- Partner with engineers to improve output reliability and performance
Defect Detection and Resolution
- Identify, document, and track bugs and quality issues across the stack
- Work closely with engineers to troubleshoot and resolve defects early in the development cycle
- Prioritize issues based on impact and risk to end users
Release Readiness and Quality Gatekeeping
- Validate that all features meet acceptance criteria and quality standards before release
- Assess production readiness, including performance, reliability, and edge cases
- Serve as the final sign-off authority for releases within the pod
Collaboration and Continuous Improvement
- Partner with Lead Engineers, Applied AI Engineers, and Product Managers to improve development and testing practices
- Engage with operating company users to understand real-world expectations and feedback
- Continuously improve testing methodologies, tools, and frameworks
- 4+ years of experience in QA engineering, software testing, or a related role
- Strong experience with test automation frameworks and tools for API, backend, and UI testing
- Proficiency in at least one programming language such as Python, JavaScript, or similar
- Experience building and maintaining automated test suites and integrating with CI/CD pipelines
- Experience testing APIs and working with REST or similar architectures
- Familiarity with modern software development practices, including version control and agile methodologies
- Experience testing AI-enabled applications, including LLM outputs, prompt behavior, or RAG pipelines
- Ability to define evaluation criteria for non-deterministic systems and assess output quality
- Strong analytical and problem-solving skills, with attention to detail
- Ability to clearly communicate quality risks, tradeoffs, and recommendations
- Experience collaborating closely with engineers, product managers, and cross-functional teams
- Comfort working in a fast-paced environment with evolving requirements
- Curiosity and interest in understanding engineering workflows within the AEC industry
