Skip to content
Smart Lead Tech
Home
Services
Our work
Saudi Arabia
About us
Blog
Contact us
🇸🇦العربيةContact now
Smart Lead Tech

Smart software solutions

Follow us

Company

  • About us
  • Our work
  • Our services
  • Saudi Arabia
  • Restaurant POS
  • Contact us
  • Book a call

Quick links

  • Blogs
  • Pricing
  • Refund policy
  • Privacy policy
  • Terms and conditions
  • Data deletion

Contact us

  • Egypt — Head Office

    Office 6, Bldg 23, Makkah Al-Mukarramah St., Omraneya, Giza 3540203, Egypt
  • Saudi Arabia — Branch

    Bldg 2646, Secondary 123, Street 7697, Ishbiliyah Dist., Jubail 35416, Kingdom of Saudi Arabia
  • Email address

    help@smartleadtech.com
  • Phone number 🇪🇬

    +20 102 201 4671
  • Phone number 🇸🇦

    +966 53 044 7112

© 2026 Smart Lead Tech. All rights reserved.

  • Smart Lead Tech Information Technology LLC·Commercial Register ⁦61239⁩·Tax ID ⁦774-391-316⁩

  • Smart Lead Tech Software Establishment·CR ⁦7054902585⁩·VAT

The operations handbook: how do you know your AI feature still works two months later?

The feature shipped, the demo worked, the team celebrated. Eight weeks later the model provider silently rolled a new version, your customers’ dialect shifted with a new season, and a colleague added a line to the prompt…

04/07/2026 — 9 min read

A monitoring dashboard showing LLM quality scores over time

The feature shipped, the demo worked, the team celebrated. Eight weeks later the model provider silently rolled a new version, your customers’ dialect shifted with a new season, and a colleague added a line to the prompt. Nobody knows quality dropped because nobody measures it. This handbook is about measuring.

The principle: prompts are code, and code needs tests

Every change to a prompt, model or tool passes through a fixed eval set before merge. In BrandBoost AI the set is 300 real requests with reference outputs approved by the marketing team. CI rejects the merge if the aggregate score drops by more than two points.

The four layers of measurement

  1. Deterministic checks: is the output valid JSON? Does it contain the required fields? Is the length within limits? Cheap, fast, and catches 40% of breakages.
  2. Domain rules: in Real Estates Matcher the model must never suggest a property outside the seeker’s budget — a simple rule checked programmatically after every generation.
  3. LLM-as-judge: a second model grades tone or accuracy against a written rubric. Useful, but calibrated regularly against human judgement.
  4. Sampled human review: 2% of real outputs weekly, with fixed questions, recorded on the same dashboard.
If you do not have one number describing your AI feature’s quality this week versus last week, you are not operating it; you are hoping.

Monitoring in production

  • Full trace per request: model, version, prompt, tools called, latency, cost.
  • User-behaviour signals: rephrasing, cancelling, tapping “talk to a human”. In Kaleem we watch the “exercise retry rate” as a signal that a generated sentence sounded unnatural.
  • Drift alerts: a shift in average reply length or refusal rate beyond two standard deviations wakes the team.

A short story about silent drift

One month, BrandBoost AI’s cost rose 60% with no change in usage. The cause: a model update made replies a third longer. Without the cost-per-request panel we would have found out on the invoice. We added a max-length domain rule and everything returned to normal within an hour.

Tools you do not need to buy

PostgreSQL for traces, a simple Grafana board, and an evals/ folder in the repo with one command that runs everything. Commercial tools are excellent later; discipline first.

Share

Projects related to this article

  • AI SystemsFlutter

    Kaleem — AI-Powered Speech Therapy for Kids

    A playful, AI-powered speech-therapy app for children, with interactive exercises and a friendly mascot to keep them engaged.

    Show more
    Kaleem — AI speech therapy for children
  • Real EstateLaravel

    Real Estates Matcher — AI-Powered Property Matching

    A smart real-estate platform that matches demand and supply using AI and data analytics.

    Show more
    Real Estates Matcher — smart demand-supply matching
  • Real EstateLaravel

    Tatweer — Real-Estate & Engineering Services Platform

    A platform built from scratch to serve landowners, engineers, consultants and contractors — with project, contract and payment management.

    Show more
    Tatweer — integrated real-estate and engineering services

Related Blogs

  • A set of niche app icons: family tree, animal shelters, tourism, trivia

    Niche apps and small communities: where AI genuinely helps and where it spoils the fun

    Not every app is a platform with a million users. Some of the best things we built serve a small community with a big passion: Family Tree System for bilingual genealogy, Petology for animal shelters and adoption, Fly‑Fl…

    11/04/2025 — 8 min read

  • A software engineer interview with an open code screen and an AI assistant

    How we hire engineers in the AI era: no algorithm puzzles, no tool bans, and one interview that reveals everything

    We dropped the “reverse a linked list” test in 2024. It measured something we no longer needed, and missed everything we did. Since then we have hired dozens of engineers for projects in eight countries, and this is the …

    09/05/2026 — 8 min read

  • A junior engineer sitting next to a senior in front of one screen

    Is the junior developer dead? A contrarian view from a company that hires juniors on purpose

    The popular theory: AI does what the junior used to do, so why hire one? Big companies cut graduate programmes, and forums are full of graduates without a single interview. I will say something contrarian: the junior did…

    09/04/2026 — 7 min read