AI in production:
Who takes care of it after Go-Live?

During development, many factors can be controlled. The team knows which data is being used, defines performance criteria, and evaluates the model against a fixed test dataset.

In production, these conditions change.

Machines undergo maintenance or modifications. Sensors are replaced, materials change, and software updates affect the available data. In visual quality inspection, different lighting conditions or camera positions can also influence detection accuracy.

Over time, these changes can cause incoming data to differ from the data used during training.

In machine learning, changes in the distribution of input data are referred to as data drift. When the relationship between input data and the predicted outcome changes, this is known as concept drift.

For business teams, the main concern is whether the results remain reliable.

Technical availability provides only limited information about that.

Traditional monitoring checks whether a service is available, how quickly it responds, and whether errors occur. AI systems need these checks too, but they don't measure prediction quality.

In the quality inspection example, the model could process every request within the expected response time for weeks while causing an increasing amount of scrap.

Detecting this requires additional information. Relevant indicators include the quality and distribution of input data, changes in predictions, and the frequency of manual corrections.

Where actual inspection results are available, they can be compared with the model's predictions.

The relevant metrics depend on the application.

In visual quality inspection, model decisions can often be compared with manual checks. Predictive maintenance is more complicated. If a machine receives preventive maintenance after a warning, it is often impossible to establish whether the predicted failure would actually have occurred.

Monitoring should therefore be considered during the design of the application. Without the necessary data, evaluating model performance later requires additional effort.

Effective monitoring also requires clear responsibilities and procedures for responding to unusual results.

If the quality inspection system in our hypothetical example suddenly starts rejecting too many components, several departments will become involved.

Production notices the increased rejection rate. IT checks the infrastructure but finds no technical failure. The data science team discovers that the training dataset doesn't adequately represent the new product variant.

The cause has now been identified. Resolving it requires additional training data, model validation, and approval of a new version. The company must also decide how quality inspection will continue during this work.

These tasks involve different areas of responsibility.

Business teams evaluate the consequences for the production process. Data science investigates model performance and develops improvements. IT or platform teams manage infrastructure and deployment.

The organizational setup differs between companies. What matters is having documented responsibilities, clear decision-making procedures, and overall ownership that continues after the development project ends.

Suppose the model has been retrained using images of the new product variant. Its detection accuracy has improved during testing.

Before deploying the new version, the team needs to check how it performs on existing product variants. An updated model can achieve better results on new data while becoming less reliable on older components.

This requires suitable test data and a controlled approval process.

Once deployed, it should also be possible to identify which model version was running at a particular point in time. If problems occur, teams need a way to restore a previously approved version.

This is where Machine Learning Operations, commonly known as MLOps, becomes relevant.

MLOps covers the methods and tools used to manage machine learning models throughout their lifecycle. These include versioning, reproducible training processes, automated testing, monitoring, and controlled deployments.

The required setup depends on the application. A model producing monthly internal forecasts has different operational requirements from several AI systems directly integrated into manufacturing processes.

Alongside technical reliability, companies need to consider regulatory requirements.

The EU AI Act distinguishes between AI systems based on their intended use and associated risks. Certain high-risk applications are subject to requirements concerning areas such as technical documentation, logging, human oversight, and post-market monitoring.

The rules are being introduced in stages. Some provisions already apply. Regulation (EU) 2026/1744 has postponed the application dates of certain high-risk requirements.

For industrial companies, the classification of each system is important. AI supporting internal production planning isn't automatically subject to the same requirements as a safety-related AI component in regulated machinery.

Regardless of legal classification, traceable model changes, documented decisions, and appropriate monitoring are also useful for technical operations.

Many operational problems can be avoided if the requirements for running the system are considered during the pilot.

• Model performance: Development teams and business teams define metrics that help identify changes in model quality. The information required to calculate them must remain available.

• Monitoring: Unusual data or predictions are recorded and reported to the responsible employees. Defined procedures guide investigation and resolution.

• Changes: New data sources, modified production conditions, and model updates go through suitable testing and approval procedures.

• Fallback procedures: A defined alternative process is available when the AI system cannot provide reliable results. In quality inspection, this could involve increasing manual checks. Other applications may use an existing rule-based procedure.

These procedures should be developed with the employees who will use the system every day. They understand the practical constraints and exceptions within the production process.

AI projects are frequently evaluated based on model performance and expected business value. Investment decisions also need to account for ongoing operating costs.

These include maintaining data pipelines, regularly evaluating model performance, deploying new versions, and providing the people required for long-term operation.

The effort depends on the application. A model producing monthly forecasts requires a different level of support from a system influencing decisions around the clock in a manufacturing facility.

At craftworks, we consider these requirements when developing industrial AI applications. This includes integration with existing systems, appropriate monitoring, and the ability to make controlled model updates.

A successful pilot demonstrates that a model works under certain conditions. The quality of ongoing operations determines whether the solution continues to deliver value.

Planning to move an AI model into production?

We help companies develop, integrate, and operate industrial AI solutions. We're happy to discuss the technical and organizational requirements of your use case.