Articles

Accessibility for all

The Llama 3.1 8B model represents a major leap forward for consumer-grade hardware. It outperforms GPT-3.5 on many benchmarks while being able to run locally and at no cost. This advancement places recent state-of-the-art performance in the hands of individual developers and researchers, empowering them to innovate without the need for expensive infrastructure.

Key Improvements

Llama 3.1 comes with several significant enhancements:

• 128K context length across all models: This allows for better handling of longer inputs, enabling more complex tasks and extended conversations.
• Multilingual support for eight languages: This broadens the model’s usability across different linguistic contexts, making it more versatile and inclusive.
• Enhanced reasoning and tool use capabilities: These improvements make the model more adept at logical reasoning and utilizing external tools effectively.
• Improved instruction-following and chat performance: The model now better understands and executes instructions, providing more accurate and coherent responses in chat applications.

What this means for the future

The release of Llama 3.1, particularly the 405B model, marks a significant milestone in open-source AI. It promises to accelerate innovation, enable new applications, and push the boundaries of what’s possible with locally-run models. As this trend continues, we can expect even more powerful and accessible AI tools to emerge in the near future.

Stay tuned as the community begins to explore and build upon these groundbreaking models!

Provision of computing resources

Over the last decade, container technology has proven to be a profitable concept for the efficient utilization of resources. Here, individual processes are encapsulated from one another using Linux kernel features, which facilitates the shared use of computing resources by independent processes. It is essential for the use of container technology that individual containers can be restarted at any time (e.g. on another computer), as this achieves an orchestrated high utilization of the available computing time. The de facto standard for orchestrating containers is the open source system Kubernetes, which provides an interface for developers to easily deploy containers (and other resources), if configured well.

As a good configuration of a Kubernetes cluster is a relatively complex task, a specialized team should be set up to deal with the network architecture, communication between containers, secret management and basic logging and monitoring in connection with Kubernetes.

Network connectivity

The inner simplicity of microservices comes at the cost of a great deal of complexity being shifted to communication between services. It is therefore important that microservices can establish a network connection to each other and to the internet. The connection should be automatically secured by mTLS. This automation can also be best ensured by a central infrastructure team.

It should also be easy for the microservice teams to set up basic allow lists for accessing services and, if necessary, some network rules should be made binding by the infrastructure team for compliance reasons. Communication between microservices is much easier to achieve if the infrastructure is hosted by a single cloud provider.

Persistence

Microservice containers should be able to be restarted at any time without losing data. This implies that persistent data storage must take place outside the microservices, typically in databases or persistent volumes. In addition to general difficulties such as authentication and authorisation as well as connectivity to the data storage systems, there are specialized challenges such as the creation of data backups and snapshots to ensure the recoverability of persistent data.

For different needs, different types of databases should be made available for microservice teams to choose from, such as an SQL database, a document database and a cache system. This is important to ensure technology openness in the implementation of microservices so that microservice teams can choose the best technology for their use case. The provision of databases or volumes should be as automated as possible, which requires close collaboration with the Kubernetes team.

API design

Depending on the microservice, the requirements for the APIs for communication with the microservice can be very different. Nevertheless, it can make sense to set common requirements for the API structure (e.g. HATEOAS or gRPC), which are only not taken into account in justified exceptional cases. For HTTP APIs, OpenAPI can be a good solution for documentation.

If necessary, a decision can be made to make the provision of OpenAPI specifications for microservices mandatory so that client developers can rely on the existence of these specifications. In any case, a common schema for the documentation of APIs (including error cases) should be created to make it easier to find relevant documentation.

Messaging

Asynchronous communication via messaging systems can significantly help to make the overall system more resilient. The temporary inconsistency in data storage caused by asynchronicity is deliberately accepted (“eventual consistency”) in order to achieve faster requests and decoupling of different microservices.

To ensure that communication between different microservices via messaging works, the messaging system itself should be provided and managed by a central team. As with the APIs, the format of the messages should be documented using a common schema.

Authentication and authorisation

Authentication and authorisation are central tasks that cannot be handled by individual microservice teams. A centralized solution, for example based on OpenID, should be provided for this purpose.

Summary

A microservice architecture can be a good solution, especially for complex systems, in order to quickly implement improvements to the system and new features. However, this creates dependencies between microservices and therefore also between the microservice teams.

The cross-cutting concepts described here help to ensure consistency, security and efficiency in a microservice environment.

Microservices are being characterized by the following properties:

• Single Purpose: As with Unix principles, a microservice should fulfill exactly one task well.

• Encapsulation: Microservices have sole ownership of their data. They interact with the outside world via well-defined interfaces.

• Ownership: A single team (ideally consisting of 5-9 people) is responsible for a microservice over its entire lifetime.

• Autonomy: The team responsible for the microservice can build and deploy the microservice at any time without consultation of other stakeholders. The team is free to make implementation decisions.

• Multiple versions: It is possible for different versions of a microservice to exist at the same time.

• Choreography: There is no centralized system that orchestrates a workflow. Instead, each microservice is able to independently provide itself with the information it needs for its functionality.

• Eventual consistency: A temporary inconsistency of data between microservices is accepted as long as the data eventually becomes consistent again.

Challenges of microservices

Due to their internal simplicity, microservices are generally more scalable and easier and quicker to change than monoliths, in which the entire logic is contained in a single program. Due to the small teams, microservice teams feel (and are) much more responsible for the success of their microservices, which often leads to better results and decisions.

Microservices facilitate omnichannel solutions through shared backend functionality that can be consumed by different user interfaces.

Increased complexity

At the level of individual microservices, we create a simple and beautiful world through the limited scope of tasks and encapsulation. The inner simplicity of microservices comes at the price of increased complexity in terms of communication between microservices and a certain degree of redundancy: as the microservice teams are fully responsible for their services (ownership) and can also deploy them independently (autonomy), the teams must be technically capable of actually carrying out this deployment. This includes both cloud infrastructure expertise and the resources required to operate the infrastructure.

In more concrete terms, we can imagine a mailing service that is responsible for sending informational emails about a product to customers. The service must be built during development and then executed. The computers used for this must have the necessary connectivity to perform these tasks. Once the service is running, other services must be able to initiate the email process. Possible ways of doing this include an event system from which the mailing service independently filters out the events relevant to it (in the sense of the choreography property) and processes them into emails.

However, such an event system or messaging system must firstly be provided and secondly be usable by different teams and their respective microservices. Another conceivable scenario would be to address the mailing service via a REST interface. In this case, the client must be able to communicate with the mailing service via a network and this connection must be secured so that only authorized services have access to the interface.

Summary

The challenges described show that there are a number of points where we should ensure standardization across services. We will take a closer look at some of these points in the next article. Elements of the software architecture that are (potentially) relevant for several building blocks of the architecture (in our specific case, microservices) are called cross-cutting concepts.

To avoid redundancy, such cross-cutting concepts should be dealt with in the overarching architecture documentation, regardless of the individual components. It is important to define cross-cutting concepts at an early stage, as a migration at a later date will take up a lot of capacity across all teams.

In the following, we want to delve into the exciting future of MLOps, exploring emerging trends poised to reshape the technical landscape and the challenges that companies must address to ensure successful deployments of Machine Learning.

Conquering Challenges: Building a Robust MLOps Foundation

While the future of MLOps holds immense promise, several challenges must be addressed to ensure successful ML deployments. Here are some key areas to consider:

• Standardization and Interoperability:  The lack of standardization across MLOps tools and frameworks can create silos and hinder collaboration. Promoting interoperability between tools and establishing best practices for MLOps workflows is crucial for creating a more unified and efficient ecosystem. A pioneering approach to this problem is the Open Inference Protocol, an industry-wide effort that aims to establish a standardized communication protocol between so-called inference servers (e.g., Seldon MLServer, NVIDIA Triton Inference Server) and orchestrating frameworks such as Seldon Core or KServe.

• Talent Shortage:  The demand for skilled MLOps professionals outstrips the available supply. According to statista, the number of vacancies for IT specialists in companies in Germany rose to a record high of 149,000 in 2023, and Index Research reports that employers advertised almost 44,000 jobs for AI experts from January to April 2023. Organizations must invest significantly in training programs, talent acquisition strategies, and competitive employee compensation to narrow this gap and establish a strong MLOps team. This process includes recognizing the essential skills and expertise needed for successful MLOps implementation, such as data science, software engineering, cloud computing, and DevOps. Furthermore, it involves forming interdisciplinary teams that encompass these diverse domains.

• Monitoring and Observability:  Effectively monitoring the performance and health of ML models in production is critical for catching issues early and ensuring model reliability. Developing robust monitoring frameworks and integrating them into MLOps pipelines is essential. Aporia, an ML platform that focuses on the observability of ML models, can be leveraged to achieve these objectives.

• Model Governance:  Establishing clear governance frameworks for managing the lifecycle of ML models is crucial. This includes defining roles and responsibilities, ensuring model versioning and control, and setting guidelines for model deployment and retirement. The enterprise platforms Domino Data Lab and Dataiku are examples of the solutions and platforms that holistically reflect these features and many others of the ML lifecycle.

• Explainability and Bias Detection:  As mentioned earlier, ensuring model explainability and detecting potential biases are critical aspects of responsible AI. Organizations must invest in tools and techniques to understand how models arrive at their decisions and identify and mitigate any fairness issues.