# embedding_metadata

key: description
string_value:
Uncover the pivotal role of Data Engineering in advancing AI. This piece walks you through the critical strategies and real-world examples that highlight Data Engineering as the foundational force behind cutting-edge AI developments, driving forward a new era in technology.


key: chroma:document
string_value:
The Artificial Intelligence (AI) industry has seen unprecedented growth in recent years, reshaping how we interact with technology. The growth rate accelerated in 2023, with the release of human-like conversation chatbots like ChatGPT, and the plethora of subsequent technologies based on Large Language Models (LLMs) and generative algorithms. As a consequence, the industry is in the midst of an AI-driven revolution, which has the potential to lead the way for future-shaping business opportunities.


key: chroma:document
string_value:
The fuel powering all of this AI machinery is data. However, data is only useful if properly processed and integrated into robust and scalable systems. In other words, data processing infrastructures constitute the foundations of high-quality AI products, and mastering the Data Engineering skills needed for their design and implementation holds paramount value.


key: chroma:document
string_value:
This lesson has been ingrained in our practice at Tryolabs, borne out of more than 10 years of dedicated work in AI and advanced analytics consulting. Crafting bespoke Data Engineering strategies stands at the heart of our solutions, embodying the essential expertise we deliver to our clients. We've not only navigated the complexities of data-driven challenges but also unlocked tailored, innovative solutions across a diverse range of industries.


key: chroma:document
string_value:
In this blog post, we are sharing the essential technical skills but more so the strategic approaches required for crafting scalable and robust data infrastructures. We're doing so through the lens of two projects undertaken by Tryolabs. These examples serve to illustrate how theory and practice intertwine, offering a comprehensive look at the challenges and solutions encountered in our quest to leverage the power of AI.

Exhibit 1: Web traffic analytics for SEO scaling


key: chroma:document
string_value:
Exhibit 1: Web traffic analytics for SEO scaling

Back in 2018, this project marked a pivotal moment in our venture into the realm of large-scale Data Engineering. The mission was to architect and deploy a data infrastructure capable of supporting high-volume data reports on web traffic for a leading client in the digital marketing space.

Navigating high-volume data for strategic SEO


key: chroma:document
string_value:
Navigating high-volume data for strategic SEO

We embarked on developing an analytics tool for Search Engine Optimization (SEO), a core component of our client's digital marketing services. The goal was to empower web users—whether individuals or businesses— to boost their online visibility, drawing significantly more traffic by refining their web content strategies.


key: chroma:document
string_value:
Faced with the need to process vast volumes of information, our task was to distill actionable data products from the ocean of online traffic data—covering keywords, pages, and domains search frequencies. This wasn't just about handling data; it was about unlocking strategies for SEO success by understanding what makes content appealing to search engines and potential visitors.

Figure 1.1. Illustration of the SEO tool for web traffic analytics


key: chroma:document
string_value:
Figure 1.1. Illustration of the SEO tool for web traffic analytics

The Data Engineering challenge To get an idea of the required data processing scale, close to 100 million keywords needed to be processed every day, and monthly aggregations required processing over two billion keywords in less than 48 hours every month.

Tackling this challenge was an ambitious task that demanded a robust, efficient, and scalable data management strategy.

Crafting the solution: Data Engineering in action

Our solution to this challenge was a meticulously planned data processing strategy that encompassed the entire data lifecycle, from acquisition to actionable data products. The approach was threefold:


key: chroma:document
string_value:
Data centralization. Data processing starts by gathering raw traffic data, primarily from Google's SERP, into a unified storage place, setting the stage for all subsequent data transformations. Extraction and processing. With raw data centralized, a carefully crafted data process is put in place to convert crude data into ripe data products relevant to the SEO tool. Storage and access. The final piece of the puzzle was storing this processed data in a way that made it easily accessible for analysis and reporting. This meant not only choosing the right storage solutions but also making design choices ensuring fast, efficient queries.


key: chroma:document
string_value:
Using the Data Engineering lexicon, our infrastructure utilized a combination of data laking, ETL pipelines, and data warehousing (points 1, 2, and 3 above, respectively), ensuring seamless access to data products.

Data Engineering components of the solution Data laking - gathering files containing Google generated data (e.g. SERP) about any given keyword.

ETL pipeline - extract keyword and associated SERP data from the raw gathered files, transform the data to a custom format and load to various databases.


key: chroma:document
string_value:
Data warehousing - the ETL pipeline produces different types of processed data depending on the querying complexity of the required analytical reports, calling for different warehousing solutions.

To bring this vision to life, we leveraged the power and flexibility of Amazon Web Services (AWS). The AWS ecosystem offers a suite of managed and serverless products aligned with our need for scalability and efficiency. This choice allowed us to focus on optimizing our Data Engineering processes without being bogged down by infrastructure management.


key: chroma:document
string_value:
Architecture design key decisions Requirements Provision of raw data (batches) by 3rd party should trigger data processing. Data processing should be easily scalable. Data products should support both simple and complex queries as well as expensive aggregations. Design decisions AWS S3 bucket for raw data storage with event triggers feeding an SQS Queue. ECS-based data processing triggered by listening to the SQS Queue. Warehousing: DynamoDB and Elasticsearch (plus aggregations using AWS Batch).


key: chroma:document
string_value:
This diagram illustrates the architecture of the implemented data infrastructure. It highlights the specific AWS services employed at each step of the Data Engineering process and the rationale behind our architectural choices.

Figure 1.2. Scheme of the data infrastructure design indicating the AWS products/services used, highlighting their corresponding Data Engineering task.

Key architectural insights

Delving into the architecture illustrated in Figure 1.2 provides deeper details about the inner workings of the solution, showcasing how each component contributes to an efficient data infrastructure.

Data laking


key: chroma:document
string_value:
Data laking

An S3 bucket was used as data lake, where raw data files, each containing 100 Google search results for a given keyword (along its metadata), were securely stored upon upload by a third-party provider. The latter triggered an S3 event, queueing a message in an SQS queue, readying the system for the next data processing phase.

ETL pipelines

The ETL pipelines were designed with a dual approach to cater event-based and batch data processing needs.

Event-based custom pipeline


key: chroma:document
string_value:
The ETL pipelines were designed with a dual approach to cater event-based and batch data processing needs.

Event-based custom pipeline

An ECS cluster hosts a docker microservice that continuously listens to the SQS queue. Upon new messages arrival, it downloads the new raw data file from the S3 bucket, process the keyword-related data, and loads it, after custom formatting, into DynamoDB and Elasticsearch.

Batch (monthly) aggregation pipelines

A first AWS Batch Job is used to process the DynamoDB table pages_data to obtain aggregations at page level, writing into the DynamoDB table pages_aggregated , useful for page-level analytics reports.


key: chroma:document
string_value:
A second AWS Batch Job is used to process the DynamoDB table pages_aggregated (first batch job) to obtain aggregations at domain level, writing into the DynamoDB table domains_aggregated , useful for domain-level analytics reports.

Data warehousing

The event-based ETL pipeline loads transformed data to two types of storage services:


key: chroma:document
string_value:
Data warehousing

The event-based ETL pipeline loads transformed data to two types of storage services:

DynamoDB keywords stores keyword metadata (simple queries) pages_data a huge table partitioned by domain:page and sorted by keyword, that will be used downstream for additional processing (page/domain level aggregations) Elasticsearch keywords stores keyword metadata (complex queries) SERP stores the SERP data for each keyword Elasticsearch storage resources were needed to process queries of high complexity not possible with DynamoDB.


key: chroma:document
string_value:
The first AWS Batch Job aggregates data at page level and loads the output to DynamoDB table pages_aggregated which, being partitioned by domain and sorted by website path, makes access to page level aggregated data more efficient and cost effective.

The second AWS Batch Job aggregates data at domain level and loads the output to DynamoDB table domains_aggregated which, being partitioned by domain and sorted by month, makes access to domain level aggregated data more efficient and cost effective.

This data infrastructure allowed for the scalable generation of a set of key data products that supported the analytical reports required by the client’s SEO tool (see Figure 1.3).


key: chroma:document
string_value:
Key data products Elasticsearch tables SERP & keywords : support complex keyword based queries (e.g. top keywords) and complex text searches (e.g. content leads, which are the top ranking pages for a given keyword). DynamoDB tables keywords : supports fast and scalable basic reports on keywords related data pages_aggregated and domains_aggregated : support fast and scalable reports on domain and website traffic evolution.

Figure 1.3. Usage of the main data products in a diversity of analytics reports of the SEO tool.

Takeaways


key: chroma:document
string_value:
Figure 1.3. Usage of the main data products in a diversity of analytics reports of the SEO tool.

Takeaways

The implemented infrastructure provided the necessary data products to fulfill the analytics needs of the client, achieving the required throughput (e.g. over 2 billion keywords being processed in less than 48 hrs every month). This was a consequence of appropriate design decisions, some of which are worth highlighting.

Efficient partitioning and sorting keys


key: chroma:document
string_value:
Efficient partitioning and sorting keys

Appropriate design of the partitioning and sorting keys in DynamoDB tables played a central role in minimizing table scans, consequently reducing querying costs. This resulted in money-wise efficient data products that were able to support the analytics and reporting needs of the client’s SEO tool.

Deep understanding of DynamoDB

Detailed knowledge of not only partitioning and sorting keys, but also of read and write capacity units and pricing characteristics allowed for appropriate decisions on monthly quote reservations, and table writing mode (only once per record), resulting in performance maximization while keeping low costs.


key: chroma:document
string_value:
Choosing the appropriate data warehouse

An appropriate combination of DynamoDB and Elasticsearch allowed for serving data products supporting different querying complexity levels, providing the required data processing flexibility in an affordable manner.

Choosing the appropriate computational scaling method


key: chroma:document
string_value:
Choosing the appropriate computational scaling method

At the beginning of the project, Spark was an appealing option for data processing parallelization, allowing to leverage a rich environment and set of tools, and reducing maintainability (and future scalability issues). However, (in our hands) a custom parallelization approach resulted in higher throughput (probably a Spark initialization overhead issue). Although this should be revisited if working on a similar project nowadays (remember this project took place six years ago), it is a relevant learning to properly weight the available computational scaling methods, before blindly jumping into mainstream tools.


key: chroma:document
string_value:
Exhibit 2: Enhancing online retail with image semantic search

As part of a long-lasting engagement with one of our clients, this project constituted a capstone in our journey towards integrating Data Engineering skills and AI expertise.

The challenge was to build a semantic search service for retail items, providing a robust data infrastructure capable of supporting various Machine Learning applications designed to tackle distinct business challenges.

Developing a unified semantic search engine


key: chroma:document
string_value:
Developing a unified semantic search engine

Our mission was to create an image-based semantic search service that could accurately and efficiently identify items similar to a given target, utilizing both image data and item metadata as inputs. This service was envisioned as a central hub, offering support to a suite of ML-driven products, including recommendation systems, dynamic pricing models, and automatic attribute tagging tools. These products work in synergy to address a broad spectrum of business needs (as illustrated in Figure 2.1).


key: chroma:document
string_value:
The challenge extended beyond the mere development of these services; it involved ensuring the system could handle the high volume of semantic search queries triggered by user interactions across the ML products. Addressing this required a focused effort on building scalable, real-time data processing capabilities. This endeavor was critical not only for enabling efficient semantic (and keyword-based) searches but also for optimizing the processing of retail items. The outcome was a system that could support timely and effective online product listings, significantly enhancing the retail experience and profitability.


key: chroma:document
string_value:
Figure 2.1. Illustration of the image semantic search service providing the data infrastructure for multiple Machine Learning based products. In this fictitious example, a user requests a business task that requires finding items similar to a floppy disk.

The Data Engineering challenge Supporting a semantic search service for retail item retrieval, covering over 3.5 million items, and handling close to 80 requests per second, from multiple business processes triggered by an end user that expects a latency below 500 milliseconds.

Fulfilling this endeavor was a challenging task requiring cloud orchestration savviness and proper integration of AI experience into data architecture design.


key: chroma:document
string_value:
Building the solution: an AI and Data Engineering mix

The approach we adopted to solve this challenge was a carefully crafted data operation that transformed items metadata from multiple concurrent sources and items images, into entries of a vector database (the core data product) that efficiently serves hybrid keyword/semantic queries. This solution contained four main components:


key: chroma:document
string_value:
Data centralization The first data processing step is gathering all retail item images in a centralized storage location (supporting unstructured data), which works as the reference source for subsequent transformations resulting in vector encodings indexed in the vector database. Streaming and batch processing Once images are centrally stored, they are extracted and transformed into vector encodings using neural nets and custom transformations. This is done both in batch and streaming modes. Both modes are also used for items metadata processing, carefully considering a key requirement: the need to integrate multiple concurrent metadata sources. Storage supporting semantic retrieval The


key: chroma:document
string_value:
carefully considering a key requirement: the need to integrate multiple concurrent metadata sources. Storage supporting semantic retrieval The final step of this data operation is storing the items metadata and vector encoding in a vector database, providing efficient and fast hybrid keyword and semantic retrieval. Monitoring With the main components of the data infrastructure up and running, it is critical to ensure adequate performance levels are continuously delivered. To that end, a monitoring service was put in place to track and fire alerts about the health of the vector database.


key: chroma:document
string_value:
In Data Engineering terms, this approach required the orchestration of data laking, neural network inference integrated into streaming and batch ETL pipelines, data warehousing, and data product health monitoring (points 1, 2, 3, and 4 above, respectively), resulting in a robust data operation.

Data Engineering components of the solution Data laking : gathering raw input images of all retail items.

Batch ETL pipeline : items metadata processing, and neural net vector encoding of item images extracted from the data lake.

Streaming ETL pipeline : near real-time processing of multiple source items metadata, and neural net vector encoding of new item images added to the data lake.


key: chroma:document
string_value:
Data warehousing : storing ETL pipelines products — processed items metadata and vector encodings — into a vector database supporting hybrid keyword-semantic search and approximate kNN methods.

Monitoring: keeping track of the data warehouse health to ensure the items retrieval service fulfills the performance requirements.

To construct this architecture we combined the AWS, the Google Cloud Platform (GCP), and third-party tools. This allowed us to take advantage of the different strengths offered by such platforms and services, while fulfilling client-specific requirements.


key: chroma:document
string_value:
Architecture design key decisions Requirements Integrates (near real-time) items metadata coming from different divisions in the client Supports hybrid keyword-semantic search queries Low latency serving to multiple internal ML-products Design decisions AWS S3 bucket for images laking, and a Kafka server for multi-source real-time data processing GCP Dataflow for batch pipeline, and a combination of Cloud Scheduler and Cloud Run for streaming pipeline Warehousing: Elasticsearch (items metadata and image vector encoding) and BigQuery (items metadata)


key: chroma:document
string_value:
Here you can see the architecture of the implemented data operation, indicating the particular platform-dependent services used in each piece of the Data Engineering puzzle, along the principles underlying the chosen design.

Figure 2.2. Scheme of the data infrastructure design indicating the AWS, GCP and third-party services used, highlighting their corresponding Data Engineering task.

Key architectural insights

Digging into the architecture illustrated in Figure 2.2 sheds light on the features of the implemented solution in finer detail, depicting the interplay between each component, which resulted in a robust data infrastructure.

Data laking


key: chroma:document
string_value:
Data laking

An AWS S3 bucket was used as data lake to store all retail item images, providing a secure and consistent reference point for data extraction by streaming and batch pipelines.

ETL pipelines


key: chroma:document
string_value:
ETL pipelines

An on-demand batch pipeline, implemented using GCP Dataflow (Apache Beam), extracts items images from the AWS S3 bucket, loads a cost effective resnet-50 convolutional neural net (stored on GCP Cloud storage), and uses the latter plus custom normalization (enhances similarity distance accuracy), to generate vector encodings from the images. Items metadata and vector encodings are pushed to an Elasticsearch vector database. In addition, items metadata is also stored in BigQuery to facilitate custom analytics and Elasticsearch backfilling.


key: chroma:document
string_value:
A streaming pipeline receives items metadata messages published by different divisions within the client to a Kafka server, which is connected to Cloud Run (triggered using Cloud Scheduler), for near real-time data processing. The latter includes i) the addition of new items, applying the same transformations and storage as used in the batch pipeline, and ii) the continuous integration of items metadata updates published by different teams in the client.

Data warehousing


key: chroma:document
string_value:
Data warehousing

A resnet-50 convolutional neural net is stored in a GCP Cloud Storage bucket. This model is used to generate image vector encodings supporting semantic search. Items metadata is stored in GCP BigQuery to support custom analytics and Elasticsearch backfilling. Items metadata and image vector encodings are stored in an Elasticsearch vector database supporting hybrid keyword-semantic search, and using the Hierarchical Navigable Small World (HNSW) approximate kNN method for efficient similarity-based retrieval.

Monitoring

The status of the vector database is monitored using Datadog, keeping track, for example, of index rates and search rates.


key: chroma:document
string_value:
Monitoring

The status of the vector database is monitored using Datadog, keeping track, for example, of index rates and search rates.

This system robustly supports the generation of key data products, the most relevant being the Elasticsearch vector database providing a hybrid keyword-semantic search service to a battery of ML-products (see Figure 2.1).

Key data products Elasticsearch vector database: Provides hybrid keyword-semantic search service to multiple ML-based products, allowing for efficient retrieval of business-relevant retail items. BigQuery tables: Support custom analysis on items metadata and vector database backfilling.

Takeaways


key: chroma:document
string_value:
Takeaways

The implemented system provided the necessary data products to fulfill the infrastructure requirements of the client. Appropriate design decisions were key for this result, some worth highlighting.

Processing items metadata coming from multiple sources


key: chroma:document
string_value:
An aspect of high importance when designing the data infrastructure for this client was that the metadata of a given item is produced in parallel by multiple teams within the client’s enterprise. Moreover, integration and processing of such metadata needed to be done in near real time. This requirement could be appropriately fulfilled by combining a Kafka server with the Cloud Scheduler and Cloud Run services in GCP. An improvement over the implemented streaming pipeline would be to use Dataflow instead of Cloud Scheduler and Cloud Run, simplifying the design and leveraging a service that is tailored to handle both streaming and batch pipelines.


key: chroma:document
string_value:
Relevance of hybrid keyword-semantic search in retail

An advantage of vector databases like Elasticsearch is their ability to combine keyword-based and semantic-based (vector encodings) queries. This is particularly relevant for a retail company, where retrieving items based only on image similarity might not be sufficient to capture the nuances of the items being targeted. Complementing the semantic search with specific text keywords can help cope with such limitations.

Serving multiple ML-based products


key: chroma:document
string_value:
Serving multiple ML-based products

The core use of the semantic search service was the support of queries from multiple ML-based products achieving specific throughput and latency thresholds. This requirement was successfully met thanks to the scalability features and efficient approximate kNN search algorithm offered by the Elasticsearch vector database. Such a data infrastructure provided the foundations for key ML services, including recommendations and automatic attribute tagging systems, working in concert to increase business productivity.

Looking ahead


key: chroma:document
string_value:
Looking ahead

It’s clear that the journey of integrating advanced Data Engineering with AI is both a challenging and immensely rewarding endeavor. The projects we've shared exemplify not just the technical and strategic depths we navigate at Tryolabs but also our commitment to pushing the boundaries of what's possible in AI and analytics.

The evolution of the AI industry demands continuous innovation and adaptability. Our experiences underscore the importance of solid Data Engineering foundations to not only meet current needs but also to anticipate future challenges and opportunities.


key: chroma:document
string_value:
Stay tuned for for more insights and stories from the forefront of AI and Data Engineering, which will serve as useful references to your path to building AI-based innovative technologies.


key: description
string_value:
Temporian is an open-source Python library for efficient temporal data processing. Developed in collaboration with Google, it addresses the challenges of working with time series data. Learn how its specialized design minimizes bugs and boosts productivity.


key: chroma:document
string_value:
Today marks a significant milestone for us at Tryolabs as we introduce Temporian.

In collaboration with Google, we've designed this tool to address the multifaceted challenges of temporal data processing head-on. Let's explore the inspiration, functionality, and future of Temporian.


key: chroma:document
string_value:
In the world of data science, our experience working with clients has made it abundantly clear that temporal data is everywhere, powering a wide range of use cases and applications. From sales forecasting, fraud detection, and marketing optimization to processing signals from electronic sensors, the challenges presented by temporal data are very diverse. Whether dealing with long time sequences spanning years or precise millisecond measurements, efficiently handling temporal events is an essential asset.


key: chroma:document
string_value:
Typically, generic data science tools are employed to tackle temporal data manipulation. While many of these tools are versatile enough and can be adapted to perform mostly any data transformation, the number of steps and intricacies required to handle some particular cases often leads to large codebases, especially as projects grow in complexity. This quickly becomes a challenge, often leading to finicky data manipulations, code subtleties, hidden behavior, and bugs. On top of that, inefficient computation becomes commonplace when using tools and operations that were not designed for their original purpose.


key: chroma:document
string_value:
Recognizing the need for a specialized solution for this domain, we partnered with Google to develop Temporian: a world-class, open-source Python library for preprocessing and feature engineering of temporal data.


key: chroma:document
string_value:
At its heart lies a clever use of proper data structures and formats, specially designed to handle temporal data in all its forms. By representing data as a generic collection of events with corresponding timestamps—potentially originating from multiple sources, having multiple attributes, and being non-uniformly sampled—Temporian seamlessly adapts to a wide array of temporal data formats. Furthermore, it leverages highly optimized C++ operations underneath, and we have an active development lane to leverage Apache Beam's capabilities for running operations on large-scale data.


key: chroma:document
string_value:
Temporian being purpose-built for temporal data manipulation brings numerous benefits, including streamlined temporal data operations and code that is clear, intuitive, and easy to maintain. As a result, it minimizes the likelihood of bugs or hidden behavior, also providing substantial computational efficiency gains.


key: chroma:document
string_value:
Not less important, it integrates with the most popular data science and Machine Learning tools. Its user-centered documentation provides comprehensive tutorials, enabling effortless initiation with Pandas and TensorFlow. From exploratory data analysis in interactive notebooks to deploying robust high-volume processing pipelines, Temporian is geared to facilitate the entire spectrum.

The story behind our collaboration journey


key: chroma:document
string_value:
How did the paths of Tryolabs and Google converge in the vast landscape of Machine Learning and data science? The answer lies in a shared aspiration and a mutual recognition of each other's strengths. At Tryolabs, forecasting is one of our main areas of expertise—having applied it to solve real-world use cases that range from Predictive Maintenance of Oil Pipes to Pricing Optimization - and as such, we know first-hand that a purpose-made temporal data preprocessing tool would bring tons of value to the table, especially when it comes to ease of development and time to market of such solutions. Add to that Google’s need for a tool capable of processing zettabytes of data from millions of


key: chroma:document
string_value:
of development and time to market of such solutions. Add to that Google’s need for a tool capable of processing zettabytes of data from millions of events worldwide in real-time, and you get both a perfect goal to strive for, and two engineering teams with more than enough expertise to take a swing at it!


key: chroma:document
string_value:
On top of that, we are no strangers to open-source development, with our home-grown Norfair being one very successful example. Neither is it the first time that we collaborate with Google, since that time we streamed together a live-chat event with some of the Swift for TensorFlow’s core developers, sharing our experience as early users of the library. They were incredibly humble and awesome, and we enjoyed the experience so much that we came back for more.


key: chroma:document
string_value:
Temporian’s journey began almost a year ago, when the teams got together to brainstorm what would be the most valuable features in the solution we aimed to build. From then we underwent a large phase of design, followed by the development of an MVP which was then validated via UX testing with real users. After having made adjustments based on the received feedback, and polishing up and publishing the documentation for the library, we finally decided to publish it for its first-ever users to have a go with it. This is only the start - there’s a long road ahead for Temporian, and we’ll be there for it!


key: chroma:document
string_value:
The library’s development was not without its challenges—needing, for example, to strike a precise balance in exposing a public API that was simple and intuitive for new users but flexible and powerful enough for advanced ones, to support several execution backends for each available operation, and to quickly become proficient in tools that we’d never crossed paths with. Although challenging, overcoming those and many other issues was little but a learning instance that we gladly embraced—and a breeze to go through when surrounded by teammates with the collective experience of our own.

Inside Temporian: a deep dive into its capabilities


key: chroma:document
string_value:
Inside Temporian: a deep dive into its capabilities

So: what exactly does Temporian do? And more importantly, what sets it apart from the myriad of other widely adopted and maintained data preprocessing libraries?

A very simple example of a Temporian script. The snippet calculates the weekly sales for each store, visualizes the output with a plot, and exports the data to a CSV file.

Unlike most data preprocessing libraries, Temporian was designed and built with temporal data—and temporal data only—in mind.


key: chroma:document
string_value:
Unlike most data preprocessing libraries, Temporian was designed and built with temporal data—and temporal data only—in mind.

That is, Temporian was built to handle the timestamps associated with each of your data points natively, and that enables it to run certain operations on them much, much more efficiently, both in terms of time and memory - in some cases reaching x1000 speedups compared to the same preprocessing written directly in NumPy! Note that this is by no means because of NumPy being slow or inefficient—it just wasn’t built for temporal data (and, as a matter of fact, Temporian heavily relies on it under the hood!).


key: chroma:document
string_value:
These results are only made possible by Temporian’s custom data structure, called the EventSet , which arranges temporal data in a way that makes it efficient to operate on it, and to great portions of Temporian’s operations being implemented in C++ and highly optimized.

Temporian’s EventSet isn’t just able to represent time series (which can be defined as uniformly-sampled temporal data, or a series of values that happen sequentially and with equal intervals between them), but also multivariate and hierarchically-structured data, and data with non-uniform and non-aligned sampling, which by itself sets it apart from any alternative libraries that we know of.


key: chroma:document
string_value:
You may be wondering exactly what operations Temporian provides. The library’s API Reference hosts the complete list of available operators (which is growing by the day!), but in short, Temporian offers a battery of time-related functions, that allow users to compute features as simple as the day of the month an event occurred on, to as complex as the moving standard deviation of the sales of each product in your online store, in relation to the lagged cumulative sum of sales across all products since launch.


key: chroma:document
string_value:
To top it all off: a Temporian program can be developed iteratively in an interactive development environment, such as a Jupyter notebook, and then saved to a file to be run in Apache Beam on production-grade workflows! This is only possible thanks to Temporian tracking the operations a user performs on their data and maintaining an underlying graph of operations - all in a transparent way to most users!

Real-world scenarios

Temporian can be helpful in any domain where temporality plays a role… Which is just about all of them?


key: chroma:document
string_value:
Real-world scenarios

Temporian can be helpful in any domain where temporality plays a role… Which is just about all of them?

Jokes aside, temporal data is as commonplace in AI as you get, and can come up in the form of transactions, user events, logs, product sales, medical data, sensor signals, or weather patterns - just to name a few. And in each of those use cases, Temporian can lend a huge hand when it comes to consuming the raw temporal data and outputting something a Machine Learning model can work with!


key: chroma:document
string_value:
Not sold yet? Check out how Temporian deals with a real-world scenario in the M5 Competition tutorial, which uses Temporian to preprocess and feature engineer hierarchical sales data from Walmart stores across the US. Its output is then fed to a Gradient Boosted Trees model to reach near state-of-the-art performance on forecasting future sales!

This tutorial gives a good overview of Temporian’s versatility. The competition’s data is loaded into three Temporian EventSet s, which are used to compute a variety of temporal features:

Past sales : the lagged sales values from the last couple of days, computed using lag() .


key: chroma:document
string_value:
Past sales : the lagged sales values from the last couple of days, computed using lag() .

: the lagged sales values from the last couple of days, computed using . Moving statistics : the average and standard deviation of sales in the last couple of weeks, computed using simple_moving_average() and moving_standard_deviation() .

: the average and standard deviation of sales in the last couple of weeks, computed using and . Calendar data : the day of the week, day of the month, and month associated with each date we want to forecast on, computed using calendar_day_of_week() , calendar_day_of_month() , and calendar_month() .


key: chroma:document
string_value:
: the day of the week, day of the month, and month associated with each date we want to forecast on, computed using , , and . Sales aggregated per department : the sum of sales in each product’s department during the last month, computed using drop_index() , moving_sum() , and propagate() .

: the sum of sales in each product’s department during the last month, computed using , , and . Special events: the number of days remaining until key calendar events, such as Christmas or Father’s Day, computed using propagate() , leak() , and resample() .


key: chroma:document
string_value:
Note that Temporian not only makes it easy for the developer to write the code to compute these features but also executes it extremely efficiently, delegating as much processing as possible to C++ and minimizing memory usage, which can make or break your program when dealing with large amounts of data!

If you want other examples of Temporian at work (or just can’t wait to dive right into the code), check out the Getting Started guide for a simple step-by-step example, or one of the featured tutorials for Detecting payment card fraud, Heart Rate Analysis, and Loan Outcomes Prediction!

Kickstarting your Temporian experience


key: chroma:document
string_value:
Kickstarting your Temporian experience

The best place to start your journey as a Temporian user is the library’s docs, which provide all kinds of resources for new and advanced users, including a Getting Started tutorial, a quick intro to how the library works, a full tour of its concepts and capabilities, and the complete API Reference for the library’s components.

If instead you’re more of a hands-on learner and can’t wait to get started: Temporian is available on PyPI and can be installed with a simple pip install temporian (though we do recommend checking out some of the introductory guides first!).

Embracing open-source, community, and the future


key: chroma:document
string_value:
Embracing open-source, community, and the future

This exciting collaboration between Tryolabs and Google holds greater significance since we're creating an open-source library, accessible for use and improvement by anyone worldwide. We know for a fact that the best solutions emerge when the collective intelligence of the community comes together, and thus, we welcome experts and developers alike to give it a try and join forces with us!


key: chroma:document
string_value:
Now is the perfect time to get involved as a user, developer, or early adopter, helping us showcase the library in real-world scenarios. With an increasingly stable API and comprehensive, well-tested documentation, at this point you can confidently explore and leverage Temporian's capabilities. Moreover, user feedback and feature requests will play an important role in shaping the library's future, ensuring that it continuously evolves to meet the needs of data scientists, engineers, researchers, and all potential users.


key: chroma:document
string_value:
Looking ahead, our roadmap is ambitious and promising. We will continue to develop Temporian, expanding its capabilities with more operators, enhancing Apache Beam and cloud providers integration, and supporting real-time streaming applications. Staying focused on temporal data preprocessing and feature engineering, we aim to become the go-to library for industry projects at all scales.

We are thrilled to be on this journey, creating a top-notch tool that bridges a crucial gap in the data science landscape. We invite you all to join us in making Temporian the perfect solution for temporal data processing!


key: description
string_value:
Discover effective strategies for controlling Large Language Models (LLMs). Learn about prompting, logit filtering, parameter tuning, and guardrails to improve LLM outputs.


key: chroma:document
string_value:
In the ever-evolving landscape of natural language processing, the advent of Large Language Models (LLMs) has ushered in a new era of possibilities and challenges. While these models showcase remarkable capabilities in generating human-like text, the potential for unintended consequences raises concerns: hallucinations—instances where the model generates information that is factually incorrect or entirely fabricated, formatting problems, the use of harmful and biased language, and security issues to name a few. Generative AI is a double-edged sword and what makes LLMs helpful might also introduce significant risks.


key: chroma:document
string_value:
To mitigate some of these issues, researchers are working on what they call model alignment, using strategies such as Reinforcement Learning from Human Feedback (RLHF) and fine-tuning. However, those strategies are often implemented before making the LLM available to the public. In this post, we will delve into techniques that can be used even when the model has been developed. For a more in depth exploration of these techniques, refer to this survey.


key: chroma:document
string_value:
Understanding the need to address these challenges, developers often seek alternative ways to limit the outputs of LLMs to ensure the practical usability of LLM-generated content in diverse applications, enhance accuracy and security, and maintain ethical standards. As we explore strategies for controlling LLM outputs, we aim to strike a balance between harnessing the power of these models and addressing the inherent challenges they pose.

We’ve researched alternatives to address output limitation so you don’t have to. At a high level, there are 3 strategies that act at different stages of the process to ‘control’ an LLM:


key: chroma:document
string_value:
Effective prompting Parameters and guided generation Post-generation guardrails

Strategy 1: effective prompting

Prompting is the easiest strategy we’ll cover. Prompting, in essence, is about designing the initial query or statement in such a way that it effectively communicates our intent to the model, thereby influencing its response to align with our desired outcome. As the field advances, its focus expands beyond crafting prompts to encompass the design of systems where model queries serve as integral components.


key: chroma:document
string_value:
Utilizing prompts represents a potent approach to augment the adaptability and manageability of large language models. A meticulously crafted prompt holds the potential to significantly amplify the productivity of LLMs. Some strategies that use prompting are templates, example libraries, or prompt optimization tools that structure user-inputted prompts. The advantages of these strategies are facilitating effortless interaction for non-technical users as well as optimizing interaction efficiency.


key: chroma:document
string_value:
Chain of Thought (CoT) is one example of how a prompt can be adapted to align with an expected result. It’s a method that encourages the LLM to justify its “reasoning” and usually involves including examples on the prompt that better illustrate the intended answer.

Awesome ChatGPT Prompts is an open-source collection of curated prompt templates for ChatGPT, each tailored for unique applications and use cases. LangGPT follows a similar approach. As an example, here’s a prompt template to turn ChatGPT into a Linux Terminal:


key: chroma:document
string_value:
PromptSource is a toolkit for creating, sharing, and using natural language prompts. Prompts are saved in standalone structured files and are written in a simple templating language called Jinja. An example of a prompt available in PromptSource for SNLI is:

PromptLang is a custom programming language specifically designed for use in GPT-4 prompts and AI interactions. Promptify is used to solve Natural Language Processing (NLP) problems by generating different NLP task prompts for popular generative models like GPT, PaLM, and more.

Strategy 2: parameters and guided generation


key: chroma:document
string_value:
Moving on to the second category of strategies, we focus on the techniques employed when calling the LLM. This stage involves the use of specific methods that can further refine the model's output during the generation process. One such powerful tool is logit filtering. Before covering this strategy, let’s discuss how text generation works under the hood. After getting the prompt, the model generates logits (raw, unnormalized predictions) for each possible output token (which isn’t necessarily a word). These logits are then passed to a softmax function, generating a probability for each possible output and a probability distribution over the entire model’s vocabulary.


key: chroma:document
string_value:
By accessing the logits themselves, we can manipulate them and scope the response, for example by:

Setting some logits to zero: useful to filter out words, topics, etc.

Applying a weighted mask: useful to allow for more control and aligns with a probabilistic approach.

Increasing/decreasing desired logits: useful to adjust the chances of some tokens being generated over others.

Take into account that directly manipulating logits can sometimes lead to unnatural responses if done too aggressively


key: chroma:document
string_value:
Take into account that directly manipulating logits can sometimes lead to unnatural responses if done too aggressively

LM-Format-Enforcer is a library that enforces the output format of language models by filtering tokens. It works by combining a character-level parser and a tokenizer prefix tree into a smart token filtering mechanism. When the language model generates a token, the character level parser gets advanced according to the new characters, ready to filter the next timestep.

Here is an example using the regular expression format Michael Jordan was Born in (\d)+.


key: chroma:document
string_value:
Here is an example using the regular expression format Michael Jordan was Born in (\d)+.

idx generated_token leading_token generated_score leading_score generated_token_idx leading_token_idx 0 ▁ ▁ 1.000000 1.000000 29871 29871 1 Michael ▁Sure 0.000027 0.959473 24083 18585 2 ▁Jordan ▁Jordan 1.000000 1.000000 18284 18284 3 ▁was ▁was 1.000000 1.000000 471 471 4 ▁Born ▁born 0.000008 1.000000 19298 6345 5 ▁in ▁in 0.994629 0.994629 297 297 6 ▁ ▁ 0.982422 0.982422 29871 29871 7 1 1 1.000000 1.000000 29896 29896 8 9 9 1.000000 1.000000 29929 29929 9 6 6 1.000000 1.000000 29953 29953 10 3 3 1.000000 1.000000 29941 29941 11 . . 0.999512 0.999512 29889 29889 12 < /s > < /s > 0.981445 0.981445 2 2


key: chroma:document
string_value:
The model "wanted" to start the answer using Sure , but the format enforcer forced it to use Michael . Afterward, almost all of the leading scores are within the allowed token set, meaning the model likely did not hallucinate due to the token forcing. The only exception was timestep 4 - " Born" was forced while the LLM wanted to choose "born". This is a hint for the prompt engineer, to change the prompt to use a lowercase “b” instead.


key: chroma:document
string_value:
The obvious limitation of this method is that it requires a Python API to process the output logits of the LLM. This means that, at least for now, it cannot be used with OpenAI ChatGPT and similar API-based solutions because those APIs don’t offer that kind of access.


key: chroma:document
string_value:
ReLLM (an acronym for ‘Regular Expressions for Language Model Completions’) is another library that works at the token level. After providing a prompt and a regular expression, ReLLM filters non-matching tokens pre-generation. For each token, ReLLM tests every possible completion against a partial regex. For the potential completions that do not match the pattern, ReLLM masks the logits so that the language model does not generate them. Here is an example of GPT2 responses with and without ReLLM:

ParserLLM extends ReLLM to handle context-free grammar and a parser generator to determine valid next tokens for an LLM generation.


key: chroma:document
string_value:
ParserLLM extends ReLLM to handle context-free grammar and a parser generator to determine valid next tokens for an LLM generation.

Another key strategy when calling the LLM is tuning certain parameters and leveraging the features exposed by the LLM. It is worth taking the time to familiarize yourself with the documentation and parameters of the model you intend to use.


key: chroma:document
string_value:
One example of this is OpenAI’s function calling, which “allows you to more reliably get structured data back from the model”. Another strategy is using the JSON mode, although OpenAI’s documentation includes the following disclaimer: “While this does work in some cases, occasionally the models may generate output that does not parse to valid JSON objects”.

Another strategy is using OpenAI’s Moderation endpoint, free to use when monitoring the inputs and outputs of OpenAI APIs, which classifies the following categories: hate, threatening, harassment, self-harm, sexual, minors, and violence. The endpoint returns the following fields:


key: chroma:document
string_value:
flagged : Set to true if the model classifies the content as violating OpenAI's usage policies, false otherwise.

: Set to if the model classifies the content as violating OpenAI's usage policies, otherwise. categories : Contains a dictionary of per-category binary usage policies violation flags. For each category, the value is true if the model flags the corresponding category as violated, false otherwise.


key: chroma:document
string_value:
: Contains a dictionary of per-category binary usage policies violation flags. For each category, the value is if the model flags the corresponding category as violated, otherwise. category_scores : Contains a dictionary of per-category raw scores output by the model, denoting the model's confidence that the input violates the OpenAI's policy for the category. The value is between 0 and 1, where higher values denote higher confidence. The scores should not be interpreted as probabilities.


key: chroma:document
string_value:
Most LLMs also provide parameters that can be tuned. Parameters such as temperature , top-k and top-p sampling can be used to somewhat control the model’s output. The temperature strikes a balance between randomness and predictability, adjusting the probability distribution used for token selection. A higher temperature increases randomness by flattening the distribution, while a lower temperature concentrates probability on a smaller set of tokens, resulting in more deterministic output. Top-k just considers the K most likely tokens and ignores all others, then samples one of them by considering their likelihood scores. Top-p dynamically sets the size of the shortlist of tokens whose sum


key: chroma:document
string_value:
all others, then samples one of them by considering their likelihood scores. Top-p dynamically sets the size of the shortlist of tokens whose sum of likelihoods does not exceed a certain value p.


key: chroma:document
string_value:
Another interesting parameter is known as logit_bias . It can be used to exclude words (or actually, tokens) from the generated response. The way it works is the following: a user can specify a bias (positive or negative) to add/subtract to the logits of a specific token generated by the model before sampling.


key: chroma:document
string_value:
As stated in OpenAI's API documentation, the impact of specific values will differ depending on the individual model used. Typically, values ranging between -1 and 1 tend to decrease or increase the likelihood of a token being selected, respectively. In contrast, extreme values such as -100 or 100 are likely to result in either a complete ban or an exclusive selection of the relevant token.


key: chroma:document
string_value:
More advanced techniques can be used to mitigate certain behaviors. For example, DoLa (Decoding by Contrasting Layers) is a decoding strategy for reducing hallucinations with pretrained LLMs. The approach obtains the next-token distribution by contrasting the differences in logits obtained from projecting the later layers versus earlier layers to the vocabulary space, exploiting the fact that factual knowledge in an LLMs has generally been shown to be localized to particular transformer layers.

Bear in mind that a caveat of Strategy 2 is the need to have a more granular access to the LLM (e.g. logits), because APIs don’t tend to expose everything that might be needed.


key: chroma:document
string_value:
Strategy 3: post-generation control

Prompts are a flexible way to implement limitations, but they cannot guarantee that the same model will generate the same output for the same prompt.

Security guardrails are a way of letting the user add structure, type, and quality guarantees to the outputs of LLMs. Examples of guardrails include semantic validation (checking for bias or harmful content in generated text, checking for bugs in generated code, etc.) and enforcing structure and type guarantees (e.g. JSON).


key: chroma:document
string_value:
This technique works with the LLMs output, providing a posteriori methods that guarantee that the output will match certain criteria: regular expressions, follow a JSON schema, be parseable into a Pydantic model, etc.

Examples

Guardrails does pydantic-style validation of LLM outputs (including semantic validation), takes corrective actions (e.g. re-asking LLM or filtering out the invalid output) when validation fails, and enforces structure and type guarantees (e.g. JSON). Under the hood, it provides a file format ( .rail ) for enforcing a specification on an LLM output, and a lightweight wrapper around LLM API calls to implement this spec.


key: chroma:document
string_value:
Let’s review some validators from the library to better understand the kind of checks that can be applied. Some are used to check code (e.g. BugFreePython , BugFreeSQL , DetectSecrets , ExcludeSqlPredicates , SqlColumnPresence ). Others have to do with text format (e.g. EndsWith , LowerCase , OneLine , RegexMatch , TwoWords ) and some validate the text’s content itself (e.g. Provenance , IsProfanityFree , CompetitorCheck ). Users can also create their own custom validator that enforces what the application is intended to do.

Guardrails logs all of the steps it takes to enforce the RAIL spec, which can later be inspected. Here’s a diagram with the full flow:


key: chroma:document
string_value:
Guardrails logs all of the steps it takes to enforce the RAIL spec, which can later be inspected. Here’s a diagram with the full flow:

One validator called OnTopic can be used to verify that a response is related to a provided topic (or one of many provided topics). Under the hood, it uses an ensemble of Zero-Shot classification model combined with an LLM to determine if the condition is met. Here’s an example that illustrates how to use it:

We recently collaborated with Guardrails in the definition and implementation of some Validators, including OnTopic.

Alternatively, NeMo Guardrails (an open-source toolkit developed by NVIDIA) and Guardrail ML offer similar features.


key: chroma:document
string_value:
Alternatively, NeMo Guardrails (an open-source toolkit developed by NVIDIA) and Guardrail ML offer similar features.

In addition to guardrails, there are countless strategies when processing the response. One example is SelfCheckGPT, a zero-resource hallucination detection solution that can be applied to black-box systems. The basic idea behind it is that if an LLM knows a given concept, sampled responses are likely to be similar and contain consistent facts. By sampling multiple responses from an LLM, one can measure information consistency between the different responses and determine if statements are factual or hallucinated.


key: chroma:document
string_value:
Output parsers are a simple way to get more structured information from the LLM’s response or enforce an expected format. LangChain, a framework for developing applications powered by language models, offers an implementation and this example to illustrate how to use it:

There are other similar libraries (e.g. Guidance, JsonFormer, Outlines, Griptape) that also work by using different techniques to enforce an output format.

Conclusions

Each of these techniques has its unique strengths and weaknesses, and their applicability depends on the specific requirements of the task at hand and the kind of output filtering intended.


key: chroma:document
string_value:
Prompt tuning is a powerful tool for guiding the model's output. By carefully crafting the input prompt, one can influence the model's responses. However, this method requires a deep understanding of how the model interprets prompts and may not always yield consistent results due to the inherent randomness in the model's output.

Logit filtering is a more advanced technique that involves manipulating the model's logits to prevent it from generating certain words or phrases. This method can be highly effective, but it requires a deep understanding of the model's inner workings and can be computationally expensive.


key: chroma:document
string_value:
Parameter tuning, such as adjusting the temperature and top-k sampling, provides a way to control the randomness and diversity of the model's output. While these methods offer a degree of control, they do not guarantee the elimination of harmful or inappropriate content.

Guardrails are a proactive approach to limit the model's output by setting predefined rules or boundaries. This method can be effective in preventing the model from generating certain types of content. However, it may also limit the model's creativity and versatility, and there's always a risk that some unwanted content may slip through.


key: chroma:document
string_value:
Parsing involves analyzing the model's output after it has been generated, allowing for the removal or modification of unwanted content. This method can be effective, but it is reactive rather than proactive, meaning that harmful or inappropriate content may be generated before it is caught and removed.


key: chroma:document
string_value:
In conclusion, the choice of strategy for controlling and limiting the output of LLMs depends on the specific needs and constraints of the task at hand. A combination of these techniques may often be the best approach, providing a balance between harnessing the power of LLMs and mitigating their potential risks. Also keep in mind that if you want to limit a particular type of output, there may be more strategies for that specific behavior. For example, this survey gathers many hallucination mitigation techniques.


key: chroma:document
string_value:
It's important to note that the field of controlling and limiting the output of LLMs is an active area of research. As we continue to explore the capabilities and challenges of these models, new techniques and tools are being developed continuously. This ongoing innovation is expanding our toolkit for managing LLMs, enabling us to better align these models with our goals and values. The strategies discussed in this post represent the current state of the art, but we can expect to see further advancements in the near future. As we navigate this evolving landscape, the key is to remain adaptable, continually learning and applying new methods as they emerge.


key: description
string_value:
Discover the pivotal role of MLOps in AI-driven organizations. Dive deep into its significance, learn how it bridges AI dynamics with business objectives, and explore key stages to optimize AI deployments for tangible results. Embark on the MLOps journey with us


key: chroma:document
string_value:
As Artificial Intelligence (AI) continues to revolutionize industries and redefine operational strategies, the challenge emerges: how can enterprises efficiently deploy, manage, and scale these AI solutions? Navigating the intricacies of AI development and deployment can be a daunting task. This is where MLOps shines. Merging the best of Machine Learning with operational practices, MLOps is more than just a technical methodology—it's the strategic compass for AI-driven enterprises. In this exploration, we'll delve into the significance of MLOps, highlighting its pivotal role in translating the vast potential of AI into tangible business results. Embark with us on this illuminating journey.


key: chroma:document
string_value:
Why MLOps is Crucial for AI-Driven Organizations

AI promises transformative power. From automation to predictive analytics, its potential applications are vast and varied. But harnessing that power in a way that aligns with business goals requires more than just stellar algorithms or accurate models. It requires a structured process that integrates AI advancements seamlessly into an organization's workflow. This is where MLOps comes to the fore.

Connecting AI Objectives with Business Ambitions


key: chroma:document
string_value:
Connecting AI Objectives with Business Ambitions

Before delving deep into the technicalities, it's paramount to understand the bigger picture: How does AI align with the overarching business objectives? Think of AI as the engine and MLOps as the steering wheel. While AI powers the drive, MLOps ensures the journey is in the right direction, aligning with the destination, the organization's business ambitions.

With MLOps, organizations can:

Establish clear metrics that tie AI performance to business outcomes.

Streamline the feedback loop, ensuring AI developments continually cater to evolving business needs.


key: chroma:document
string_value:
Streamline the feedback loop, ensuring AI developments continually cater to evolving business needs.

Foster collaboration between data scientists and business stakeholders, ensuring both are on the same strategic page.

Ensuring reproducibility, efficiency, and speed of AI deployments

MLOps isn't just about direction; it's also about efficiency and momentum. As AI-driven solutions scale, ensuring consistent performance becomes a challenge. Variability in data, environmental conditions, and deployment platforms can introduce unintended consequences or inefficiencies.

Key benefits of MLOps in this arena include:


key: chroma:document
string_value:
Key benefits of MLOps in this arena include:

Reproducibility : With standardized processes and environments, models can be deployed and re-deployed consistently, ensuring stable performance over time.

: With standardized processes and environments, models can be deployed and re-deployed consistently, ensuring stable performance over time. Efficiency : MLOps introduces practices like automated testing and continuous integration, minimizing manual interventions and reducing the chances of errors.


key: chroma:document
string_value:
: MLOps introduces practices like automated testing and continuous integration, minimizing manual interventions and reducing the chances of errors. Speed: Through streamlined pipelines, AI solutions can be swiftly moved from development to deployment, ensuring businesses can rapidly capitalize on AI insights and advantages.


key: chroma:document
string_value:
In essence, MLOps serves as the bridge between the dynamism of AI and the structured ambitions of business enterprises. It's the linchpin ensuring AI doesn't just operate in a vacuum but is intricately and strategically tied to the broader organizational tapestry. By instilling reproducibility, efficiency, and speed into AI deployments, MLOps ensures that AI-driven organizations are not just running, but running in the right direction, at the right pace.

The MLOps Journey: Key Stages Explained

Stage 1: Getting your Data in order


key: chroma:document
string_value:
The MLOps Journey: Key Stages Explained

Stage 1: Getting your Data in order

The first step of any MLOps journey starts at the heart of every Machine Learning project: the data. Quality data is the foundation of successful AI, and initial efforts to enhance its usability can drastically boost the efficacy of your Data Science teams.

Common data challenges: Lack of centralization: Data dispersed across numerous databases creates a barrier to efficient data integration and interpretation.


key: chroma:document
string_value:
Data dispersed across numerous databases creates a barrier to efficient data integration and interpretation. Poor data quality: Unreliable datasets plagued with missing values, free-text categorical descriptors, and other basic issues compromise the validity of results.

Unreliable datasets plagued with missing values, free-text categorical descriptors, and other basic issues compromise the validity of results. Lack of documentation: Without a centralized repository detailing data specifics, teams over-rely on the tacit knowledge of seasoned members to navigate the data maze.


key: chroma:document
string_value:
Without a centralized repository detailing data specifics, teams over-rely on the tacit knowledge of seasoned members to navigate the data maze. No reusability: If each model requires unique SQL queries and lacks standard preprocessing, the wheel is being reinvented every time – an inefficient and error-prone process.

Strategies for Data enhancement:

Centralization: Use a Data Warehouse like BigQuery or Snowflake to consolidate data. When this isn't viable, Data Catalogs like Alation, Amundsen, or AWS Glue can provide an organized directory of information across services.


key: chroma:document
string_value:
Use a Data Warehouse like BigQuery or Snowflake to consolidate data. When this isn't viable, Data Catalogs like Alation, Amundsen, or AWS Glue can provide an organized directory of information across services. Data quality improvement: Regular manual checks by Data Scientists can flag potential issues. Open communication channels between data creators (typically Engineering and Data Engineering teams) and consumers (like Data Science teams) allow for collaborative issue resolution.


key: chroma:document
string_value:
Regular manual checks by Data Scientists can flag potential issues. Open communication channels between data creators (typically Engineering and Data Engineering teams) and consumers (like Data Science teams) allow for collaborative issue resolution. Prioritize documentation: Data documentation is as pivotal as code documentation. Templates like Datasheets for Datasets, the Aether Data Documentation Template, or Data Nutrition Labels offer frameworks to record essential data characteristics and provenance.


key: chroma:document
string_value:
Data documentation is as pivotal as code documentation. Templates like Datasheets for Datasets, the Aether Data Documentation Template, or Data Nutrition Labels offer frameworks to record essential data characteristics and provenance. Promote reusability: Encourage shared, stored queries. While a simple .sql file in a repository might suffice, creating views in your Data Warehouse can be even more beneficial. Regularly allocate time for team members to generalize and share these queries. Aim to shift data cleaning to the ETL phase, providing tables with pre-cleaned data.


key: chroma:document
string_value:
Although vital, this blogpost doesn't delve into data security, compliance, and user privacy. However, mastering these foundational data steps allows for progressing to more intricate facets of your MLOps infrastructure.

Stage 2: Experiment and model tracking

As data teams progress from ensuring data quality and usability, the next pivotal stage in the MLOps journey is experiment and model tracking. This stage revolves around the systematic documentation, monitoring, and management of experiments and models. But why is this process so essential, and what challenges might one face?

Challenges in experimentation:


key: chroma:document
string_value:
Difficulty reproducing results: Without structured tracking, re-creating experiments to achieve the same results becomes a maze. Losing track can translate to lost time, effort, and critical insights, acting as significant hindrances. Collaboration barriers: The absence of a centralized system can lead to disjointed team efforts. Imagine multiple data scientists working independently on their Jupyter notebooks without versioning, causing a significant overlap and rework. Inefficient experimentation: Tracking the myriad of experiments, understanding their outcomes, and discerning the reasons for their success or failure can get chaotic. An unorganized approach can slow down the


key: chroma:document
string_value:
understanding their outcomes, and discerning the reasons for their success or failure can get chaotic. An unorganized approach can slow down the experimentation process considerably. Inability to track model performance: Choosing the best model becomes an uphill task when there’s no systematic way to compare the performance metrics of different models.


key: chroma:document
string_value:
As organizations scale, these challenges can snowball, leading to considerable bottlenecks and stymied development. However, addressing these issues head-on can smooth out the workflow significantly.

Solutions and approaches

Standardized experimentation environment: The foundation lies in ensuring that the entire Data Science team operates in a standardized setting. Tools like pyenv coupled with poetry or pip-tools can aid in managing Python dependencies. Moreover, integrating Docker into the workflow helps create containerized environments, making them shareable and consistent across various stages. Environment management with Pyenv


key: chroma:document
string_value:
Embrace experiment tracking tools: Tools dedicated to tracking experiments can be game-changers. They meticulously document everything from hyperparameters and input data to performance metrics, enabling data scientists to draw comparisons, spot trends, and assimilate learnings from past experiments. Examples of experiment-tracking tools include Weights & Biases, Comet ML, Studio by Iterative.ai, and even TensorBoard. You may also track your experiments directly on GCP Vertex AI or AWS SageMaker Experiments features.


key: chroma:document
string_value:
Tools dedicated to tracking experiments can be game-changers. They meticulously document everything from hyperparameters and input data to performance metrics, enabling data scientists to draw comparisons, spot trends, and assimilate learnings from past experiments. Examples of experiment-tracking tools include Weights & Biases, Comet ML, Studio by Iterative.ai, and even TensorBoard. You may also track your experiments directly on GCP Vertex AI or AWS SageMaker Experiments features. Adopting a model registry: Think of a Model Registry as a centralized library for your trained models. Such a registry helps streamline model lifecycle management, making collaboration a breeze and ensuring that


key: chroma:document
string_value:
library for your trained models. Such a registry helps streamline model lifecycle management, making collaboration a breeze and ensuring that insights and models are retained and not lost in the shuffle. While GCP Vertex AI and AWS Sagemaker offer built-in registry capabilities, they often require a custom interaction layer to suit specific organizational needs.


key: chroma:document
string_value:
In this stage, the importance of structured and systematic tracking of experiments and models cannot be understated. Adopting the right tools and practices ensures that the organization can scale its Machine Learning efforts without getting entangled in its own web of experiments.

Stage 3: Standardizing production process

Upon streamlining data quality and ensuring a well-documented trail of experiments and models, teams often hit a new roadblock: transitioning models from proof of concept to production swiftly and efficiently.

Production challenges


key: chroma:document
string_value:
Production challenges

Lack of reusable code: Every model's deployment seems like reinventing the wheel, with custom code and infrastructure required every single time.

Every model's deployment seems like reinventing the wheel, with custom code and infrastructure required every single time. Maintainability concerns: Ensuring active models function flawlessly in production becomes a challenge with the increasing complexity of code and infrastructure.


key: chroma:document
string_value:
Ensuring active models function flawlessly in production becomes a challenge with the increasing complexity of code and infrastructure. Delay in deployment: The gap between acknowledging a model's efficacy and its first live prediction in production feels extended and cumbersome.

To swiftly navigate these challenges, the key lies in consistency in development environments and strategic model-serving approaches.

Model serving approaches

There are two predominant methods for serving Machine Learning models. These methods, each with its own unique approach, are integral to the process of making predictions and interpreting results. Let's take a deeper look:


key: chroma:document
string_value:
Batch Predictions: This method involves a series of systematic steps. Initially, scripts or jobs pull data, which is then preprocessed. The model is loaded, predictions are made, and these predictions are diligently stored for future reference.


key: chroma:document
string_value:
This method involves a series of systematic steps. Initially, scripts or jobs pull data, which is then preprocessed. The model is loaded, predictions are made, and these predictions are diligently stored for future reference. Online Prediction APIs: This process is a bit more immediate. The model's raw features are inputted, undergoing only light preprocessing. Predictions are swiftly made, and the results are immediately presented, providing real-time insights.


key: chroma:document
string_value:
Guidelines for Model Serving Instead of reinventing the wheel each time, standard practices streamline the decision-making process concerning model formats, infrastructure, and serving frameworks. Platforms like SageMaker Models (AWS), VertexAI endpoints (GCP), MLFlow Models (Databricks), or NVIDIA Triton offer an almost plug-and-play experience, simplifying model deployment to a few clicks or commands.


key: chroma:document
string_value:
Standard Production Environments One may wonder: how can we align AI models with the plethora of deployment options? The answer resides in the standardization of production environments, a technique that, when correctly mirrored to development environments, can result in notable time and effort savings. The use of dependency managers (examples include poetry and pip-tools) adequately caters to a majority of development needs. However, when transitioning into production, a more robust solution is often required for consistency and predictability. In this respect, Docker stands out as a reliable choice, effectively minimizing unwelcome surprises during deployment.


key: chroma:document
string_value:
The same way you would use a dependency manager to standardize a development environment, we recommend using Docker for a standard production environment.

Myriad strategies exist to standardize and expedite the transition from PoC to production. Identifying the perfect blend that complements the team's capabilities and the tech stack is pivotal. Prioritizing consistency in both development and production environments ensures that models trained are seamlessly integrated into the standardized serving process.

Stage 4: Model and Data Monitoring


key: chroma:document
string_value:
Stage 4: Model and Data Monitoring

As organizations standardize the serving process of models, the logical next step is scalability. Initially, manual oversight might suffice for monitoring performance, but true scalability demands automation in model and data monitoring. Without this automated vigilance, a model's efficiency can plummet, adversely affecting end-users, oftentimes unbeknownst to the organization until complaints start pouring in.

Recognizing the need for monitoring


key: chroma:document
string_value:
Recognizing the need for monitoring

External feedback: Are most of your insights on model performance stemming from external feedback? If so, it’s a red flag. Efficient systems should internally detect and rectify issues before end-users even notice.

Are most of your insights on model performance stemming from external feedback? If so, it’s a red flag. Efficient systems should internally detect and rectify issues before end-users even notice. Emerging biases: If the model's outputs display a consistent bias towards specific segments, it's indicative of data bias, which can be a subtle, yet detrimental issue.


key: chroma:document
string_value:
If the model's outputs display a consistent bias towards specific segments, it's indicative of data bias, which can be a subtle, yet detrimental issue. System failures: Just as in traditional software, an ML system's failure should be promptly detected and rectified. Efficient monitoring can be a savior here.

Solutions to consider

Data monitoring: Data is the lifeblood of models. Monitoring it can preemptively flag issues like data drift, biases, or missing values. It ensures the model's training data remains representative, maintaining the integrity of its outputs. Moreover, detecting shifts in data distribution can hint towards the need for model retraining.


key: chroma:document
string_value:
Data is the lifeblood of models. Monitoring it can preemptively flag issues like data drift, biases, or missing values. It ensures the model's training data remains representative, maintaining the integrity of its outputs. Moreover, detecting shifts in data distribution can hint towards the need for model retraining. Model monitoring: This is a continuous process of overseeing key performance metrics, often set in collaboration with stakeholders. Early identification of performance dips or biases allows timely interventions – from retraining to fine-tuning. Furthermore, resource utilization insights can lead to cost optimizations.


key: chroma:document
string_value:
This is a continuous process of overseeing key performance metrics, often set in collaboration with stakeholders. Early identification of performance dips or biases allows timely interventions – from retraining to fine-tuning. Furthermore, resource utilization insights can lead to cost optimizations. Observability: Beyond monitoring, observability aids in understanding the "why" behind any issues, facilitating quicker resolutions. This includes deep dives into logs, metrics, and system traces.


key: chroma:document
string_value:
Generic cloud monitoring and observability tools, such as Datadog, AWS CloudWatch, GCP Cloud Operations, or Azure Monitor, provide broad oversight. Sentry and Bugsnag excel in issue detection and troubleshooting. For specialized ML monitoring, Whylabs stands out, offering in-depth insights for data and ML systems. The beauty lies in the harmonious coexistence of these tools. Instead of choosing one over the other, they can be synergistically integrated to craft a comprehensive monitoring and observability solution.

Take the next step in your MLOps journey


key: chroma:document
string_value:
Take the next step in your MLOps journey

We've only scratched the surface of what's possible in the Data Science ecosystem. Beyond these preliminary stages, a myriad of strategies, nuances, and insights await you, insights that could revolutionize your organization's approach to AI and data.

At Tryolabs, we pride ourselves on being more than mere service providers. We are your partners and your strategic allies on the road to MLOps excellence. Our team, boasting of seasoned experts equipped with extensive knowledge and experience, is committed to ensuring your MLOps journey is smooth, strategic, and successful.

Why trust Tryolabs? Let's delve into what makes us stand out:


key: chroma:document
string_value:
Why trust Tryolabs? Let's delve into what makes us stand out:

Expertise: Our team comprises seasoned professionals who have honed their skills over years of practice. Not only experts in MLOps, but also in various aspects of Machine Learning, and data science. Experience: With a rich and diverse portfolio of complex MLOps projects under our belt, we are adept at navigating and addressing varied challenges. Strategic insight: We don't just execute projects, we strategize. We take the time to understand your business, goals, and specific needs, allowing us to provide you with the most effective and efficient solutions.


key: chroma:document
string_value:
As your MLOps partner, Tryolabs brings deep expertise, real-world experience, and commitment to guide you every step of the way. We understand the complexities of MLOps and have the solutions to help you succeed.

Ready to start your MLOps journey? Here are 3 ways we can help:


key: chroma:document
string_value:
Immerse yourself in our comprehensive white paper on MLOps. This guide provides actionable strategies on team building, governance, and more. It's the roadmap you need to achieve data-driven excellence. Download the Tryolabs MLOps White Paper now. Have questions? Want hands-on guidance? Our experts are ready to advise and accompany you. Reach out to us at hello@tryolabs.com and we'll help make your MLOps implementation smooth and successful. Bring the learning to your team with a customized MLOps Talk. Our in-house sessions deliver practical insights to skill up your staff. Book an AI Talk today and let's explore MLOps together.


key: chroma:document
string_value:
With Tryolabs as your guide, you don't have to navigate the MLOps path alone. Take the first step now and let our experts help drive your success. The future of AI is bright when we work together!


key: link
string_value:
https://tryolabs.com/blog/2023/05/19/mastering-pricing-strategies-vol-2-exploring-complex-market-scenarios


key: description
string_value:
Explore complex scenarios with our advanced Market Simulator in this post. Learn how to balance exploration and exploitation with Bootstrap integration for improved market adaptability.


key: chroma:document
string_value:
If you’re new to the topic of market simulations, we highly recommend reading the first blog in our series to get familiar with the basics. In that post, we briefly introduced the concepts of demand forecasting and pricing optimization. We also explained why this tool is critical to providing the best pricing solutions. Moreover, we took a deep dive into the inner workings of our simulation process and the mathematical definition of our demand function. Finally, we used the simulator to test a learning model capable of making pricing decisions and learning from them within a market scenario.


key: chroma:document
string_value:
This second post will explore more complex market scenarios to test and evaluate a learning model. We’ll mainly focus on balancing exploration and exploitation while optimizing. We’ll further discuss how adding Bootstrap to the learning model’s output enables that balance.


key: chroma:document
string_value:
In this section, we'll quickly review our market simulator's building blocks. If you're already familiar with it, feel free to skip to the next section. As a starting point, let's revisit the mathematical definitions that determine our market. The first definition we want to review is the underlying expected demand Q ˉ \bar{Q} Qˉ​, a function of price and, optionally, other exogenous factors. Furthermore, once we define our expected demand, we can also calculate the expected profit function as a function of the expected demand, the price, and the given cost. It's also important to remember that our historical data is modeled by sampling a Poisson distribution, using Q ˉ \bar{Q} Qˉ​ as the


key: chroma:document
string_value:
the given cost. It's also important to remember that our historical data is modeled by sampling a Poisson distribution, using Q ˉ \bar{Q} Qˉ​ as the corresponding λ \lambda λ for each price.


key: chroma:document
string_value:
Figure 1. a) Three important equations to keep in mind, defining, from top down: expected demand, expected profit, and realized demand. b) The expected demand curve is represented in blue, the expected profit is represented in red, the realized demand can be seen as the green dots, and the true optimum profit is shown as a blue dot.


key: chroma:document
string_value:
We will also briefly reiterate how the simulation works. It's a dynamic, fixed-duration timestep (think daily) market simulation where we use all the historical data available to train a specific demand forecasting model at each step. Once we obtain the values for the exogenous factors, this trained model will allow us to forecast the expected demand ( Q ˉ ∗ \bar{Q}^* Qˉ​∗) for any price. These exogenous factors are a part of the market state, so the simulator actually produces their values. With the model estimating Q ˉ ∗ \bar{Q}^* Qˉ​∗ and the simulated exogenous factors X X X, we can define a price range (the x-axis in the plot) and obtain our forecasted demand and profit. By maximizing


key: chroma:document
string_value:
simulated exogenous factors X X X, we can define a price range (the x-axis in the plot) and obtain our forecasted demand and profit. By maximizing that forecasted profit, we find our optimum price. Finally, the optimum price is set in the market, and the (simulated) market responds with its corresponding realized sales units, generating a new data point for our dataset in the process. The simulation now moves to the next time step and repeats the same process as before. In the following figure, we can see the same simulation animation as in the previous blog.


key: chroma:document
string_value:
Figure 2. Animation of a market simulation loop with up to 350 iteration steps.

In our previous post, we discussed the assembly of the market simulator and modeled a single-item market scenario with a specific expected demand curve. By setting the market conditions, we were able to see how well the learning model, XGBoost, predicted the underlying expected demand. The simulator allowed us to understand the model's strengths and weaknesses since we knew precisely what the objective demand function was. Figure 2 illustrates that the model learned how to provide the optimal price in the given situation, as the suggested optimum further approached the true optimum with each iteration.


key: chroma:document
string_value:
Upping the game

Our last blog post also showed how XGBoost successfully handled a pricing optimization problem given a particular scenario with fast sales in our market simulation. We now want to see how the model fares in other, more challenging scenarios.

Versatility is key


key: chroma:document
string_value:
Versatility is key

We designed the demand function’s definition with flexibility in mind. So modeling various demand curves is simply a matter of adjusting the values for α , β {\alpha}, {\beta} α,β, and γ {\gamma} γ. By doing so, we get to model different conditions of interest. We can change the speed at which our simulated item can sell, the location of the elastic region of the demand, and how steep this region is. We can also model both perishable and non-perishable products.


key: chroma:document
string_value:
In our previous post, we worked with a demand function that described a perishable product with a high daily sales volume that would saturate at lower prices, seeing no sales increase after a specific low price was reached.



That means that no further price reduction will impact the number of units you sell after hitting a certain price. This demand curve is displayed on the right in Figure 3a. Figure 3.a) Fast sales saturating demand curve.

Another demand function we were interested in testing out is one with both slow sales and the previously explained market saturation condition.


key: chroma:document
string_value:
Another demand function we were interested in testing out is one with both slow sales and the previously explained market saturation condition.



This slow demand curve is shown in Figure 3b. Notice that the y-axis is much smaller now. In particular, the expected demand, shows sales of less than 1 unit per period.



This type of demand is often referred to as 'sparse' because sales don't occur at every time step and the item's natural sales velocity is slow. Figure 3.b) Sparse sales saturating demand curve.


key: chroma:document
string_value:
The last demand function we wanted to try out exhibits an exponentially decaying pattern, as shown in Figure 3c. With this particular model, we observe a continual surge in sales with every reduction in price, and the saturation effect is no longer present.



This type of demand model corresponds to a non-perishable product that the market would stockpile as the price decreases. Figure 3.c) Exponential demand curve.

A bit of history


key: chroma:document
string_value:
A bit of history

Now we have three distinct scenarios that we want to test our model on. But we also wanted to understand the impact of the historical data's price placement on our model's ability to learn. Figure 4 clearly shows what we mean by this. We have four different historical datasets that vary in where the historically tried prices are located. We've analyzed some simulations with a significant standard deviation in historical prices, but our experience has shown that this behavior is rather uncommon in real-life datasets. So we won't discuss those simulations here.

Figure 4. Different demand function and historical dataset combination scenarios to evaluate our model in.


key: chroma:document
string_value:
Figure 4. Different demand function and historical dataset combination scenarios to evaluate our model in.

Putting it together

Now that we have different demand functions and historical datasets to try, we’ll combine them into the nine situations in Figure 5. Given those market realities, we’ll be testing how good our demand forecasting model is at predicting the expected demand in the daily repricing situation of our simulator. We’ll also carefully analyze if the lack of exploration becomes an insurmountable problem for our model in any scenario.

Different historical datasets we will use to combine with our fast sales demand function to evaluate our model in.


key: chroma:document
string_value:
Different historical datasets we will use to combine with our fast sales demand function to evaluate our model in.

Different historical datasets we will use to combine with our slow sales demand function to evaluate our model in.

Low price historical dataset exponential demand function scenario to evaluate our model in.


key: chroma:document
string_value:
Low price historical dataset exponential demand function scenario to evaluate our model in.

Once we run all the pertinent simulations with the configurations mentioned above, we can see that the proposed demand forecasting model had some issues. In particular, all the scenarios with sparse sales were performing poorly, as shown in the summary in Figure 6. In the displayed table, we can see two metrics to consider in determining whether or not the strategy used was successful in a given scenario. These measures correspond to the percentage of cumulative profit losses concerning the actual expected optimum incremental profit and the rate of steps with less than 5% of profit losses.


key: chroma:document
string_value:
The first value was introduced in the first blog post of this series and was calculated as the percentual difference between the accumulated expected profit, had we always used the accurate optimum price, and the accumulated profit made using the predicted optimum price. A perfect score here would be 0, indicating we lost no profit even without knowing the actual demand curve.

The second measure we consider is the percentage of iteration steps in which we experienced a loss of less than 5% of the total achievable profit for that step. In this case, a perfect score would be 100% indicating all steps lost less than 5% of the attainable profit in each step.


key: chroma:document
string_value:
Ideally, we want our strategy to minimize the first value mentioned and maximize the second. We use these values to evaluate our strategy's overall success in the different scenarios. As you can see, it's not performing at its best at the moment…

Figure 6. Results for the different demand and historical data combined scenarios.

But one of the advantages of having the simulator is that we can now inspect model behavior in a controlled environment and, knowing what should have happened, understand where the model went astray.


key: chroma:document
string_value:
We'll now zoom into one of the unsuccessful scenarios to determine what problems we can identify. The situation in Figure 7 corresponds to a market with sparse sales and historical data located just below the optimum price. Looking at the animation, we understand why this scenario is so demanding for the learning model. In Figure 7, we see that sparse sales create a 0 inflation problem. That means that we're going to have (price, sales) points with 0 sales for two very different reasons: a) because the price is too high for there to be any sales, and b) the product hasn't sold because not enough time has passed yet. That creates a challenging training set because the model can't distinguish


key: chroma:document
string_value:
and b) the product hasn't sold because not enough time has passed yet. That creates a challenging training set because the model can't distinguish between these two situations, and because we optimize for maximum profit, that region is of very little interest because its forecasted profit is rather low. As that price region isn't of interest, the model has very little incentive to explore those prices, failing to improve its estimates, thus missing the optimum price entirely.


key: chroma:document
string_value:
Figure 7. Market simulation of a sparse type of demand and historical prices situated below the known optimum price.

We exemplified the problem with the situation shown in Figure 7, but all the failing scenarios present variations of the same problem. Fortunately, this isn't an unknown phenomenon. This is a widely known problem in Reinforcement Learning and is known as the Exploration-Exploitation dilemma.

Issues with Exploration & Exploitation


key: chroma:document
string_value:
To ensure we identified the problem correctly, we designed an experiment on the other edge of the spectrum. Instead of doing pure exploitation, we’ll go down the road of pure exploration. This is not what we recommend in actual production, as price exploration has a cost, but as a theoretical experiment, it’s helpful to understand the characteristics of these approaches. So, instead of optimizing for the maximum forecasted profit, we built an agent that randomly chooses a new price from the entire price range. This strategy would undoubtedly force exploration and obtain better price space coverage. A more extensive coverage should enable the model to better grasp the expected demand curve.


key: chroma:document
string_value:
exploration and obtain better price space coverage. A more extensive coverage should enable the model to better grasp the expected demand curve. So we tried it, and voila! As you can see in Figure 8, exploration allows the model to improve it's estimates of the demand curve across the entire price range. The corresponding animation shows that the light blue curve is very similar to the blue curve and as time advances it becomes more and more so.


key: chroma:document
string_value:
Figure 8. Animation of the same scenario as Figure 7 but with a strategy in which a random price is chosen at each time step.

We needed to implement a sound exploration approach into our modeling strategy that would allow us to understand the demand curve in the regions where we have no data, but that's also efficient and doesn't waste time exploring prices that produce fewer gains or losses. And this exploration approach should work across all scenarios. Considering this problem and discussing potential techniques, we found many similarities with the Multi-Armed Bandit (MAB) problem.


key: chroma:document
string_value:
As a brief overview, the MAB problem involves finding the best resource allocation strategy among multiple options with uncertain outcomes. To exemplify, say you have a set of slot machines, each giving a different payout. You want to determine through trial and error which slot machine to pull to maximize your gains. Typically, one would do this by pulling each slot machine's lever multiple times and building an estimate of the distribution of the expected reward of each try. After pulling each lever, you update the distribution estimates and pick the slot machine with the highest expected payout. You can quickly see that this strategy will have the same exploration problem as we do in our


key: chroma:document
string_value:
pick the slot machine with the highest expected payout. You can quickly see that this strategy will have the same exploration problem as we do in our simulator. If we happen to receive higher samples from a slot machine that initially has a lower expected payout, we will continue to use that one and refrain from testing the others.


key: chroma:document
string_value:
We can see this exemplified in Figure 9. The second slot machine had the largest expected payout when the image was taken. But if you look at the actual payouts, you can see that the first slot machine will give the largest payout. One popular way to solve this problem is the Upper Confidence Bound algorithm.

Figure 9. Four different slot machines with their corresponding expected payouts in blue, true payouts in red, and their confidence intervals.


key: chroma:document
string_value:
The Upper Confidence Bound algorithm is based on the principle of Optimism in the Face of Uncertainty. Before we explain that, it's important to recall that the sample mean estimator has a particular variance that depends on the sample size. The more we pull on a specific slot machine's lever, the more we can trust our sample mean estimator. In the image above, this translates to smaller confidence bands around the expected result as the confidence interval is reduced. Optimism in the Face of Uncertainty results from choosing the lever with the highest payout + uncertainty. In practical terms, this means we take the upper bound of the box we can see in Figure 9 instead of the hightop-most


key: chroma:document
string_value:
highest payout + uncertainty. In practical terms, this means we take the upper bound of the box we can see in Figure 9 instead of the hightop-most blue line.


key: chroma:document
string_value:
If the newly chosen lever has a bigger payout, its distribution will shift upwards and gradually increase its likelihood of being picked. If it turns out we were wrong and that slot machine pays less, then the sample mean's variance will decrease, making the box smaller and the lever less likely to be picked. By rewarding pulling on levers with high variance, we encourage exploration and the discovery of new possibilities.


key: chroma:document
string_value:
That sounds like a great idea, but there are a few caveats, starting with the exogenous factors. Usually, you would use the number of times a price has been tried in the past to determine sample variance, but we have multiple factors contributing to demand. It's not simply a question of having tried a particular price but also about the context in which it was tested for a specific factor. Another issue is that the price is continuous, just as the exogenous factors could be.

The question then becomes: can we build a confidence interval for our forecasted expected demand that depends on both covariates and a continuous price?


key: chroma:document
string_value:
We can answer in the affirmative thanks to a widely known technique from the field of statistics: Bootstrap.

Bootstrap


key: chroma:document
string_value:
Bootstrap is a method that allows you to obtain an estimate for any distribution parameter from a statistic whose values you can observe. This strategy works by resampling the original data with replacement, producing several Bootstrap samples. You then calculate the statistic of interest for the bootstrapped samples and use those to get a feel for the uncertainty of the parameter of interest. If we consider our model's forecasts as a Random Variable that depends on the samples we got in our training set, we can estimate the distribution of these targets, i.e. the forecasted expected demand, using Bootstrap. Namely, you'll obtain a high quantile of the distribution of your forecasted


key: chroma:document
string_value:
of these targets, i.e. the forecasted expected demand, using Bootstrap. Namely, you'll obtain a high quantile of the distribution of your forecasted expected demand (can you see where this is going?).


key: chroma:document
string_value:
Figure 10. Bootstrap sampling strategy used.

The strategy for bootstrapping the output of a machine-learning model is displayed in Figure 10. It involves training a model with each Bootstrap sample, getting the forecasted expected demands from each model, and then using all the different forecasts as their distribution estimate. We then use a high quantile to calculate the upper confidence bound. In our case, we chose the 0.8 quantile. Figure 11 shows this strategy aiding our model to explore and converge to the optimum price.

Figure 11. Animation of the same scenario as Figure 7 but using Bootstrap to force exploration.


key: chroma:document
string_value:
We trained 30 XGBoost models for this simulation, obtaining 30 different forecasts for each price point, following the Bootstrap technique described before. If you look at the plot above, you can see we added a few shaded areas with boundaries corresponding to several quantiles (50%, 60%, 70%, 80%, 90%, and 100% quantiles) of the Bootstrap estimated distributions. As expected, the bands are wider at the extremes where we have fewer samples. Because observations are less frequent there, we have fewer chances of one ending up in the Bootstrap samples, hence making forecasts less stable and allowing us to quantify uncertainty. This process enhances the model's ability to make predictions in


key: chroma:document
string_value:
samples, hence making forecasts less stable and allowing us to quantify uncertainty. This process enhances the model's ability to make predictions in areas with limited data and also enables us to measure the level of uncertainty associated with those predictions.


key: chroma:document
string_value:
Let's be optimistic and optimize for the 80% quantile of the forecasted demand (and hence for profit since the profit function is a linear function), achieving the exploration effect we want. Looking at the above, it's easy to see that the price dispersion the model is trying is much higher now. The model tests out different sections of the demand curve and learns from them, eventually discovering the optimum price. Also, it's not trying prices at random. It's exploring prices that make sense. And most of the time, the prices that produce profits are very close to the optimum.


key: chroma:document
string_value:
Side note: If you paid attention to the simulation’s animations, you might have noticed that the expected demands look pretty smooth compared to the usual XGBoost output. Don’t worry; your eyes aren’t deceiving you. We just found XGBoost’s output a bit too angular for a demand curve, so we smoothed it out a little.


key: chroma:document
string_value:
Choosing the number of models you’ll be bootstrapping is not trivial. If you use too few, you may not get enough exploration, but it will be less computationally expensive. On the other hand, the more models you use, the better the results, but the computational costs increase proportionally. You should keep this in mind when using Bootstrap for your projects. We recommend trying different quantities of models to determine the best trade-off between time, resources, and the results obtained for your project(s).


key: chroma:document
string_value:
We used this strategy in all the market simulations we mentioned earlier, as shown in Figure 12. Pretty awesome, right? All these scenarios are played out in parallel for multiple learning agent candidates. This allows us to observe all these variant models' behavior and understand their strengths and weaknesses. With this approach, we can not only select the most effective model but also identify any issues with its performance to make necessary improvements. This leads to quicker R&D cycles and more efficient development.

Figure 12. Gif with all the scenarios playing at once.


key: chroma:document
string_value:
Figure 12. Gif with all the scenarios playing at once.

To emphasize the findings, we reconstructed the chart we presented earlier with the updated results, as illustrated in Figure 13. We can see that the model reaches the optimum price across all simulation scenarios now. The difference from the results shown in Figure 6 is massive. And we even managed to limit the percentage of cumulative profit losses to less than 5% in all cases!

Figure 13. Results for the different demand and historical data combined scenarios.

Wrapping up


key: chroma:document
string_value:
Figure 13. Results for the different demand and historical data combined scenarios.

Wrapping up

If you made it this far, thank you! It shows you're as interested in these pricing problems as we are! Let's provide a brief recap of everything we've learned so far.

Pricing pitfalls in live production can be difficult to anticipate without evaluation. A market simulator can help by testing scenarios and bringing potential problems to the surface for better decision-making.


key: chroma:document
string_value:
Determining the optimal price point for a market will involve exploration. It's impossible to predict the outcome of a price range without testing. It's critical to try out different options to better understand potential outcomes.

Using statistical techniques like bootstrapping maximizes data utility and quantifies uncertainty, leading to smarter explorations and informed pricing decisions.

We were able to validate the above claims across a varied range of scenarios using the simulator, ensuring that our models are better prepared for production use and maximizing our clients' success.

Final thoughts


key: chroma:document
string_value:
Final thoughts

Although Bootstrap played a crucial role in our approach and ensured that all scenarios converged, the pace of convergence was not quite up to our standards - it was a bit slow for our tastes. To achieve the final results above, we tweaked our pricing agent even further - particularly regarding its approach to unknown regions.

If you're interested in getting more information about these developments, whether you have a specific market scenario you'd like to test out using our simulator or if you're interested in evaluating a particular model, we'd be delighted if you got in touch with us.


key: description
string_value:
Learn how to optimize your pandas code for large datasets with these top five tips. From vectorizing operations to embracing NumPy, our expert advice will help you get the most out of your pandas workflow.


key: chroma:document
string_value:
If you've ever worked with tabular data, you probably know the process: import the data into pandas, clean and transform it, and use it as input for your models. But when the time comes to scale up and take your code to production, there's a good chance your pandas pipeline starts to break down and runs slowly.

But there's no need to panic! Given our extensive experience handling large datasets and using pandas in over 80% of our projects, Tryolabs has gained valuable insights into making your pandas code run faster.

Tip 1: Vectorize like there’s no tomorrow. Using vectorized operations eliminates the need to manually iterate through rows.


key: chroma:document
string_value:
Tip 1: Vectorize like there’s no tomorrow. Using vectorized operations eliminates the need to manually iterate through rows.

Tip 2: Iterate like a semi-pro. When vectorization is impossible, consider looping over the data frame using methods such as list comprehension with zip , lru_cache or parapply .

Tip 3: Embrace NumPy. To enhance performance, consider working directly with the Numpy array through .to_numpy() ; it can provide a significant bump in speed.

Tip 4: Know your dtypes. Selecting the appropriate dtype is critical for memory and computation optimization.


key: chroma:document
string_value:
Tip 4: Know your dtypes. Selecting the appropriate dtype is critical for memory and computation optimization.

Tip 5: Long-term storage. Pandas can read many formats such as CSV, parquet, pickle, JSON, Excel, etc. We recommended using the parquet format, a compressed, efficient columnar data representation.

We'll also explain what can slow your pandas down and share a few bonus tips surrounding caching and parallelization. Keep reading, and you'll become a pandas pro in no time.

Panda brought to life by the power of Midjourney.


key: chroma:document
string_value:
Panda brought to life by the power of Midjourney.

The best way to demonstrate the value of our recommendations is to test the techniques with real datasets. If you are curious about the specifics, take a look at the following section; otherwise, feel free to skip it.

Benchmarking setup We conducted the benchmarks in the following sections on two anonymized data frames from actual projects. And we've made those datasets available to share. Download the data frames to run the tests yourself. Data frame 1


key: chroma:document
string_value:
Data frame 2 Please note that depending on your computer's specifications, you may have trouble opening the data frames. The creator of pandas, Wes McKinney, stated that as a rule of thumb, it is recommended to have 5 to 10 times the amount of RAM as the dataset size. To avoid these issues in running the benchmarks, we used a high-performance desktop computer with a 10-core/20-thread i9 10900K, 128GB of RAM, and a fast SSD running Ubuntu 20.04. Regarding the software used, we chose Python 3.10, pandas 1.4.2, and pyarrow 8.0.0.

Let’s start with the best tip we have to share.

Tip 1: Vectorize like there’s no tomorrow


key: chroma:document
string_value:
Let’s start with the best tip we have to share.

Tip 1: Vectorize like there’s no tomorrow

Vectorized operations in pandas allow you to manipulate entire data frame columns using just a few lines of code rather than manually looping through each row individually.


key: chroma:document
string_value:
The importance of vectorization cannot be overstated, and it's a fundamental aspect of working with pandas. It's so crucial, in fact, that we even created a "catchy" rhyme to remind ourselves of its importance: "Hector Vector is a better mentor than Luke the for loop." In all seriousness, however, pandas lives and breathes vector operations, as should you. We strongly recommend you embrace vectorization and make it a core part of your pandas workflow.

How does it work?

A key element of vectorized operations is broadcasting shapes. Broadcasting allows you to manipulate objects of different shapes intuitively. Examples are shown below.


key: chroma:document
string_value:
Array a with 3 elements is multiplied with scalar b which results in an array of the same shape as a Source Array a of shape (4, 1) is added with array b (3,) resulting in an array of shape (4, 3). Source

Much has been written about this (including these articles about NumPy Optimization and Array Programming with NumPy), and it's essential in deep learning, where massive matrix multiplications happen all the time. But we'll limit ourselves to 2 short examples.

First, imagine you want to count a given integer's occurrences in a column. Below are 2 possible ways of doing it.

In testing df1 and df2 , we got a speedup of just 82x by using the count_vectorized method over the count_loop .


key: chroma:document
string_value:
In testing df1 and df2 , we got a speedup of just 82x by using the count_vectorized method over the count_loop .

Now say you have a DataFrame with a date column and want to offset it by a given number of days. Below, you’ll find two ways of doing that. Can you guess the speedup factor of the vectorized operation?

By using vectorized operations rather than loops for this costly operation, we got an average speedup of 460x!

One small note on the offset_loop function: we construct d outside the list comprehension. Otherwise, it would get built in each iteration, which would double its runtime (this is noteworthy in its own right).


key: chroma:document
string_value:
But you get the point, vectorized operations in actual data are a lifesaver. Remember, it isn't the few microseconds you can shave off a single operation that counts; the total time saved over multiple operations makes the difference. This can take a pipeline from "impossible to run" to "runs in a reasonable amount of time.”

Now that you know how vectorized operations can significantly improve your code's performance, it's time to implement it. Look at your current projects and see where you can apply this technique. You may be surprised at how much of a difference it can make in speed and efficiency.


key: chroma:document
string_value:
Although vectorized operations in pandas are highly recommended for optimal performance and efficiency, it's important to note that not all operations can be vectorized, and there may be instances where alternative methods, such as iteration, will be necessary.

Tip 2: Iterate like a semi-pro

You might be wondering why you’re not able to iterate like a fully-fledged pro. The answer is simple: pros vectorize and rookies iterate.

We know it’s not that simple in practice. You can’t vectorize all operations all the time, but hear us out: whenever you can, vectorize ruthlessly; you’ll thank yourself later.


key: chroma:document
string_value:
There will be times when there’s no alternative to looping over the millions of rows in a data frame. You may choose to perform these complex operations yourself or solicit help from an external service. Whatever the case, we’ll go through various iteration methods.

Key takeaways: Vectorize if possible. Use a list comprehension with zip . Look into lru_cache if your columns have several repeats. Look into parapply .

The benchmark

We’ll apply the following function to the datasets:


key: chroma:document
string_value:
The benchmark

We’ll apply the following function to the datasets:

It removes all the words appearing in the phrase words_to_remove from the given remove_from phrase and eliminates HTML tags while keeping the words with a length greater than or equal to min_include_word_length .

Think of remove_from as a long description from which you want to remove some words that already appear in another data column.

For instance, say you have the following data frame:

Applying remove_words to all rows would result in the following list:

for loops

The first and most intuitive way to iterate would be to use a Python for loop.


key: chroma:document
string_value:
for loops

The first and most intuitive way to iterate would be to use a Python for loop.

There’s overhead everywhere: accessing single values of df and dynamically creating a list with append mean this will be the slowest method and we’ll use it as our baseline for comparison. But you want to avoid this at all costs.

apply

An easy next step would be to use pandas' apply method, which handles the looping internally. You’d expect this to be blazing fast, but it’s only about 1.9x faster than baseline. The code is below. But because better methods are readily available, you should avoid this as well.


key: chroma:document
string_value:
In each iteration of df.apply, the provided callable gets a Series whose index is df.columns and whose values are the row’s. This means that pandas has to generate that series in each loop, which is costly. To cut the costs, it’s better to call apply on the subset of df you know you’ll use, like so:

This simple tweak makes the operation 2.1x faster than the baseline, but we’d still recommend avoiding this method.

list-comp + itertuples

Using itertuples combined with the beloved list comprehension to iterate is definitely better. itertuples yields (named) tuples with the data on the rows. Notice that we subset df again. This is, on average, 4.6x faster than baseline.

list-comp + zip


key: chroma:document
string_value:
list-comp + zip

A slightly different approach that yields about the same performance (4.6x faster than baseline) is to use a list comprehension again but iterate over the desired columns with zip. zip takes iterables and yields tuples where the i-th tuple has the i-th element of all the given iterables in order. This approach looks like this:

list-comp + to_dict

A slower variant of this approach is to use to_dict(orient="records") to iterate over dicts of rows of the dataset. This is about 3.9x faster than the baseline. It was achieved using the following code:


key: chroma:document
string_value:
This is not a hard rule but, from observation, we determined that using to_dict almost tripled the memory footprint of our program, so be judicious when using it.

Bonus method 1: caching


key: chroma:document
string_value:
In addition to the iteration techniques we've discussed, two other methods can help improve your code's performance: caching and parallelization. Caching can be particularly useful if your pandas function is called multiple times with the same arguments. For example, if remove_words is applied to a dataset with many repeated values, you can use functools.lru_cache to store the results of the function and avoid recalculating them each time. To use lru_cache , simply add the @lru_cache decorator to the declaration of remove_words , then use your preferred iteration method to apply the function to your dataset. This can significantly improve your code's speed and efficiency. Take, for example,


key: chroma:document
string_value:
preferred iteration method to apply the function to your dataset. This can significantly improve your code's speed and efficiency. Take, for example, the following code:


key: chroma:document
string_value:
Adding this decorator produces a function that "remembers" the output for inputs it previously encountered, removing the need to run all the code again. You can read more about it in Python's official documentation here. Don’t forget to play with the maxsize parameter to explore the speed vs memory tradeoff.

You may notice that the cached function’s performance for df1 is similar to its non-cached counterpart. This goes to show that caching isn’t the be-all and end-all. In our case, df1 's structure doesn’t have as much repetition, so caching doesn’t help as much as it does fo df2 where the speedup is 46x.

Bonus method 2: parallelization


key: chroma:document
string_value:
Bonus method 2: parallelization

The last ace up our sleeve is using pandarallel to parallelize our function calls across multiple independent chunks of df. The tool is easy to use: you simply import and initialize it and then change all your .applys for .parallel_applys.

In our case, we measured an 12-fold improvement.

However powerful, parallel_apply is no silver bullet. It pickles each chunk of the df into /dev/shm (which is effectively RAM); therefore, you might run out of memory!

If this happens, you can limit pandarallel 's memory usage by initializing it with fewer processes, which will also impact performance.


key: chroma:document
string_value:
If this happens, you can limit pandarallel 's memory usage by initializing it with fewer processes, which will also impact performance.

Why not combine both? Well, we tried and saw no practical benefit. It either went as fast as bare parallel_apply or slower than a bare list comprehension with cache. This might have something to do with how parallelization is achieved and the fact that the cache might not be shared. However, we would need to pursue our investigation further to draw any firm conclusions.

Results

A summary of the results is shown below:


key: chroma:document
string_value:
Results

A summary of the results is shown below:

Description df1 df1 df2 df2 Description Time [s] Speedup Time [s] Speedup loop 546.6 1 245.9 1 apply 289.3 1.9 135.7 1.8 apply (only used cols) 263.9 2.1 121.7 2.0 itertuples (only used cols) 116.1 4.7 54.7 4.5 zip (only used cols) 115.6 4.7 54.2 4.5 to_dict (only used cols) 137.8 4.0 64.0 3.8 cached zip (only used cols) 118.1 4.6 5.3 46.4 parapply (only used cols) 44.5 12.3 19.3 12.7 cached parapply (only used cols) 45.5 12.0 10.6 23.2

Remember to use the above tips to optimize performance when iterating with pandas in the future.

Tip 3: Embrace NumPy


key: chroma:document
string_value:
Remember to use the above tips to optimize performance when iterating with pandas in the future.

Tip 3: Embrace NumPy

Pandas is built on top of NumPy, which is known for its performance and speed in handling large arrays and matrices of numerical data. This helps make pandas efficient and fast when working with large datasets. Now that you’re an expert in vectorizing operations embracing numpy is a natural next step. This section is short and sweet: if you need a bit of extra performance, consider going down to numpy by using .to_numpy() and the numpy methods directly.


key: chroma:document
string_value:
Benchmarking this method is close to impossible. Results will vary wildly between the use case and the implementation details of your code. In our tests, we observed speedups from anywhere between 2x to an absurd 3000x.

Continuing with the above examples, what if we tried counting or offsetting directly in numpy? Have a look at the following code:

As you can see, it’s easy to accomplish: add .to_numpy() and use numpy objects. Regarding the count, this gives a total 185x speedup (2.2x over the vectorized pandas method). In regards to the offset, this gives a whopping 1200x speedup over the loop operation.


key: chroma:document
string_value:
We hear you! These numbers are somewhat artificial because no one in their right mind would offset a day column like this. But this may happen to beginners familiar with other programming languages, and these examples should serve as a cautionary tale.

These speedups are made possible by compounding the benefits of vectorization and the direct use of numpy. This also eliminates the overhead added by pandas' killer feature set.


key: chroma:document
string_value:
Be warned that sometimes this isn’t as simple as placing .to_numpy() after each series in your operation. Case in point: a team member reported that using .to_numpy() in datetime data with timezones actually dropped the timezone. This should be fine if all the data is at the same timezone and you’re calculating differences. Still, it’s something to keep in mind.


key: chroma:document
string_value:
Another team member experienced some weirdness with dates as well. When calling .to_numpy() on a Series of date times, they would get an array of datetime64[ns] . However, when calling .to_numpy() on a single pd.Timestamp , an int was returned. Hence, trying to subtract a single date from an array of dates will return gibberish unless you do .to_numpy().astype(int) .

Given these speedup factors and despite its caveats, you want this tool in your arsenal, and you shouldn't hesitate to use it.

Tip 4: Know your dtypes


key: chroma:document
string_value:
Tip 4: Know your dtypes

In pandas DataFrames, the dtype is a critical attribute that specifies the data type for each column. Therefore, selecting the appropriate dtype for each column in a DataFrame is key.

On the one hand, we can downcast numerics into types that use fewer bits to save memory. Conversely we can use specialized types for specific data that will reduce memory costs and optimize computation by orders of magnitude.

We’ll talk about some of the most prevalent types in pandas, like int , float , bool , and strings. But first, here’s a a primer on the dreaded object .

The object type


key: chroma:document
string_value:
The object type

Regarding datatypes, pandas has many types that efficiently map to numpy types at the fast C level. However, when there’s no easy mapping, it falls back on python objects. For the uninitiated, object is the parent class of all objects in the language [source] . This leads to inefficiencies due to how memory is handled.

The fact that numpy's core is written in C means that an array points directly to all its values located in a contiguous block of memory. This allows for much faster execution time as the cache memory can leverage the spatial locality of the data. You can read more about it here.


key: chroma:document
string_value:
Because pandas stores strings as objects, it has to fall back on slow Python arrays. In contrast to numpy, a Python `list` has a pointer to a memory-contiguous buffer of **pointers**, which point to objects stored in memory, but which also reference data stored in other locations. Hence, accessing this data during runtime will be quite slow in comparison. Source


key: chroma:document
string_value:
Because pandas stores strings as an array of objects, it has to fall back on slow Python arrays. In contrast to numpy, a Python list has a pointer to a memory-contiguous buffer of pointers, which point to objects stored in memory, but which also reference data stored in other locations. Hence, accessing this data during runtime will be quite slow in comparison.

A Python list and how it handles data. Source

In a nutshell, pandas is constrained in part by the ecosystem. But how can we make the best of the tools we have right here, right now? Well, it’s a different story for numerical and string data.

Numerical types


key: chroma:document
string_value:
Numerical types

Regarding ints and floats, downcasting is the key to saving memory. Pandas supports 8, 16, 32, and 64-bit signed and unsigned integers and 16, 32, and 64-bit floats. By default, it opts to use 64-bit variants for both types.

The trick is to choose the smallest type that can comfortably hold your data. You should consider both current and new valid data. You don't want to limit yourself to unsigned ints in your pipeline if signed ints are a real possibility.

In practice, and for your current dataset, you can easily do this with pandas' own pd.to_numeric like so:


key: chroma:document
string_value:
In practice, and for your current dataset, you can easily do this with pandas' own pd.to_numeric like so:

Just like that, we reduced df 's size by 1/3. But your mileage will vary based on your data. As a side note: pd.to_numeric only downcasts floats to float32 , it doesn’t use float16 even though it is compatible with pandas.

Regarding data types, we found that bool should be favored if you have no nulls. It occupies the same amount of memory as an uint8 , but makes the column’s content much clearer.

“What if I have NaN values?” We hear you. In principle, you can cast your ints and bools to float32 and call it a day. You’ll take a memory hit, but NaN s will be handled natively.


key: chroma:document
string_value:
Somewhat recently, pandas has added its own Nullable integer and Nullable boolean data types. They can be more memory efficient than float32 but with the caveat that neither .values nor .to_numpy() return correct numpy arrays with np.nan (because it is itself a float, not an int). Internally, these arrays hold a _data array with the appropriate numpy dtype and a _mask , where True means that the value in _data is missing. This can be an issue if you’re trying to use numpy. The latter and their experimental nature are why we can't fully recommend their use for now.

This is also why we don’t recommend using .convert_dtypes just yet because it uses nullable types.

String types


key: chroma:document
string_value:
This is also why we don’t recommend using .convert_dtypes just yet because it uses nullable types.

String types

With numbers out of the way, let's dive into strings. Strings are used everywhere, but they're the bane of our existence when manipulating data. As mentioned before, they're stored with the inefficient object type, and that's why they use lots of space and are slower than more specialized types.


key: chroma:document
string_value:
But there’s a glimmer of hope with StringDType. Like the other nullable types we mentioned, StringDType is still experimental, so we don't recommend using it in production code for now. Looking ahead, it promises correct handling of missing values and performance improvements over object .


key: chroma:document
string_value:
If your column has low cardinality (many repeated values), the smartest move will be to use pd.Category . nternally, a categorical column holds a mapping from the actual column value (i.e., the contents of your column before conversion) to some arbitrary integer. Then, the column is saved as an int type, exploiting the advantages of contiguous memory and efficient storage. Furthermore, Run Length Encoding is used to reduce the storage size even further (if an integer is repeated, it only saves the integer and the number of instances rather than all the integers).

This results in excellent compression and processing speed, as seen in the results subsection.

Results


key: chroma:document
string_value:
This results in excellent compression and processing speed, as seen in the results subsection.

Results

Now that you have a better understanding of the different dtypes and their efficiency, let's look at some of the tests we ran and their results:

We tried using pandas' convert_dtypes in both df1 and df2 . We ran a custom convert_dtypes that doesn’t use pandas' nullable types. We converted applicable string columns into categoricals. We downcasted numeric types into their smaller types.

Of course, these results will vary according to your data, but we wanted to highlight the benefits of choosing the correct data type. Here are the results in table form:


key: chroma:document
string_value:
Description df1 df1 df2 df2 Description Size [GB] Ratio Size [GB] Ratio Size on RAM (unoptimized) 24.8 1 12.0 1 pandas' convert_dtypes 24.9 1.004 12.5 1.04 Our convert_dtypes 24.7 0.996 12.6 1.05 Convert to Category 13.8 0.556 2.3 0.19 Downcasting 12.5 0.504 1.7 0.14

We managed to reduce the dataset size to about 1/2 and 1/8, respectively, as soon as pandas finished loading it.

This is important because we can now process the data in more ways than before simply because we have more free RAM. And our entire system will likely be snappier as well.

Also note that pandas' convert_dtypes enlarges the dataset in both cases.


key: chroma:document
string_value:
Also note that pandas' convert_dtypes enlarges the dataset in both cases.

Again, you might be asking, why bother with convert_dtypes if it barely saves any memory?” Or, for that matter, how come these people always know the questions I’m about to ask? Well, our job is to predict the future, so we've gotten quite good at it. And we're glad you asked because the speed improvements completely dwarfed the memory reduction benefit. Try running the following code to compare your times:


key: chroma:document
string_value:
With our data, we saw speedups of between 12x to 50x. For the mean, the speedups were 12x to 25x. For the standard deviation, they were 20x to 50x. For value_counts , they were about 15x. That’s not bad for such a small amount of work!

Categoricals, in particular, reap the benefits twofold. Not only is the memory usage reduction reason enough to use them, but they make operations (like groupby or str.lower() ) much faster. Notably, .str.lower() was about 22.5x faster on categorical columns than on regular object columns (or StringDType columns for that matter).


key: chroma:document
string_value:
groupby s are another can of worms. In our testing, when grouping with 1 column, categoricals were about 1.5x faster. However, when grouping with multiple columns, where at least one of them is categorical, pandas defaults to creating an index that is the cartesian product between all categories in said columns. You can imagine how this explodes in size and complexity quickly.


key: chroma:document
string_value:
But don’t despair. There’s a solution! Okay, you can despair just a little because the solution isn’t a catch-all. Some categories are mutually exclusive; therefore, their combination, which would appear in the cartesian product, would never be observed in the data. You can use the observed=True kwarg of groupby to instruct pandas to execute the operation in this way.

IIn our tests, we measured a 2.5x speedup using observed=True , while we got a slowdown of about 2x when we didn’t. If you notice this doesn’t work well for you, converting to object , performing the groupby , and converting back to categorical could be the way to go.

Tip 5: Long-term storage


key: chroma:document
string_value:
Tip 5: Long-term storage

CSV, parquet, pickle, JSON, XML, HDF5, Feather, ORC, STATA, SAS, SPSS, or, dare I say it, Excel (yuck!) are some of the formats that pandas supports. Argghhh! There are too many options! How are we supposed to choose?

We might start, as any responsible engineer would, by reviewing the pros and cons of each format. We’d then test a few that stand out in the real world and come to some conclusion about which format we’d favor and when. Life is tough, though, and with looming deadlines imposing time constraints, the selection process looks more like this:


key: chroma:document
string_value:
Fear not! We've done our homework - and we promise that our method is more scientific than Homer's. We recommend you take a good look at parquet. You don't need to constantly be making these decisions anymore.

Parquet was created to help developers reap the benefits of a "compressed, efficient columnar data representation." It supports different encodings and can compress on a per-column basis. That fits the ML practitioner's use case like a glove!

It sounds awesome; sign me up! Right? Go on, try to do a simple df.to_parquet . I’ll wait…


key: chroma:document
string_value:
It sounds awesome; sign me up! Right? Go on, try to do a simple df.to_parquet . I’ll wait…

Already back? Did you get an ImportError ? Parquet is technically a format, which is more like a spec than an implementation. So, to use it, you need to install another library that implements the format. Currently, pandas supports pyarrow and fastparquet .

But you promised no more decisions! Well, we believe pandas’ use of pyarrow by default is the right choice. We found it to be more reliable than fastparquet . For example, when trying to save and then immediately read df2 , fastparquet complained about an overflow.


key: chroma:document
string_value:
Below is a table with read/write durations and the resulting sizes of the datasets saved in different ways (with pyarrow ). It boils down to this: use parquet and change the compression method to fit your needs. Uncompressed or snappy parquet ought to be your choice during development, resorting to gzip compression for long-term archival.


key: chroma:document
string_value:
Description df1 df1 df1 df2 df2 df2 Description write [s] read [s] size [GB] write [s] read [s] size [GB] pickle 17.7 9.9 6.5 0.9 0.6 1.3 pickle gzip 532.2 40.1 3.8 99.8 3.0 0.2 pickle lzma 2670.5 158.0 3.2 187.9 7.7 0.1 pickle bz2 455.3 218.1 3.5 127.9 19.5 0.2 parquet 22.3 16.0 6.4 3.6 1.3 0.4 parquet gzip 299.5 24.7 4.0 21.5 1.5 0.2 parquet snappy 25.5 15.5 5.9 4.0 1.3 0.3


key: chroma:document
string_value:
CSV is probably in the top 5 data distribution formats, and that's in part thanks to its ease of use. With a simple df.to_csv , you can create a file that anyone (and their dogs) can open in the program of their choice. But, and that's a big but, CSV is severely limited in terms of volume, size, and speed. When you hit these limitations, it's time to start exploring parquet.

The latest pickle version is fast but has substantial security and long-term storage issues. The words "execute arbitrary code during unpickling" should send shivers down your spine, and those are the exact words in the massive, red warning in pickle's official documentation.


key: chroma:document
string_value:
You might think, "well, that's a non-issue; I'll only unpickle stuff I've pickled," but that won't save you either. When unpickling, the object's definition at that time is used to load it from storage. That means that if your objects change, your pickles could break. In the example below, which raises an AttributeError , we show how removing an attribute from class C after saving the pickle results in the loaded object losing the attribute despite it being there when it was saved.

Now imagine pandas changes something in the DataFrame class. You might run into issues when unpickling older DataFrames. Favor (uncompressed) parquets over pickles to avoid this.


key: chroma:document
string_value:
We recently came across a tool called Lance that aims to simplify machine learning workflows. It supports a columnar data format called Lance Format, which claims to be 50 to 100 times faster than other popular options such as Parquet, Iceberg, and Delta. Although we have yet to thoroughly test it, the potential speed increase is intriguing, and we encourage you to give it a try.

Conclusion

Phew! That was a loaded blog post, full of insights and valuable information. We hope you apply some of these tips in the real world and supercharge your codebase to be faster and more powerful.

Here are some of the key concepts to keep in mind:


key: chroma:document
string_value:
Here are some of the key concepts to keep in mind:

Python’s dynamic nature makes it slower than compiled languages. This issue is exacerbated in scientific computing because we run simple operations millions of times. Don’t loop over your data. Instead, vectorize your operations (remember Hector Vector). You can use numpy directly by calling .to_numpy() on the Dataframe, which can be even faster. Choose the smallest possible numerical dtypes, use categoricals, and prefer float32 over the newish nullable dtypes (for now). Use parquet to store your data. Use snappy compression (or none at all) during development, resorting to gzip for long-term archival.


key: description
string_value:
Exploring the expansive universe of Python libraries in 2023, our curated selection goes beyond the usual suspects to uncover hidden gems and innovative tools. Dive in!


key: chroma:document
string_value:
Welcome to the 9th and not-yet-quite-a-decade edition of our yearly Top Python Libraries list! As we journey through the constantly evolving landscape of Python development, it's time again to spotlight the standout libraries and tools that have caught our attention this year. Since our first edition in 2015, we’ve dedicated ourselves to exploring the depths of the Python ecosystem, and this year is no exception. In fact, it's special — 2023 marks the boom of Generative AI and Large Language Models (LLMs), a trend that has significantly influenced our picks.

2021, 2020, 2019, 2018, 2017, 2016, 2015. See the other posts in this series: 2022


key: chroma:document
string_value:
2021, 2020, 2019, 2018, 2017, 2016, 2015. See the other posts in this series: 2022

Our criteria remain consistent: we're on the lookout for libraries and tools that have been launched or risen to prominence over the past year. To qualify for our top picks, they need to be well-maintained, innovative, and, of course, cool enough to spark your interest. This year, you'll notice an emphasis on libraries that align with the LLM wave, reflecting how these technologies are reshaping the Python (or should we say programming?) landscape.


key: chroma:document
string_value:
But fear not, diversity in our selection is still key! Although getting increasingly more difficult every year, we tried to ensure that our full list is not just for AI enthusiasts but also contains gems that can be valuable for a wide range of Python developers, including some niche applications.

Our aim with this post is also to ignite conversations about the remarkable libraries and tools that we might have overlooked.

So, with eager anticipation, let's dive into the world of Python in 2023!

The 10 main picks

1. LiteLLM — call any LLM using OpenAI format, and more


key: chroma:document
string_value:
The 10 main picks

1. LiteLLM — call any LLM using OpenAI format, and more

Imagine a world where you're not tethered to a specific model provider, where switching between LLMs is as easy as changing one or zero lines of code, contrary to rewriting your entire stack using tools like LangChain. LiteLLM makes this a reality.

LiteLLM's intuitive and non-intrusive design allows seamless integration with various models, providing a unified format for inputs and outputs, regardless of the underlying LLM. This flexibility is a game-changer for developers who need to adapt quickly to the rapidly changing landscape of large language models.


key: chroma:document
string_value:
But LiteLLM isn't just about flexibility; it's about efficiency and control. Features like streaming responses and mapping exceptions to OpenAI’s format demonstrate its robustness. The ability to track and manage costs across different LLMs is particularly noteworthy, ensuring that your AI explorations don't lead to unexpected bills. This level of financial control is vital in a domain where API calls can quickly accumulate costs.

Furthermore, LiteLLM comes with a proxy API server, enhancing its versatility in integrating on codebases without changing a single line of code ✨.


key: chroma:document
string_value:
Another differentiating feature is the built-in load balancing and rate limiting features. These capabilities ensure optimal performance and resource utilization, especially when dealing with high volumes of requests across multiple inference calls. Load balancing efficiently distributes incoming API calls, preventing any single model provider from becoming a bottleneck. Meanwhile, the rate limiting function plays a crucial role in maintaining stable and controlled access to the models, safeguarding against excessive usage and potential overloads (forget HTTP 429).


key: chroma:document
string_value:
Of course, the devil is in the details: prompting will work differently for different LLM providers, so you will need to do some work if you truly want to utilize different providers. But it’s good to have a layer that you can rely on to make the switch as easy as possible.

Cheers and congrats to the LiteLLM authors. Whether you're a seasoned AI developer or just starting out playing with Generative AI, LiteLLM is a library that belongs in your toolkit!

2. PyApp — deploy self contained Python applications anywhere


key: chroma:document
string_value:
2. PyApp — deploy self contained Python applications anywhere

Python's surge in popularity over the years is no secret. Its blend of simplicity and effectiveness, coupled with a vast and friendly community, has made it one of the most widely used programming languages globally. Yet, distributing Python applications remains a long-standing challenge. Unlike other languages that allow distribution of standalone compiled binaries, Python has struggled in this area, especially when targeting non-technical users. Traditionally, Python applications are shared via Docker container images, a method not always accessible to all users.


key: chroma:document
string_value:
Enter PyApp: a utility library that simplifies the distribution and installation of Python applications. It achieves this by embedding Python within a self-installing package that is compatible across all operating systems. But it doesn't stop there! PyApp also supports self-updating capabilities, and can be customized to accommodate a variety of use cases.


key: chroma:document
string_value:
With PyApp, specifying dependencies for pip installation via requirements.txt files is a breeze, and additional artifacts can be seamlessly embedded into the package using .in files. The library handles the installation process, creating a dedicated directory for Python and the app's required dependencies. On subsequent runs, PyApp checks for the existence of this installation directory, optimizing command-line interface responsiveness.

It differentiates from other tools you might know like PyInstaller in the fact that PyApp doesn’t do dependency discovery but rather relies on an explicit declaration of dependencies which seems to solve some issues users have been facing.


key: chroma:document
string_value:
In a nutshell, PyApp presents a straightforward yet potent solution for delivering Python applications to your users. It streamlines distribution and installation procedures, eliminating the hurdles typically associated with these processes. As we look forward to 2024, PyApp is undoubtedly a tool worth adding to your Python arsenal, making the distribution of Python apps as easy as pie!

3. Taipy — build UIs for data apps, even in production


key: chroma:document
string_value:
3. Taipy — build UIs for data apps, even in production

In the wise words of fashion designer Giorgio Armani, "Elegance is not about being noticed, it's about being remembered". This philosophy is the guiding star of Taipy, a Python library that empowers data scientists to weave engaging narratives from their data.


key: chroma:document
string_value:
In the realm of data science, the art of storytelling can be as crucial as the underlying models. And having the right tool can make all the difference. That is Taipy’s game. Taipy is a low-code Python library that enables data scientists to build interactive Web UIs for their machine learning products, without the need to master web stack tools. It's all about allowing data scientists to focus on what they do best - extracting insights from data.


key: chroma:document
string_value:
Taipy's magic lies in its ability to bind variables and expressions to the state of visual components in the UI. This binding happens in-place when declaring the visual components, drastically reducing boilerplate code needed to develop your app. And speaking of visual components, Taipy offers a rich array of options, including plots, sliders, and image grids, to help you build next-gen UIs.


key: chroma:document
string_value:
But wait, didn’t we have Streamlit (our pick #7 of 2019) for that? Well, that’s where Taipy shines. Unlike Streamlit, which reruns code for the entire page upon detecting an input change, Taipy employs callbacks to update only the components affected by the change (much like shiny does for R for those coming from our brother language) enhancing responsiveness and accelerating development.


key: chroma:document
string_value:
But the innovation doesn't stop there. Taipy also provides tools for visually composing machine learning pipelines, speeding up development while offering an easy way to explain at a high level the steps your model takes. Importantly, data nodes, the inputs to these visual pipelines, can be dynamically changed, enabling interactive exploration and what-if scenarios.

Whether you choose to self-host your Taipy applications or use Taipy Cloud for serverless deployment, you're in for a seamless experience. Taipy helps you unveil the stories hidden in your data and create memorable, interactive machine learning workflows. Check it out today!


key: chroma:document
string_value:
Here’s their 2023 PyData Seattle presentation and here’s their launch post on HackerNews; be sure to check the discussion for some interesting information nuggets on the lib.

4. MLX — machine learning on Apple silicon with NumPy-like API

In 2021 Apple introduced it’s M1 architecture, now further refined into the current M3, a revolutionary piece of hardware that brought along performance and efficiency improvements, as well as a unified memory buffer that can be accessed by the CPU, GPU and NPU (Apple’s Neural Engine) without the need to copy information between processing units.


key: chroma:document
string_value:
In 2023, the spotlight is on Apple's MLX, a groundbreaking array framework specifically designed for machine learning on Apple silicon. Brought to you by Apple's own machine learning research team, MLX is setting new standards for the development and deployment of machine learning models on Mac hardware.

MLX is designed to retain familiarity: its Python API is a close cousin of NumPy's, and it also offers a fully-featured C++ API that mirrors its Python counterpart. Higher-level packages like mlx.nn and mlx.optimizers follow the footsteps of PyTorch, making the construction of complex models a breeze.


key: chroma:document
string_value:
But MLX is more than just a familiar face. It introduces innovative features like composable function transformations, enabling automatic differentiation, automatic vectorization, and computation graph optimization. It also employs lazy computation, materializing arrays only when needed, and dynamic graph construction, making debugging more intuitive and avoiding slow compilations due to changes in function argument shapes.


key: chroma:document
string_value:
MLX's multi-device capability allows operations to run on any supported processing unit, be it CPU, GPU or NPU. And here's the game-changer: MLX's unified memory model. Unlike other frameworks, MLX arrays exist in shared memory, enabling operations on any supported device without the need for time-expensive data transfers.

This user-friendly yet efficient framework is designed by machine learning researchers for machine learning researchers. Inspired by the likes of NumPy, PyTorch, Jax, and ArrayFire, MLX aims to make it easy for researchers to extend and improve the framework, fostering rapid exploration of new ideas.


key: chroma:document
string_value:
As we move towards 2024, MLX is definitely a toolkit to keep an eye on. Whether you're an Apple aficionado, a machine learning enthusiast, or both, MLX is poised to revolutionize your machine learning journey and utilize the capabilities of Apple silicon to the fullest.

5. Unstructured — the ultimate toolkit for text preprocessing

In the vast landscape of AI, a model is only as good as the data it feeds on. But what happens when your data is a chaotic jumble of raw documents, scattered across various formats? Enter Unstructured, a comprehensive solution for all your text preprocessing needs.


key: chroma:document
string_value:
Unstructured is the master key to unlock the potential of your raw data, capable of digesting anything thrown at it — PDFs, HTML files, Word documents, even images. It's like a gourmet chef for your data, slicing and dicing it into palatable pieces for your AI models.

Unstructured offers a smorgasbord of functions, neatly categorized into six types:


key: chroma:document
string_value:
Partitioning Functions that transform raw documents into structured and standardized formats. Cleaning Functions that scrub away unwanted text, leaving behind only the relevant nuggets. Staging Functions that format your data, readying it for ML tasks like inference or labeling. Extraction Functions that mine for valuable information within documents, like emails, IP addresses, or pattern-matched texts. Chunking Functions that break down documents into bite-sized sections, perfect for similarity searches and RAG. Embedding Functions that convert preprocessed text into vectors, offering easy-to-use interfaces.


key: chroma:document
string_value:
But that's not all. Unstructured also comes equipped with Connectors, a set of tools for seamless integration with various storage platforms. From GCS and S3 to Google Drive, One Drive, Notion, and many more. These Connectors work both ways, allowing you to read and write information to and from a plethora of sources.

Unstructured is your go-to Python library for effortless text extraction from documents. It simplifies text cleansing, handling everything from bullet removal to emoji management and language translation. Its extractor functions parse cleansed text, retrieving valuable information. You can use built-in extractors or custom templates for structured text.


key: chroma:document
string_value:
The library's staging functions save preprocessed documents in diverse formats, compatible with various tools like Label Studio and Prodigy. It also features a chunker that groups texts by their title building cohesive, related sections and embedding functions for easy text vectorizing using popular LLM models.

With Unstructured in your toolkit, your journey into the world of Large Language Models is set to be a rewarding one. Get ready to unlock the full potential of your data in 2024!

6. ZenML and AutoMLOps — portable, production-ready MLOps pipelines

Alright, we're bending the rules a bit here. Two libraries, one entry. But trust us, it’ll be worth it.


key: chroma:document
string_value:
Alright, we're bending the rules a bit here. Two libraries, one entry. But trust us, it’ll be worth it.

MLOps is a crucial link between machine learning model development and operations. It enables a smooth transition from experimental to production-ready models. Without MLOps, organizations face navigating a minefield of inefficiencies, wasted resources, and unreliable predictions. It ensures model reproducibility, scalability, and monitoring, transforming brilliant ideas from mere theories into valuable solutions. Thus, effective MLOps is the backbone of successful machine learning initiatives.


key: chroma:document
string_value:
However, setting up an effective MLOps pipeline can often feel like a Herculean task. The sheer range of services and skills required can make the process daunting and time-consuming. Furthermore, custom developments often become so intertwined with application logic that maintaining application code becomes a challenging endeavor.

These picks are designed to alleviate these issues and put MLOps best practices front and center of your development.


key: chroma:document
string_value:
These picks are designed to alleviate these issues and put MLOps best practices front and center of your development.

ZenML provides a fast-lane for transitioning from experimental code to a structured, production-ready pipeline with minimal changes. This vendor-agnostic framework excels at decoupling application code from infrastructure tooling, allowing you to specify different service providers for each component of your infrastructure.


key: chroma:document
string_value:
With ZenML, your pipeline’s steps are annotated and linked together to form the actual pipeline. Then, you configure a Zen Stack, specifying the necessary tooling and infrastructure. This stack consists of two main components: the orchestrator and the artifact store. The orchestrator, which could be a cloud execution service or a simple Python interpreter, runs your pipeline's code. The artifact store is where your data will reside. And the best part? ZenML supports numerous integrations, offering additional functionality like feature stores, data validation, and experiment trackers.


key: chroma:document
string_value:
In the other corner, we have Google's AutoMLOps, which shares ZenML's approach of annotating model development code as components and pipelines. However, AutoMLOps takes it a step further by automatically generating the necessary code to containerize your components, provision them to Google’s cloud, and deploy those containers to the allocated resources. All of this follows an automated, reproducible CI/CD process that helps accelerate your path towards MLOps maturity level 2.


key: chroma:document
string_value:
The deployed pipelines can then make use of Google’s extended MLOps suite of functionalities, ranging from data processing and storage, to experimentation, training and offline evaluation to online serving with configurable auto-scaling capabilities and testing as well as monitoring for drift both in your input data and in your models’ predictions. This enables use of advanced capabilities like continuous training and deployment, enhancing your applications’ accuracy and catching errors before they can have a real impact on the end-user or business.


key: chroma:document
string_value:
Uniquely tailored to leverage Google Cloud Services and Vertex AI, AutoMLOps is your magic carpet ride through the cloud. While it's true that this tool is unabashedly Google-centric, it's a small concession for the vast benefits you'll reap. If you're open to navigating the Google Cloud, prepare to have a significant portion of your journey streamlined, making your cloud-based endeavors more efficient and enjoyable than ever.

With these two powerhouses in your toolkit, you're well-equipped to navigate the complexities of MLOps, ensuring your machine learning initiatives are not only successful but also efficient and manageable.


key: chroma:document
string_value:
Say goodbye to the days of wrestling with infrastructure and application code, and say hello to the era of streamlined, effective MLOps.

Have you crossed paths with Whisper, the trailblazing speech recognition model from OpenAI? It's like a multilingual maestro, adept at a concert of tasks such as speech transcription, timestamping, speech-to-text translation and voice detection — all in various languages.


key: chroma:document
string_value:
Now, let me introduce you to its turbo-charged cousin, WhisperX. It's like Whisper after a double shot of espresso, boasting more accurate timestamping, multiple speaker detection, and a reduction in hallucinations (not the psychedelic kind, but the kind where it starts pouring words when no-one is speaking) by enhancing the voice activity detection. The best part? It does all this while being faster and less memory-hungry.


key: chroma:document
string_value:
WhisperX has a secret weapon called voice activity detection preprocessing. It's like a bouncer that only lets in audio segments that contain voice activity, and it's much cheaper than letting everyone in for a full automatic speech recognition party. This approach also ensures that the audio is sliced into neat chunks, avoiding any awkward cuts mid-speech and boosting the reliability of whole word detection. Plus, the timestamps are tidied up to match these boundaries, making them even more accurate. The final touch? A phoneme classifier model that aligns word boundaries based on a fixed, per-language dictionary. It's like having a grammar stickler on your team.


key: chroma:document
string_value:
Curious to see it in action? Check out an English sample of the results on their GitHub repository.

The beauty of WhisperX is that the chunks can be processed in parallel, like a well-oiled assembly line, which enables batched inference. And thanks to a switch to the Faster Whisper backend, the RAM usage for the whisper-large model has been trimmed down to less than 8GB. This means it can now play nice with a wider range of graphics cards while also boasting a whopping 70x speedup on the same model. Like putting your speech recognition on a high-speed treadmill.

For the curious minds who love to dive into the nitty-gritty, the authors have generously shared their pre-print paper.


key: chroma:document
string_value:
For the curious minds who love to dive into the nitty-gritty, the authors have generously shared their pre-print paper.

8. AutoGen — LLM conversational collaborative suite

AutoGen: Agent Collaboration with Human Intervention. Source: AutoGen Multi-Agent Conversation Framework

Ever dreamed of a personal team of software engineers ready to bring your app ideas to life? Meet Microsoft’s AutoGen, your magic wand that conjures up conversational agents working in harmony to achieve a shared goal. It's like the Avengers, but for software development.


key: chroma:document
string_value:
AutoGen's simplest setup stars two key players: the AssistantAgent and UserProxyAgent . The AssistantAgent is your AI sidekick, designed to execute tasks independently. Meanwhile, the UserProxyAgent is your digital stand-in, allowing you to step in when human intervention is needed but also being able to act on your behalf. It's like having your cake and eating it too!

The AssistantAgent performs tasks, while the UserProxyAgent provides feedback to the AssistantAgent on whether the task is succesfully completed. Once the task is set, these agents spring into action, working autonomously until they either require human oversight or they've successfully completed the task.


key: chroma:document
string_value:
This framework shines in use cases like code generation. The AssistantAgent crafts the code, while the UserProxyAgent tests it. If bugs appear, the UserProxyAgent provides feedback, and the cycle repeats. It's like owning a personal code factory.

The beauty of AutoGen lies in its scalability. You can introduce additional agents to fulfill various roles, akin to building your software development team. Need a tester agent or a product manager? AutoGen has got you covered. For inspiration, check out ChatDev, a project that built an entire software team using a similar concept.


key: chroma:document
string_value:
AutoGen also boasts several Enhanced Inference features that empower your workflow. It unifies the API for several models, enabling you to link multiple LLMs behind a single agent: if one model fails the next in line takes the mantle, ensuring robustness. Furthermore, it provides a prompt results cache for faster, cost-effective inference, chat logging for debugging and prompt templating so that common instructions (such as explain your reasoning step by step) can be reused across multiple tasks.


key: chroma:document
string_value:
But that's not all. Do you want to enhance your agents with RAG? Just add Retrieve to their class names and provide a docs path et voila! Your agents can now tap into custom databases providing context-specific information. Want an agent that learns on the go? Try explaining via prompts some stuff to the TeachableAgent and it will store the information you provide in a MemGPT-style embeddings DB, allowing it to recall and reason over this context in future interactions. An extremely powerful usage for this is having your agents reason about what worked in their interactions and produce a summary recipe of the steps that did work, then teach the TeachableAgent that recipe so it knows how to


key: chroma:document
string_value:
worked in their interactions and produce a summary recipe of the steps that did work, then teach the TeachableAgent that recipe so it knows how to do it next time.


key: chroma:document
string_value:
To cap it all off, AutoGen offers robust security measures. The AutoGen team strongly advises running your agents' code inside a Docker container. You know, just in case you stumble upon the agent combo that unlocks AGI. 😉😉. With AutoGen, the sky's indeed the limit.

9. Guardrails — babysit LLMs so they behave as intended

Have you ever had a fantastic prompt, carefully crafted through a meticulous prompt-engineering process only to find your Large Language Model breaking its chains and responding with a non-structured answer? Or an LLM not following your instructions and mentioning your competitor?


key: chroma:document
string_value:
The flexibility of LLMs provides an immensely vast range of applications but also makes its integration into a greater pipeline challenging. Enter Guardrails: a library designed to specify structure and type, as well as validating and correcting the outputs of large language models.

Guardrails works by defining a .rail specification, a flavor of XML, that’s intended to be human readable and allows for defining the structure of the requested output, validators to ensure the values meet a given quality criteria, and corrective actions such as re-asking the LLM or filtering out the invalid output if the validations are not met.


key: chroma:document
string_value:
Here’s an example that specifies the quality criteria (generated length, URL reachability) in the format fields of the RAIL spec, re-asks for explanation and filters the follow_up_url if not valid.


key: chroma:document
string_value:
Guards such as these can be initialized from a RailSpec as well as from PyDantic type definitions, or even strings for less complex use cases. The structure can then be used to define the fields that the LLM must produce as well as their types, including nested objects defined within your logic. Several validators are provided out of the box ranging from simple string min/max length or regex validations to syntactic bug-free Python or SQL code, profanity-free text and many [more] (https://docs.guardrailsai.com/api_reference/validators/).

With Guardrails in your stack, your LLM applications are bound to be more reliable and robust, making it easier to integrate them into your flows.


key: chroma:document
string_value:
With Guardrails in your stack, your LLM applications are bound to be more reliable and robust, making it easier to integrate them into your flows.

10. Temporian — the “Pandas” built for preprocessing temporal data

Time is of the essence. Time is money. Time is running out… You’ve heard it before right? Temporal data is ubiquitous in most human activity, that's why many popular libraries allow representing and manipulating time-series data: a series of values with equal time intervals between them.


key: chroma:document
string_value:
However, the classical time series approach falls short for representing many kinds of real-world temporal data that is not equally spaced, such as user clicks on a website or sales happening in several of a retailer’s stores.

The go-to approach in this scenario has always been to use a general data preprocessing tool, such as pandas, and cram data into a tabular format by resampling and aggregating it — losing both flexibility and granularity in the whole feature engineering process.


key: chroma:document
string_value:
This is why Tryolabs has partnered with Google to develop a state-of-the-art library for temporal data manipulation. Temporian allows for safe, simple and efficient preprocessing and feature engineering of temporal data through a Python API backed by a C++ implementation for blazing-fast speeds. Temporian supports multivariate time-series, multivariate time-sequences, event logs, and cross-source event streams.


key: chroma:document
string_value:
At the core of Temporian is the EventSet , which contains timestamped records called Events that hold feature values. EventSets can contain multivariate time series with uniform or non-uniform timestamps, as well as independent series belonging to different entities. Handy operators like lag, resample, and window functions help fill in gaps and align timestamps for arithmetic operations.

To get a sense of the API, here’s a simple Temporian snippet that computes the daily revenue for each store in a sales dataset, sampled at 10pm every workday:


key: chroma:document
string_value:
Temporian proposes a new paradigm through which to work with temporal data. And being tailor-made for it, plus its core computations running as highly-optimized C++ code, enables it to make commonplace temporal operations safer, easier to write, and much, much faster to execute — with benchmarks showing speedups of upwards of 100x against equivalent pandas code.

If you are curious, check out the Tutorials in the documentation, which will walk you through real examples using the library. And of course, all feedback is more than welcome :)

Runner-ups

The following libraries almost made it to our top picks and we feel they deserve special recognition:


key: chroma:document
string_value:
Runner-ups

The following libraries almost made it to our top picks and we feel they deserve special recognition:

DataGradients — say hello to your personal assistant for computer vision dataset analysis . DataGradients, a brainchild of Deci AI, is an open-source Python library that comes packed with solutions for common tasks in vision dataset management . Whether it's identifying corrupt images, wrong labels, biases, extracting summaries for a deeper understanding of available data, or providing guidance for hyperparameter selection, DataGradients has got you covered.


key: chroma:document
string_value:
— say hello to your personal assistant for . DataGradients, a brainchild of Deci AI, is an open-source Python library that comes packed with solutions for common tasks in . Whether it's identifying corrupt images, wrong labels, biases, extracting summaries for a deeper understanding of available data, or providing guidance for hyperparameter selection, DataGradients has got you covered. functime — a comprehensive library for time-series forecasting and feature extraction on large panel datasets. Offering unique preprocessing options and cross-validation splitters. It stands out for its incredible speed and efficiency, capable of processing 100,000 time-series in seconds , thanks to its


key: chroma:document
string_value:
splitters. It stands out for its incredible speed and efficiency, capable of processing 100,000 time-series in seconds , thanks to its smart use of Polars for parallel feature engineering. But functime is more than just speed; it's a powerful ally equipped with proven machine learning algorithms and supports exogenous features across all forecasters. It's also automated, managing lags and hyperparameter tuning with FLAML. With over 100 time-series feature extractors available in its custom ts namespace, functime offers an efficient, user-friendly platform that is transforming the way we approach time-series data.


key: chroma:document
string_value:
— a comprehensive library for time-series forecasting and feature extraction on large panel datasets. Offering unique preprocessing options and cross-validation splitters. It stands out for its incredible speed and efficiency, , thanks to its smart use of Polars for parallel feature engineering. But functime is more than just speed; it's a powerful ally equipped with proven machine learning algorithms and supports exogenous features across all forecasters. It's also automated, managing lags and hyperparameter tuning with FLAML. With over 100 time-series feature extractors available in its custom namespace, functime offers an efficient, user-friendly platform that is transforming the way we


key: chroma:document
string_value:
feature extractors available in its custom namespace, functime offers an efficient, user-friendly platform that is transforming the way we approach time-series data. LoRaX — an inference server that scales to 1000s of fine-tuned LLMs, by Predibase. LoRaX (”LoRA eXchange”) is designed to serve a multitude of fine-tuned models on a single GPU , significantly cutting down the costs associated with serving while maintaining impressive throughput and latency. Key features include Dynamic Adapter Loading, allowing just-in-time loading of any fine-tuned LoRA adapter without affecting concurrent requests; Heterogeneous Continuous Batching, which combines different adapter requests to maintain


key: chroma:document
string_value:
LoRA adapter without affecting concurrent requests; Heterogeneous Continuous Batching, which combines different adapter requests to maintain stable latency and throughput; and Adapter Exchange Scheduling, which efficiently manages adapter prefetching and offloading between GPU and CPU. It supports a range of Large Language Models like Llama and Mistral as base models, and is compatible with LoRA adapters trained using PEFT and Ludwig libraries.


key: chroma:document
string_value:
— an inference server that scales to 1000s of fine-tuned LLMs, by Predibase. LoRaX (”LoRA eXchange”) is designed to serve a , significantly cutting down the costs associated with serving while maintaining impressive throughput and latency. Key features include Dynamic Adapter Loading, allowing just-in-time loading of any fine-tuned LoRA adapter without affecting concurrent requests; Heterogeneous Continuous Batching, which combines different adapter requests to maintain stable latency and throughput; and Adapter Exchange Scheduling, which efficiently manages adapter prefetching and offloading between GPU and CPU. It supports a range of Large Language Models like Llama and Mistral as base


key: chroma:document
string_value:
manages adapter prefetching and offloading between GPU and CPU. It supports a range of Large Language Models like Llama and Mistral as base models, and is compatible with LoRA adapters trained using PEFT and Ludwig libraries. outlines — guided text generation with LLMs, especially focusing on constrained text generation. Allows for model guidance such as, multiple choice, type constraints and dynamic stopping by using a very efficient regex-guided generation, leading to faster inference when compared to other guidance approaches. Generate valid JSON 100% of the time! Check out their HN Launch Blog post for more insights.


key: chroma:document
string_value:
— with LLMs, especially focusing on constrained text generation. Allows for model guidance such as, multiple choice, type constraints and dynamic stopping by using a very efficient regex-guided generation, leading to faster inference when compared to other guidance approaches. Generate valid JSON 100% of the time! Check out their HN Launch Blog post for more insights. Pearl — a gift from Meta's Applied Reinforcement Learning (RL) team. It's a production-ready AI Agent Library that prioritizes cumulative long-term rewards over immediate feedback . These agents can adapt to environments with limited observability, sparse feedback, and high stochasticity, making them a fitting choice for a


key: chroma:document
string_value:
. These agents can adapt to environments with limited observability, sparse feedback, and high stochasticity, making them a fitting choice for a wide range of real-world problems. Pearl's modular implementation of state-of-the-art RL algorithms is a testament to its versatility and forward-thinking approach.


key: chroma:document
string_value:
— a gift from Meta's Applied Reinforcement Learning (RL) team. It's a AI Agent Library that . These agents can adapt to environments with limited observability, sparse feedback, and high stochasticity, making them a fitting choice for a wide range of real-world problems. Pearl's modular implementation of state-of-the-art RL algorithms is a testament to its versatility and forward-thinking approach. PEFT — brought to you by Hugging Face, PEFT or Parameter-Efficient Fine-Tuning methods, is your ticket to fine-tuning pre-trained language models for a wide array of downstream applications without having to change all the model's parameters, saving on computational resources while also retaining


key: chroma:document
string_value:
a wide array of downstream applications without having to change all the model's parameters, saving on computational resources while also retaining the good properties of more generalistic models and improving performance on your specific task.


key: chroma:document
string_value:
— brought to you by Hugging Face, PEFT or Parameter-Efficient Fine-Tuning methods, is your ticket to for a wide array of downstream applications without having to change all the model's parameters, saving on computational resources while also retaining the good properties of more generalistic models and improving performance on your specific task. vLLM — A high-throughput and memory-efficient inference and serving engine for LLMs. It improves LLM serving performance by 3.5x over previous SOTA, without changing model architecture. Since output size isn’t known beforehand LLMs typically reserve very large and wasteful long blocks of contiguous memory as large as the model output size. The


key: chroma:document
string_value:
size isn’t known beforehand LLMs typically reserve very large and wasteful long blocks of contiguous memory as large as the model output size. The secret (actually not secret, it’s open sourced) sauce is an algorithm called PagedAttention where they physically split the key-value caches that LLMs use into fixed length, short and dynamically allocated blocks. The dynamic assignment prevents memory waste, allowing more prompts to fit into VRAM and hence enabling faster throughput. At the time of writing, vLLM has been adopted as the serving solution by many of the big players, becoming a de-facto standard serving solution.


key: chroma:document
string_value:
The Hidden Gems: Unveiling the Long Tail of 2023 Python Libraries

While our top picks have certainly captured the limelight, there's a whole constellation of Python libraries that didn't make it to the main stage but are equally deserving of your attention. With more than 120 innovative Python libraries analyzed, each with its own unique capabilities and cutting-edge features, if we were to do justice to each one of them, we'd be here until the next Winter Olympics!

To help you navigate this treasure trove, we've neatly categorized these libraries and distilled their key features into succinct, one-liner summaries.

Causal inference


key: chroma:document
string_value:
Causal inference

CausalTune — a library for automated tuning and selection for causal estimators.

— a library for automated tuning and selection for causal estimators. CausalPy — A Python package for causal inference in quasi-experimental settings.

— A Python package for causal inference in quasi-experimental settings. PyWhy-LLM — experimental library integrating LLM capabilities to support causal analyses.

Chatblade — ChatGPT on the command line, providing utility methods to extract JSON or Markdown from ChatGPT responses.

— ChatGPT on the command line, providing utility methods to extract JSON or Markdown from ChatGPT responses. Elia — A terminal ChatGPT client built with Textual.


key: chroma:document
string_value:
— A terminal ChatGPT client built with Textual. Gorilla CLI — powers your command-line interactions with a user-centric tool. Simply state your objective, and Gorilla CLI will generate potential commands for execution.

— powers your command-line interactions with a user-centric tool. Simply state your objective, and Gorilla CLI will generate potential commands for execution. LLM — A CLI utility and Python library for interacting with Large Language Models, both via remote APIs and models that can be installed and run on your own machine. By the author of Datasette.

Chainlit — “the Streamlit for ChatGPT”, create ChatGPT-like UIs on top of any Python code in minutes!


key: chroma:document
string_value:
Chainlit — “the Streamlit for ChatGPT”, create ChatGPT-like UIs on top of any Python code in minutes!

— “the Streamlit for ChatGPT”, create ChatGPT-like UIs on top of any Python code in minutes! pydistcheck — Linter that finds portability issues in Python package distributions (wheels and sdists).

— Linter that finds portability issues in Python package distributions (wheels and sdists). pyxet — lightweight interface for the XetHub platform, a blob-store with a filesystem like interface and git capabilities.

Computer vision

deepdoctection — orchestrates document extraction and document layout analysis tasks using deep learning models.


key: chroma:document
string_value:
Computer vision

deepdoctection — orchestrates document extraction and document layout analysis tasks using deep learning models.

— orchestrates document extraction and document layout analysis tasks using deep learning models. FaceFusion — Next generation face swapper and enhancer.

— Next generation face swapper and enhancer. MetaSeg — packaged version of the Segment Anything Model (SAM).

— packaged version of the Segment Anything Model (SAM). VTracer — open source software to convert raster images (like jpg & png) into vector graphics (svg)

Data and Features

Adala — Adala - Autonomous DAta (Labeling) Agent framework.


key: chroma:document
string_value:
Data and Features

Adala — Adala - Autonomous DAta (Labeling) Agent framework.

— Adala - Autonomous DAta (Labeling) Agent framework. Autolabel — Label, clean and enrich text datasets with LLMs.

— Label, clean and enrich text datasets with LLMs. balance — simple workflow and methods for dealing with biased data samples when looking to infer from them to some target population of interest. See launch blog post. By META.


key: chroma:document
string_value:
— simple workflow and methods for dealing with biased data samples when looking to infer from them to some target population of interest. See launch blog post. By META. Bytewax — Python framework that simplifies event and stream processing. Because Bytewax couples the stream and event processing capabilities of Flink, Spark, and Kafka Streams with the friendly and familiar interface of Python, you can re-use the Python libraries you already know and love.


key: chroma:document
string_value:
— Python framework that simplifies event and stream processing. Because Bytewax couples the stream and event processing capabilities of Flink, Spark, and Kafka Streams with the friendly and familiar interface of Python, you can re-use the Python libraries you already know and love. Featureform — feature store. Turn your existing data infrastructure into a feature store.

— feature store. Turn your existing data infrastructure into a feature store. Galactic cleaning and curation tools for massive unstructured text datasets. Ben (48/100) on X


key: chroma:document
string_value:
cleaning and curation tools for massive unstructured text datasets. Ben (48/100) on X Great Expectations — helps data teams build a shared understanding of their data through quality testing, documentation, and profiling.

Data Visualization

PyGWalker — turn your pandas DataFrame into a Tableau-style User Interface for visual analysis.

— turn your pandas DataFrame into a Tableau-style User Interface for visual analysis. Vizro — a toolkit for creating modular data visualization applications. By McKinsey.

Embeddings and Vector DBs

Epsilla — a high performance Vector Database Management System, focused on scalability, high performance, and cost-effectiveness of vector search.


key: chroma:document
string_value:
Epsilla — a high performance Vector Database Management System, focused on scalability, high performance, and cost-effectiveness of vector search.

— a high performance Vector Database Management System, focused on scalability, high performance, and cost-effectiveness of vector search. LanceDB — open-source database for vector-search built with persistent storage, which greatly simplifies retrieval, filtering and management of embeddings.


key: chroma:document
string_value:
— open-source database for vector-search built with persistent storage, which greatly simplifies retrieval, filtering and management of embeddings. SeaGOAT — local search tool that leverages vector embeddings to enable to search your codebase semantically.

— local search tool that leverages vector embeddings to enable to search your codebase semantically. Text Embeddings Inference — A blazing fast inference solution for text embeddings models.

Federated learning

Flower — A Friendly Federated Learning Framework.


key: chroma:document
string_value:
Federated learning

Flower — A Friendly Federated Learning Framework.

— A Friendly Federated Learning Framework. MetisFL — federated learning framework that allows developers to easily federate their machine learning workflows and train their models across distributed data silos without ever collecting the data in a centralized location.

Generative AI

AudioCraft — library for audio processing and generation with deep learning. By Meta.

— library for audio processing and generation with deep learning. By Meta. Image Eval — A toolkit for evaluating your favorite image generation models. LinkedIn Launch Post.


key: chroma:document
string_value:
— A toolkit for evaluating your favorite image generation models. LinkedIn Launch Post. imaginAIry — Pythonic generation of stable diffusion images.

— Pythonic generation of stable diffusion images. Modular Diffusion — Python library for designing and training your own Diffusion Models with PyTorch.

— Python library for designing and training your own Diffusion Models with PyTorch. SapientML — Generative AutoML for Tabular Data.

LLM Accuracy Enhancements

AutoChain — AutoChain: Build lightweight, extensible, and testable LLM Agents

— AutoChain: Build lightweight, extensible, and testable LLM Agents Auto-GPT — An experimental open-source attempt to make GPT-4 fully autonomous.


key: chroma:document
string_value:
— AutoChain: Build lightweight, extensible, and testable LLM Agents Auto-GPT — An experimental open-source attempt to make GPT-4 fully autonomous.

— An experimental open-source attempt to make GPT-4 fully autonomous. Autotrain-Advanced — faster and easier training and deployments of state-of-the-art machine learning models.

— faster and easier training and deployments of state-of-the-art machine learning models. DSPy — framework for solving advanced tasks with language models (LMs) and retrieval models (RMs). DSPy unifies techniques for prompting and fine-tuning LMs — and approaches for reasoning and tool/retrieval augmentation. By Stanford NLP.


key: chroma:document
string_value:
— framework for solving advanced tasks with language models (LMs) and retrieval models (RMs). DSPy unifies techniques for prompting and fine-tuning LMs — and approaches for reasoning and tool/retrieval augmentation. By Stanford NLP. GPTCache — GPTCache is a library for creating semantic cache to store responses from LLM queries.

— GPTCache is a library for creating semantic cache to store responses from LLM queries. Neural-Cherche — fine-tune neural search models such as Splade, ColBERT, and SparseEmbed on a specific dataset, and run efficient inference on a fine-tuned retriever or ranker.


key: chroma:document
string_value:
— fine-tune neural search models such as Splade, ColBERT, and SparseEmbed on a specific dataset, and run efficient inference on a fine-tuned retriever or ranker. MemGPT — Teaching LLMs memory management for unbounded context 📚🦙.

— Teaching LLMs memory management for unbounded context 📚🦙. nanoGPT — The simplest, fastest repository for training/finetuning medium-sized GPTs.

— The simplest, fastest repository for training/finetuning medium-sized GPTs. Promptify — common prompts that work well to leverage LLMs for a variety of scenarios.

— common prompts that work well to leverage LLMs for a variety of scenarios. SymbolicAI — Compositional Differentiable Programming Library.


key: chroma:document
string_value:
— common prompts that work well to leverage LLMs for a variety of scenarios. SymbolicAI — Compositional Differentiable Programming Library.

— Compositional Differentiable Programming Library. zep — a long-term memory store for LLM / Chatbot applications. Easily add relevant documents, chat history memory & rich user data to your LLM app's prompts.

LLM App Building

autollm — Ship RAG based LLM web apps in seconds.

— Ship RAG based LLM web apps in seconds. Chidoriv — reactive runtime for building AI agents. It provides a framework for building AI agents that are reactive, observable, and robust. It supports building agents with Node.js, Python, and Rust.


key: chroma:document
string_value:
— reactive runtime for building AI agents. It provides a framework for building AI agents that are reactive, observable, and robust. It supports building agents with Node.js, Python, and Rust. FastChat — open platform for training, serving, and evaluating large language model based chatbots.

— open platform for training, serving, and evaluating large language model based chatbots. GPTRouter — smoothly manage multiple LLMs and image models, speed up responses, and ensure non-stop reliability. Similar to LiteLLM, our top pick!


key: chroma:document
string_value:
— smoothly manage multiple LLMs and image models, speed up responses, and ensure non-stop reliability. Similar to LiteLLM, our top pick! guidance — a guidance language for controlling large language models.

— a guidance language for controlling large language models. haystack — end-to-end NLP framework that enables you to build NLP applications powered by LLMs, Transformer models, vector search and more.

— end-to-end NLP framework that enables you to build NLP applications powered by LLMs, Transformer models, vector search and more. Instructor — interact with OpenAI’s function call API from Python code, with Python structs / objects.


key: chroma:document
string_value:
— interact with OpenAI’s function call API from Python code, with Python structs / objects. Jsonformer — A Bulletproof Way to Generate Structured JSON from Language Models

— A Bulletproof Way to Generate Structured JSON from Language Models Langroid — easily build LLM-powered applications. Set up Agents, equip them with optional components (LLM, vector-store and methods), assign them tasks, and have them collaboratively solve a problem by exchanging messages.


key: chroma:document
string_value:
— easily build LLM-powered applications. Set up Agents, equip them with optional components (LLM, vector-store and methods), assign them tasks, and have them collaboratively solve a problem by exchanging messages. LLM App — build innovative AI applications by providing real-time human-like responses to user queries based on the most up-to-date knowledge available in your data sources.

— build innovative AI applications by providing real-time human-like responses to user queries based on the most up-to-date knowledge available in your data sources. maccarone — AI-managed code blocks in Python, lets you delegate sections of your Python program to AI ownership.


key: chroma:document
string_value:
— AI-managed code blocks in Python, lets you delegate sections of your Python program to AI ownership. magentic — prompt LLMs as simple Python functions using decorators.

— prompt LLMs as simple Python functions using decorators. Semantic Kernel — integrate cutting-edge LLM technology quickly and easily into your apps. Microsoft’s “version” of LangChain.

aider — command line tool that lets you pair program with GPT-3.5/GPT-4, to edit code stored in your local git repository.

— command line tool that lets you pair program with GPT-3.5/GPT-4, to edit code stored in your local git repository. ChatGDB — Harness the power of ChatGPT inside the GDB debugger!


key: chroma:document
string_value:
— Harness the power of ChatGPT inside the GDB debugger! Dataherald — natural language-to-SQL engine built for enterprise-level question answering over structured data. HN launch post.

— natural language-to-SQL engine built for enterprise-level question answering over structured data. HN launch post. FauxPilot — open-source GitHub Copilot server.

— open-source GitHub Copilot server. GPT Engineer — Specify what you want it to build, the AI asks for clarification, and then builds it.


key: chroma:document
string_value:
— open-source GitHub Copilot server. GPT Engineer — Specify what you want it to build, the AI asks for clarification, and then builds it.

— Specify what you want it to build, the AI asks for clarification, and then builds it. gpt-repository-loader — command-line tool that converts the contents of a Git repository into a text format that can be interpreted by LLMs.

— command-line tool that converts the contents of a Git repository into a text format that can be interpreted by LLMs. ipython-gpt — extension that allows you to use ChatGPT directly from your Jupyter Notebook or IPython Shell.


key: chroma:document
string_value:
— extension that allows you to use ChatGPT directly from your Jupyter Notebook or IPython Shell. Jupyter AI — generative AI extension for JupyterLab.

— generative AI extension for JupyterLab. PlotAI — use ChatGPT to create plots in Python and Matplotlib directly in your Python script or notebook.

— use ChatGPT to create plots in Python and Matplotlib directly in your Python script or notebook. sketch — AI code-writing assistant for pandas users that understands the context of your data, greatly improving the relevance of suggestions.

LLM Development

distilabel — AI Feedback framework for scalable LLM alignment.


key: chroma:document
string_value:
LLM Development

distilabel — AI Feedback framework for scalable LLM alignment.

— AI Feedback framework for scalable LLM alignment. language-model-arithmetic — controlled text generation via language model arithmetic.

— controlled text generation via language model arithmetic. Lit-GPT — Hackable implementation of state-of-the-art open-source LLMs based on nanoGPT. Supports flash attention, 4-bit and 8-bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training.


key: chroma:document
string_value:
— Hackable implementation of state-of-the-art open-source LLMs based on nanoGPT. Supports flash attention, 4-bit and 8-bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Lit-LLaMA — Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training.

— Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. LMQL — a query language for programming (large) language models.

LLM Experimentation


key: chroma:document
string_value:
LLM Experimentation

ChainForge — open-source visual programming environment for battle-testing prompts to LLMs.

— open-source visual programming environment for battle-testing prompts to LLMs. Langflow — UI for LangChain, designed with react-flow to provide an effortless way to experiment and prototype flows.

— UI for LangChain, designed with react-flow to provide an effortless way to experiment and prototype flows. PromptTools — a set of open-source, self-hostable tools for experimenting with, testing, and evaluating LLMs, vector databases, and prompts. HN launch post.

LLM Metrics

DeepEval — a simple-to-use, open-source evaluation framework for LLM applications.


key: chroma:document
string_value:
LLM Metrics

DeepEval — a simple-to-use, open-source evaluation framework for LLM applications.

— a simple-to-use, open-source evaluation framework for LLM applications. Fiddler Auditor — a tool to evaluate the robustness of language models.

— a tool to evaluate the robustness of language models. ragas — Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines.

— Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines. tvalmetrics — Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.

LLM Serving


key: chroma:document
string_value:
LLM Serving

Aviary — an LLM serving solution that makes it easy to deploy and manage a variety of open source LLMs. By the authors of Ray.

— an LLM serving solution that makes it easy to deploy and manage a variety of open source LLMs. By the authors of Ray. GPT4All — an ecosystem to run powerful and customized large language models that work locally on consumer grade CPUs and any GPU (ex pygpt4all/pyllamacpp) with python bindings.

— an ecosystem to run powerful and customized large language models that work locally on consumer grade CPUs and any GPU (ex pygpt4all/pyllamacpp) with python bindings. LLM Engine — engine for fine-tuning and serving large language models. By Scale AI.


key: chroma:document
string_value:
— engine for fine-tuning and serving large language models. By Scale AI. LLM Gateway — gateway for secure & reliable communications with OpenAI and other LLM providers.

— gateway for secure & reliable communications with OpenAI and other LLM providers. punica — Serving multiple LoRA finetuned LLM as one.

— Serving multiple LoRA finetuned LLM as one. Ollama — Get up and running with Llama 2 and other large language models locally.

— Get up and running with Llama 2 and other large language models locally. OnPrem.LLM — tool for running on-premises large language models with non-public data.


key: chroma:document
string_value:
— tool for running on-premises large language models with non-public data. OpenLLM — An open platform for operating large language models (LLMs) in production. Fine-tune, serve, deploy, and monitor any LLMs with ease. By BentoML.

— An open platform for operating large language models (LLMs) in production. Fine-tune, serve, deploy, and monitor any LLMs with ease. By BentoML. OpenLLMetry — Open-source observability for your LLM application, based on OpenTelemetry.

— Open-source observability for your LLM application, based on OpenTelemetry. privateGPT — Interact privately with your documents using the power of GPT, 100% privately, no data leaks.


key: chroma:document
string_value:
IncarnaMind — Connect and chat with your multiple documents (pdf and txt) through GPT and Claude LLMs in a minute.

— Connect and chat with your multiple documents (pdf and txt) through GPT and Claude LLMs in a minute. Puncia — leveraging AI and other tools, it will tell you everything about a web domain or subdomain, like finding hidden subdomains.

— leveraging AI and other tools, it will tell you everything about a web domain or subdomain, like finding hidden subdomains. scrapeghost — experimental library for scraping websites using OpenAI's GPT API.

MLOps, LLMOps, DevOps


key: chroma:document
string_value:
MLOps, LLMOps, DevOps

phoenix — ML Observability in a Notebook - Uncover Insights, Surface Problems, Monitor, and Fine Tune your Generative LLM, CV and Tabular Models.

LLaVAv — Visual Instruction Tuning - Large Language-and-Vision Assistant built towards multimodal GPT-4 level capabilities.

— Visual Instruction Tuning - Large Language-and-Vision Assistant built towards multimodal GPT-4 level capabilities. Multimodal-Maestro — effective prompting for Large Multimodal Models like GPT-4 Vision, LLaVA or CogVLM.

— effective prompting for Large Multimodal Models like GPT-4 Vision, LLaVA or CogVLM. Nougat — the academic document PDF parser that understands LaTeX math and tables.


key: chroma:document
string_value:
— the academic document PDF parser that understands LaTeX math and tables. UForm — Pocket-Sized Multi-Modal AI For Semantic Search & Recommendation Systems.

Python ML

difflogic — A Library for Differentiable Logic Gate Networks by Felix Petersen.

— A Library for Differentiable Logic Gate Networks by Felix Petersen. TensorDict — a dictionary-like class that inherits properties from tensors, such as indexing, shape operations, casting to device etc. The main purpose of TensorDict is to make code-bases more readable and modular by abstracting away tailored operations.

Performance and scalability


key: chroma:document
string_value:
Performance and scalability

AITemplate — Python framework which renders neural network into high performance CUDA/HIP C++ code. Specialized for FP16 TensorCore (NVIDIA GPU) and MatrixCore (AMD GPU) inference.

— Python framework which renders neural network into high performance CUDA/HIP C++ code. Specialized for FP16 TensorCore (NVIDIA GPU) and MatrixCore (AMD GPU) inference. AutoGPTQ — easy-to-use LLMs quantization package with user-friendly APIs, based on GPTQ algorithm.


key: chroma:document
string_value:
— easy-to-use LLMs quantization package with user-friendly APIs, based on GPTQ algorithm. composer — PyTorch library that enables you to train neural networks faster, at lower cost, and to higher accuracy. Implements more than two dozen speedup methods that can be applied to your training loop in just a few lines of code.

— PyTorch library that enables you to train neural networks faster, at lower cost, and to higher accuracy. Implements more than two dozen speedup methods that can be applied to your training loop in just a few lines of code. fastLLaMa — Python wrapper to run Inference of LLaMA models using C++.


key: chroma:document
string_value:
— Python wrapper to run Inference of LLaMA models using C++. hidet — open-source deep learning compiler, written in Python. It supports end-to-end compilation of DNN models from PyTorch and ONNX to efficient cuda kernels.

— open-source deep learning compiler, written in Python. It supports end-to-end compilation of DNN models from PyTorch and ONNX to efficient cuda kernels. LPython — compiler that aggressively optimizes type-annotated Python code. It has several backends, including LLVM, C, C++, and WASM. LPython’s primary tenet is speed. Launch blog post.


key: chroma:document
string_value:
— compiler that aggressively optimizes type-annotated Python code. It has several backends, including LLVM, C, C++, and WASM. LPython’s primary tenet is speed. Launch blog post. Petals — Run 100B+ language models at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading.

— Run 100B+ language models at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading. TokenMonster — Determine the tokens that optimally represents a dataset at any specific vocabulary size

Python Programming

Optimization / Math

Lineax — a JAX library for linear solves and linear least squares. Launch Tweet.


key: chroma:document
string_value:
Python Programming

Optimization / Math

Lineax — a JAX library for linear solves and linear least squares. Launch Tweet.

— a JAX library for linear solves and linear least squares. Launch Tweet. pyribs — a bare-bones Python library for quality diversity optimization.

Reinforcement Learning

cheese — adaptive human in the loop evaluation of language and embedding models.

— adaptive human in the loop evaluation of language and embedding models. imitation — Clean PyTorch implementations of imitation and reward learning algorithms.


key: chroma:document
string_value:
— Clean PyTorch implementations of imitation and reward learning algorithms. RL4LMs — modular RL library to fine-tune language models to human preferences. By AI2.

— modular RL library to fine-tune language models to human preferences. By AI2. trlX — distributed training of language models with Reinforcement Learning via Human Feedback (RLHF).

Time Series

aeon — A unified framework for machine learning with time series.

Video Processing

VapourSynth — video processing framework with simplicity in mind. Python docs.


key: description
string_value:
The ultimate list of the top Artificial Intelligence (AI) & Machine Learning conferences to attend in 2024.


key: chroma:document
string_value:
There's a record amount of exciting Artificial Intelligence (AI) and Machine Learning (ML) conferences worldwide and keeping track of them may prove to be a challenge.

This list provides an overview with upcoming AI & ML conferences and should help you decide which one to attend, sponsor or submit talks to.

For some conferences we added remarkable speakers and discount codes. Those with the tag "Tryolabs' Pick" indicate our personal favorites. Drop us a line here if you think of a conference that should be added to the list.

Want to be notified when new conferences are added to this list? Sign up and the monthly updates will be delivered right to your inbox.

Last update: April 2024


key: chroma:document
string_value:
Last update: April 2024

Categories

Continent

Thinking of implementing a machine learning project in your organization? We have compiled 11 essential questions to ask before kicking off a ML initiative. Go to the list


key: description
string_value:
Dive into AI innovation at the Impulse Data & AI Conference 2024 in Mexico City. Discover how Latin America is shaping the global AI landscape through groundbreaking discussions and real-world applications, from embedding engineering to ethical AI use and MLOps.


key: chroma:document
string_value:
Welcome to a journey through the heart of Latin America's AI innovation, captured at the Impulse Data & AI Conference held in Mexico City. This two-day event not only served as a confluence of minds eager to decode the future of artificial intelligence but also as a testament to the region's growing influence on the global AI stage. As both a speaker and an attendee, I had the privilege to immerse myself in the groundbreaking discussions and innovations that unfolded. Through this post, I aim to share the knowledge and insights gained, offering you a window into the future of AI as envisioned by some of the field's most forward-thinking pioneers.

Key themes and innovations


key: chroma:document
string_value:
Key themes and innovations

The conference was a kaleidoscope of AI's vast potential, with themes ranging from the technical intricacies of machine learning models to the ethical considerations of AI applications. Among the standout topics were the transformative potential of large language models (LLMs), the imperative of ethical AI use, and the strategic importance of MLOps practices in realizing AI's full potential. These discussions underscored a collective pursuit to not only advance AI technology but to do so with a keen awareness of its societal impact.

Insights from notable sessions


key: chroma:document
string_value:
Insights from notable sessions

The heart of the conference lay in its rich agenda of talks, where industry leaders shared their expertise and visions for the future of AI. Here are highlights from some of the sessions that resonated most deeply with me:

The power of embeddings : James A. Rolfsen from Rappi illuminated the pivotal role of embedding engineering in enhancing search and personalization. His insights into the profitability and technical strategies behind embeddings offered a compelling look at how AI can tailor digital experiences to individual user needs.


key: chroma:document
string_value:
: James A. Rolfsen from Rappi illuminated the pivotal role of embedding engineering in enhancing search and personalization. His insights into the profitability and technical strategies behind embeddings offered a compelling look at how AI can tailor digital experiences to individual user needs. Graph Neural Networks (GNNs) unleashed : Jaime D. Acevedo's exploration of GNNs and their application in fraud detection, credit risk estimation, and recommendation systems provided a fascinating glimpse into how AI can add layers of context to data. The real-world examples from companies like Snapchat and Amazon highlighted the practical value of these models in solving complex business problems.


key: chroma:document
string_value:
: Jaime D. Acevedo's exploration of GNNs and their application in fraud detection, credit risk estimation, and recommendation systems provided a fascinating glimpse into how AI can add layers of context to data. The real-world examples from companies like Snapchat and Amazon highlighted the practical value of these models in solving complex business problems. A new lens on data: Christian Ramirez introduced the audience to Topological Data Analysis (TDA), an emerging field that promises to revolutionize how we approach high-dimensional data sets. His presentation on using TDA for anomaly detection and problem-solving in machine learning models underscored the innovative ways in which AI can


key: chroma:document
string_value:
His presentation on using TDA for anomaly detection and problem-solving in machine learning models underscored the innovative ways in which AI can enhance our understanding of data.


key: chroma:document
string_value:
Each of these sessions not only deepened our understanding of AI's technical frontiers but also sparked conversations about its practical applications across industries.

Case studies and real-world applications

The conference wasn't just about theoretical knowledge; it was also a showcase for AI's real-world impact:

AI in the sky : Nicolás Venegas' discussion on integrating AI within LATAM Airlines revealed how technology is reshaping the airline industry. From optimizing fuel efficiency to revolutionizing revenue management and cargo optimization, his presentation underscored AI's role in enhancing operational efficiency and reducing costs.


key: chroma:document
string_value:
: Nicolás Venegas' discussion on integrating AI within LATAM Airlines revealed how technology is reshaping the airline industry. From optimizing fuel efficiency to revolutionizing revenue management and cargo optimization, his presentation underscored AI's role in enhancing operational efficiency and reducing costs. Visionary healthcare: E. Ulises Moya shared an inspiring case study from Jalisco, where AI systems are being used for patient referral with diabetic retinopathy. This initiative, recognized by the Global Partnership on AI (GPAI), showcased how AI could offer life-changing benefits in healthcare by improving diagnosis and treatment processes.

The MLOps imperative


key: chroma:document
string_value:
The MLOps imperative

As AI continues to evolve at a breakneck pace, the conversation around effectively integrating AI into real-world applications has never been more critical. This event illuminated this crucial dialogue through a focused exploration of MLOps, and it was brought to life through practical insights.

Navigating the MLOps landscape at Nissan

Rodrigo Olivares opened the discussion with a candid look into the MLOps challenges encountered at Nissan. His talk underscored the critical nature of MLOps in transforming AI models from experimental prototypes into robust, scalable solutions that drive business value. Key highlights from his session included:


key: chroma:document
string_value:
Operational challenges : Olivares shared the real-world difficulties of deploying AI models without a structured MLOps framework, leading to bottlenecks and inefficiencies that hindered timely implementation.

: Olivares shared the real-world difficulties of deploying AI models without a structured MLOps framework, leading to bottlenecks and inefficiencies that hindered timely implementation. Collaborative solutions : The necessity for a collaborative ecosystem was emphasized, bridging the gap between data scientists, engineers, and IT staff to foster a productive environment for AI deployment.


key: chroma:document
string_value:
: The necessity for a collaborative ecosystem was emphasized, bridging the gap between data scientists, engineers, and IT staff to foster a productive environment for AI deployment. Path to success: Offering a beacon of hope, Olivares discussed strategies that Nissan employed to overcome these obstacles, focusing on building a supportive MLOps culture and investing in continuous learning and development.

The MLOps journey unveiled: Tryolabs roadmap to operational excellence


key: chroma:document
string_value:
The MLOps journey unveiled: Tryolabs roadmap to operational excellence

My presentation aimed to provide a structured pathway through the complexities of MLOps, drawing from my experience in the field. The "Tryo-way" roadmap I introduced is designed to guide teams from the initial stages of data preparation to the continuous monitoring and improvement of deployed models.

Here's an overview of the journey:


key: chroma:document
string_value:
Data organization: Highlighting the cornerstone of any successful AI project, I delved into effective data management practices essential for laying a solid foundation. Model development and tracking: I discussed the significance of establishing rigorous processes for model experimentation and tracking, ensuring transparency and reproducibility throughout the model lifecycle. Streamlining production: The transition from experimental models to production environments was explored, with a focus on standardization and automation to facilitate scalability and maintainability. Continuous monitoring and iteration: Lastly, the importance of ongoing vigilance in monitoring model performance and


key: chroma:document
string_value:
scalability and maintainability. Continuous monitoring and iteration: Lastly, the importance of ongoing vigilance in monitoring model performance and adapting to new data or changing conditions was emphasized, ensuring that AI solutions remain effective and relevant over time.


key: chroma:document
string_value:
For those looking to explore each stage of the MLOps journey in greater depth, I highly recommend our previous blog post, where we dissect the nuances and strategies pivotal to mastering MLOps.

Generative AI: opportunities, challenges, and horizons

The conference convened a distinguished panel of experts in Generative AI, offering profound insights into its capabilities, ethical considerations, and the trajectory of future developments. Here's an enhanced perspective on the discussions that unfolded:


key: chroma:document
string_value:
Sasha Luccioni's session, "Generative AI Models: The Good, the Bad, and the Bias," served as a cornerstone for understanding the dual-edged nature of generative AI. She adeptly highlighted the transformative applications of technologies like ChatGPT and Dall-E 2 in enhancing user interactions through search and customer service. Yet, Luccioni conscientiously r aised the flag of caution , urging a thoughtful consideration of generative AI's limitations. Her presentation was a clarion call for the adoption of human-centered, ethical frameworks in AI development, emphasizing the importance of navigating the innovation landscape responsibly .


key: chroma:document
string_value:
, urging a thoughtful consideration of generative AI's limitations. Her presentation was a clarion call for the adoption of human-centered, ethical frameworks in AI development, emphasizing the importance of . Mabel Gerónimo shifted the narrative towards the practical implications of Generative AI in the realm of software development, focusing on GitHub's AI assistant, Copilot. By unveiling the underestimated benefits of generative AI in coding—ranging from heightened productivity to expedited deployment cycles—she painted a picture of a future where AI tools become indispensable allies in the development process . Her foresight into the evolution of AI-assisted tools offered a roadmap for


key: chroma:document
string_value:
where AI tools become indispensable allies in the development process . Her foresight into the evolution of AI-assisted tools offered a roadmap for staying at the cutting edge of technological advancement.


key: chroma:document
string_value:
. Her foresight into the evolution of AI-assisted tools offered a roadmap for staying at the cutting edge of technological advancement. Daniel Vaughan, bringing two decades of experience and authorship of pivotal O'Reilly publications, delved into the nuanced impact of Generative AI on the data science discipline. Highlighting the irreplaceable value of human skills such as analytical thinking and causal reasoning, Vaughan's discussion was a poignant reminder that despite AI's advances, certain human capabilities remain uniquely ours. His vision for the future of data science education—emphasizing critical thinking over mere programming prowess—suggests a paradigm shift towards nurturing


key: chroma:document
string_value:
for the future of data science education—emphasizing critical thinking over mere programming prowess—suggests a paradigm shift towards nurturing skills that AI cannot replicate.


key: chroma:document
string_value:
These discussions underscored a collective recognition of Generative AI's immense potential to redefine industries and human interactions. Yet, they also underscored the imperative for ethical vigilance and the cultivation of uniquely human skills that complement AI's capabilities. As we stand on the precipice of this new technological era, these insights serve as both a guide and a warning—prompting us to forge ahead with innovation, yet always with an ethical compass in

Wrapping-up


key: chroma:document
string_value:
Wrapping-up

The event was packed with technical talks, networking opportunities, and insightful discussions. As a speaker, I was able to share my experiences and insights, and as an attendee, I was able to learn from some of the great minds in the field. The diversity of topics covered, from the latest AI trends to practical data science applications, made the event a rich learning experience for everyone involved.


key: chroma:document
string_value:
Impulse Data & AI 2024 was more than just a conference; it was a celebration of the AI and data science community. It was a place where professionals, enthusiasts, and the general public could come together to learn, share, and inspire each other. Despite my initial nerves, I left the event feeling more connected to my field and more excited about the future of AI and data science.


key: description
string_value:
Unlock the full potential of Automated Machine Learning with our ultimate guide. Dive into AutoML's core concepts, advantages, and challenges, and explore a detailed comparison of leading tools.


key: chroma:document
string_value:
The quest for efficient and user-friendly solutions has led to the emergence of a game-changing concept: Automated Machine Learning (AutoML). AutoML is the process of automating the tasks involved in the entire Machine Learning lifecycle, such as data preprocessing, feature engineering, model selection, and hyperparameter optimization. It aims to make Machine Learning easier to understand, faster to prototype, and more effective, for a broader range of people, including experienced data scientists but also professionals with limited knowledge in the field.


key: chroma:document
string_value:
In this blog post, we will explore the dynamic world of AutoML, highlighting its advantages, as well as the challenges and limitations it presents. Moreover, we will delve into an exploration and comparison of the most notable tools in this domain.


key: chroma:document
string_value:
For the sake of comparison, we will use Computer Vision tasks to guide our evaluation. This will provide a concrete, illustrative example through which we can compare and evaluate the capabilities of different AutoML providers. This decision, however, should not be seen as limiting the scope of our analysis strictly to the realm of Computer Vision. Instead, it acts as a lens, offering clarity and insight into the broader landscape of AutoML tools.

Advantages of AutoML


key: chroma:document
string_value:
Advantages of AutoML

AutoML covers every section of the end-to-end Machine Learning pipeline, from data preparation and feature engineering to model selection, hyperparameter tuning, performance evaluation, and even some aspects of model deployment. By automating these complex tasks, more people and businesses can gain access to this technology without understanding the underlying algorithms.

Low entry barrier for non-experts


key: chroma:document
string_value:
Low entry barrier for non-experts

Building on this democratization effort, AutoML's user-friendly approach empowers individuals from diverse backgrounds, such as business analysts, domain experts, and decision-makers. AutoML removes the need for in-depth knowledge of algorithms, coding languages, or intricate model tuning and the collaboration flourishes, opening up new avenues for innovation in industries where data-driven decision-making was once limited to a select few.

Enables businesses without dedicated Data Science teams


key: chroma:document
string_value:
Enables businesses without dedicated Data Science teams

While we don't recommend entirely disposing of a Data Science team, this can benefit particularly small businesses, where establishing a dedicated team presents resource challenges. AutoML becomes a game-changer, enabling businesses to harness Machine Learning capabilities with the reduced necessity for a specialized team. This reduces costs and accelerates the integration of Machine Learning into everyday operations.


key: chroma:document
string_value:
AutoML proves to be essential not only for beginners but also for advanced data scientists. Even experts can benefit from its time-saving capabilities, allowing them to focus on strategic aspects like problem framing, domain understanding, and interpreting model outputs.

Speeding up model development


key: chroma:document
string_value:
The time-intensive nature of manual Machine Learning tasks often hinders models’ rapid development and deployment. AutoML addresses this challenge by expediting model development through quick prototyping and experimentation. Automation enables the exploration of various algorithms and configurations, facilitating the identification of optimal models in a fraction of the time it would take through traditional methods. The combination of automation and ensemble also enhances the adaptability to complex relationships within the data that are challenging for a single model to capture comprehensively, even in some cases yielding better performance than any single model on its own.


key: chroma:document
string_value:
Efficient use of time AutoML platforms efficiently navigate the vast search space of potential models and configurations, allowing data scientists and practitioners to focus on higher-level tasks, such as interpreting results and refining strategies. This aspect enhances the overall efficiency of the Machine Learning pipeline.

Challenges and limitations of AutoML

While AutoML brings forth a paradigm shift in democratizing Machine Learning, it is crucial to acknowledge and address the challenges that accompany this transformative technology.

Handling domain-specific knowledge


key: chroma:document
string_value:
Although it's designed to be user-friendly, AutoML tools may struggle to incorporate domain-specific knowledge effectively. Understanding the intricacies of a specific industry or problem often requires human intuition and expertise. Integrating domain-specific insights into the automated processes of data preprocessing, feature engineering and model selection, remains challenging, as it involves capturing nuanced patterns and context that may be beyond the scope of automated algorithms. In our experience, the most accurate models combine hand-coded domain-specific features derived from expert knowledge in the field with automated model exploration, parameterization, and ensembling. It’s a


key: chroma:document
string_value:
domain-specific features derived from expert knowledge in the field with automated model exploration, parameterization, and ensembling. It’s a perfect case of human-machine collaboration.


key: chroma:document
string_value:
Limited customization

AutoML platforms often prioritize simplicity and generalization to cater to a broad audience. However, this emphasis on ease of use may limit the degree of customization available to advanced users. Tailoring models to specific needs or integrating proprietary algorithms can be challenging within the constraints of some AutoML frameworks, potentially restricting their adoption in specialized domains.

Interpretability and explainability


key: chroma:document
string_value:
Interpretability and explainability

As AutoML systems become more sophisticated, the interpretability and explainability of the generated models become critical concerns. Understanding why a particular model makes a specific prediction is essential, especially in domains where decisions have significant consequences. The inherent complexity of some automated models may make it challenging to provide clear explanations, raising ethical and practical concerns in applications where transparency is crucial and, in many cases, requiring additional tools to deduce what may be going on inside the black box.


key: chroma:document
string_value:
There are also challenges and limitations associated with efficiently using resources. The extreme simplification of the development process achieved with AutoML often results in over-complex and heavier-to-maintain model architectures.

Training and prediction time

AutoML might still require a considerable amount of computational resources and time. In most cases, manual model development with carefully chosen configurations will outperform or match the performance of automated solutions in reduced training time.


key: chroma:document
string_value:
Some highly complex models generated by AutoML processes may have longer prediction times, especially if they involve deep neural networks. It's essential to strike a balance between model complexity and deployment efficiency, considering applications’ specific needs.

Resource intensive

Certain models selected by AutoML may be resource-intensive, particularly on RAM/VRAM and computationally expensive during training and prediction phases. It's important to assess whether the chosen model is aligned with the available computational resources and meets the necessary criteria for prediction time.


key: chroma:document
string_value:
Costs might be another challenge since training and running AutoML models tend to be more expensive than simpler models. In the long run, this could be an issue due to the scaling and accumulated cost of the resources over time.


key: chroma:document
string_value:
Choosing the right AutoML tool for your specific needs becomes increasingly crucial. This comprehensive dimensional analysis delves into key aspects that can guide your evaluation of AutoML tools. Each dimension provides valuable insights into the capabilities and limitations of the tools, enabling you to make an informed decision tailored to your requirements. Although other factors, such as accuracy or overall performance, could also be considered, they require a deeper analysis for a future review.


key: chroma:document
string_value:
As stated at the beginning of this blog post, Computer Vision will serve as our comparative framework. This choice allows us to systematically analyze and highlight the strengths and limitations of different AutoML providers. The findings, primarily grounded in the field of Computer Vision, are meant to guide choices across a range of potential use cases, providing a broad perspective on selecting and utilizing AutoML tools in various scenarios.


key: chroma:document
string_value:
Dimension Key aspects Support for vision tasks Evaluate range of supported tasks like segmentation, pose estimation, and identify limitations. Customization Examine flexibility in providing custom code and declarative changes to training behavior. Ease of use Analyzing how accessible the tool can be for users who do not have high expertise in Machine Learning model development. Cost Analyze the cost associated with the utilization of each tool. Data management Evaluate capabilities for labeling, collaboration, assistance mechanisms, and EDA support. Production readiness Assess the ability to export model files, create production-ready endpoints, and avoid vendor lock-in. MLOps readiness


key: chroma:document
string_value:
support. Production readiness Assess the ability to export model files, create production-ready endpoints, and avoid vendor lock-in. MLOps readiness Explore the capability to create end-to-end pipelines, monitor performance, and support continuous training. Community and support Consider active community presence, availability of official support, and developers' background. Documentation Evaluate the comprehensiveness and clarity of available documentation.


key: chroma:document
string_value:
Below, you can see the comparative ratings for the different tools. As you can see, a star-based scale was used, where a higher number of stars indicates a better comparative performance in that dimension.

Did we miss a standout tool or feature? Don't be shy - drop us a message with your recommendations!

AutoGluon is a free and open-source Python library that can handle diverse datasets, including images, text, and tabular data. With an emphasis on simplicity, Autogluon delivers integrated support for automatic deployment to the Amazon Web Services (AWS) ecosystem while operating in a cloud context, along with a smooth local execution experience.


key: chroma:document
string_value:
When it comes to flexibility and customization, AutoGluon is an excellent choice. This tool supports a wide range of vision tasks, including object detection, image classification, image-image matching, image-text semantic matching and semantic segmentation. Additionally, AutoGluon's hyperparameter tuning capabilities, along with its data augmentation settings, allow users to fine-tune their models for optimal performance.


key: chroma:document
string_value:
However, there are also limitations to consider. AutoGluon lacks specific features related to labeling and data analytics. Additionally, MLOps functionalities, such as monitoring or model tracking, are not covered. Users may need to rely on other tools to fulfill these needs.

Furthermore, it is important to note that AutoGluon operates as a Python library, which requires some coding skills. This characteristic makes it less codeless compared to other alternatives in the market.


key: chroma:document
string_value:
As part of Google Cloud's Machine Llearning platform, Vertex AI AutoML stands out for its user-friendly approach, allowing users to train different types of data for Machine Learning applications without requiring coding and using a carefully developed interface to make it intuitive and easy to use.

A major strength of Vertex AI AutoML is its seamless integration with Google Cloud's infrastructure, optimizing Machine Learning workflows for efficiency and scalability. Deploying models to production within the Google environment is streamlined, and the platform offers an array of tools for MLOps, including model versioning and monitoring pipelines for continuous integration.


key: chroma:document
string_value:
While Vertex AI AutoML provides a range of vision tasks, including object detection and classification for images, as well as video data tasks like action recognition, classification, and object tracking, it does have certain limitations. Users may find that their use cases are somewhat restricted, leading them to explore alternative solutions for greater flexibility in model development.


key: chroma:document
string_value:
An important final consideration is that the platform relies heavily on the Google environment, which could be a limitation if your infrastructure is on another cloud provider, or you don't want to depend on a specific cloud provider to develop and deploy your Machine Learning service. This service runs on the cloud; therefore, it is important to thoughtfully consider the associated costs. The pricing structure depends on the model type and running time.

Driverless AI builds upon the strengths of the H2O.ai open-source platform, offering a simplified development experience through its user-friendly interface.


key: chroma:document
string_value:
This tool has a codeless interface and offers high levels of customization and adaptability. Users can choose to use Python for further customization and it provides a variety of configuration options to fine-tune models. Furthermore, the tool includes data augmentation capabilities and a comprehensive explainability toolkit. These features are highly valuable for understanding and improving the quality of the model.


key: chroma:document
string_value:
From the perspective of limitations, it primarily caters to classification and regression tasks and lacks extensive support for vision-related functionalities. Moreover, it falls short in areas such as continuous training, model versioning, and monitoring within a production environment.

Being an integral part of the Azure platform, Azure AutoML simplifies the transition from model development to production, offering seamless integration with MLOps practices. Tasks such as putting models into production, automatic retraining, and monitoring are streamlined within the Azure ecosystem.


key: chroma:document
string_value:
One of the notable advantages of Azure AutoML is its flexibility in catering to both code-first and no-code experiences. This allows users to choose an approach that aligns with their preferences and expertise.

The tool has extensive capabilities, especially in the field of Computer Vision tasks. It supports multi-class image classification, multi-label image classification, object detection, and instance segmentation. Additionally, the platform provides customization features that enable users to customize the search space of model hyperparameters, including options such as model architectures, learning rates, optimizers, and more options.


key: chroma:document
string_value:
However, there are certain limitations to consider. While Azure AutoML supports data augmentation for Computer Vision tasks, users have limited control over the specific transformations applied. The platform applies predefined transformations based on the task, with minimal flexibility for users to fine-tune or customize these augmentations according to their specific requirements.

Finally, it is important to consider the cost associated with each use case and the potential reliance on the Azure environment that may come with this proprietary solution built on the Azure cloud.


key: chroma:document
string_value:
One of the primary strengths of Roboflow is its specialization in Computer Vision, offering comprehensive functionalities for annotation and preprocessing. The platform's user-friendly interface makes data augmentation a straightforward process, allowing users to effortlessly select and apply transformations with a simple click. Roboflow supports a wide range of Computer Vision tasks, including object detection, classification, instance segmentation, semantic segmentation, and keypoint detection.


key: chroma:document
string_value:
Notably, Roboflow stands out for its ease of use, providing a Python API and a web interface for users to create datasets, upload data, modify annotations, split datasets, and add preprocessing and augmentation transformations with minimal effort. During the training process, users can monitor metrics such as loss and accuracy in real time, estimate training times, and receive email notifications upon job completion.


key: chroma:document
string_value:
However, users should be aware of the associated costs, as while free plans are available, there are also paid plans with additional features. As of the information available, Roboflow lacks features related to continuous training, model monitoring, and model versioning, which may be essential for certain use cases and workflows.


key: chroma:document
string_value:
Ludwig is an open-source Machine Learning platform that primarily focuses on LLMs (Large Language Models), but it also covers other domains like Computer Vision and predictive modeling. Ludwig stands out with its key features of being low code, declarative, and highly customizable. It is designed to facilitate Machine Learning tasks with minimal coding, providing a user-friendly experience while offering significant flexibility in customization.


key: chroma:document
string_value:
Ludwig presents several advantages for users. It supports a variety of data augmentation transformations, such as horizontal and vertical flip, rotation, blur, brightness, and contrast. Users enjoy complete freedom in selecting and applying these transformations based on their specific needs. The platform's high degree of customization extends beyond data augmentation, allowing users to set hyperparameters for optimization and define search spaces for hyperparameter tuning.

However, it is important to remember that while low-code, the tool still requires a basic understanding of programming to be used effectively.


key: chroma:document
string_value:
However, it is important to remember that while low-code, the tool still requires a basic understanding of programming to be used effectively.

Unlike some other platforms, Ludwig does not provide labeling solutions. Users are required to handle this aspect separately. Additionally, Ludwig does not have built-in functionalities for model versioning, continuous training, or monitoring of deployed models. Users may need to supplement Ludwig with additional tools or practices to address these aspects of the Machine Learning lifecycle.


key: chroma:document
string_value:
Amazon SageMaker Canvas simplifies Machine Learning for users with limited coding skills by offering a no-code interface and exploratory data analysis (EDA) features. These capabilities allow users to visually explore data before model building, leading to informed decisions about feature engineering and model selection.


key: chroma:document
string_value:
A key strength of the platform, particularly for AWS infrastructure users, is its seamless integration with the AWS environment. This facilitates easy model deployment and leverages the scalability and reliability of AWS services. Furthermore, SageMaker Canvas extends beyond the modeling stage, offering MLOps tools for labeling, reproducibility, automation, and monitoring.


key: chroma:document
string_value:
However, it's important to note that SageMaker Canvas may not be the best choice for complex Computer Vision tasks. The platform only offers image classification functionalities, which may not be sufficient for a wide range of Computer Vision scenarios. Additionally, the expected cost of this service should be considered depending on the intended use case.

Final thoughts


key: chroma:document
string_value:
Final thoughts

In conclusion, the emergence of AutoML represents a significant advancement in the accessibility of Machine Learning, allowing a wider audience to leverage the potential of AI. However, it is important to acknowledge that while AutoML is a valuable tool, it is not a one-size-fits-all solution, and its effectiveness varies depending on the specific use case.

Furthermore, the market offers a wide variety of tools that cater to different needs. In the Computer Vision domain, industry leaders such as Vertex AI, Azure AutoML, and Sage Maker Canvas provide platform-like solutions prioritizing seamless integration and easy deployment.


key: chroma:document
string_value:
On the other hand, more specialized tools like H2O and Roboflow offer user-friendly interfaces without being tied to a specific cloud service provider. Additionally, free and open-source low-code tools like Ludwig and AutoGluon are available, allowing users to engage in model development with simplicity and customization.


key: chroma:document
string_value:
Ultimately, it's about selecting the tool that best matches your needs to fully embrace the extensive benefits AutoML offers. If this discussion has sparked your interest and you're curious about how AutoML can be tailored to the unique needs of your organization or business, we're here to help. Reach out to us to discover how these cutting-edge technologies can drive innovation, efficiency, and growth in your projects.


key: description
string_value:
Explore the advantages of fine-tuning LLMs for scalable, cost-effective GenAI inference in our comprehensive guide. Learn how to leverage GPT-4 for initial high-quality data labeling, and subsequently using Predibase to fine-tune more economical models for specific tasks.


key: chroma:document
string_value:
Data is the new oil, fueling the AI revolution. From user-tailored shopping assistants to AI researchers, to recreating the King, the applicability of AI models knows no bounds. Yet these models are only as good as the data we feed them.

State-of-the-art generative AI models offer the potential to automate information extraction processes, providing a unified and adaptable approach that can ingest vast amounts of raw data in a short timespan. However, integrating a large language model into a production pipeline can be both costly and challenging.


key: chroma:document
string_value:
In this blog post, we'll explore an effective way to leverage a powerful model like GPT-4 to produce titles for unstructured text, and then fine-tune your own open-source LLM with Predibase to perform accurate, controlled, and cost-effective inference. This task falls under the umbrella of automatic summarization (i.e., computationally extracting a small subset of information, representative of a larger document), but these ideas are applicable to the whole category of information extraction problems. So, if you're interested in unlocking the full potential of your data, keep on reading.

The power of Large Language Models (LLMs)


key: chroma:document
string_value:
The power of Large Language Models (LLMs)

LLMs are AI models that have been trained on vast amounts of data, enabling them to both digest and produce human-like text. Their ability to understand and generate text makes them perfect for the task of information extraction. Instead of having teams of manual laborers perform a mechanical and low-engagement task, a single data scientist can interact (”prompt”) with the LLM, asking it to produce the required information. Inspecting the produced information, the scientist can apply techniques such as prompt-engineering and retrieval-augmented generation (RAG) to refine the model’s results, further improving the quality of automated results.


key: chroma:document
string_value:
These tasks can take the form of automatic summarization, but also of any other Natural Language Processing task such as sentiment analysis, classification and the wider category of information extraction in general. With the advent of Multimodal LLMs, source information doesn’t even need to be text, images, audio and any kind of digitized data can be processed.

We now have the information extractor that we need. We can’t quite call it a day yet though. While state-of-the-art LLMs such as GPT-4 are incredibly powerful, their size and complexity make them expensive to use, reducing their applicability on projects with high query volumes.

The strategy: training cheaper models with LLMs


key: chroma:document
string_value:
The strategy: training cheaper models with LLMs

The idea is simple yet powerful: use an expensive LLM to perform an initial, high-quality and zero-shot, low-volume first pass, then use this data as labels to fine-tune a cheaper LLM, that instead of being a generalist like GPT-4 focuses only on your specific problem.


key: chroma:document
string_value:
This approach gives you the best of both worlds. You get the high-quality data labeling capabilities of an expensive LLM on your specific task, while also reaping much lower ongoing costs because of the smaller, more specific and cheaper model for your actual business operations. Inference costs of smaller models is orders of magnitude smaller than those of proprietary API calls, with typical cost reductions in the 40X-200X range.

Another advantage of fine-tuning is that you won’t need as specific prompts, leading to needing fewer input tokens and therefore lower cost.


key: chroma:document
string_value:
A third, often neglected advantage is that these models are Open Source. You can export the weights of your finetuned model and use readily available Open Source libraries, like llama.cpp or vLLM, to serve them where you see fit. This allows for secure prompting without the need for sending your private data to third parties and also to design your own scaling solutions avoiding rate limits and ensuring required throughputs. Also, LLM API providers will make changes to their underlying models and possibly deprecate API versions, by self hosting your own language model this risk is avoided entirely.


key: chroma:document
string_value:
This strategy is cost-effective, scalable, private, and achieves high-quality outputs, making it an ideal solution for businesses of all sizes.

Case study: leveraging GPT-4 for fine-tuning an inexpensive LLM for AI news summarization

In this section we’ll show a real life case study on how the problem of information extraction can be resolved in a high quality cost effective fashion.

The task we’ll solve is creating an AI agent capable of understanding News articles related to AI and producing meaningful titles for them. This falls under the umbrella of Automatic Summarization.

The dataset


key: chroma:document
string_value:
The dataset

For this, we compiled a database from the "AI News Shorts" section of MarkTechPost, a reputable source for current AI trends.

AI News database.

This dataset, consisting of numerous articles, posed a significant challenge for labeling, due to its sheer volume and the dense, technical nature of the content.

Putting generative AI into action: by using GPT-4 Turbo, specifically the gpt-4-1106-preview model, we generated titles using this customized prompt:


key: chroma:document
string_value:
You are a highly skilled marketing AI consultant trained in language comprehension, summarization and bizdev. You have to read the following AI news text and extract a concise, catchy title, no more than 60 characters long. Include in the title the name of the development and why it's relevant: this may be because of technological advancements, societal impact, business applications, investment opportunities, security and privacy, AI ethics, because the author is well known (ie Google, Meta, Apple, etc), or other factors. These are some good example titles: Code Llama, an Open-Source LLM for coding

DeciCoder, a 1B Params FOSS Code-Gen LLM by Deci

Google Unveils Multimodal Gemini" AI


key: chroma:document
string_value:
DeciCoder, a 1B Params FOSS Code-Gen LLM by Deci

Google Unveils Multimodal Gemini" AI

TII released Falcon 180B

Microsoft unveils custom AI chips

SynthID, identifying AI-generated images

Meta advances the development of social embodied AI agents

Replit now offers free access to AI Assistant

Milestone in AI breast cancer detection

Microsoft brings Python to Excel

GraphCast, faster, more accurate, global weather forecasting

NExT-GPT, An any-to-any Multimodal LLM

Grok, a Hitchhiker's Guide to the Galaxy based AI, by xAI

Q-ALIGN, Aligning AI Image Assessment with Human Judgment


key: chroma:document
string_value:
Grok, a Hitchhiker's Guide to the Galaxy based AI, by xAI

Q-ALIGN, Aligning AI Image Assessment with Human Judgment

LLMs Crack Visual Code, MIT CSAIL's Text-To-Image Synthesis Include only the summarized title in the response. AI News Text: {text}

The approach we just took is an application of synthetic data generation, where we had an AI model create the information for us.

For the sake of this example, we used this prompt to generate 400 titles that we manually curated one by one. During this process we found 59% of the generated titles needed no corrections while the remaining 41% needed some form of adjustment.


key: chroma:document
string_value:
We took note of the “errors” the model made (in this context we wanted the model to generate user-friendly titles so we considered specialized acronyms and jargon as errors) and found titles like these:

FlowVid: Boosting V2V Synthesis Efficiency & Quality DL3DV-10K: Pioneering NVS with Real-World Data MoE LLMs Optimized for Consumer Hardware

Readers familiar with AI jargon may realize that V2V refers to Video-2-Video, NVS to Neural View Synthesis and MoE to Mixture of Experts, but for newcomers these acronyms may be difficult to read.

We also found the model had some issues identifying two kinds of news articles: tool compilations and dataset releases.


key: chroma:document
string_value:
We also found the model had some issues identifying two kinds of news articles: tool compilations and dataset releases.

For the tool compilations, things such as “Top 40 Generative AI tools (December 2023)”, GPT-4 produced a title related to the first tool on the list, instead of realizing the article was a compilation. For the linked article, the first title GPT-4 produced was this:

Multimodal GPT-4 Elevates AI Capabilities & Safety


key: chroma:document
string_value:
Multimodal GPT-4 Elevates AI Capabilities & Safety

In the case of dataset releases, such as a paper introducing DL3DV-10K—a large-scale scene dataset for 3D Vision—the main interest lies in the new data's availability for benchmarking aspects of AI. However, the titles generated often missed the mark, focusing incorrectly, as seen here:

DL3DV-10K: Pioneering NVS with Real-World Data

Having identified these issues we reformulated the prompt to this (modifications highlighted in bold italic):


key: chroma:document
string_value:
You are a highly skilled marketing AI consultant trained in language comprehension, summarization and bizdev. You have to read the following AI news text and extract a concise, catchy title, no more than 60 characters long. Avoid overly specialized tech jargon and little known acronyms, but use buzzwords and keywords related to the area of application. Include in the title the name of the development and why it's relevant: this may be because of technological advancements, societal impact, business applications, investment opportunities, security and privacy, AI ethics, because the author is well known (ie Google, Meta, Apple, etc), or other factors. These are some good example titles: Code


key: chroma:document
string_value:
and privacy, AI ethics, because the author is well known (ie Google, Meta, Apple, etc), or other factors. These are some good example titles: Code Llama, an Open-Source LLM for coding


key: chroma:document
string_value:
Grok, a Hitchhiker's Guide to the Galaxy based AI, by xAI

Q-ALIGN, Aligning AI Image Assessment with Human Judgment

LLMs Crack Visual Code, MIT CSAIL's Text-To-Image Synthesis Include only the summarized title in the response. Don't enclose your title in quotes. If the article refers to the release of a dataset, identify the dataset's relevance. If the article isn't a news article but instead a compilation of developments, such as: 40 AI code tools you can try today, identify it and produce an according compilation title. AI News Text: {text}

And proceeded to generate titles for the remaining 600 rows in the dataset.

Fine-tuning an LLM for the specific task


key: chroma:document
string_value:
And proceeded to generate titles for the remaining 600 rows in the dataset.

Fine-tuning an LLM for the specific task

Finetuning a language model is a supervised form of training, requiring a labeled dataset, on which a subset of the models’ parameters are retrained to learn a specific task. This form of training is far cheaper to compute, since it doesn’t involve modifying the entire set of model weights, and is well suited for tuning a model to a specific kind of output.


key: chroma:document
string_value:
Typical use cases of fine-tuning include adopting a writing style, efficient text compression, producing structured output such as JSON, and adopting task specific lexicon when the original training dataset didn’t have enough variety.


key: chroma:document
string_value:
State-of-the-art code for parameter efficient fine-tuning algorithms can be found in HuggingFace’s excellent PEFT library. While the basic flow of implementing your own code to do fine-tuning with PEFT is pretty straightforward, you just instantiate a LoraConfig object with a few training parameters, and tell it to run off of a base model, executing this code is no small feat. You can see in PEFT’s own examples that fine-tuning a model as small as 3B parameters will use more than 14GB of video RAM. This can be alleviated with CPU offloading, but then training times will suffer. And this is just for the 3B parameter model, 7B models are shown to require as much as 55GB VRAM. Access to


key: chroma:document
string_value:
but then training times will suffer. And this is just for the 3B parameter model, 7B models are shown to require as much as 55GB VRAM. Access to server-grade compute GPUs alleviates these issues.


key: chroma:document
string_value:
While all this is doable, we’ll follow a more streamlined approach, that will both save us time and spare us trouble in the future.

Fine-tuning with Predibase

We want to go the easy route this time, so we’ll use Predibase for both the fine-tuning and the inference. Predibase is an LLM infrastructure provider for Open Source models, enabling its customers to serve customized language models at a fraction of the costs other services offer.


key: chroma:document
string_value:
Moreover, their platform streamlines the fine-tuning process using state-of-the-art approaches such as LoRA and QLoRA, making experimentation and deployment as simple as a few clicks. Their compute instances include NVIDIA A100 80GB VRAM GPUs to finetune your models on.

As of the writing of this post, inference of a smaller 7B model on their infrastructure will cost 50 times less per input token than GPT-4 Turbo, and 150 times less per output token. You can simulate your savings by making a copy of this handy calculator we made. With a system similar to ours making 100k requests/day you can save ~$1200 per day from API calls only.


key: chroma:document
string_value:
For the task at hand, we wanted to push the capabilities of the platform, so we trained 4 different model variants (we actually trained 10, but we’ll share the most interesting here):

llama2-13b-chat , an instruction tuned 13B parameters version of Meta’s Llama2.

, an instruction tuned 13B parameters version of Meta’s Llama2. mistral-7b-instruct , an instruction tuned 7B param surpassing llama2-13b in several benchmarks.

, an instruction tuned 7B param surpassing llama2-13b in several benchmarks. zephyr-7b-beta , an already finetuned version of mistral-7b using Direct Preference Optimization.


key: chroma:document
string_value:
, an already finetuned version of mistral-7b using Direct Preference Optimization. mixtral-8x7b-instruct-v0-1 , this one is the odd one out since it’s in a different weight class and its serving costs makes it less attractive for use. It’s a 47B parameter model but the interesting thing is it’s achieving similar performance to much heavier models such as Llama2-70B and GPT-3.5 so we wanted to see how it fared in the comparison.

Training these models using Predibase is a breeze. You upload your dataset (or connect one of the many available cloud storage provider integrations) to create a data connection:


key: chroma:document
string_value:
Once your data is connected, creating a new model task is as simple and streamlined as selecting your parameters:

Predibase allows for both full custom model training (adjusting all model weights) as well as fine-tuning. Start by creating a model repository to organize your workflows. Then select your dataset, the base model you want to fine tune and introduce the prompt template. The prompt template uses a simple wildcard logic where you can replace any content with a dataset column by enclosing it in curly brackets. For example {text} gets replaced with the value of the text column for each row in the selected dataset.


key: chroma:document
string_value:
Finally, you choose the target column the model should aim to replicate, and select whether you want to use QLoRA with 4 or 8 bit quantization, or no quantization LoRA. We tried all and found very little performance difference between them, hence we suggest going with the cheaper and faster QLoRA 4bit quantization, at least for quick experimentation.

And that’s it! Your model is trained and ready to serve with this code snippet:

Inquisitive readers might have realized that we get the model for “TWIAI Titles” .

TWIAI stands for This Week in AI, our weekly curated list of significant AI news.


key: chroma:document
string_value:
TWIAI stands for This Week in AI, our weekly curated list of significant AI news.

That’s right, we use tools such as these to automatically catalog and summarize the content, but don’t worry, human curators revise, prioritize and decide which news are worthy of attention.

Evaluating model performance: beyond manual inspection

With cost-effective inference models at our disposal for information retrieval tasks, a critical question arises: How can we ascertain the quality of the generated titles?

The challenge of evaluating LLMs


key: chroma:document
string_value:
The challenge of evaluating LLMs

Evaluation of LLMs remains a complex area of ongoing research. The challenge lies in quantitatively capturing behaviors such as "understanding the meaning" or "adhering to the prompt," which are inherently qualitative. Although manual inspection is often considered the most reliable method for assessing LLM outputs, striving for a degree of objectivity in our evaluations is essential.

Predibase’s built-in metrics


key: chroma:document
string_value:
Predibase’s built-in metrics

Predibase enhances this process with a suite of built-in metrics for model evaluation. These metrics, include a combined loss that accounts for various factors like ROUGE , BLEU , perplexity , and others, offering a foundational layer for objective analysis and model training.

Predibase Learning Curves

Custom evaluation with BERT-Score


key: chroma:document
string_value:
Predibase Learning Curves

Custom evaluation with BERT-Score

Yet, to align our evaluation closer with our specific needs—ensuring the generated titles accurately reflect the articles' main points—we turned to bert-score, a contextual similarity metric. BERT-score assesses how closely the meanings of two texts align by calculating microsoft/deberta-xlarge-mnli embeddings (a model derived from BERT) for each text's tokens. By employing cosine similarity, it measures the closeness of tokens in encoding space, with proximity indicating similarity in meaning.


key: chroma:document
string_value:
For each input token, cosine similarities are calculated, and the most closely related output token is identified. Finally similarities are averaged across all input tokens to produce the final metric. The authors propose 3 pivotal metrics: precision (the relevance of the generated title to the original text), recall (how comprehensively the title covers the original content), and the F1 score (a harmonic mean of precision and recall). A perfect score of 1 in these metrics signifies an ideal match. For our problem we’ll be focusing on precision as our primary metric of interest. This choice stems from our goal to determine how well the generated title retains the original text's core


key: chroma:document
string_value:
as our primary metric of interest. This choice stems from our goal to determine how well the generated title retains the original text's core information.


key: chroma:document
string_value:
Analyzing the results

Now, let's delve into the data. We begin by comparing the performance metrics of GPT-4 against 400 manually curated labels:

The plot shows on the x-axis the score (Precision/Recall/F1) and on the y-axis how many of the produced titles achieved that score, red bars represent the non-curated GPT-4 titles and green bars represent the manually curated titles. The more scores we have closer to the right, the better.


key: chroma:document
string_value:
Don’t take these face-value, but in principle, all curated score distributions are shifted to the right, indicating that both our own subjective assessment of better titles produced by the curation and the objective metric, agree that the curation captures more of the original meanings than the zero shot titles that GPT-4 came up with.

Moving on to the finetuned models, we’ll focus on the precision metric only, and compare GPT-4 (in red) to all finetuned models:

Test Precision: zero-shot GPT-4 titles vs finetuned small language models.


key: chroma:document
string_value:
Test Precision: zero-shot GPT-4 titles vs finetuned small language models.

You can see that, save mistral-7b which produced a few empty titles resulting in 0 Precision, all finetuned precision distributions are slightly shifted to the right. mixtral-8x7b-instruct-v0-1 is the best performing model in terms of this average precision but the much smaller and chepear zepyhr-7b-beta is so close behind that we’re opting for it. You can find average and standard deviation of precision for each of these in the Test Precision Stats sheet.

Subjective evaluation and conclusion


key: chroma:document
string_value:
Subjective evaluation and conclusion

We said that manual evaluation was best for these models. So let’s take a look at some of the examples produced by zephyr and GPT-4. This is a sneak peek of the data, you can find the full data in the Test Manual Comparison sheet of this doc:

Subjective evaluation.

This conclusive evidence aligns our objective metrics with subjective evaluations, consistently demonstrating that the smaller model frequently generate titles of comparable or superior quality.

Conclusion


key: chroma:document
string_value:
Conclusion

Thus concludes our comprehensive exploration: from sourcing a publicly available dataset and initially employing a high-cost, generalist LLM for rapid, zero-shot auto-labeling, creating an initial synthetic dataset, to refining quality through manual curation, and ultimately fine-tuning a smaller LLM for efficient and precise inference in production settings.


key: chroma:document
string_value:
Further work could involve refining the prompt into shorter a less specific version, for additional savings now that our model has been finetuned. Another avenue could be enriching the synthetic data generation process by building a retrieval augmented generation system that would feed articles and titles similar to the one being summarized instead of the generic examples we provided in the prompt.

Balancing AI performance and cost


key: chroma:document
string_value:
Balancing AI performance and cost

Navigating the complexities of AI integration into your business operations doesn't have to be a daunting task. With the right approach, leveraging advanced LLMs for data labeling and subsequently fine-tuning less expensive models allows for the exploitation of GenAI's potential in a manner that is both impactful and cost-efficient. At Tryolabs, our mission is to transform this potential into your competitive advantage.


key: chroma:document
string_value:
We are committed to crafting bespoke AI solutions that not only meet but exceed your unique business requirements. Our expertise lies in identifying and executing the most effective strategies for each project, ensuring that cost-effectiveness is not just an option but a priority. Let us guide you through the process of making AI work for you — efficiently and economically.

Ready to take the next step? Let’s talk!


key: description
string_value:
NuPIC is transforming NLP inference on CPUs. Learn how this innovative platform achieves remarkable speedups rivaling GPU performance. We dive into real-world impact across use cases like semantic search and text classification.


key: chroma:document
string_value:
In the ever-evolving landscape of AI, the demand for powerful Large Language Models (LLMs) has surged. This has led to an unrelenting thirst for GPUs and a shortage that causes headaches for many organizations. But what if there was a way to use LLMs efficiently without relying solely on scarce GPUs?

Numenta, a Bay Area-based company with nearly two decades in neuroscience research — possibly recognized for the influential book A Thousand Brains: A New Theory of Intelligence published by its Co-Founder — has recently launched its first commercial product: NuPIC.


key: chroma:document
string_value:
The name stands for “Numenta Platform for Intelligent Computing,” and it's an LLM acceleration platform that applies neuroscience principles to achieve remarkable speedups on CPU-based LLM inference, promising even to surpass GPU performance in many scenarios.

Source: Numenta.


key: chroma:document
string_value:
Source: Numenta.

Through a partnership with Numenta, we got early access to NuPIC and in this blog post, we will explore the platform’s architecture and then delve into its capabilities, showing you how to solve a classical use case of semantic search. To test the CPU vs GPU claims, we will do comparative benchmarks against Hugging Face's BERT on CPU and GPU and set the stage for discussing its pros, cons, and potential applications.

Join us on this exploration as we examine NuPIC, the claims surrounding its performance, and its potential role in LLM inference.


key: chroma:document
string_value:
Join us on this exploration as we examine NuPIC, the claims surrounding its performance, and its potential role in LLM inference.

Full disclosure: Tryolabs has partnered with Numenta to support the development of NuPIC. Our team worked closely with Numenta engineers to design and implement key infrastructure for NuPIC's modules, including the Inference Server and Training Module. This hands-on involvement has provided us with valuable technical expertise and experience using NuPIC. While we aim to provide an objective view of the platform, our collaboration with Numenta is relevant to disclose upfront.

But what is NuPIC?


key: chroma:document
string_value:
But what is NuPIC?

NuPIC aims to accelerate LLM inference on CPUs. But what does that look like in practice? It currently consists of two docker containers, which allow easy integration into any AI inference infrastructure:

1. NuPIC Optimized Inference Server

This server runs on a CPU and presents a simple interface to get LLM model inference for different NLP tasks, such as classification, Q&A, or embeddings. Key features:

Key features:

Handles multiple simultaneous requests , as each inference step runs in a single core, and several simultaneous models without additional configuration.


key: chroma:document
string_value:
, as each inference step runs in a single core, and several simultaneous models without additional configuration. Achieves better throughput than an NVIDIA A100 GPU when running on a 4th gen Intel Xeon CPU (according to this case study) by leveraging the AVX-512 and AMX instruction sets.

NuPIC is delivered with a set of optimized BERT and BERT Large models.

2. NuPIC Training Module

This module runs on GPUs and allows fine-tuning the optimized models on your own data.

Key features:

Improves application accuracy.

Output models can be transferred to the Inference Server to start using in your application.


key: chroma:document
string_value:
Key features:

Improves application accuracy.

Output models can be transferred to the Inference Server to start using in your application.

The NuPIC Training Module includes a Weight & Biases integration for experiment tracking.

With NuPIC's architecture in mind, let's see it in action for a semantic search use case.

Using NuPIC's SBERT for semantic search

Imagine you're a developer building a content-rich website for a travel agency. Your goal is to empower users to find their dream destinations effortlessly, making the most out of the intent in their words.

Setting the scene: the power of semantic search


key: chroma:document
string_value:
Setting the scene: the power of semantic search

In the world of travel, words carry a world of meaning. You want users to type in their travel desires, whether it's "beach paradises," "cultural escapades," or "adventurous getaways”, and receive tailored recommendations that capture the essence of their desires. Enter semantic search – the art of understanding not just keywords but the intent behind them. But the question is: how to perform this intent understanding?

The challenge: machine intent-understanding


key: chroma:document
string_value:
The challenge: machine intent-understanding

Traditional keyword-based search engines often fall short when capturing semantic nuances; that’s why an AI approach is more suitable here, given the recent rise of LLMs and their interpretation power.


key: chroma:document
string_value:
A BERT-based model is appropriate for this task, as they are specifically trained to represent text with embeddings — a mathematical representation that encapsulates the contextual meaning of the text. In particular, Sentence-BERT, or SBERT for short, is a powerful language model specifically trained to better understand the structure of whole sentences, much more than its predecessors. Thus making it doubly appropriate for retrieving meaningful embeddings and paving the way for a more intelligent and intuitive search experience.

The solution: NuPIC's SBERT


key: chroma:document
string_value:
The solution: NuPIC's SBERT

Pre-trained magic : NuPIC's SBERT model was pre-trained on a vast text corpus, including travel guides, blogs, and reviews. This means it understands the intricacies of travel lingo, from the excitement of discovering hidden gems to the tranquility of sunset beach strolls.


key: chroma:document
string_value:
: NuPIC's SBERT model was on a vast text corpus, including travel guides, blogs, and reviews. This means it understands the intricacies of travel lingo, from the excitement of discovering hidden gems to the tranquility of sunset beach strolls. Embeddings unleashed : Let's say a user searches for "secluded mountain retreats.” NuPIC's SBERT converts this query into an embedding that captures the essence of this phrase.


key: chroma:document
string_value:
: Let's say a user searches for "secluded mountain retreats.” NuPIC's SBERT converts this query into an that captures the essence of this phrase. Mapping meanings : Your website's content, from travel package descriptions to user reviews, is also transformed into embeddings using SBERT and stored into a vector dataset. Now, it's not just about matching keywords; it's about matching meanings.


key: chroma:document
string_value:
: Your website's content, from travel package descriptions to user reviews, is also transformed into embeddings using SBERT and stored into a vector dataset. Now, it's not just about matching keywords; Tailored recommendations: You can provide excellent recommendations by retrieving the closest embeddings to the user request! But you're not just providing recommendations based on keyword matches; you're offering suggestions that resonate with the user's intent. If users seek the serenity of mountains, they won't be bombarded with options for bustling city breaks.

Implementation steps


key: chroma:document
string_value:
Implementation steps

Now, we'll guide you through implementing your search functionality using NuPIC's SBERT model. From infrastructure setup to real-time embeddings and personalized recommendations, let's explore the journey to enriching user interactions and understanding their intent.

0. Getting access to NuPIC

As of September 2023, NuPIC is available, and you can contact Numenta to request a demo.

1. Setting up the inference server: deploy NuPIC

To get started, you'll need a Linux machine equipped with an Intel CPU that supports the AMX instruction set, at least 6GB of RAM, and has Docker installed.


key: chroma:document
string_value:
Then, you must download NuPIC’s Inference Server container using the Numenta-provided download.sh script and run it with the run_inference_server.sh script.

This server will play a vital role in generating embeddings and facilitating semantic searches.

2. Building your knowledge base: vector dataset population

Vector Datasets, such as Chroma or Pinecone, are a great solution to store, manage, and query embeddings. You must configure your vector dataset and keep all the travel guide summaries as embeddings.


key: chroma:document
string_value:
You'll need to convert the travel guide summaries into embeddings using NuPIC's SBERT model to achieve this. Thankfully, it’s very straightforward: here is an example code of how to generate the embeddings using the NuPIC client in Python for a fixed set of sentences:

The generated embeddings capture the contextual meaning of the text and lay the foundation for your semantic search functionality.

3. On-the-fly semantics: real-time embedding generation

Enable your backend system to instantly convert user input into embeddings. For this purpose, take advantage of NuPIC's Python client, which simplifies interaction with the NuPIC Inference Server.


key: chroma:document
string_value:
By making a single API call using this client, your system can swiftly generate embeddings in real-time in the same way as in the previous step.

4. Matching meanings, not just keywords: content retrieval


key: chroma:document
string_value:
With the user input transformed into an embedding, proceed to query the vector database with it to get the content closest to the input embedding. This content aligns with the user's intent and allows you to pull the corresponding travel guide information. Now, present these tailored recommendations in an appealing and user-friendly manner on your website. By following these steps, you can harness the capabilities of LLMs to craft a semantic search experience that's both intuitive and intelligent. This approach transcends traditional keyword-based search engines, allowing you to offer your users personalized interactions that truly grasp the essence of their request.


key: chroma:document
string_value:
This semantic search example is just one simple use case of NuPIC. You can improve this same example by merging duplicate requests or prioritizing them by importance. Or you can move and enhance other applications, such as Question & Answering (Q&A), and speed up the retrieval of important information in contracts or legal documents. You can also use techniques like Retrieval-Augmented Generation (RAG) for the BERT models, to add specific domain knowledge and reduce hallucination in an LLM.

Benchmarking NuPIC against BERT


key: chroma:document
string_value:
Benchmarking NuPIC against BERT

Now, let's put NuPIC to the test by comparing its performance against Hugging Face's BERT on CPU and GPU. We'll conduct a straightforward experiment to test the efficiency and speed of NuPIC.

You can find all the code and instructions in this repo.

Experiment setup

We'll use the Financial Sentiment Analysis dataset and measure the time required to generate embeddings for each model:

NuPIC's SBERT Large Hugging Face's BERT Large (CPU) Hugging Face's BERT Large (GPU)

All CPU experiments ran on a 4th Generation Intel Xeon from a m7i.2xlarge AWS EC2 instance, while the GPU experiment ran on a NVIDIA A100 .

NuPIC's SBERT


key: chroma:document
string_value:
NuPIC's SBERT

First, we need to import and setup the NuPIC's Client to connect to our running NuPIC Inference Server. Then, we iterate over the sentences and use NuPIC's SBERT to get the embedding. Finally, we measure the time it takes with the following script:

Hugging Face's BERT on CPU

To make a fair comparison to NuPIC, we will use Hugging Face’s BERT Large model, which has almost the same architecture as the numenta-sbert-2-v2 model we used before. We will only use one thread for inference as the NuPIC Inference Server assigns one thread for each request.


key: chroma:document
string_value:
First, let's import the necessary libraries. Next, load the BERT model and tokenizer required. Finally, measure the time it takes to run the inference from text to tokens and to embedding on the whole dataset.

Hugging Face's BERT on GPU

Lastly, let's measure the inference time for Hugging Face's BERT on GPU, by simply replacing the "cpu" with "cuda" at the beginning of the previous script. Remember that this experiment also uses a batch size of 1 for inference.

Results and comparison

Now that we've measured the inference time for NuPIC's SBERT, Hugging Face's BERT both on CPU and GPU, we can compare the results, which show the efficiency of NuPIC's SBERT for CPU-based NLP tasks.


key: chroma:document
string_value:
NuPIC HF (CPU) HF (GPU) Execution time 21.2ms 302.9ms 23.6ms Max hourly throughput * ~1.358.000 ~95.000 ~1.220.000 Cost per 1M requests * * 1.18 16.84 26.80

The CPU experiments were run on an AWS EC2 m7i.2xlarge instance, and the GPU experiment was run on Google Colab with an NVIDIA A100 . * With a maximum of 8 concurrent requests or a batch size of 8. * * Using AWS as a reference, with a p4d.24xlarge as GPU instance.


key: chroma:document
string_value:
In our experiments, NuPIC's SBERT demonstrates a remarkable 15x speed for CPU inference time and a 10% speed improvement when compared with a GPU, and between 14 and 22 times cost reduction, making it a compelling choice for CPU-based LLMs, especially in scenarios where GPU resources may be limited or costly to deploy.


key: chroma:document
string_value:
An important point to remark is that NuPIC handles each request independently and does not require any additional batching to make an efficient use of the computational resources as the GPU would. Another important point is that NuPIC can handle several models simultaneously while keeping the same throughput, while GPUs have significant throughput drops whenever you need to use a different model. This means that in practice, the throughput gains of NuPIC can be much higher than what this benchmark shows.


key: chroma:document
string_value:
How to Scale NuPIC We tested this to ensure NuPIC's scalability using infrastructure as a code: We began by creating a template instance with the NuPIC inference server, ensuring that every new instance inherits the same configuration and settings.

Then, we set up the autoscaling service to allow us to adjust the computational resources to match real-time demand.

And finally, we set the required network services as the load balancer, firewall, and proxies. The result? A scalable LLM solution that effortlessly handled surges in workload without breaking a sweat.


key: chroma:document
string_value:
So, there you have it, folks! NuPIC stands out as it achieves impressive 15x speedups and allows for an easy integration, allowing to deliver practical and efficient LLM solutions running on a CPU. Now let's point out some Pros and Cons of NuPIC.

Pros and Cons of NuPIC

NuPIC offers notable benefits but also has some limitations to consider, as summarized below:


key: chroma:document
string_value:
Pros and Cons of NuPIC

NuPIC offers notable benefits but also has some limitations to consider, as summarized below:

Pros Cons Privacy and control: runs in your infrastructure Current version has variants of BERT out of the box Much faster CPU inference speeds Newer platform with less integration support Better price/performance ratio than GPUs Documentation still being worked on Avoids GPU shortage and scaling issues Manual process for existing models Fine-tuning module

Pros


key: chroma:document
string_value:
Pros

Privacy and control . Runs fully in your infrastructure so you maintain control over data and models. Whether you operate in the cloud or prefer an on-premise environment, NuPIC's privacy policy allows you to make strategic decisions aligned with your organization's unique needs.


key: chroma:document
string_value:
. Runs fully in your infrastructure so you maintain control over data and models. Whether you operate in the cloud or prefer an on-premise environment, NuPIC's privacy policy allows you to make strategic decisions aligned with your organization's unique needs. CPU efficiency . Optimized to achieve much faster LLM inference on CPUs vs GPUs. In our benchmarking experiments, NuPIC achieved much faster LLM inference speeds on CPUs compared to BERT on both CPU and GPU.


key: chroma:document
string_value:
. Optimized to achieve much faster LLM inference on CPUs vs GPUs. In our benchmarking experiments, NuPIC achieved LLM inference speeds on CPUs compared to BERT on both CPU and GPU. Cost-effectiveness . Better price performance ratio compared to expensive GPUs. A cost-effective solution could be achieved without compromising performance by using CPUs for LLM.

. Better price performance ratio compared to expensive GPUs. A cost-effective solution could be achieved without compromising performance by using CPUs for LLM. Availability . Avoids GPU shortage issues. Can scale CPU resources more easily.


key: chroma:document
string_value:
. Avoids GPU shortage issues. Can scale CPU resources more easily. Fine-tuning module. NuPIC includes a simple training module that allows you to fine-tune NuPIC models with your own data while keeping all the performance optimization.

Cons

It is a Paid Framework . As we’ve said before, you’ll need to contact Numenta’s team to get your copy of NuPIC. You’ll need to do the math based on the pricing and the money you’ll be saving on GPUs (which are expensive).


key: chroma:document
string_value:
. As we’ve said before, you’ll need to contact Numenta’s team to get your copy of NuPIC. You’ll need to do the math based on the pricing and the money you’ll be saving on GPUs (which are expensive). Current version has variants of BERT out of the box, with GPT models likely coming in the future. If you are currently working with a different model architecture, you should consider moving to a BERT-like model to get the full optimization that NuPIC offers.


key: chroma:document
string_value:
with GPT models likely coming in the future. If you are currently working with a different model architecture, you should consider moving to a BERT-like model to get the full optimization that NuPIC offers. Emerging community . As a newer platform, NuPIC has an emerging user community. While resources are still limited today, early adopters can help shape future growth through real-world usage and feedback.


key: chroma:document
string_value:
. As a newer platform, NuPIC has an emerging user community. While resources are still limited today, early adopters can help shape future growth through real-world usage and feedback. Manual process for existing models. You can't just plug-and-play existing models yet. You will need the help of Numenta to put your model through a manual process to convert it to NuPIC format.

Closing thoughts


key: chroma:document
string_value:
Closing thoughts

In this exploration and testing of NuPIC, we've managed to examine the results of neuroscience being applied to AI, achieving notable LLM inference speeds on CPUs. With experiments and usage examples, we've benchmarked NuPIC's capabilities to enhance inference speed, specifically for semantic search.

NuPIC does make a leap in reducing hardware limitations for language models. Be it conversational AI, personalized recommendations, or other language-based AI applications. This is a new tool, so what could the future bring along? Exciting times to be in the field of AI inference!


key: chroma:document
string_value:
At Tryolabs, we have hands-on experience with NuPIC and are eager to guide you on this new frontier of efficient LLM inference. Our team stays on the cutting edge of AI advances so we can deliver strategic guidance and custom solutions leveraging the latest innovations like NuPIC. Contact us to learn how we can help you build transformative AI applications enhanced by next-generation technology.


key: description
string_value:
An introduction to Price Optimization, its opportunities with Machine Learning, and how retailers can use automated pricing solutions to increase revenue.


key: chroma:document
string_value:
The challenge of setting the right price

Setting the right price for a good or service is an old problem in economic theory. There are a vast amount of pricing strategies that depend on the objective sought. One company may seek to maximize profitability on each unit sold or on the overall market share, while another company needs to access a new market or to protect an existing one. Moreover, different scenarios can coexist in the same company for different goods or customer segments.


key: chroma:document
string_value:
In this blog post, we’ll present the problem of price optimization for retail – which has its own particularities – and how retailers can take advantage of the tremendous power of Machine Learning technology to build effective pricing solutions.

These are some of the crucial questions that retailers recurrently face: What price should we set if we want to make the sale in less than a week?

What is the fair price of this product, given the current state of the market, the period of the year, the competition, or the fact that it is a rare product?


key: chroma:document
string_value:
Given that these days it is very easy for a customer to compare prices thanks to online catalogs, specialized search tools or collaborative platforms, retailers must pay close attention to several parameters when setting prices. Factors such as competition, market positioning, production costs, and distribution costs, play a key role for retailers in order to make the right move. Check this example for a deep dive into a real-life sales data analysis for an online retailer.


key: chroma:document
string_value:
Machine Learning can be of great help in this case and have an enormous impact on KPIs. Its power lies in the fact that the developed algorithms can learn patterns from data, instead of being explicitly programmed. Machine Learning models can continuously integrate new information and detect emerging trends or new demands.

The use of Machine Learning is a very attractive approach for retailers. Instead of using, for example, aggressive general markdowns (which is often a bad strategy), they can benefit from predictive models that allow them to determine the best price for each product or service.

Let's talk about building new solutions Let's talk

What is price optimization?


key: chroma:document
string_value:
Let's talk about building new solutions Let's talk

What is price optimization?

Briefly, price optimization uses data analysis techniques to pursue two main objectives:

Understanding how customers will react to different pricing strategies for products and services, i.e., understanding the elasticity of the demand. Finding the best prices for a given company, considering its goals.

Pricing systems have evolved since the early 1970s until now, from applying very simple strategies, such as a standard markup to base cost, to being capable of predicting the demand of products or services and finding the best price to achieve the set KPI.


key: chroma:document
string_value:
Understanding consumer preferences can have several uses besides optimizing prices. If this picks your interest, take a look at this article where we modeled customer demand to perform assortment optimization in a large B2B Consumer Packaged Goods company, delivering a 10 million dollar profit increase in the first six months, representing a 2% uplift of total profits.

Price optimization techniques can help retailers evaluate the potential impact of sales promotions or estimate the right price for each product if they want to sell it in a certain period of time.

Current state-of-the-art techniques in price optimization allow retailers to consider factors such as:

Competition

Weather


key: chroma:document
string_value:
Current state-of-the-art techniques in price optimization allow retailers to consider factors such as:

Competition

Weather

Season

Special events / holidays

Macroeconomic variables

Operating costs

Warehouse information

to determine:

The initial price

The best price

The discount price

The promotional price

Using different kind and sources of data to find the prices that improve profits.

Price optimization vs dynamic pricing

Even though sometimes these two concepts are used as synonyms, they represent different concepts. The main difference is that dynamic pricing is a particular pricing strategy, while price optimization can use any kind of pricing strategy to reach its goals.


key: chroma:document
string_value:
For example, using a dynamic pricing strategy, retailers can dynamically alter the prices of their products in order to match their competitor's price. This strategy would imply changing prices very frequently but not necessarily being this the best strategy possible. Price optimization techniques focus on finding the price that maximizes a defined cost function (e.g., the company's margin), considering many different factors to suggest such price or price range for different scenarios. Depending on the particular use case, this can indeed be performed in a dynamic way, and thus combining dynamic pricing + optimization is the go-to option for many scenarios.


key: chroma:document
string_value:
In some highly competitive scenarios where anticipating demand is hard, incorporating dynamic pricing hand crafted rules into a pricing system can allow to take quick advantage of certain market situations. Stay tuned for the next blogpost in our pricing series in which we’ll show a case study where we combined Dynamic Pricing with Continuous Experimentation to increase profits in a large tech powered Amazon e-tailer.

Price optimization vs automatic pricing

Moreover, it is important to differentiate price optimization from automatic pricing as they primarily solve two different pain points: sub-optimal pricing strategy vs. excessive cost of pricing.


key: chroma:document
string_value:
The main difference is that we focus on a price automation solution when pricing is a pain point for the company in terms of costs. By automatically pricing the items we are not changing the pricing strategy itself but we are changing the pricing process making it cheaper and faster. On the other hand, when we think of a price optimization solution we change the pricing strategy in order to maximize an objective function, subject to different business constraints.


key: chroma:document
string_value:
This is to say that by implementing a price optimization solution we are automating our pricing process but not vice versa; not necessarily all price automation solutions optimize the pricing strategy. Both price automation and price optimization solutions could be understood as dynamic pricing if the frequency of price changes is high.


key: chroma:document
string_value:
Autopricing on its own can have a huge impact on a company’s bottom line, due to the capacity of Machine Learning models to simultaneously consider vast amounts of information from disparate sources. If you want to take a look at how we leveraged Machine Learning to incorporate non-traditional sources of information into a Real Estate pricing solution, check this article in our pricing series.

Price automation with and without Machine Learning


key: chroma:document
string_value:
Price automation with and without Machine Learning

Finally, price automation can be developed with or without Machine Learning. The difference between these two approaches is that without Machine Learning the pricing rules are pre-defined while with Machine Learning rules are obtained in a data-driven way.

For example, a price automation system without using Machine Learning would take the form of a pre-defined set of rules such as:

Mark up all products in FOOD CATEGORY by 15%

If BRAND is in MARKETING CAMPAIGN add a 10% discount

If KEY VALUE ITEM set price equal to competition and lower it by 5%

Adjust all prices to end with “.99”


key: chroma:document
string_value:
If KEY VALUE ITEM set price equal to competition and lower it by 5%

Adjust all prices to end with “.99”

On the other hand, a price automation solution with Machine Learning implies training a model capable of automatically price items the way they would be priced by a human expert at scale. The model could take in historical data and different characteristics of the product as well as unstructured data such as images and text and would learn the pricing rules with no explicit coding, adapting to changes in the environment in a much richer and dynamic way.

Key differences between different pricing concepts


key: chroma:document
string_value:
Key differences between different pricing concepts

Feature Dynamic Pricing Price Optimization Price Automation with ML Prices change frequently Yes Yes Yes Prices are set automatically Yes Yes Yes Main goal is to reduce pricing process cost - - Yes Main goal is to optimize pricing strategy - Yes - Demand Forecasting - Yes - Estimate Store/SKU price elasticity of demand - Yes - "What if" Pricing Scenario Capability - Yes -

What Machine Learning can do for retail price optimization


key: chroma:document
string_value:
What Machine Learning can do for retail price optimization

The pricing strategies used in the retail world have some peculiarities. For example, retailers can determine the prices of their items by accepting the price suggested by the manufacturer (commonly known as MSRP). This is particularly true in the case of mainstream products. Another simple strategy is keystone, which consists in defining the sale price as double the wholesale price or cost of the product.


key: chroma:document
string_value:
While these and other strategies are widely used, Machine Learning enables retailers to develop more complex strategies that work far better to achieve their KPIs. Machine Learning techniques can be used in many ways to optimize prices. Let's have a look at a typical scenario.

For example, a widely adopted pricing strategy technique that enhances this technology is dynamic pricing. Using this strategy, retailers can dynamically alter the prices of their products based on current market demand. However, changing the prices dynamically with no objective function in mind may lead to suboptimal results. This is why we suggest using dynamic pricing jointly with price optimization techniques.


key: chroma:document
string_value:
A typical scenario

Imagine an e-commerce or brick-and-mortar retailer who wants to estimate the best prices for new products for the next season. The competition is hard, so their prices and promotions need to be taken into consideration. Therefore, the retailer adopts a widely used strategy: competitive pricing. Simply put, this strategy defines the price of a product or service based on the prices of the competition.

Let’s see the steps needed to develop a Machine Learning solution for this use case.

Process of defining prices in retail with price optimization using Machine Learning.

1. Gather input data


key: chroma:document
string_value:
Process of defining prices in retail with price optimization using Machine Learning.

1. Gather input data

First of all, we need data. To train Machine Learning models, it is necessary to have different types of information (structure or unstructured data):

Transactional : a sales history that includes the list of the products purchased and, eventually, the customers who purchased them.

: a sales history that includes the list of the products purchased and, eventually, the customers who purchased them. Description of the products : a catalog with relevant information about each product such as category, size, brand, style, color, photos and manufacturing or purchase cost.


key: chroma:document
string_value:
: a catalog with relevant information about each product such as category, size, brand, style, color, photos and manufacturing or purchase cost. Data on past promotions and past marketing campaigns.

and past marketing campaigns. Customer Reviews : reviews and feedback given by customers about the products.

: reviews and feedback given by customers about the products. Data on the competition : prices applied to identical or similar products.

: prices applied to identical or similar products. Inventory and supply data .

. In the case of physical stores: information about their geographical location and that of the competitors.


key: chroma:document
string_value:
. In the case of physical stores: information about their geographical location and that of the competitors.

For further reference take a look at this post that highlights actionable steps to get your data ready for price optimization.

Depending on the set KPIs and the way of modeling the solution, some of this data may not be necessary. For example, if there is little or no information about customers, which is sometimes the case for brick-and-mortar retailers, the model can nonetheless be trained.


key: chroma:document
string_value:
In contrast, information about the competition is crucial for a competitive pricing strategy. In many cases, it is even possible to connect via APIs to this information or monitor it online. Checkout a real-life data collection example here.

2. Define goals and constraints

The next step is to define the strategic goals and constraints.

Retailers may pursue a unique, clear objective of profit maximization. However, they may also be interested in customer loyalty (e.g. increasing the net promoter score or the conversion rate) or in attracting a new segment (e.g. young people).


key: chroma:document
string_value:
Restrictions may be of legal nature (e.g. if some type of control of sale prices is carried out), they may have to do with the reputation of the company (e.g. fearing a bad image for applying favorable prices only to a certain segment of customers) or be related to physical aspects such as the capacity of a store or the average time of supply.

Each particular scenario will impact the way the problem is modeled. It is possible, and usually very interesting, to test different scenarios for the same retailer, which implies using different models.

3. Modeling and training


key: chroma:document
string_value:
3. Modeling and training

In this step, the data previously gathered is used to train the Machine Learning models. There is a wide variety of models that can be used in price optimization. Historically, Generalized Linear Models (GLMs) have been used (in particular, logistic regression). However, for a few years, more complex and powerful methods have been developed. For instance, depending on the volume of data available, it could be possible to use Deep Learning methods or even reinforcement learning techniques.


key: chroma:document
string_value:
In this case, in which we are dealing with new products for the next season, there is an additional difficulty since there is no previous product data. The interesting thing is that the Machine Learning models will know how to find similar products and be effective despite not having specific prior data. The same happens in the case of retailers that sell rare or exotic products.

You can take a look at a real-life example of demand forecasting modeling here.

4. Execute and adjust prices


key: chroma:document
string_value:
You can take a look at a real-life example of demand forecasting modeling here.

4. Execute and adjust prices

Once the model is trained, prices can be estimated for the new products and tested. Depending on the modeling, the estimate may be an exact price or a range. The prices obtained by the model can be subsequently adjusted manually by the retailer and optimized regularly.

More opportunities of using Machine Learning for price optimization


key: chroma:document
string_value:
More opportunities of using Machine Learning for price optimization

Machine Learning can be used for other tasks related to pricing in retail. For example, given a new product, a clustering algorithm can quickly associate it with similar products to obtain a probable price segment. Another compelling possibility is to jointly predict prices and demands for items that were never sold.

More generally, Machine Learning can be a tremendous tool for insights:

In what way is the sale of pants impacted when shirts’ prices are drastically cut?

When efforts are made to sell more pens, are the related products, such as ink, notebooks or work agendas, impacted?


key: chroma:document
string_value:
When efforts are made to sell more pens, are the related products, such as ink, notebooks or work agendas, impacted?

Are customers who buy a certain computer more or less likely to buy monitors the following month?

Are clients inactive in the last year sensitive to a promotion campaign?

These are just some examples of the questions that Machine Learning models can help answer.

Advantages of price optimization with Machine Learning

In addition to automation and speed, there are several advantages to using Machine Learning to optimize prices.


key: chroma:document
string_value:
In addition to automation and speed, there are several advantages to using Machine Learning to optimize prices.

First, Machine Learning models can consider a huge number of products and optimize prices globally. The number and nature of parameters and their multiple sources and channels allow them to make decisions using fine criteria. This is a daunting task if retailers try to do it manually, or even using basic software.


key: chroma:document
string_value:
For example, it is known that changing the price of a product often impacts the sales of other products in ways that are very hard to predict for a human. In most cases, the accuracy of a Machine Learning solution will be significantly higher than that of a human. In addition, retailers can modify the KPI and immediately see how the models recalculate prices for the new goals.

Second, by analyzing a large amount of past and current data, a Machine Learning can anticipate trends early enough. This is a key issue that allows retailers to make appropriate decisions to adjust prices.


key: chroma:document
string_value:
This is of particular importance both for new products for which we have no past price reference and for existing products in dynamic markets whose price characteristics change over time. Well designed pricing algorithms can exploit this by solving a problem known as the Exploration vs Exploitation tradeoff, where the algorithm aims to balance exploiting the currently known good prices, with exploring other promising pricing regions that may arise. If you want to know more, you can take a look at this article where we apply a Reinforcement Learning agent capable of balancing exploration and exploitation to pricing. You can even play a game to try to beat our pricing algorithm!


key: chroma:document
string_value:
Alternatively, you can also follow our Mastering Pricing Strategies series: in its first volume, we explain in depth the optimization problem, and then define a simulated market on which we can evaluate these algorithms on fair grounds. Then the second volume is a deeper dive, where we show how we resolved the Exploration vs Exploitation problem for a more advanced class of algorithms that can leverage more complex data.


key: chroma:document
string_value:
Finally, in the case of a competitive pricing strategy, Machine Learning solutions can benefit from systems that can continuously crawl the web and social media to gather valuable information about prices of competitors for the same or similar products, what customers say about products and competitors, and a competitor's deals for certain products, as well as their price history over the last number of days or weeks.

A system that can learn most of what is happening in the market allows retailers to have more information than their competitors in order to make better decisions.

Price optimization for brick-and-mortar and e-commerce retailers


key: chroma:document
string_value:
Price optimization for brick-and-mortar and e-commerce retailers

While it may seem more natural to apply Machine Learning in the case of e-commerce retailers, brick-and-mortar retailers can perfectly take advantage from this technology.

In fact, price changes are less often performed in brick-and-mortar retailers and thus, having more room to improve and adjust to current demand.

Digital price tags are enabling brick-and-mortar retailers to do as many price changes as e-commerce sites. However, even without digital price tags, weekly or monthly price changes can be performed in order to match the current demand and maximize profit.


key: chroma:document
string_value:
We have proven our approach with one of the largest travel retailers in the world with over 400 stores across the globe and over 160 million clients per year. We helped them boost gross margin by 28% performing weekly price changes in-store.

Price optimization and demand forecasting with Machine Learning during a crisis


key: chroma:document
string_value:
Price optimization and demand forecasting with Machine Learning during a crisis

In May 2023, the World Health Organization officially declared the end of the COVID-19 global public health emergency. However, back in 2020, we observed its direct impact on consumer spending. Businesses all over the world had to revise their annual sales forecasts and strategies. They were not operating in a business as usual (BAU) manner, leading to the question: are we still able to use Machine Learning to predict demand in completely unexpected scenarios like these?


key: chroma:document
string_value:
Our answer is “yes, but with new things to consider”. In a BAU scenario, Machine Learning models are likely to leverage historical sales and correlated external data to bring insights such as seasonality, relevant sales dates, and competitors' reactions. During a crisis, as the market is not behaving as usual, the historical insights are likely to fall short to predict future sales. To fight back, we’d need to increase the importance of shorter-term information (e.g., daily sales), in the understanding that the recent past is much more suitable to predict the future. Practically, this means adjusting the feature engineering process to weigh the shorter-term sales lags rather than the


key: chroma:document
string_value:
to predict the future. Practically, this means adjusting the feature engineering process to weigh the shorter-term sales lags rather than the historical ones.


key: chroma:document
string_value:
Additionally, the demand forecasting problem will also require the incorporation of more real-time market data than before as well as external macroeconomic and social data. On the real-time data, this means regularly updating available market data such as sales data, customer churn, sales intent (e.g. added to cart items, traffic to competitors’ sites), competitors' prices, among others. On the macroeconomic level, data such as consumer spending, unemployment, GDP and even community mobility segmented by cities/regions could also be considered, although these are mostly reported on a monthly basis. Stock market indicators (S&P 500, Dow Jones) could potentially be considered too, as a proxy


key: chroma:document
string_value:
although these are mostly reported on a monthly basis. Stock market indicators (S&P 500, Dow Jones) could potentially be considered too, as a proxy of real-time macroeconomic trends. Finally, there might also be positive results by incorporating social data, if we take the pandemic as a use case, we could use the reported COVID cases or current government policies (i.e. lockdown duration), to generate scenario forecasting and consider them for modeling future demand.


key: chroma:document
string_value:
To summarize, in any abruptly new economic scenario, our analysis is still that Machine Learning could be greatly leveraged to build accurate demand forecasts and optimize the pricing strategy. The key adaptations to a BAU scenario would be to incorporate more real-time data (market and macroeconomic data) + adapt the models to consider nearer-term lags vs. historical data. It’s also worth noting that business understanding and human judgment will still play a key role in the creation of this solution.

Companies using Machine Learning for price optimization


key: chroma:document
string_value:
Companies using Machine Learning for price optimization

Price optimization has been used, with significant success, in industries such as hospitality, airline, car rental, and e-commerce retail.

One of the first success stories occurred in the early 2000s, when Hilton Hotels Corp and InterContinental Hotels Group decided to eliminate fixed rates in favor of a fluid scheme, including dynamic pricing strategies. In those years the prices of the rooms were modified once or twice a day. The current computational power allows prices to change practically in real time.


key: chroma:document
string_value:
The hotel industry continues to employ dynamic pricing strategies, based entirely on Machine Learning. Currently, Airbnb proposes a dynamic price tool that recommends prices to its hosts, considering parameters such as seasonality, the day of the week or special events, and also more sophisticated factors such as photos of the property to be rented or the prices applied in the neighborhood. Other companies such as eBay and Uber have adopted similar approaches.


key: chroma:document
string_value:
Amazon is another of the big players when talking about dynamic pricing strategies. To give an idea, in 2012 Amazon was changing prices much more often than its competitors and in 2013, they were performing as much as 2.5 million price changes per day. Changing prices in such a dynamic way is informally known as the Amazon effect.

In the retail world, the most popular examples have been in e-commerce, but brick-and-mortar retailers have not been left behind. Although it is difficult to know precisely all the retail companies using Machine Learning to optimize their prices and operating processes, there are nevertheless some known success stories.


key: chroma:document
string_value:
Companies like Ralph Lauren and Michael Kors use Machine Learning to offer fewer markdowns and better manage their inventory, seeking to increase profit margins, even at the risk of losing a little revenue. Another well-known case is that of Zara, which uses Machine Learning to minimize promotions and adapt quickly to the changing trends. There are many other success stories, such as Morrisons –one of the largest supermarket chains in the United Kingdom–, bonprix –an international fashion company based in Germany– or Monoprice –an American B2B and B2C electronics retailer–, among others. While there is no information available on the exact modeling of the problems, it is known that these


key: chroma:document
string_value:
and B2C electronics retailer–, among others. While there is no information available on the exact modeling of the problems, it is known that these companies are taking advantage of the power of Machine Learning to increase their revenues and improve operations.


key: chroma:document
string_value:
Get the latest news about price optimization Subscribe to our pricing with ML newsletter.

Final thoughts

Nowadays the world is moving towards changing prices more often and using state-of-the-art data driven pricing strategies is a must. In a study performed by Bain & Company they show that top performers across industries are nearly twice as likely to price dynamically.

Whether it's about an e-commerce marketplace or a brick-and-mortar retail store, both are embracing the benefits of dynamic pricing and price optimization.


key: chroma:document
string_value:
Price optimization helps retailers understand how customers will react to different price strategies for products and services, and set the best prices. Machine Learning models can take key pricing variables into account (e.g. purchase histories, season, inventory, competitors’ pricing) to find the best prices, even for vast catalogs of products or services, that can achieve the set KPIs.

These models don’t have to be programmed. They learn patterns from data and are capable of adapting themselves to new data. They allow retailers to quickly test different hypotheses and make the best decision.


key: chroma:document
string_value:
What is probably most important to keep in mind is that the use of Machine Learning in the retail world keeps widening, and all signs point to the fact that this trend will continue in the coming years.

The question is no longer whether to apply dynamic pricing or not. But the question is how to do so in order to remain profitable.

Dive deeper into pricing strategies with Tryolabs

Here at Tryolabs, we specialize in Machine Learning solutions for retail companies. Since 2010, we have been working with several retailers, which let us better understand the opportunities, challenges and available solutions within the industry.


key: chroma:document
string_value:
Want to stay ahead in the competitive retail landscape? Explore our Pricing Collection for a curated set of content tailored to inform and inspire.

For a more hands-on, tailored experience, consider our in-company AI Talk on Price Optimization. Our experts visit your teams, delivering insights and strategies on how AI-driven dynamic pricing can elevate your business.

Ready to harness the power of data-driven pricing? Contact us, and together we'll chart your path to pricing excellence.


key: link
string_value:
https://tryolabs.com/blog/2023/04/13/mastering-pricing-strategies-vol-1-intro-to-our-market-simulator


key: description
string_value:
Explore our innovative Market Simulator for demand forecasting and price optimization, designed to help businesses make informed decisions, test models, and adapt to ever-changing market conditions for enhanced performance


key: chroma:document
string_value:
Demand forecasting and price optimization can help businesses make informed decisions regarding production quotas, inventory, and pricing. It can be a daunting task to obtain real-world data to inform those decisions. We discussed the ever-growing role of Machine Learning in solving these kinds of problems in a previous blog post. And we also provided practical examples of one of our working pipelines. Now, in this post, we will focus on a tool that can assist you in developing these strategies: our Market Simulator.

Why build a market simulator?


key: chroma:document
string_value:
Why build a market simulator?

Let's start with a quick overview. Demand forecasting is the process of predicting the future demand for a product or service. It involves analyzing historical data, market trends, and consumer behavior to estimate the demand for a particular product or service. Price optimization, for its part, is the process of determining the optimal price for a product or service. It involves considering factors such as demand forecasts, competitors' prices, and production costs to determine the optimal price point to maximize profits. These are the issues most often encountered by our partners when they reach out to us for help.


key: chroma:document
string_value:
Throughout this journey, we've dealt with many different problems. And while each case was unique, they all presented some common denominators we managed to isolate and incorporate into our own set of internal pricing tools. One of these tools - our pricing suite - takes all the lessons learned along the way into account and can be easily repurposed from project to project, allowing our teams to provide effective and tailored solutions much faster. We'll be sharing more about this tool in our upcoming blog, so keep your eyes on this space! Experience has also proven that staying competitive in competitive markets requires significant research, so we started to develop our Market Simulator.


key: chroma:document
string_value:
We designed this tool to obtain accurate and risk-free scenarios to test our models and apply our solutions. We aim to loosely represent many known market situations of interest and evaluate how each model performs in the given business case. Our simulator can receive the market conditions to model along with the model we want to test. And this enables us to comprehensively analyze the given model's performance within the selected context.


key: chroma:document
string_value:
Our goal when we started, was to develop a tool to help us further understand how our pricing models learn and adapt to an ever-changing reality. The tool's primary purpose is to evaluate if we were appropriately leveraging our pricing model's strengths and managing their weaknesses. It's invaluable for experimenting with new ideas to enhance our solutions further. It has proven essential in our efforts to continuously improve our strategies and provide the best pricing models available. In the next section, we'll get into the nitty-gritty of our design process, so it might get a bit more technical, but hang in there; we promise it'll be worth it!


key: chroma:document
string_value:
To sum it all up, with a market simulator, we can: Emulate different market conditions

Test any desired model

Quickly evaluate and tweak all new approaches in a risk-free environment

The ins and outs of building the simulator


key: chroma:document
string_value:
We built a controlled simulation of a single-item market, with unlimited supply, to accomplish the above. We specify the actual underlying demand function and configure our models to be able to find it based on their observation of market outcomes. The selected model is presented with limited historical data and must make a pricing decision based only on that data. We then set the chosen price in the market and observed the simulated customers’ response to that price (i.e., the number of units they purchased). That, in turn, becomes a new data point that can help us further understand demand, so we repeat the cycle, training our model repeatedly over time.


key: chroma:document
string_value:
Our market simulation is built from a single random variable (RV): the customer’s demand ( Q ) (Q) (Q). This RV takes positive integer values, so we modeled it following a Poisson distribution (we’ll provide more details about this later). What characterizes this distribution is its expected value, which depends on the features that conform to the market state. These features always include price, and can include any other feature that may be of interest like seasonal effects, seller ratings, shipping times, etc. For simplicity of this exposition, we’re going to model the expected number of units sold in the market ( Q ^ \hat{Q} Q^​) depending on the price ( p p p), using the following


key: chroma:document
string_value:
we’re going to model the expected number of units sold in the market ( Q ^ \hat{Q} Q^​) depending on the price ( p p p), using the following equation:


key: chroma:document
string_value:
Q ^ ( p ) = α 1 + e ( β × p + γ ) ( 1 ) \hat{Q}(p) = \frac{\alpha} {1 + e^{(\beta \times p + \gamma)}} \qquad \qquad (1) Q ^ ​ ( p ) = 1 + e ( β × p + γ ) α ​ ( 1 )

Where:

α \alpha α , also known as the market size parameter, controls the overall height of the curve,

, also known as the market size parameter, controls the overall height of the curve, β \beta β controls the slope of the elastic region,

controls the slope of the elastic region, γ \gamma γ controls the location of the inflection point.


key: chroma:document
string_value:
controls the slope of the elastic region, γ \gamma γ controls the location of the inflection point.

Figure 2: a) Here we can see how α affects the overall height of the curve. b) Here we can see how β affects the slope of the elastic region. c) Here we can see how γ affects the location of the inflection point.

If we wanted to model factors other than price affecting sales, instead of just a constant, α \alpha α becomes α ( X ) \alpha(X) α(X), any function of the covariates X X X.

In particular, we can effectively model α ( X ) \alpha(X) α(X) as a simple linear combination of the covariates, with the coefficients we choose representing the importance of each covariate:


key: chroma:document
string_value:
α ( X i ) = α 0 + ∑ i = 1 N F α i × X i ( 2 ) \alpha(X_i) ={\alpha_0+\sum_{i=1}^{N_F}\alpha_i \times X_i} \qquad \qquad (2) α ( X i ​ ) = α 0 ​ + i = 1 ∑ N F ​ ​ α i ​ × X i ​ ( 2 )

Where:

X i X_i X i ​ is a vector of size N F N_F N F ​ of additional features affecting demand

The power of simplicity: While simple, this formulation is rather flexible, allowing for the representation of multiple market realities as shown below.


key: chroma:document
string_value:
For example, to represent perishable products, values of α = 1 , β = 1 , γ = − 8 \alpha=1, \beta=1, \gamma=-8 α = 1 , β = 1 , γ = − 8 allow formulating a demand curve with a terminal velocity, where prices below 2 saturate demand and you don’t get any more sales even while continuing to decrease prices (Figure 3.a).


key: chroma:document
string_value:
allow formulating a demand curve with a terminal velocity, where prices below 2 saturate demand and you don’t get any more sales even while continuing to decrease prices (Figure 3.a). On the contrary, values of α = 2 , β = 0.5 , γ = − 0.5 \alpha=2, \beta=0.5, \gamma=-0.5 α = 2 , β = 0.5 , γ = − 0.5 produce a curve that keeps increasing sales units as price decreases (Figure 3.b).

Figure 3: a) Example of the demand curve with α=1, β=1, γ=-8. b) Example of the demand curve with α=2, β=0.5, γ=-0.5.

Simulating demand


key: chroma:document
string_value:
Figure 3: a) Example of the demand curve with α=1, β=1, γ=-8. b) Example of the demand curve with α=2, β=0.5, γ=-0.5.

Simulating demand

We obtain the simulated expected demand as a function of price, the blue curve shown below, by plotting the above-stated demand function. And it shows that a slight shift in price near the inflection point can cause a substantial change in demand. The further away we get from the inflection point, the less sensitive demand is to price fluctuations. This formulation corresponds then to a demand curve with terminal velocity.

Figure 4: The expectation of the simulated demand function. We should notice it is modeled as a logit.


key: chroma:document
string_value:
Figure 4: The expectation of the simulated demand function. We should notice it is modeled as a logit.

Calculating the expected profit function is straightforward, as we now have an established expected demand function. Taking a specific cost of production or acquisition, c, into account, we can define our profit as a function of the expected demand as follows:

P r o f i t ( q ) = ( p − c ) × q ( 3 ) Profit(q) = (p - c) \times q \qquad \qquad (3) P ro f i t ( q ) = ( p − c ) × q ( 3 )


key: chroma:document
string_value:
P r o f i t ( q ) = ( p − c ) × q ( 3 ) Profit(q) = (p - c) \times q \qquad \qquad (3) P ro f i t ( q ) = ( p − c ) × q ( 3 )

Notice that profit is also a random variable because it depends on the demand. And since profit is a linear function of demand, the linearity of expectation allows us to calculate its expected value easily.

P r o f i t ‾ ( q ) = ( p − c ) × q ‾ ( 4 ) \overline{Profit}(q) = (p - c) \times \overline{q} \qquad \qquad (4) P ro f i t ​ ( q ) = ( p − c ) × q ​ ( 4 )

Now that we've defined the expected profit in terms of the expected demand, we can add its plot to the previous one and determine the optimal price - our (expected!) profit.


key: chroma:document
string_value:
Notice that the observed profit may be higher than the expected profit for a particular realization. In the long run, the variability should be canceled out, and the expected gain should be met.

Figure 5: Here we can see the simulated expected demand function as a constant blue curve and its profit function as a constant red curve. We can see the profit function's y-axis to the right of the plot. We are also highlighting the profit function's maximum with a blue dot. That blue dot is at the maximum profit achievable with our expected demand function. The price value corresponding to this point is thus the optimal price in this scenario.


key: chroma:document
string_value:
This approximates a given market situation for a particular product with an established price range and some X i X_i Xi​ features. In other words, we defined the expected sales rate for a given market situation X X X, and price p p p , (the target variable of all the demand curve regression problems). The data we have produced so far represents the ground truth we want to get as close to as possible with our chosen model’s predictions. But to generate any predictions, we are going to need data.


key: chroma:document
string_value:
Every demand forecasting project is built with data. So we now need to create historical data we can work with later. We could simply randomly sample Q ^ \hat{Q} Q^​. But, in the real world, we can't observe the expected sales rate of a product directly. What we see instead is a noisy random realization of said rate. To emulate this, we should model our historical data as a sampling from a Poisson distribution using Q ^ \hat{Q} Q^​ as the average rate at each price. In Figure 6, we can see the resulting historical data points in green. In this case, we sampled prices to cover the whole price range, but in practice, this likely isn’t going to be the case.


key: chroma:document
string_value:
Figure 6: Poisson sampled data from the known simulated expected demand function.

Understanding demand curves

Let's quickly recap: We now have a known ground truth expected demand function, a known expected profit function with its corresponding optimal value, and some noisy observational data to work with. What's next?


key: chroma:document
string_value:
The next step is to estimate the demand curve from our noisy observations using an appropriate demand forecasting model. To try and solve this regression problem, we'll give it a go with our old friend, XGBoost. So we train our regression model of choice on the available historical data (the green dots) and use that data to predict an estimated demand curve ( Q ^ ∗ \hat{Q}^* Q^​∗). The regression model will target the number of units sold, as a function of price and any other covariates of interest we have access to. An example of the resulting predicted demand is displayed in light blue below. Once we have obtained an estimated demand function, we can calculate the corresponding forecasted


key: chroma:document
string_value:
predicted demand is displayed in light blue below. Once we have obtained an estimated demand function, we can calculate the corresponding forecasted profit as mentioned in Eq 4.


key: chroma:document
string_value:
In the following plot, we can see the aforementioned estimated profit function as a pink curve with an orange dot at its maximum value. The maximum estimated profit value observed corresponds to the predicted optimal price. With a more complex optimization problem, we could also consider things other than profit, such as how long a product took to sell and the stock available. But to keep things simple, we'll just focus on profit optimization and recommend that the price be set at the aforementioned predicted optimal price.


key: chroma:document
string_value:
Notice that the suggested price given by the model is somewhat higher than the actual optimal price. That's because our information is imperfect: While close to the actual underlying expected demand, our estimate has a margin of error, as all Machine Learning models do. In this case, it's underestimating the demand near the optimal price points, making the profit smaller than expected. We can also see regions where it overestimates the demand, making the estimated profit bigger than it should be.

Figure 7: Forecasted demand (light blue curve) tries to estimate the expected demand (blue curve), while forecasted profit (pink curve) tries to estimate the expected profit (red curve).


key: chroma:document
string_value:
With all we've covered so far, we can create sales for any given period and in different scenarios. Our goal was to simulate market dynamics in a daily re-pricing scenario. All that's left is choosing the suggested price and observing how it fares. To that end, we created an algorithm where on t = 0 t=0 t=0, we use our historical data to predict the optimal price relative to the historical data available. Remember that the optimal price, in this case, is the price that will maximize the expected profit, which is deduced from the model's demand curve estimate. Once we obtain the optimal price, we set it as the sale price for the item and simulate the observed demand by sampling from a


key: chroma:document
string_value:
demand curve estimate. Once we obtain the optimal price, we set it as the sale price for the item and simulate the observed demand by sampling from a Poisson distribution with the known expected value given by Eq 1 at the recommended price point. By chaining all this together, we end up with the following procedure:


key: chroma:document
string_value:
For every t t t in the simulation horizon:

Train the demand model with all the historical data up to t t t and obtain Q ^ ∗ \hat{Q}^* Q ^ ​ ∗ , the demand forecast Simulate market features X t X_t X t ​ for the current time step t t t Optimize forecasted expected profit using Q ^ ∗ \hat{Q}^* Q ^ ​ ∗ , X i X_i X i ​ and all possible price points to get p ∗ p^* p ∗ Set p ∗ p^* p ∗ in the market and simulate a number of sales by sampling from Q ^ \hat{Q} Q ^ ​ the realized sales units q q q Add the ( p ∗ , X i , q ) (p^*,\: X_i,\: q) ( p ∗ , X i ​ , q ) point as a new data point Increase time step and repeat


key: chroma:document
string_value:
If we were to repeat this process for a fixed amount of time, we would get the results shown in the animation below (Figure 8). Neat, right? With animations like this, we can easily track our model's progress through time in any scenario. We can also compare how different strategies fared in the same conditions (when fed the same data).


key: chroma:document
string_value:
As we can see from our example, we start with a misguided pricing situation. In simulation step 0, all of our historical prices are way below the optimal price, and that's reflected in the resulting profit we obtained. But, if we didn't have our ground truth to compare with, we wouldn't even know this. We could only glean the market responses to the prices we set. However, as we let our model do its job, given enough time, it will suggest a new optimal price, so we end up with a good approximation of our known Q ^ \hat{Q} Q^​ in the region of interest.


key: chroma:document
string_value:
If you’re wondering why the suggested price doesn’t go into the price > 15 region, that’s because we bound price exploration to a region not too far away from historic prices. This is to avoid the model extrapolating too much into a region where no data is available.

Figure 8: Re-pricing simulation. Here we show how our model learns the underlying demand through the historical data and suggests a new optimum price that we then use in the simulated market.


key: chroma:document
string_value:
Looking back at the first steps of our simulation, you can see that our model suggests increasingly higher prices at the beginning of the process. As we try those prices, our model gets new input. And this new input demonstrates that our profit grows as we increase our product’s price. But only up to a certain point, after which the decrease in demand counters the price increase. Once that limit is reached, our profit starts to decrease again. It takes the model a few tries, but it quickly learns that the optimal price resides somewhere in the middle. Moreover, it does so by producing higher profits in each step than all the ones tried before.

Looking at our simulation metrics


key: chroma:document
string_value:
Looking at our simulation metrics

Our 15-minute experiment enabled us to collect some interesting metrics.


key: chroma:document
string_value:
Our 15-minute experiment enabled us to collect some interesting metrics.

For starters, we can measure the profit lost at each step (Figure 9.a). This calculation shows how the profit we generated by choosing the predicted optimal price compares to the profit we could have made had we known the accurate optimal price every time. That value is meant to display the cost of having imperfect information. If we knew the actual underlying demand curve, we would always be able to achieve the real optimal profit. Because we simply don't have this curve, the best we can do is estimate it and use our estimate to forecast profit. This metric is also known as the model’s regret.


key: chroma:document
string_value:
The difference between the true profit curve and the estimated one leads to profit losses, which we want to minimize. We start with a significant profit difference, and that's accurate because we began with prices that were way off the mark. We then see a reduction in the losses a few iterations later as our model tries out new prices, learns more, and moves closer to the optimal price. It's clear that while the model started with prices that were way off, it nonetheless managed to find the interval of the true optimal price and price consistently around it, reducing the profit losses at each step and converging to almost 0 profit lost per step.


key: chroma:document
string_value:
Another plot we included is the percentage of cumulative profit losses (Figure 9.b). Here we illustrate the ratio of the total incremental profit loss up until the current simulation step. What percentage did we lose out of the total optimum profit we could have made thus far? That gives us an idea of the total profit lost throughout the simulation due to imperfect information. Even with the optimum price oscillations we end up with, we still have a very low profit loss percentage.


key: chroma:document
string_value:
We were interested in tracking two additional values as the simulation progressed: the difference between the actual and predicted optimal values of both the price and the demand (Figures 9.c and 9.d). The results obtained for the run shown in the animation above are shown in the following plots. We can see that the absolute difference gets smaller in both cases as time goes by.

Figure 9: a) Profit lost in each iteration step. b) Percentage of cumulative profit lost. c) Price Δ between the optimum price and the predicted optimum price. d) Demand Δ between the optimum demand values and the predicted optimum demand values.


key: chroma:document
string_value:
We also calculated the mean error (Figure 10.a) and the root mean squared error (Figure 10.b) for the predicted demand ( Q ^ ∗ \hat{Q}^* Q^​∗) and the expected demand ( Q ^ ) (\hat{Q}) (Q^​) using a separate set of datapoints, independent from the actually tried prices, that aims to cover the entire input space of the demand function. Ideally, we would have wanted a gridded sample across the entire demand input space, however, due to the curse of dimensionality the size of that grid grows exponentially with the dimension of the space, so we chose a random sample instead.


key: chroma:document
string_value:
These values can tell us how close we are to the underlying demand, considering the entire price range shown in the animation. We're off to a good start, but there is still work to be done to determine the expected demand. In this case, incomplete observations is but one of many reasons leading to errors in the estimate. The chosen model's extrapolation outside the historically tried prices - both price extremes - is a continuation of the nearest known value. Within lower-priced regions, this is manageable, but as prices grow, the predictions become consistently poor. Hence, this region skews the demand ME and RMSE values throughout the simulation, as shown in the following plots.


key: chroma:document
string_value:
A small note. If we perform our simulation, removing randomness (the Poisson samples) and instead sampling values from the expected demand directly, we get the results shown in the following animation (Figure 11). This is a much more trivial problem, since the model can learn much faster from data without noise, but in practice, this isn’t realizable. Even then, we can still see that at very high prices, we have a constant prediction that matches the sales obtained for the highest price, since trees extrapolate with a constant value. Although the model quickly hits the optimal price in this scenario, our demand RMSE and ME would still be skewed because of the difference in the price range


key: chroma:document
string_value:
the model quickly hits the optimal price in this scenario, our demand RMSE and ME would still be skewed because of the difference in the price range mentioned above (10- ∞ \infin ∞).


key: chroma:document
string_value:
Figure 11: Simulation without randomness. If we take observations from the underlying expected demand (without randomness), estimating it is trivial.

Lessons learned, lessons shared

Glad you made it this far! We hope you come away with some valuable concepts. But just to be sure, let’s recap the key takeaways from our journey:

We’ve built a simulator that’s capable of modeling different market realities from the ground up with a simple yet very expressive formulation.

We’ve formalized the problem at hand, determined how to generate the synthetic information we need, and formulated the functions we need to estimate.


key: chroma:document
string_value:
We’ve implemented a Machine Learning algorithm with a data-based feedback loop to estimate both demand and profit. Said algorithm is able to converge to the desired results for the problem at hand.

We’ve collected several performance metrics along the way, having the simulator become a “gym” for our algorithms, allowing us to compare the performance of different modeling alternatives in a controlled environment.


key: chroma:document
string_value:
So we built a simulator capable of modeling different market states from the ground up. And with it, we trained several variations of XGBoost models for estimating demand, and by tracking several relevant metrics that we detailed above, we can now assess how performant each of the variants is, and which one performs the best across multiple scenarios.

This simulator has enabled faster and more effective incorporation of additional tools into our pricing strategy. We can quickly determine if a new strategy will yield better results in different simulations in a risk-free environment. Only once we are confident it performs well do we include it in an actual plan with much higher stakes.


key: chroma:document
string_value:
If this post piqued your interest, stay tuned for the upcoming volume 2! In it we’ll explore some limitations of the basic learning models and how we can leverage the simulator to overcome them, particularly striking a good balance between exploration and exploitation as well as improving the model’s behavior in the extrapolation regions.

Do you think your organization could benefit from such a simulator? Drop us a line!


key: link
string_value:
https://tryolabs.com/blog/2023/03/01/tryolabs-and-google-join-forces-for-forecasting-code-camp-at-khipu


key: description
string_value:
Join us for Khipu.ai 2023, the AI event of the year! We're excited to be a sponsor and among the notable speakers, including experts from Apple, DeepMind, and more. Plus, don't miss our partnership with Google for a groundbreaking forecasting code camp. Register now and secure your spot!


key: chroma:document
string_value:
We're thrilled to announce that we're coming back as a sponsor for the upcoming Khipu conference! This extraordinary AI event will take place from March 6th to March 10th, 2023, in Uruguay, and we can't wait to be part of it once again.

Khipu is the largest AI conference in Latin America, bringing together students, researchers, industry participants, and distinguished international speakers. This community of like-minded individuals are working towards a common goal: to advance the field of artificial intelligence and shape the future of technology.

We're excited to join this community once again and connect with fellow enthusiasts who share our passion for AI.

Notable speakers


key: chroma:document
string_value:
We're excited to join this community once again and connect with fellow enthusiasts who share our passion for AI.

Notable speakers

We're excited to be among speakers from around the world, including:

Our CTO, Alan Descoins, is also part of this list and will be sharing his insights with attendees and learning from other industry experts. We're stoked to have him on the list of speakers! If you're at the conference, come and say hi to Alan and our team! We can't wait to connect with you and learn from the many talented folks at the event.

A partnership with Google for our Forecasting Code Camp!


key: chroma:document
string_value:
A partnership with Google for our Forecasting Code Camp!

We're thrilled to announce that we are partnering with Google for our code camp "A hands-on forecasting guide: from theory to practice." This unique collaboration will take place on Tuesday, March 7th, and it's an event that you don't want to miss!

During this Code Camp, we'll provide an overview of concepts, methods, and practical applications on the M5 forecasting challenge. You'll get hands-on experience with real-world challenges and learn from experts in the field. This is a rare opportunity to explore the fascinating world of forecasting and gain valuable insights that can benefit your work and career.


key: chroma:document
string_value:
We're proud to partner with Google for this event and excited to combine our knowledge and resources to bring you an exceptional learning experience.

Join our Code Camp waitlist! Join our waitlist now and take the first step toward mastering the art of forecasting. We'll be in touch soon to confirm your spot. We can't wait to connect with you and explore the fascinating world of forecasting together!


key: link
string_value:
https://tryolabs.com/blog/2023/01/27/fine-tuned-llms-and-distributed-training-to-elevate-your-conversational-ai-game


key: description
string_value:
Unlock the power of Large Language Models (LLMs) in this step-by-step guide to fine-tuning for specific tasks. From building a custom dataset to optimizing inference time, we'll walk you through the process of fine-tuning LLMs like GPT-3, OPT, and T5 for NLP tasks.


key: chroma:document
string_value:
Large Language Models (LLMs) are powerful tools in the field of artificial intelligence that can understand and respond to natural language. Recently, LLMs like GPT-3, OPT, or T5 have stolen the spotlight on AI. These huge deep learning models, which consist of billions of parameters, perform astonishingly well for an extensive set of natural language processing tasks, including text generation, summarization, translation, and many more. A great example of the flexibility and power of LLMs is the recently released ChatGPT, which can produce astounding human-like text responses for a wide range of prompts.


key: chroma:document
string_value:
While these large models are great for general tasks, they may need to be fine-tuned to achieve optimal performance on specific downstream tasks or when dealing with new, unseen data. Fine-tuning is when a model's parameters are adjusted to fit a particular dataset better. This can help companies use the model for tasks specific to their business.

As a treat to our amazing readers, we fine-tuned an LLM to build a chatbot that can answer natural questions about the rich content published on our blog. This article will walk you through the fine-tuning process, from building our own dataset to optimizing inference time for the fine-tuned model!

Say Hi!


key: chroma:document
string_value:
Say Hi!

Before diving into the details of our fine-tuning pipeline, we invite you to try out our chatbot for yourself! We’ve set up a 🤗 Hugging Face Space where you can ask the chatbot any questions you have about this blog post. Simply follow this link to access the space and start chatting! In addition, we’ve uploaded the fine-tuned model and its ONNX export to 🤗 Hugging Face so that you can explore them at will.


key: chroma:document
string_value:
The model powering the chatbot was trained on the task of Conversational Question Answering, which means it can answer questions regarding a specific context (in this case, the content of this blog) or previous questions or answers from the current conversation thread. Try chatting with the bot in a conversational style, just as you would with one of Tryolabs’ tech experts!

Caveats


key: chroma:document
string_value:
Caveats

Remember that this is just a small demo, so the answers may not be as accurate as expected; in the Improvements section, we’ll discuss how we can enhance the model’s responses! Also, the content of this blog was not included in the training set, which means that the chatbot will give you answers about new, unseen data!


key: chroma:document
string_value:
Response time may take between 10 to 20 seconds due to the use of 🤗 Hugging Face’s free-tier space, which only has access to two CPU cores. This response time is slow for a chatbot being used in production, but it's fast for a CPU deployment and for such large inputs (since the model needs to process the entire blog to generate an answer). We'll discuss optimizing inference time in the Optimizing with ONNX section.

The Boom of Foundation Models


key: chroma:document
string_value:
The Boom of Foundation Models

Have you ever wondered how computers can understand and respond to human language? The answer lies in a new concept called ‘Foundation Models’. Popularized under this name by the Stanford University, these are machine learning models trained on immense amounts of data (over tens of terabytes, and growing) that can be adapted to a wide range of downstream tasks in fields such as image, text, and audio, among others.


key: chroma:document
string_value:
LLMs, a type of Foundation Model, are particularly well-established in Natural Language Processing (NLP). These models are cutting-edge artificial intelligence, capable of producing human-like language with remarkable accuracy. They can be used to generate text for various tasks such as translation, summarization, paraphrasing, question answering, and even assisted coding.


key: chroma:document
string_value:
These models come in different flavors and are often open-source. They can vary in size, the tasks they were pre-trained on, efficiency, and other characteristics. For example, the smallest LLMs have only a few million parameters, while the largest ones, such as the recently released Switch Transformers, have over a trillion.

Some of the most widely adopted LLMs are:

BERT , developed by Google

, developed by Google BLOOM , developed by BigScience Workshop

, developed by BigScience Workshop GPT-J , the open-source version of GPT-3 developed by OpenAI

, the open-source version of GPT-3 developed by OpenAI OPT , developed by Meta

, developed by Meta T5, developed by Google


key: chroma:document
string_value:
, the open-source version of GPT-3 developed by OpenAI OPT , developed by Meta

, developed by Meta T5, developed by Google

A comprehensive list of open-source language models can be found on 🤗 Hugging Face’s list of supported models. Thanks to 🤗 Hugging Face and the ML community for providing this huge open-source repository!


key: chroma:document
string_value:
One recent example of the impressive capabilities of Foundation Models is ChatGPT, arguably one of the most incredible advances in AI history. This model, developed by OpenAI, is a fine-tuned version of GPT-3.5 (one of the latest versions of the GPT-3 model family). ChatGPT can be used through a simple chat interface to perform various tasks, including summarization, text generation, code generation, and question-answering on virtually any topic. What sets ChatGPT apart is its ability to produce highly detailed, complete, and human-like responses, admit mistakes and adjust replies based on the user's needs.


key: chroma:document
string_value:
As the name suggests, Foundation Models can serve as the foundation for many applications, but they require fine-tuning and adjustments to get the most out of them. So keep reading to learn how to get them to their full potential!

Fine-tuning an LLM

1. Building a Dataset

There is a wide range of public datasets that you can use to fine-tune your model, but if you desire to improve the results for your specific task, you will probably need to build your own dataset. This might be a crucial task to achieve good results, albeit time-consuming.


key: chroma:document
string_value:
Our goal was to build a chatbot that could answer questions about Tryolabs’ blog posts, so we created our own custom dataset, TryoCoQA, to fine-tune a Foundation Model that can interact with Tryolabs’ readers, just like yourself!


key: chroma:document
string_value:
Like the CoQA dataset for conversational question answering, TryoCoQA consists of questions and answers about a specific context in a conversational manner, with some questions and answers referencing previous points in the conversation. We aimed for natural language questions with a variety of writing styles and vocabulary, and answers that can be short spans of text extracted from the context or free-form text, hoping for the model to produce more human-like responses with high-quality content.

The dataset and guidelines for building it are available in the following GitHub repository.

2. Choosing a Foundation Model


key: chroma:document
string_value:
The dataset and guidelines for building it are available in the following GitHub repository.

2. Choosing a Foundation Model

Selecting the right open-source LLM to fine-tune can be a tough choice. There’s a handful of them, and while they may perform quite well for a broad range of general tasks, it can be challenging to predict how well they will perform on our specific task before fine-tuning. Think of it like buying a new pair of dancing shoes - they may look great and feel comfortable, but you won't know how well they'll perform until you hit the dance floor.


key: chroma:document
string_value:
In addition to wanting the model to perform well on our specific task, we also wanted to consider other factors, such as resource efficiency and the size of the dataset required for fine-tuning. We were looking for a model that could run on minimal resources and be fine-tuned with a small dataset.


key: chroma:document
string_value:
Given our goal of building a QA chatbot for our blog posts and the constraints we have outlined, we zeroed in on T5 and its variants as a promising choice. T5 stands for “Text-To-Text Transfer Transformer.” Its philosophy is that it models every NLP task as a text-to-text problem. For example, in the task of Document Classification, T5 outputs a simple text containing the name of the predicted class for the document rather than outputting logits for all possible classes. In addition to this creative approach to NLP problems, we also liked that T5’s architecture is similar to the original transformer encoder-decoder architecture proposed in the ‘Attention Is All You Need’ paper.


key: chroma:document
string_value:
One variant of T5 that stood out was FlanT5. It’s been fine-tuned on a wide range of downstream tasks and over 450 datasets, and it even performs well on zero-shot tasks, meaning it can obtain good results for a new task or dataset without training on it. Just what we needed! With a minimal dataset, one could fine-tune FlanT5 for a few epochs and obtain excellent results.


key: chroma:document
string_value:
Of course, ML is not always that simple: our use case required processing whole blog posts, which can be very long. T5 and FlanT5 both use the popular self-attention mechanism, which has a quadratic increase in memory consumption with input size! In other words, to process a whole blog post — or worse, a batch of them — we would need an astronomical amount of memory.


key: chroma:document
string_value:
Fortunately, another variant of T5, called LongT5, saved the day! LongT5, proposed by Google, is especially designed to process large inputs. It uses a different attention mechanism called TGlobal (Transient Global) Attention Mechanism, which requires far less memory and allows LongT5 to excel at numerous tasks that other transformer architectures can’t handle due to memory shortage, such as scientific paper summarization or QA about Wikipedia articles.


key: chroma:document
string_value:
So LongT5 seemed like the right choice for efficient training and inference on our dataset, but it did pose a small problem. In contrast with FlanT5, LongT5’s available checkpoints weren’t fine-tuned on many tasks, so it didn’t perform very well on zero-shot scenarios. To address this, we fine-tuned LongT5 on a few publicly available datasets before training on our own data!


key: chroma:document
string_value:
In summary As you can see, selecting which model to train for your particular problem can be challenging as it generally involves many factors. So always try to consider the following: How well the model fits your specific task . You can dig deeper into the datasets and functions it was pre-trained on and do some zero-shot or few-shot rounds of evaluation on your own tasks and data.


key: chroma:document
string_value:
. You can dig deeper into the datasets and functions it was pre-trained on and do some zero-shot or few-shot rounds of evaluation on your own tasks and data. The size of your inputs and outputs. Some models may not scale well for large inputs and outputs due to high memory consumption or a substantial number of operations.

Some models may not scale well for large inputs and outputs due to high memory consumption or a substantial number of operations. The size of your dataset. If you have a small dataset, choose a more powerful model capable of zero-shot or few-shot learning. Nevertheless, the more data you have, the better results you will achieve.


key: chroma:document
string_value:
If you have a small dataset, choose a more powerful model capable of zero-shot or few-shot learning. Nevertheless, the more data you have, the better results you will achieve. How much computing power is available for training and inference. This can be a significant factor in determining which models you can use, as larger models may not fit in your memory, or training may become painfully slow. You can use libraries like 🤗 Accelerate for distributed training to make the most out of your hardware. All this can seem overwhelming, but don’t worry; here at Tryolabs, we can help you choose the right model for your needs! Contact us and get started on building your own fine-tuning pipeline.


key: chroma:document
string_value:
3. Fine-tuning strategy

Once we selected the perfect Foundation Model, the next step was to prepare our data to be able to train on it.

One of the challenges we faced was that our dataset was relatively small, which could make it harder for the model to learn how to converse and answer questions about our blog posts.

But we had a plan! To give our model a head start, we first fine-tuned it on much larger and more general datasets. This way, the model was able to learn how to retrieve answers to general questions before focusing on the specific content of our blog posts.

We selected two available datasets, SQuAD2.0 and CoQA, and trained on them following two different schemes:


key: chroma:document
string_value:
We selected two available datasets, SQuAD2.0 and CoQA, and trained on them following two different schemes:

Fine-tuning the model sequentially on each dataset, which means fine-tuning first on the SQuAD2.0 dataset, then on CoQA, and finally, on our dataset. Combining the datasets SQuAD2.0 and CoQA into one single dataset and fine-tuning on it, and when the training is over, fine-tuning sequentially on our dataset.

Combining the datasets resulted in slightly better performance, but both alternatives produced similar results.


key: chroma:document
string_value:
Combining the datasets resulted in slightly better performance, but both alternatives produced similar results.

In addition to deciding how to combine the datasets, we also needed to adapt our examples to T5's text-to-text input and output format. This involved restructuring and rewriting the context and questions in our inputs.

To refresh the reader’s memory, in the use-case of Conversational Question Answering, the goal of the model is to generate the answer to a question given a context and the previous questions and answers from the conversation. With this in mind, we formatted the inputs following this structure:


key: chroma:document
string_value:
Here, the input is a text string containing the context (i.e., the content of one of Tryolabs blog posts) followed by the last two question-and-answer pairs in the conversation and the current target question, with the target output being the answer to the target question. We chose to add just the last two question-and-answer pairs to limit the amount of conversation history the model needs to pay attention to while still being able to generate coherent responses. Note that this is a hyper-parameter you can adjust when fine-tuning your own model.

With our data prepared and fine-tuning strategy determined, the final step was setting up our infrastructure environment and training the model.


key: chroma:document
string_value:
With our data prepared and fine-tuning strategy determined, the final step was setting up our infrastructure environment and training the model.

4. 🤗 Accelerate

Fine-tuning a big Foundation Model can be computationally expensive, especially if you want to do it quickly. However, we can still train models with limited resources. There are various techniques to accommodate large models in small environments.


key: chroma:document
string_value:
In our case, we used 2 x NVIDIA GeForce RTX 3090 GPUs, which had a total of 48GB VRAM. Although this setup seems to have excellent computing power and memory capacity, it was not enough to train the model independently. To accomplish the task, we had to use different techniques supported by 🤗 Hugging Face’s distributed training and inference library, 🤗 Accelerate.

The techniques that worked for us were:

Using gradient accumulation, which lets you train on bigger batch sizes as the gradients get accumulated over several batches, and the optimization step is calculated after a certain number of them.


key: chroma:document
string_value:
Using gradient checkpointing to reduce memory consumption by forgetting the activations during the forward pass and recomputing them on the backward pass of each training round.

Picking the right batch size to balance training speed and memory consumption.

Choosing optimizers that consume less memory, such as Adafactor (the optimizer used in the original T5 and LongT5 papers).

Tweaking parameters in the data loader, such as pinning the memory to the CPU and setting the right number of workers.

Distributing the training across both GPUs to leverage the computing power available.


key: chroma:document
string_value:
Distributing the training across both GPUs to leverage the computing power available.

You can find more information on these techniques and how to use them in HuggingFace’s documentation, particularly in the “Efficient Training on a Single GPU” and “Efficient Training on Multiple GPUs” sections, as well as in 🤗 Accelerate’s documentation.

5. Evaluation

Apart from training the model, we wanted to evaluate what it was doing and how well it was performing.


key: chroma:document
string_value:
5. Evaluation

Apart from training the model, we wanted to evaluate what it was doing and how well it was performing.

Firstly, to correctly evaluate the training, we had to take a portion of the data to use as a validation split. Thankfully SQuAD2.0 and CoQA datasets already have a validation split. In the case of our dataset, we split our data into 85% training and 15% validation.

To assess its performance, we used the F1 Score by validating how many tokens appeared in common in both predictions and ground truth samples. We also used Exact match to see if the model was actually writing the same answer.


key: chroma:document
string_value:
Since we had two different training steps, we also had two additional evaluation steps. The first training, on SQuAD2.0 and CoQa, resulted in a 74.29 F1 Score on the validation split after 3 epochs. The second training, on TryoCoQa, produced a 54.77 F1 Score after 166 epochs.


key: chroma:document
string_value:
More than analyzing the quantitative metrics is required to evaluate these results and conversational models in general. It is essential to consider the qualitative aspect of the model's answers, like their grammatical correctness and coherence within the conversation context. Sometimes it is preferable to have better answers (qualitatively speaking) than a better F1. So we looked at some answers from the validation set to ensure that the model was correctly generating what we were looking for. Our analysis revealed that higher F1 scores were generally associated with greater-quality answers. As a result, we selected the checkpoint with the highest F1 score to use in constructing our


key: chroma:document
string_value:
were generally associated with greater-quality answers. As a result, we selected the checkpoint with the highest F1 score to use in constructing our demonstration chatbot.


key: chroma:document
string_value:
If you want to play around with our fine-tuned model, you can find it on 🤗 Hugging Face with the ID tryolabs/long-t5-tglobal-base-blogpost-cqa!

Faster inference with 🤗 Optimum and ONNX


key: chroma:document
string_value:
Faster inference with 🤗 Optimum and ONNX

After fine-tuning our model, we wanted to make it available to our awesome readers, so we deployed it on 🤗 Hugging Face Spaces, which offers a free tier with two CPU cores for running inference on the model. However, this setup can lead to slow inference times, and processing significant inputs like ours doesn’t make it any better. And a chatbot that takes a few minutes to answer a question doesn't strike anyone as being particularly chatty, does it? So, to improve the speed of our chatbot, we turned to 🤗 Optimum and the ONNX Runtime!


key: chroma:document
string_value:
In our previous blog post, A guide to optimizing Transformer-based models for faster inference, we used 🤗 Optimum and ONNX to achieve an 8x speed-up on inference for a Transformer model. Be sure to check it out!


key: chroma:document
string_value:
Using 🤗 Optimum’s recently released exporters feature, we were able to convert our PyTorch model to the ONNX format. This feature is handy for encoder-decoder models like the LongT5 model we trained, as it exports the three main components separately: the encoder, the decoder with the Language Modeling head, and the same decoder with pre-computed hidden states as additional inputs. According to 🤗 Optimum’s documentation, combining these three components can speed up sequential decoding, which results in faster text generation.

Our fine-tuned model, exported to ONNX into these three components, is also available on 🤗 Hugging Face with the ID tryolabs/long-t5-tglobal-base-blogpost-cqa-onnx!


key: chroma:document
string_value:
Once our model was exported to ONNX, we used 🤗 Optimum’s integration with the ONNX Runtime to optimize our model and run inference on it by using the ORTModelForSeq2SeqLM class. This class can optimize and downcast the model using ONNX’s tools and then use ONNX Runtime to run inference with this new, faster model! You can even take it one step further and quantize the model for even shorter inference time on CPU and lower memory consumption.

With these improvements, we achieved a 2x speed-up on inference time! Although the model still takes between 10 to 20 seconds to answer, this is a reasonable speed for a CPU-only deployment and processing such large inputs.


key: chroma:document
string_value:
Average inference time measured over various blogposts.

It’s worth noting that members of the ONNX community are actively working on tools for exporting and optimizing the latest ML architectures. The features mentioned in this section are fresh from the oven, thanks to the invaluable support of the ONNX team. Their work is crucial for unifying deep learning frameworks and optimizing models for faster and more cost-effective training and inference.

Improvements

We’re just scratching the surface of what’s possible. There are numerous potential improvements to keep working on to enhance the user's overall experience and the chatbot's performance. Here are some ideas to keep working on:


key: chroma:document
string_value:
Fine-tune on more public datasets for different downstream tasks. By doing so, we hope that the model learns to respond in a more human-like manner. For example, we could fine-tune on context-free QA datasets or on free-form answers so that the model not only retrieves the answers from the context but also generates original answers. We could also fine-tune on summarizing or paraphrasing tasks so that the user can also ask the chatbot to rewrite certain parts of the blog or summarize entire sections!


key: chroma:document
string_value:
By doing so, we hope that the model learns to respond in a more human-like manner. For example, we could fine-tune on context-free QA datasets or on free-form answers so that the model not only retrieves the answers from the context but also generates original answers. We could also fine-tune on summarizing or paraphrasing tasks so that the user can also ask the chatbot to rewrite certain parts of the blog or summarize entire sections! Increase the size of our dataset. We built a small dataset for this demo, so it's not surprising that the model doesn't generalize well to new unseen blogs. We could potentially train a more accurate model by providing it with more training data.


key: chroma:document
string_value:
We built a small dataset for this demo, so it's not surprising that the model doesn't generalize well to new unseen blogs. We could potentially train a more accurate model by providing it with more training data. Optimize training using ONNX Runtime. We could achieve better results in the same amount of time if we could train for more epochs or fit a larger version of LongT5 in the same amount of memory. To do this, we could leverage the optimization power of ONNX Runtime for training PyTorch models, which would allow us to accelerate training speeds and optimize memory usage. Thanks to the recently released 🤗 Optimum integration with ONNX Runtime for training, this can be done in just a


key: chroma:document
string_value:
speeds and optimize memory usage. Thanks to the recently released 🤗 Optimum integration with ONNX Runtime for training, this can be done in just a few lines of code!


key: chroma:document
string_value:
Takeaways

With the ever-increasing popularity of LLMs, it can seem almost impossible to train these models without having access to millions of dollars in resources and tons of data. However, with the right skills and knowledge about Foundation Models, Deep Learning, and the Transformer architecture, we showed you that fine-tuning these huge models is possible, even with few resources and a small dataset!

Fine-tuning is the key to unlocking the full potential of Foundation Models for your business. It allows you to take a pre-trained model and adapt it to your specific needs without breaking the bank.


key: chroma:document
string_value:
And the best part? We're here to help you every step of the way. If you're ready to see how fine-tuning can benefit your business, don't hesitate to reach out, and let's get started on your fine-tuning journey today!


key: description
string_value:
There are so many amazing Python libraries and tools out every year that it's hard to keep track of them all. That's why we share with you our hand-picked selection of our best picks.


key: chroma:document
string_value:
Welcome to the 8th edition of our Top Python Libraries list!

We are excited to present this year's picks for the most innovative developments in the Python ecosystem. From this edition, we are expanding our list to include not only libraries per-se, but also tools that are built to belong in the Python ecosystem — some of which are not written in Python as you’ll see.

2021, 2020, 2019, 2018, 2017, 2016, 2015. See the other posts in this series: 2023


key: chroma:document
string_value:
2021, 2020, 2019, 2018, 2017, 2016, 2015. See the other posts in this series: 2023

The rules for selection are the same as in previous years. We are looking for libraries that were launched or gained popularity in the past year, are well-maintained, and are simply cool and worth checking out. Our picks are heavily influenced by AI and data science libraries, but we also include a number of libraries that can be useful for non-data science purposes.

In addition to our main picks, we have also included an "Extra picks" section with a longer tail of libraries that we are sure many of you will find interesting.


key: chroma:document
string_value:
We hope that this list will showcase not only the best of what the Python community has to offer but also spark discussion about the libraries and tools that we may have missed. And unavoidably we do miss some every year, that's why we add the Misses from 2021 section as well.

Without further ado, let's get started!

The 10 main picks

1. Ruff — a fast linter

Was there anyone who did not use a linter in 2022?


key: chroma:document
string_value:
Without further ado, let's get started!

The 10 main picks

1. Ruff — a fast linter

Was there anyone who did not use a linter in 2022?

Through the years, the community has come to acknowledge that linters are an essential part of the software development process. They analyze source code for potential errors and style issues, providing valuable feedback and suggestions for improvement. Ultimately, they help developers write cleaner, more efficient code. In order to make the most of this process, it is important to have a linter that is fast and effective. This is where Ruff comes in.


key: chroma:document
string_value:
Ruff is an extremely fast Python linter that is written in Rust. It is 10-100x faster than existing linters and can be installed via pip.

Linting the CPython codebase from scratch.

In addition to linting, Ruff can also be used as an advanced code transformation tool, capable of upgrading type annotations, rewriting class definitions, sorting imports, and more.

It is a powerful tool that aims to replace a variety of other tools, including Flake8, isort, pydocstyle, yesqa, eradicate, and even a subset of pyupgrade and autoflake, all while executing at lightning fast speeds.

Definitely a big highlight to add to your arsenal going into 2023!

2. python-benedict — a dict on steroids


key: chroma:document
string_value:
Definitely a big highlight to add to your arsenal going into 2023!

2. python-benedict — a dict on steroids

Dictionaries are a vital data structure in Python, but working with complex dictionaries can be a challenge. The built-in dict type is powerful, but it lacks many features that make it easier to access and manipulate nested values or convert dictionaries to and from different data formats. If you've found yourself struggling to work with dictionaries in Python, python-benedict could be the solution you've been looking for.


key: chroma:document
string_value:
benedict is a subclass of the built-in dict type, meaning that it is fully compatible with existing dictionaries and can be used as a drop-in replacement in most cases.

One of the key features of benedict is its support for keylists and keypaths. This makes it easy to access and manipulate values in complex dictionaries without having to manually dig through nested levels. For example:


key: chroma:document
string_value:
In addition to its keylist and keypath support, benedict also provides a wide range of I/O shortcuts for working with various data formats. You can easily read and write dictionaries to and from formats such as JSON, YAML, and INI, as well as more specialized formats like CSV, TOML, and XML. It also has support for multiple I/O operations backends such as filepath (read/write), url (read-only),and s3 (read/write).

It has a gazillion more features, so make sure to check out the readme.

3. Memray — a memory profiler


key: chroma:document
string_value:
It has a gazillion more features, so make sure to check out the readme.

3. Memray — a memory profiler

Optimizing the memory usage of your systems is vital for improving their performance and stability. Memory leaks can cause programs to consume increasing amounts of memory, degrade overall system performance and eventually crash. Although garbage-collected and generally easy to work with, Python does not shield you from these issues. Nothing prevents circular references or unreachable objects (for more on this, read on how garbage collection works); even more so if we talk about C/C++ extensions.


key: chroma:document
string_value:
A memory profiler can help you identify and fix these issues, making it an essential tool for optimizing the memory usage of your programs. This is where Memray comes in handy!

It is a memory profiler that tracks memory allocations in Python code, native extension modules, and the Python interpreter itself, providing a comprehensive view of memory usage. Memray generates various reports, including flame graphs, to help you analyze the collected data and identify issues such as leaks and hotspots. It is fast and works with Python and native threads, making it a versatile tool for debugging memory issues in multithreaded programs.

Memray live mode.


key: chroma:document
string_value:
Memray live mode.

Memray can be used as a command line tool or as a library for more fine-grained profiling tasks. Its live mode allows you to interactively inspect memory usage while a script or module is running, making it useful for debugging complex memory patterns.

A must-have tool by Bloomberg for the serious Python programmer.

4. Codon — a Python compiler using LLVM


key: chroma:document
string_value:
A must-have tool by Bloomberg for the serious Python programmer.

4. Codon — a Python compiler using LLVM

We all love Python for its simplicity and ease of use, but sometimes we need a little extra runtime speed. Even with all the optimizations, Python interpreters (like CPython) can only go so far. When performance needs to be pushed even further, compilers come in. They convert our Python code into machine code that can be directly executed by the processor, skipping the interpreter step and giving us some serious performance boosts.


key: chroma:document
string_value:
Codon is a high-performing ahead-of-time Python compiler that can even compete with C/C++ in terms of speed, with typical speedups reported to be 10-100x or more (on a single thread). It can be used within a larger Python codebase using the @codon.jit decorator or by calling plain Python functions and libraries from within Codon using Python interoperability.


key: chroma:document
string_value:
There is no free lunch here, so you will likely have to make some modifications to your Python code so that it can be compiled by Codon. But the restrictions that Codon imposes over the language are what ultimately allow for the performance gains. Nevertheless, the compiler will guide you by producing detailed error messages that identify and help you resolve incompatibilities.

Diagram of how Codon works.

It's definitely worth checking out if you're looking to speed up your code. The FAQ is very informative for getting a quick glance at what this tool will do for you.

5. 🦜️🔗 LangChain — building LLM-powered apps


key: chroma:document
string_value:
5. 🦜️🔗 LangChain — building LLM-powered apps

Unless you have been living under a rock, you are well aware that generative AI has been taking the world by storm this year. A large chunk of this is Large Language Models (LLMs).

A lot of code that we have written over these past years to solve AI problems can be just thrown away and replaced with a LLM (like GPT-3 or its evolutions — InstructGPT or ChatGPT — T5, or any other). Also, we have witnessed the birth of a new programming language for the LLM interface: text-based prompts.

LangChain comes in to help leverage the full power of LLMs.


key: chroma:document
string_value:
LangChain comes in to help leverage the full power of LLMs.

First: in any serious use of LLM, one does not typically need to think of prompts as a one-off thing but rather a combination of things: templates, user input, and input/output examples that the LLM can take as reference. LangChain helps you streamline this “prompt management” by providing interfaces for building different prompts based on the individual components straightforwardly.


key: chroma:document
string_value:
Second, for building the prompts, sometimes you need to inject external knowledge (or even other models). For example, imagine you need to perform a database query to extract a customer’s name for a personalized email. This is the concept of chains, and LangChain provides a unified interface for this.

Then there’s the concept of fetching and augmenting data so the LLM can work over your own data, contrary to the “generic” data over which the models are trained.

There are more things you can do with LangChain, such as being prepared to switch to another model/provider without changing your code, building agents with memory, etc.


key: chroma:document
string_value:
Certainly, an innovative tool that we expect will grow a lot in 2023!

6. fugue — distributed computing done easy

If you're familiar with Pandas or SQL, you know that these tools are great for working with small to medium-sized datasets. But when dealing with larger amounts of data, you often need a distributed computing framework like Spark to process it efficiently. The thing is, Spark is a whole different beast from Pandas or SQL. The syntax and concepts are completely different, and it can be a challenge to migrate your code from one to the other. That's where Fugue comes in.


key: chroma:document
string_value:
Fugue is a library that makes it easier to work with distributed computing frameworks like Spark, Dask, and Ray. It provides a unified interface for distributed computing that lets you execute Python, pandas, and SQL code on Spark, Dask, and Ray with minimal rewrites.

The best starting point is using Fugue’s transform() function. It lets you parallelize the execution of a single function by bringing it to Spark, Dask, or Ray. See the following example where the map_letter_to_food() function is brought to the Spark execution engine:


key: chroma:document
string_value:
Fugue allows you to maintain one single codebase for Spark, Dask and Ray projects where logic and execution are completely decoupled, saving programmers from having to learn each of the different frameworks.

The library also has several other interesting features and a growing community, so make sure to check it out!

7. Diffusers — generative AI


key: chroma:document
string_value:
7. Diffusers — generative AI

2022 will always be remembered as the year when generative AI breached the frontiers of the AI community and expanded to the outside world. This was largely powered by Diffusion models, which gained much attention thanks to their impressive capabilities in generating high-quality images. DALL·E 2, Imagen, and Stable Diffusion are just a few examples of diffusion models that have made waves this year. Their results inspired discussion and admiration for their ability to generate images that push the boundaries of what was previously thought possible — even by AI experts.


key: chroma:document
string_value:
Hugging Face's Diffusers library is a collection of tools and techniques for working with diffusion models, including the Stable Diffusion model, which has proven particularly effective at generating highly realistic and detailed images. The library also includes tools for optimizing the performance of image-generation models and analyzing the results of image-generation experiments.

Starting to use this library for text-to-image generation may be as simple as dropping these few lines of code:

But Diffusers doesn’t stop with images. Features for audio generation and even molecule generation (!) are coming soon.


key: chroma:document
string_value:
But Diffusers doesn’t stop with images. Features for audio generation and even molecule generation (!) are coming soon.

It looks like most open-source implementations of Diffusion models will leverage the building blocks provided by this library. Sweet!

8. LineaPy — convert notebooks to production pipelines


key: chroma:document
string_value:
8. LineaPy — convert notebooks to production pipelines

Jupyter notebooks are an extremely useful tool for prototyping and developing code. They allow you to easily mix code, plots, media, and interactive widgets in a single document, making it easy to document and understand your code as you develop it. They are the perfect playground for experimenting with code and testing out ideas, which can be especially handy for data analysis and machine learning tasks. But as the ideas progress and one is happy with the result and wants to put code into production, challenges start to show.


key: chroma:document
string_value:
Code within notebooks may not always be structured in a way that is easy to deploy to a production environment. Should you take the notebook’s code and rewrite it elsewhere, modularize it and make it follow the best practices? This is the most common approach, but the drawback is that if, later down the road, you want to improve or debug something, you lost the interactivity of the notebook. So now you have to maintain both a notebook and separate production code, which is far from ideal.

There is a path to do better, thanks to LineaPy.


key: chroma:document
string_value:
There is a path to do better, thanks to LineaPy.

LineaPy is a Python library that helps you move quickly from prototyping to creating robust data pipelines. It takes your messy notebook and helps clean up and refactor code, making it easier to run in an orchestration system or job schedulers, such as cron, Apache Airflow, or Prefect.

It also helps with reproducibility: it has the concept of “artifacts” that encapsulate both data and code that can help you trace back how values were produced. At a high level, LineaPy traces the sequence of code execution to form a comprehensive understanding of the code and its context.


key: chroma:document
string_value:
The best part? It’s very simple to integrate, and can run in a Notebook environment with just two lines of code!

Make sure to check out the documentation, which does a great job showcasing the problems the tool solves with very comprehensive examples.

9. whylogs — model monitoring

As AI models make their way to produce actual value for a business, their behavior in production must be monitored continuously to make sure that the value is sustained over time. That is, one must have a way to tell that model predictions are reliable and accurate and that the inputs that are being fed to the model do not deviate significantly from the type of data that was used to train it.


key: chroma:document
string_value:
But model monitoring is not just limited to AI models — it can be applied to any kind of model, including statistical and mathematical models. Hence the usefulness of this pick.

whylogs is an open-source library that enables you to log and analyze any kind of data. It provides a range of features, starting with the ability to generate summaries of datasets: whylogs profiles.


key: chroma:document
string_value:
The profiles capture statistics over the original data, such as distribution, missing values, and many other configurable metrics. They are computed locally using the library and can even be merged to allow for analysis over distributed and streaming systems. They form a representation of the data that conveniently does not need to expose the data itself, but only metrics derived from it — which is good for privacy.

Profiles are straightforward to generate. For example, to generate a profile from a Pandas DataFrame you would do:


key: chroma:document
string_value:
Profiles are straightforward to generate. For example, to generate a profile from a Pandas DataFrame you would do:

But profiles are only as useful as what you decide to do with them. For this, visualization is a must. You can install the viz module using pip install "whylogs[viz]" which can create interactive reports for you. Here’s a sample drift report inside a Jupyter Notebook:

Sample drift report using whylogs viz module.

You can also set constraints so that you are notified when data doesn’t match the expectations, effectively doing some kind of testing on your data/model. After all, isn’t this what we were set to do from the beginning?


key: chroma:document
string_value:
If you prefer managed solutions (remember you do not need to share your data, but only statistics!), you can opt for the parent company’s SaaS ML monitoring and AI observability platform, which will make setting all this up very easy and also has a free starter edition. Nice!

10. Mito — spreadsheet inside notebooks

In the era of data science, many people are transitioning from analyzing data manually in spreadsheets to wring code to do that. But there’s no denying that spreadsheets are an attractive tool, providing a streamlined interface for editing and immediate feedback that allows for fast iterations.


key: chroma:document
string_value:
Many times, we process data using spreadsheets. But if we want to do the same thing again over new data, we must start from scratch! Code is so much better at this, saving us precious time.

Could we get the best of both worlds? Meet Mito.

Mito comes in handy as it is a library that allows you to work with data in a spreadsheet-like interface within Jupyter Notebooks. It allows you to import and edit CSV and XLSX files, generate pivot tables and graphs, filter and sort data, merge data sets, and perform a variety of other data manipulation tasks.

And the most important feature: Mito will generate Python code that corresponds to each of your edits!

How Mito works.


key: chroma:document
string_value:
And the most important feature: Mito will generate Python code that corresponds to each of your edits!

How Mito works.

Mito also supports the use of Excel-style formulas and provides summary statistics for columns of data. It’s designed to be the first tool in the data science toolkit, meaning it’s built to be a user-friendly tool for data exploration and analysis.

Tools like Mito lower the barrier of entry to the data science world, allowing people familiar with software such as Excel or Google Sheets (nearly everyone?) to start contributing code fast, and therefore deserves a spot in our top 10 picks this year.

Extra picks aka the long tail — don’t miss these

Non-AI


key: chroma:document
string_value:
Extra picks aka the long tail — don’t miss these

Non-AI

Neutron — native Python apps along with CSS and HTML for frontend design.

CustomTkinter — modern and customizable Python UI-library based on Tkinter.

Pyxel — retro game engine for Python.

Feathr — an enterprise-grade, high-performance feature store for your AI models. By LinkedIn.

Hugie — CLI interface for working with Hugging Face’s Inference Endpoints API.

InvokeAI — a Stable Diffusion toolkit, fork of the original but with several more features like an interactive Web UI.

Kangas — exploring, analyzing, and visualizing large-scale multimedia data.


key: chroma:document
string_value:
Kangas — exploring, analyzing, and visualizing large-scale multimedia data.

Lance — alternative to Parquet. 100x faster for random access, automatic versioning, optimized for ML data. Apache Arrow and DuckDB compatible.

Marqo — an open-source tensor search engine that seamlessly integrates with your applications, websites, and workflow.

Mercury — convert Jupyter Notebooks to web apps and share with non-technical users.

MLEM — a tool to package, serve, and deploy any ML model on any platform. By Iterative AI (creators of DVC).

Stable Diffusion web UI - browser interface based on Gradio library for Stable Diffusion, with a lot of features.

LLMs and the CLI


key: chroma:document
string_value:
Stable Diffusion web UI - browser interface based on Gradio library for Stable Diffusion, with a lot of features.

LLMs and the CLI

All of these are handy Python tools to basically ask GPT-3 (or its variants) to output commands for the shell, based on human-language prompts:

Torch-X

TorchMultimodal — PyTorch domain library for training SoTA multi-task multimodal models at scale. Modular and composable building blocks plus examples. See launch blog post. By Meta.

TorchRec — for recommendation systems, providing common sparsity and parallelism primitives, enabling researchers to build state-of-the-art personalization models and deploy them in production. See launch blog post. By Meta.


key: chroma:document
string_value:
TorchScale — allow researchers and developers to scale up Transformers efficiently and effectively.

TorchView — visualize PyTorch models as graphs using Graphviz.

Computer Vision

pybboxes — convert between different formats of bounding boxes for computer vision applications.

super-gradients — easily train or fine-tune SOTA computer vision models with one open source training library. By Deci.ai.

video-transformers — fine-tuning models for video classification.

NLP

Lingua — accurate language detection for +75 languages both in long and short texts.


key: chroma:document
string_value:
NLP

Lingua — accurate language detection for +75 languages both in long and short texts.

PyChatGPT — Python client for the unofficial ChatGPT API, which doesn’t require a headless browser. This isn’t official and is bypassing the ever-evolving protections of the OpenAI site, so you can call ChatGPT programmatically. Use it while it works; it might break soon!

SetFit — an efficient and prompt-free framework for few-shot fine-tuning of Sentence Transformers.

Distributed / large model training / scale

ColossalAI — efficient distributed learning.

DeepSpeed — a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. By Microsoft.


key: chroma:document
string_value:
DeepSpeed — a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. By Microsoft.

voltaML — open-source lightweight library to accelerate your machine learning and deep learning models. Optimize, compile and deploy your models to your target CPU and GPU devices, with just one line of code.

Interpretability

FastTreeSHAP — fast implementation of the TreeSHAP algorithm in the SHAP package, for interpreting tree-based models. Blog post.

surgeon-pytorch — inspect and extract intermediate layers of PyTorch models.

Others


key: chroma:document
string_value:
surgeon-pytorch — inspect and extract intermediate layers of PyTorch models.

Others

anomalib — anomaly detection library with SOTA algorithms and features such as experiment management, hyper-parameter optimization, and edge inference.

bayesian-statistics-toolbox — a library of functions for running sophisticated Bayesian analyses in a simple, straight forward manner.

open-metric-learning — a PyTorch-based framework to train and validate the models producing high-quality embeddings.

poniard — find out the best scikit-learn model by fitting them all and comparing them (not AutoML!).

Misses from 2021 (sorry!)

Argilla — data-centric NLP, build datasets in less time.


key: chroma:document
string_value:
Misses from 2021 (sorry!)

Argilla — data-centric NLP, build datasets in less time.

AutoViz — visualize any dataset, any size with a single line of code.

Contextualized Topic Models (CTM) — use pre-trained representations of language (like BERT) to support topic modeling.

FairScale — large-scale distributed training on PyTorch.

FiftyOne — open-source tool for building high-quality datasets and computer vision models. Get hands-on with your data, visualize labels, evaluate models, explore, find annotation mistakes, and more.

Ivy — attempt to unify all ML frameworks. Currently supports JAX, TensorFlow, PyTorch, and Numpy. Soon, automatic code conversions and multi-framework support.


key: chroma:document
string_value:
Jury — evaluate performance of NLP models (see blog post).

Ploomber — fast way to build data pipelines and deploy anywhere.

Shapash — make machine learning interpretable and understandable by everyone. Live demo.

Small-Text — state-of-the-art active learning for text classification.

WeaSEL — based on the End-to-End Weak Supervision paper (NeurIPS 2021), allows you to train your favorite neural network for weakly-supervised classification.


key: chroma:document
string_value:
WeightWatcher — predict performance of neural networks without access to training or test data, solely based on theory and statistical properties of the intermediate layers. Original version from earlier than 2021, but it’s getting a lot of development and traction this year!


key: description
string_value:
Learn how to optimize your Transformer-based model for faster inference in this comprehensive guide that covers techniques for reducing the size and time required for execution.


key: chroma:document
string_value:
Have you ever suffered from high inference time when working with Transformers? In this blog post, we will show you how to optimize and deploy your model to improve speed up to x10!

If you have been keeping up with the fast-moving world of AI, you surely know that in recent years Transformer models have taken over the state-of-the-art in many vital tasks on NLP, Computer Vision, time series analysis, and much more.


key: chroma:document
string_value:
Although ubiquitous, many of these models are very big and slow at inference time, requiring billions of operations to process inputs. Ultimately, this affects user experience and increases infrastructure costs as heaps of memory and computing power are needed to run these models. Even our planet is affected by this, as more computing power demands more energy, which could translate to more pollution.


key: chroma:document
string_value:
One solution for accelerating these models can be buying more powerful hardware. However, this isn’t a very scalable solution. Also, it may be the case that we can’t just use bigger hardware as we may want to run these models on the edge: mobile phones, security cameras, vacuum cleaners, or self-driving vehicles. By nature, these embedded devices are resource-constrained, with limited memory, computing power, and battery life. So, how can we make the most out of our hardware, whether a mobile phone or a huge data center? How can we fit our model in less memory, maintaining accuracy? And how can we make it faster or even run in real time?


key: chroma:document
string_value:
The answer to these questions is optimization. This blog post presents a step-by-step guide with overviews of various techniques, plus a notebook hosted on Google Colab so you can easily reproduce it on your own!

Our code is available here.

You can apply many optimization techniques to make your models much more efficient: graph optimization, downcasting and quantization, using specialized runtimes, and more. Unfortunately, we couldn’t find many comprehensive guides on how to apply these techniques to Transformer models.


key: chroma:document
string_value:
We take a fine-tuned version of the well-known BERT model architecture as a case study. You will see how to perform model optimization by loading the transformer model, pruning the model with sparsity and block movement, converting the model to ONNX, and applying graph optimization and downcasting. Moreover, we will show you how to deploy your optimized model on an inference server.

Try it out

We prepared and included the following live frontend on a 🤗 Hugging Face space so you can test and compare the baseline model, which is a large BERT fine-tuned on the squadv2 dataset, with an already available pruned version of this baseline, and finally, our ‘pruned + optimized’ version!


key: chroma:document
string_value:
In the interactive demo below, you will see three fields:

the Model field, where you can select any of the three models mentioned before (by default, our optimized model).

field, where you can select any of the three models mentioned before (by default, our optimized model). the Text field, where you should provide context about a topic.

field, where you should provide context about a topic. the Question field, where you can ask the AI about something in the text.

You can also try the demo here.

Question Answering (QA) models solve a very simple task: retrieve the answer to questions about a given text.

Let’s take the following text as an example:


