Latent Space Podcast 7/2/23 [Summary] AI Trends: a Latent Space x Practical AI crossover pod!

Original Link: [Practical AI] AI Trends: a Latent Space x Practical AI crossover pod!

Summary

In a crossover episode with the Latent Space podcast, host Daniel Whitenack from Practical FM joined co-hosts Alessio Fanelli and Shawn Wang. The discussion began with introductions, with Whitenack highlighting his background in computational physics, his tenure as a data scientist at SIL International, and his recent venture with Prediction Guard. The podcast touched on the rise of AI, its global community, and the practical implications and uses of AI technology in everyday life.

Daniel also shed light on episodes of Practical AI, sharing personal favorites and audience hits. Episodes like "Fully Connected" are particularly enlightening, as they delve deep into AI topics such as ChatGPT, diffusion models, and AlphaFold. Another highlight was the "AI for Africa" series, which emphasized the importance of AI being inclusive and accommodating for various global use cases.

Throughout the conversation, there was a consistent theme of focusing on the practical, day-to-day uses of AI, rather than getting caught up in the hype.

AI Roundtable: Insights, Excitements, and Challenges in the Modern AI Landscape

In a podcast episode, Shawn Wang, Alessio Fanelli, and Daniel Whitenack discuss their personal episode highlights, trends in the AI industry, and the evolution of model evaluation.

Episode Highlights:

• Alessio Fanelli's highlight was the episode with Mike Conover from Databricks discussing DALL-E. He emphasizes Mike's passion and the exciting release of the RedPajama dataset.

• Shawn Wang recalled the news-driven episode on ChatGPT Plugins where 4,000 people tuned in. He finds value in capturing real-time reactions to significant AI developments.

• Daniel Whitenack reminisces about the time when image generation was trending but now finds himself immersed in NLP and language models.

Trends in the AI Industry:

• The most popular episode was on Metaflow, a Python package for full-stack data science modeling, which resonated with listeners due to the challenges of the model lifecycle in practical applications.

• Daniel notes that the industry is transitioning from traditional MLOps to a new kind of operations (possibly termed LLMOps), focused more on leveraging pre-trained models.

• Alessio mentions that their most popular episodes revolved around model-based topics, including discussions about the limitations of traditional benchmarks and the need for better evaluations.

Model Evaluation:

• Alessio talked about the rapid evolution of benchmarks, indicating the need for better evaluation metrics for models like GPT-4.

• Shawn brings up HellaSwag's adversarially-generated benchmarks and the surprising revelation that less than 1% of the data for "Segment Anything from Meta" was human-generated.

• Daniel observes the growing trend of model-generated output and datasets, emphasizing that the scale of simulated or augmented data is astounding.

• Shawn voices concerns about "mode collapse" and the possibility of models over-optimizing for median use cases, potentially sidelining low-resource applications.

• Daniel believes that linguistic diversity in large language models can benefit downstream, lower-resource languages.

Throughout the discussion, the trio touches upon the evolving landscape of AI, the importance of benchmarks, and the challenges and potential of using models to evaluate other models.

Grassroots AI Efforts & The Changing Landscape of AI Models

Shawn Wang explores the impact and mission of Masakhane, a grassroots organization of African NLP researchers creating relevant technologies for Africa. Daniel Whitenack elaborates on the unique challenges faced by African communities, stressing the importance of building AI solutions tailored to specific needs. He cites examples of advanced tech in the U.S., such as tractors running on Kubernetes clusters, versus the contrasting requirements in Africa, like drought identification or disaster relief.

Daniel further delves into the "AI for Africa" episodes, highlighting the vast difference in tech applications across regions. He encourages global participation, pointing to Masakhane as a model for how local expertise can shape meaningful technology.

Turning to the topic of AI models, Daniel references an insightful episode from Raj Shah of Hugging Face on the capabilities of Large Language Models (LLMs). The episode discusses various axes like model accessibility, commercial usability, task specificity, and more. This discussion provides a guide for navigating the overwhelming number of models available today.

Both Shawn and Daniel discuss their strategies for staying updated in the rapidly evolving AI landscape. They touch on the role of platforms like Twitter and Hugging Face, emphasizing the importance of keeping an eye on download statistics to gauge model popularity.

Finally, Daniel highlights an interesting collaboration where grassroots efforts led to the creation of language models in six African languages. These models, discovered by Meta on Hugging Face, have now been incorporated into new models released by the tech giant. This showcases the full-circle impact of community-driven AI initiatives, further emphasizing the significance of tailored technological solutions.

AI Landscape Evolution: From Open-Source Advocacy to the Rise of AI Engineering

In a podcast episode, Alessio Fanelli highlights the shift in the AI landscape, pointing to how a proliferation of models are becoming available on platforms like Hugging Face, democratizing access and creating opportunities for more players to participate. The conversation delves into Jonathan Frankle's advocacy for keeping AI models open-source. Shawn Wang and Daniel Whitenack discuss the potential of incorporating audio enhancements to podcasts to maintain engagement. They also discuss a past episode with Kirsten Lum, who addressed the challenges of managing ML tasks in mid-sized organizations compared to larger enterprises.

A significant part of the conversation focuses on the practical implementation of AI in enterprises. Daniel underscores that while many companies experiment with tools like ChatGPT, they often don't progress past superficial usage. He emphasizes the need for businesses to dig deeper into the applications of these models, from prompt engineering to more complex operations. The discussion evolves into the topic of prompt engineering, its hype, and its genuine significance in the AI ecosystem. Shawn Wang introduces the term "AI engineering," suggesting that the discipline is evolving as a subset of software engineering, marking a shift from traditional ML roles to a fusion with general software engineering roles.

Transitioning to AI: Challenges, the Importance of UX, and the Evolution of Data Labeling

Unique Challenges of Engineers vs. Data Scientists Transitioning to AI Daniel Whitenack and Shawn Wang discuss the challenges faced by software engineers and data scientists as they transition into AI-focused roles. For software developers, confronting non-deterministic systems, particularly those they don't fully control, like blackbox models from OpenAI, presents a unique hurdle. They're also working with a model whose capabilities have not been fully tapped into yet.

From a data scientist's perspective, Whitenack highlights the instinct to jump into model training and fine-tuning, even though significant achievements can be made through data augmentation and prompting. He underscores that data scientists have dealt with uncertainties for a while, but it's different when they aren't in charge of the datasets or the training.

The Value of AI User Experience (UX) Wang introduces the concept of AI UX, emphasizing that the final step of showcasing AI output in a user-friendly manner can be as crucial as training the model. This is evidenced by OpenAI's ChatGPT, which many deem a UX breakthrough. Whitenack concurs, suggesting that while innovations in modeling and data are critical, a poor user experience can nullify those advances. He cites Gmail's smooth AI-powered autocomplete feature as an example of seamless AI integration that provides immediate value to users.

Discussion on Dataset Evolution and Curation Whitenack mentions the evolution in NLP datasets and the popularity of open-source tools like Label Studio for data labeling. Wang notes that Label Studio has introduced tools for fine-tuning generative AI models. Whitenack discusses the challenges companies face in understanding instruction-tuned models and emphasizes the importance of clear tooling around this process to make it more approachable.

Finally, Wang ponders on why multiple data-labeling companies like Label Box and Label Studio continue to emerge and flourish even when giants like Scale already exist in the market, pointing to the ongoing importance and evolution of data-centric AI.

Deciphering AI's Shift from Model-Centric to Data-Centric Paradigms

AI Customization and The Evolution of Data Use

Daniel Whitenack discusses the evolving relationship between AI and businesses, emphasizing a shift from a model-centric to a data-centric approach. Many enterprises, he notes, are now thinking about leveraging state-of-the-art models for their data. This involves either enhancing these models with their data or fine-tuning them, a perspective reflected by the rise of APIs that offer fine-tuning.

Shawn Wang raises a pertinent point on OpenAI's shift away from fine-tuning for models like GPT-3.5 and GPT-4. He also delves into the gray area of unlabeled datasets for unsupervised and self-supervised learning. The conversation further delves into the challenges of determining the right data mix for training models and highlights the lack of clarity on the best practices.

Whitenack adds insight into the world of data mixes, suggesting that the exact composition of data used by top models like those from OpenAI remains unknown. This creates a challenge for those attempting to emulate such models. He emphasizes the importance of experimenting with different mixes of data to optimize model training. Whitenack encourages AI enthusiasts to remain hands-on, leveraging the array of available tools to gain a better understanding of models and to foster innovation.

In a lightning round, Whitenack highlights his astonishment at the general-purpose capabilities of large language models, especially beyond traditional NLP tasks. He points out the huge potential for exploration in AI, especially with respect to languages other than English and Mandarin, and also in diverse modalities of communication like sign language. He ends with a call to action, urging individuals to get involved and hands-on with AI models to gain intuition and insight.

End Note: The transcript is open source and available on GitHub for improvements.