Pondhouse Data AI - Edition 8

Subscribe, if this was forwarded to you

Hey there,

We’re excited to bring you the 8th edition of our Pondhouse AI newsletter — your source for tips and tricks around AI and LLMs. Whether you want to learn about AI concepts, use AI tools effectively, or see inspiring examples, we’ve got you covered.

Let’s get started!

Cheers, Andreas & Sascha

In todays edition:

• News: AI Agents building other AI agents

• Tutorial: How to use GPT-4o to analyze images

• Tip of the Week: Save costs using Anthropics prompt caching

• Tool of the Week: Open WebUI - the best self-hosted ChatGPT alternative?

Find this Newsletter helpful?
Please forward it to your colleagues and friends - it helps us tremendously.

Top News

Automated Design of Agentic Systems (ADAS) - When Agents design Agents

A new paper from researchers at the University of British Columbia and the Vector Institute introduces the concept of Automated Design of Agentic Systems (ADAS). This innovative framework seeks to massively reduce the complexity of how we develop AI agents by shifting from manual to automated design processes.

Traditionally, building effective AI agents has required extensive human intervention, combining various techniques like chain-of-thought reasoning and tool usage. However, the ADAS approach allows a "meta" agent to autonomously create and refine new AI agents using a novel method called Meta Agent Search. This technique iteratively programs, tests, and archives new agents, progressively improving their capabilities.

Key Insights:

• Automated Creation: ADAS leverages a Turing-complete code space to automatically design agents, enabling the discovery of entirely new AI architectures and techniques.

• Performance Gains: The agents developed through ADAS significantly outperform traditional, hand-designed models in various domains, including math and reading comprehension.

• Transferability: These newly developed agents not only excel in their initial tasks but also demonstrate strong performance when applied to different models and domains.

This work highlights the potential of ADAS to improve the efficiency and effectiveness of AI development, being a significant step towards fully autonomous AI systems. The full paper is available for those interested in exploring the technical details further.

For more information, you can access the paper here.

Tutorials & Use Cases

Using GPT-4o to analyse images

GPT-4o is a powerful multimodal model capable of processing text, images, and soon audio. This tutorial explores its vision capabilities, focusing on image classification, description, and even a little data analysis.

Why use GPT-4o vision?

1. GPT-4o can process images without the need for specialized training.

2. It's effective for zero-shot image classification tasks.

3. The model can provide detailed image descriptions based on specific prompts.

4. GPT-4o can analyze complex visual data, such as dashboards.

5. It's cost-effective, with prices starting at 0.02 cents per low-resolution image.

Step by step (summary):

1. Prepare the image: Use a public URL or encode the image in base64 format.

2. Create the messages array:

   • Include a system message defining the task.

   • Add a user message with text and image data.

3. Use the OpenAI SDK: Create a chat completion request using the prepared messages.

4. Customize the prompt: Adjust the system message to tailor the output for your specific needs.

5. Process the response: Extract and use the content from the API response.

from openai import OpenAI

client = OpenAI(api_key="your_api_key")
messages = [
    {"role": "system", "content": "Classify the image as 'cat' or 'dog'."},
    {"role": "user", "content": [
        {"type": "text", "text": "Classify this image."},
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
    ]}
]
response = client.chat.completions.create(
    model="gpt-4o-2024-08-06",
    messages=messages,
    max_tokens=300
)

print(response.choices[0].message.content)

For a more complete guide - including how to use GPT-4o to analyze dashboards, read the full tutorial here.

Finally: GPT-4o fine-tuning is here - and it’s free

GPT-4o is one of the most used and still best LLMs out there. It is relatively cheap - compared to its size - and provides overall very good outputs. One thing was missing though - the ability to fine-tune this remarkable model. OpenAI changed this and provides fine-tuning for GPT-4o as part of their platform offering.

The good part: It’s free for 1 Million fine-tuning tokens per day (!) - at least until Sept. 23rd 2024.

Best Practices for fine-tuning:

• Prioritize data quality over quantity.

• Start with a moderate dataset and incrementally increase if needed.

• Regularly evaluate your model's performance.

• Use fine-tuning for altering behavior or output format, not for adding new knowledge (use RAG for that).

Read the full announcement here.

For a tutorial on how to fine-tune OpenAI models, see here.

We should not trust the reasoning skills of LLMs

Recent CSAIL research reveals that LLMs like GPT-4 excel in familiar scenarios but struggle with novel variations when it comes to reasoning.

The study found significant performance drops in tasks such as non-base-10 arithmetic and altered chess problems, suggesting LLMs rely more on memorisation than true reasoning.

Lead researcher Zhaofeng Wu emphasises the importance of these findings for improving AI adaptability.

This study highlights the importance of carefully designed AI systems for complex tasks. Rather than relying on single language models alone, organizations should develop tailored AI solutions that combine multiple approaches and account for specific reasoning requirements in their domain.

For more details, see the full article here.

Are we facing an “AI model collapse”?

Recent discussions in the AI community have raised concerns about a potential "model collapse" - a scenario where future AI systems could become progressively less intelligent due to an increase in AI-generated data on the internet.

1. The proliferation of AI-created content online could potentially "pollute" the data sources needed for AI development.

2. Research shows that AI systems trained solely on AI-generated data become less effective over time.

3. Complete model collapse is unlikely, as human and AI data will likely accumulate in parallel, more pressing concerns are:

   • Reduced human interaction in online communities

   • Difficulty in distinguishing between human and AI-generated content

   • Potential loss of socio-cultural diversity in online spaces

For more details, read the full article here.

Tip of the week

Saving up to 90% on LLM costs using Anthropics prompt caching

Anthropic has introduced prompt caching for their Claude AI models, and here's how we can benefit:

What is it?

Prompt caching allows you to store frequently used context between API calls. The ever-the-same parts of your prompt.

Benefits:

• Reduce costs by up to 90%

• Cut latency by up to 85% for long prompts

• Provide more background knowledge to Claude

When to use it:

• Conversational agents

• Coding assistants

• Large document processing

• Detailed instruction sets

• Multi-step tool use scenarios

• Q&A with long-form content

• Your prompt needs to be at least 1024 tokens

Basically anytime you have a longer prompt where parts of the prompt don’t change in between model calls.

How to use it:

• When calling the Claude API, you have a new parameter cache_control as part of the messages array. Set this to ephemeral as shown below:

    system=[
      {
        "type": "text", 
        "text": "You are an AI assistant tasked with analyzing literary works. Your goal is to provide insightful commentary on themes, characters, and writing style.\n",
      },
      {
        "type": "text", 
        "text": "<the entire contents of 'Pride and Prejudice'>",
        "cache_control": {"type": "ephemeral"}
      }
    ],

Tool of the week

Open WebUI: The open source, privacy-friendly ChatGPT alternative

ChatGPT is great, has a good user interface, works well and arguably is one of the best productivity tools invented in the last decade or so.

However, by default, ChatGPT uses each and any conversation for training OpenAI’s latest models - meaning it is a huge data privacy risk.

There are some paid plans available now, which - according to OpenAI - prevent using your data for model training. But:

• These plans are relatively expensive: Starting at 25$ per user per month, up to 60$ per user per month

• It’s still a fact, that you potentially send your most precious and secret data to an American startup - OpenAI - when using ChatGPT

But not using something like ChatGPT is also not the answer, as the productivity gains are undeniable.

That’s where Open WebUI comes into play. It’s - simply put - an open source ChatGPT clone, which runs on your very own on-premise or cloud infrastructure. On top of that it even provides more features than ChatGPT:

1. Data Privacy: Self-hosted solution ensures complete control over your data.

2. Model Flexibility: Easily switch between various AI models, not limited to just OpenAI's offerings.

3. Cost Efficiency: Pay only for actual usage, potentially more economical for organizations.

4. Extensive Customization: Modify interface, integrations, and functionalities to suit your needs.

5. Advanced Model Management: Whitelist models, create custom ones with specific prompts and tools.

6. Side-by-Side Comparison: Compare outputs from multiple models simultaneously.

7. In-Browser Python Execution: Run code snippets directly in your browser.

8. Flexible Authentication: Support for various methods, including OAUTH2 integration.

9. Powerful Extension System: Connect Open WebUI to your internal and external systems via pipelines

10. Community-Driven Extensions: Access a hub of user-created pipelines and add-ons.

11. Integrated RAG: Chat with documents using built-in Retrieval Augmented Generation.

12. Custom Action Buttons: Create unique interaction options for each AI response.

Read our in-depth introduction on our blog.

We hope you liked our newsletter and you stay tuned for the next edition. If you need help with your AI tasks and implementations - let us know. We are happy to help