How to Build Multi-Modal Agentic System For Stock Insights?

Multimodal agentic systems represent a revolutionary advancement in the field of artificial intelligence, seamlessly combining diverse data types—such as text, images, audio, and video—into a unified system that significantly enhances the capabilities of intelligent technologies. These systems rely on autonomous intelligent agents that can independently process, analyze, and synthesize information from various sources, facilitating a deeper and more nuanced understanding of complex situations.

By merging multimodal inputs with agentic functionality, these systems can dynamically adapt in real time to changing environments and user interactions, offering a more responsive and intelligent experience. This fusion not only boosts operational efficiency across a range of industries but also elevates human-computer interactions, making them more fluid, intuitive, and contextually aware. As a result, multimodal agentic frameworks are set to reshape the way we interact with and utilize technology, driving innovation in countless applications across sectors.

Learning Objectives

• Benefits of agentic AI systems with advanced image analysis
• How Crew AI’s Vision Tool enhances agentic AI capabilities?
• Overview of DeepSeek-R1-Distill-Qwen-7B model and its features
• Hands-on Python tutorial integrating Vision Tool with DeepSeek R1
• Building a multi-modal, multi-agentic system for stock analysis
• Analyzing and comparing stock behaviours using stock charts

This article was published as a part of theData Science Blogathon.

Table of contents

• Agentic AI systems with Image Analysis Capabilities
• Building a Multi-Modal Agentic System to Explain Stock Behavior From Stock Charts
• Hands-On Python Implementation using Ollama on Google Colab
• Another Example of a Multi-Modal Agentic System For Stock Insights
• Conclusions
• Frequently Asked Questions

Agentic AI systems with Image Analysis Capabilities

Agentic AI systems, fortified with sophisticated image analysis capabilities, are transforming industries by enabling a suite of indispensable functions.

• Instantaneous Visual Data Processing: These advanced systems possess the capacity to analyze immense quantities of visual information in real time, dramatically improving operational efficiency across diverse sectors, including healthcare, manufacturing, and retail. This rapid processing facilitates quick decision-making and immediate responses to dynamic conditions.
• Superior Precision in Image Recognition: Boasting recognition accuracy rates surpassing 95%, agentic AI substantially diminishes the occurrence of false positives in image recognition tasks. This elevated level of precision translates to more dependable and trustworthy outcomes, crucial for applications where accuracy is paramount.
• Autonomous Task Execution: By seamlessly incorporating image analysis into their operational frameworks, these intelligent systems can autonomously execute intricate tasks, such as providing medical diagnoses or conducting surveillance operations, all without the need for direct human oversight. This automation not only streamlines workflows but also minimizes the potential for human error, paving the way for increased productivity and reliability.

Crew AI Vision Tool

CrewAI is a cutting-edge, open-source framework designed to orchestrate autonomous AI agents into cohesive teams, enabling them to tackle complex tasks collaboratively. Within CrewAI, each agent is assigned specific roles, equipped with designated tools, and driven by well-defined goals, mirroring the structure of a real-world work crew.

The Vision Tool expands CrewAI’s capabilities, allowing agents to process and understand image-based text data, thus integrating visual information into their decision-making processes. Agents can leverage the Vision Tool to extract text from images by simply providing a URL or a file path, enhancing their ability to gather information from diverse sources. After the text is extracted, agents can then utilize this information to generate comprehensive responses or detailed reports, further automating workflows and enhancing overall efficiency. To effectively use the Vision Tool, it’s necessary to set the OpenAI API key within the environment variables, ensuring seamless integration with language models.

Building a Multi-Modal Agentic System to Explain Stock Behavior From Stock Charts

We will construct a sophisticated, multi-modal agentic system that will first leverage the Vision Tool from CrewAI designed to interpret and analyze stock charts (presented as images) of two companies. This system will then harness the power of the DeepSeek-R1-Distill-Qwen-7B model to provide detailed explanations of these companies’ stock’s behaviour, offering well-reasoned insights into the two companies’ performance and comparing their behaviour. This approach allows for a comprehensive understanding and comparison of market trends by combining visual data analysis with advanced language models, enabling informed decision-making.

DeepSeek-R1-Distill-Qwen-7B

To adapt DeepSeek R1’s advanced reasoning abilities for use in more compact language models, the creators compiled a dataset of 800,000 examples generated by DeepSeek R1 itself. These examples were then used to fine-tune existing models such as Qwen and Llama. The results demonstrated that this relatively simple knowledge distillation method effectively transferred R1’s sophisticated reasoning capabilities to these other models

The DeepSeek-R1-Distill-Qwen-7B model is one of the distilled DeepSeek R1’s models. It is a distilled version of the larger DeepSeek-R1 architecture, designed to offer enhanced efficiency while maintaining robust performance. Here are some key features:

The model excels in mathematical tasks, achieving an impressive score of92.8% on the MATH-500 benchmark, demonstrating its capability to handle complex mathematical reasoning effectively.

In addition to its mathematical prowess, the DeepSeek-R1-Distill-Qwen-7B performs reasonably well on factual question-answering tasks, scoring49.1% on GPQA Diamond, indicating a good balance between mathematical and factual reasoning abilities.

We will leverage this model to explain and find reasonings behind the behaviour of stocks of companies post extraction of information from stock chart images.

Hands-On Python Implementation using Ollama on Google Colab

We will be using Ollama for pulling the LLM models and utilizing T4 GPU on Google Colab for building this multi-modal agentic system.

Step 1. Install Necessary Libraries

!pip install crewai crewai_tools
!sudo apt update
!sudo apt install -y pciutils
!pip install langchain-ollama
!curl -fsSL https://ollama.com/install.sh | sh
!pip install ollama==0.4.2

Step 2. Enablement of Threading to Setup Ollama Server

import threading
import subprocess
import time

def run_ollama_serve():
  subprocess.Popen(["ollama", "serve"])

thread = threading.Thread(target=run_ollama_serve)
thread.start()
time.sleep(5)

Step 3. Pulling Ollama Models

!ollama pull deepseek-r1

Step 4. Defining OpenAI API Key and LLM model

import os
from crewai import Agent, Task, Crew, Process, LLM
from crewai_tools import LlamaIndexTool
from langchain_openai import ChatOpenAI
from crewai_tools import VisionTool
vision_tool = VisionTool()

os.environ['OPENAI_API_KEY'] =''
os.environ["OPENAI_MODEL_NAME"] = "gpt-4o-mini"

llm = LLM(
    
    model="ollama/deepseek-r1",
)

Step 5. Defining the Agents, Tasks in the Crew

def create_crew(image_url,image_url1):

  #Agent For EXTRACTNG INFORMATION FROM STOCK CHART
  stockchartexpert= Agent(
        role="STOCK CHART EXPERT",
        goal="Your goal is to EXTRACT INFORMATION FROM THE TWO GIVEN %s & %s stock charts correctly """%(image_url, image_url1),
        backstory="""You are a STOCK CHART expert""",
        verbose=True,tools=[vision_tool],
        allow_delegation=False

    )

  #Agent For RESEARCH WHY THE STOCK BEHAVED IN A SPECIFIC WAY
  stockmarketexpert= Agent(
        role="STOCK BEHAVIOUR EXPERT",
        goal="""BASED ON THE PREVIOUSLY EXTRACTED INFORMATION ,RESEARCH ABOUT THE RECENT UPDATES OF THE TWO COMPANIES and EXPLAIN AND COMPARE IN SPECIFIC POINTS WHY THE STOCK BEHAVED THIS WAY . """,
        backstory="""You are a STOCK BEHAVIOUR EXPERT""",
        verbose=True,

        allow_delegation=False,llm = llm
         )

  #Task For EXTRACTING INFORMATION FROM A STOCK CHART
  task1 = Task(
      description="""Your goal is to EXTRACT INFORMATION FROM THE GIVEN %s & %s stock chart correctly """%((image_url,image_url1)),
      expected_output="information in text format",
      agent=stockchartexpert,
  )

  #Task For EXPLAINING WITH ENOUGH REASONINGS WHY THE STOCK BEHAVED IN A SPECIFIC WAY
  task2 = Task(
      description="""BASED ON THE PREVIOUSLY EXTRACTED INFORMATION ,RESEARCH ABOUT THE RECENT UPDATES OF THE TWO COMPANIES and EXPLAIN AND COMPARE IN SPECIFIC POINTS WHY THE STOCK BEHAVED THIS WAY.""",
      expected_output="Reasons behind stock behavior in BULLET POINTS",
      agent=stockmarketexpert
  )
 
  #Define the crew based on the defined agents and tasks
  crew = Crew(
      agents=[stockchartexpert,stockmarketexpert],
      tasks=[task1,task2],
      verbose=True,  # You can set it to 1 or 2 to different logging levels
  )

  result = crew.kickoff()
  return result

Step 6. Running the Crew

The below two stock charts were given as input to the crew

text = create_crew("https://www.eqimg.com/images/2024/11182024-chart6-equitymaster.gif","https://www.eqimg.com/images/2024/03262024-chart4-equitymaster.gif")
pprint(text)

Final Output

Mamaearth's stock exhibited volatility during the year due to internal<br> challenges that led to significant price changes. These included unexpected<br> product launches and market controversies which caused both peaks and<br> troughs in the share price, resulting in an overall fluctuating trend.<br><br>On the other hand, Zomato demonstrated a generally upward trend in its share<br> price over the same period. This upward movement can be attributed to<br> expanding business operations, particularly with successful forays into<br> cities like Bengaluru and Pune, enhancing their market presence. However,<br> near the end of 2024, external factors such as a major scandal or regulatory<br> issues might have contributed to a temporary decline in share price despite<br> the overall positive trend.<br><br>In summary, Mamaearth's stock volatility stems from internal inconsistencies<br> and external controversies, while Zomato's upward trajectory is driven by<br> successful market expansion with minor setbacks due to external events.

As seen from the final output, the agentic system has given quite a good analysis and comparison of the share price behaviours from the stock charts with sufficient reasonings like a foray into cities, and expansion in business operations behind the upward trend of the share price of Zomato.

Another Example of a Multi-Modal Agentic System For Stock Insights

Let’s check and compare the share price behaviour from stock charts for two more companies – Jubilant Food Works & Bikaji Foods International Ltd. for the year 2024.

text = create_crew("https://s3.tradingview.com/p/PuKVGTNm_mid.png","https://images.cnbctv18.com/uploads/2024/12/bikaji-dec12-2024-12-b639f48761fab044197b144a2f9be099.jpg?im=Resize,width=360,aspect=fit,type=normal")
print(text)

Final Output

The stock behavior of Jubilant Foodworks and Bikaji can be compared based on<br> their recent updates and patterns observed in their stock charts.<br><br>Jubilant Foodworks:<br><br>Cup & Handle Pattern: This pattern is typically bullish, indicating that the<br> buyers have taken control after a price decline. It suggests potential<br> upside as the candlestick formation may signal a reversal or strengthening<br> buy interest.<br><br>Breakout Point: The horizontal dashed line marking the breakout point implies<br> that the stock has reached a resistance level and may now test higher<br> prices. This is a positive sign for bulls, as it shows strength in the<br> upward movement.<br><br>Trend Line Trend: The uptrend indicated by the trend line suggests ongoing<br> bullish sentiment. The price consistently moves upwards along this line,<br> reinforcing the idea of sustained growth.<br><br>Volume Correlation: Volume bars at the bottom showing correlation with price<br> movements indicate that trading volume is increasing alongside upward price<br> action. This is favorable for buyers as it shows more support and stronger<br> interest in buying.<br><br>Bikaji:<br><br>Recent Price Change: The stock has shown a +4.80% change, indicating positive<br> momentum in the short term.<br><br>Year-to-Date Performance: Over the past year, the stock has increased by<br> 61.42%, which is significant and suggests strong growth potential. This<br> performance could be attributed to various factors such as market<br> conditions, company fundamentals, or strategic initiatives.<br><br>Time Frame: The time axis spans from January to December 2024, providing a<br> clear view of the stock's performance over the next year.<br><br>Comparison:<br><br>Both companies' stocks are showing upward trends, but Jubilant Foodworks has<br> a more specific bullish pattern (Cup & Handle) that supports its current<br> movement. Bikaji, on the other hand, has demonstrated strong growth over the<br> past year and continues to show positive momentum with a recent price<br> increase. The volume in Jubilant Foodworks correlates well with upward<br> movements, indicating strong buying interest, while Bikaji's performance<br> suggests sustained or accelerated growth.<br><br>The stock behavior reflects different strengths: Jubilant Foodworks benefits<br> from a clear bullish pattern and strong support levels, whereas Bikaji<br> stands out with its year-to-date growth. Both indicate positive<br> developments, but the contexts and patterns differ slightly based on their<br> respective market positions and dynamics.

As seen from the final output, the agentic system has given quite a good analysis and comparison of the share price behaviours from the stock charts with elaborate explanations on the trends seen like Bikaji’s sustained performance in contrast to Jubilant Foodworks’ bullish pattern.

Conclusions

In conclusion, multimodal agentic frameworks mark a transformative shift in AI by blending diverse data types for better real-time decision-making. These systems enhance adaptive intelligence by integrating advanced image analysis and agentic capabilities. As a result, they optimize efficiency and accuracy across various sectors. The Crew AI Vision Tool and DeepSeek R1 model demonstrate how such frameworks enable sophisticated applications, like analyzing stock behaviour. This advancement highlights AI’s growing role in driving innovation and improving decision-making.

Key Takeaways

1. Multimodal Agentic Frameworks: These frameworks integrate text, images, audio, and video into a unified AI system, enhancing artificial intelligence capabilities. Intelligent agents within these systems independently process, analyze, and synthesize information from diverse sources. This ability allows them to develop a nuanced understanding of complex situations, making AI more adaptable and responsive.
2. Real-Time Adaptation: By merging multimodal inputs with agentic functionality, these systems adapt dynamically to changing environments. This adaptability enables more responsive and intelligent user interactions. The integration of multiple data types enhances operational efficiency across various sectors, including healthcare, manufacturing, and retail. It improves decision-making speed and accuracy, leading to better outcomes
3. Image Analysis Capabilities: Agentic AI systems with advanced image recognition can process large volumes of visual data in real time, delivering precise results for applications where accuracy is critical. These systems autonomously perform intricate tasks, such as medical diagnoses and surveillance, reducing human error and improving productivity.
4. Crew AI Vision Tool: This tool enables autonomous agents within CrewAI to extract and process text from images, enhancing their decision-making capabilities and improving overall workflow efficiency.
5. DeepSeek-R1-Distill-Qwen-7B Model: This distilled model delivers robust performance while being more compact, excelling in tasks like mathematical reasoning and factual question answering, making it suitable for analyzing stock behaviour.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Frequently Asked Questions

Q1. What are multimodal agentic frameworks in AI?

Ans. Multimodal agentic frameworks combine diverse data types like text, images, audio, and video into a unified AI system. This integration enables intelligent agents to analyze and process multiple forms of data for more nuanced and efficient decision-making.

Q2. What is Crew AI?

Ans. Crew AI is an advanced, open-source framework designed to coordinate autonomous AI agents into cohesive teams that work collaboratively to complete complex tasks. Each agent within the system is assigned a specific role, equipped with designated tools, and driven by well-defined goals, mimicking the structure and function of a real-world work crew.

Q3. How does the Crew AI Vision Tool enhance multimodal systems?

Ans. The Crew AI Vision Tool allows agents to extract and process text from images. This capability enables the system to understand visual data and integrate it into decision-making processes, further improving workflow efficiency.

Q4. What industries can benefit from agentic AI systems with image analysis capabilities?

Ans. These systems are especially beneficial in industries like healthcare, manufacturing, and retail, where real-time analysis and precision in image recognition are critical for tasks such as medical diagnosis and quality control.

Q5. What are DeepSeek R1’s distilled models?

Ans. DeepSeek-R1’s distilled models are smaller, more efficient versions of the larger DeepSeek-R1 model, created using a process called distillation, which preserves much of the original model’s reasoning power while reducing computational demands. These distilled models are fine-tuned using data generated by DeepSeek-R1. Some examples of these distilled models are DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Qwen-14B, DeepSeek-R1-Distill-Llama-8B amongst others.

The above is the detailed content of How to Build Multi-Modal Agentic System For Stock Insights?. For more information, please follow other related articles on the PHP Chinese website!