Interactive project report with Jira and LLM

For all projects I worked on, I used some sort of project management system where project scope was defined as a list of tasks (tickets), and progress was reported by changing task statuses.

While such project management systems offer various dashboards and reports, interpreting a long list of tasks with cryptic titles created by engineers is not trivial. To provide transparency for project sponsors and customers, I had to manually create a project report and then answer related questions.

What if instead we asked LLM for help? The idea is simple: fetch all project tasks and feed them to LLM asking for the report. Then start a chat allowing for further questions.

Collecting data

We will use Jira for this experiment as it's a popular tool with easy-to-use REST API. The example project, for which we'll create the report, is very technical - it's about creating a build script that can detect what the code uses and generate the required instructions for the build system. Such a project will surely have technical tasks with cryptic titles.

Let's start with fetching tasks. The project in the example setup is represented as a single parent ticket (an epic) with a list of child tickets (tasks). For each task, we will fetch the full history to see how the ticket status changes over time. Using Python's Jira client the implementation is simple. Note in Jira nomenclature the term issue is used instead of ticket which is reflected in the code.

jira = JIRA(server=os.environ["JIRA_SERVER"], basic_auth=(os.environ["JIRA_USER"], os.environ["JIRA_TOKEN"]))

def fetch_issues(epic_key):
    issues = []
    print("Loading epic data...", end="", flush=True)
    issues.append(jira.issue(epic_key))
    print("done")

    print("Loading tasks...", end="", flush=True)
    child_issues = jira.search_issues(f"parent = {epic_key}")
    for issue in child_issues:
        issues.append(jira.issue(issue.key, expand="changelog"))
    print("done")

    return issues

Since fetching all tickets with history takes a while, it is handy to store this data locally for further experiments. While playing with the implementation I used the below functions to save tasks to, and load them from a file:

def save_issues(filename, issues):  
    with open(filename, "x") as file:
        file.write("[")
        file.write(",".join(
            json.dumps(issue.raw) for issue in issues))
        file.write("]")

def load_issues(filename):
    with open(filename, "r") as file:
        data = json.load(file)
        return [Issue(jira._options, jira._session, raw=raw_issue)
            for raw_issue in data]

Preparing data

The next step is to prepare data for LLM. Raw Jira data in JSON format is quite verbose, we don't need all these extra fields. Let's extract basic information: subject, description, type, status, and creation date. From history, we will only extract ticket status changes along with their date and author, ignoring changes to other fields.

All this information will be stored as plain text. I have seen people using JSON or XML as LLM input, but my observation is LLMs are very good at interpreting plain text data. Plus with this approach, I don't need to worry about formatting text fields to be JSON or XML compatible. The only processing I do is to strip empty lines from the description, and the primary reason is to make it easier for me to look at results.

def strip_empty_lines(s):
    return "".join(line for line in (s or "").splitlines() if line.strip())

def issue_to_str(issue):
    return f"""
{issue.fields.issuetype}: {issue.key}
Summary: {issue.fields.summary}
Description: {strip_empty_lines(issue.fields.description)}
Type: {issue.fields.issuetype}
Status: {issue.fields.status}
Created: {issue.fields.created}
Priority: {issue.fields.priority}
"""

def changelog_to_str(changelog, changeitem):
    return f"""
Author: {changelog.author.displayName}
Date: {changelog.created}
Status change from: {changeitem.fromString} to: {changeitem.toString}
"""


def history_to_str(issue):
    if issue.changelog is None or issue.changelog.total == 0:
        return ""
    history_description = ""
    for changelog in issue.changelog.histories:
        try:
            statuschange = next(filter(lambda i: i.field == "status", changelog.items))
            history_description += changelog_to_str(changelog, statuschange)
        except StopIteration:
            pass
    return history_description

#this function assumes the first issue is an epic followed by tasks.
def describe_issues(issues):
    description = "Project details:"
    description += issue_to_str(issues[0])
    description += "\nProject tasks:"
    for issue in issues[1:]:
        description += "\n" + issue_to_str(issue)
        description += f"History of changes for task {issue.key}:"
        description += history_to_str(issue)
    return description

The epic I use for this experiment has 30 tasks that have between 1 and 15 status changes in their history. I will not quote the full output of the describe_issues function, but to give you an idea of how it looks here is a short excerpt:

Project details:
Epic: TKT-642
Summary: Create universal build script
Description: 
Type: Epic
Status: In Development
Created: 2024-05-24T10:48:33.050+0200
Priority: P4 - Low

Project tasks:

Task: TKT-805
Summary: add test reporting for e2e tests
Description: 
Type: Task
Status: In Progress
Created: 2024-09-06T09:56:33.919+0200
Priority: P4 - Low
History of changes for task TKT-805:
Author: Jane Doe
Date: 2024-09-06T10:04:15.325+0200
Status change from: To Do to: In Progress

Task: TKT-801
Summary: Sonar detection
Description: * add sonar config file detection *
Type: Task
Status: In Progress
Created: 2024-08-30T13:57:44.364+0200
Priority: P4 - Low
History of changes for task TKT-801:
Author: Jane Doe
Date: 2024-08-30T13:57:58.450+0200
Status change from: To Do to: In Progress

Task: TKT-799
Summary: Add check_tests step
Description: 
Type: Task
Status: Review
Created: 2024-08-29T18:33:52.268+0200
Priority: P4 - Low
History of changes for task TKT-799:
Author: Jane Doe
Date: 2024-08-29T18:40:35.305+0200
Status change from: In Progress to: Review
Author: Jane Doe
Date: 2024-08-29T18:33:57.095+0200
Status change from: To Do to: In Progress

Prompting

The prompt we'll use consists of two parts. First, it gives an instruction to create a report, detailing what specific information we want in the report. Then we insert ticket information prepared in previous paragraphs. LLMs tend to give verbose responses, so we ask specifically to not add any additional explanation. As the experiment is performed in the terminal, we'll also ask to make the response terminal-freindly.

def create_prompt(isses_description):
    return f"""
    Using below information from jira generate executive report that focuses on what was done and what is remaining.
    Provide project progress information. Point out problematic tasks and what may be wrong.
    Use status change history to point out tasks that were worked on for too long or have been reopened many times. 
    Check if you can detect other problems from tasks or their history.
    Provide only text content that can be displayed on terminal, without any additional explanation or markdown.
    {isses_description}
    """

The LLM I use for this experiment is Anthropic's Claude 3.5 Sonnet. In addition to sending the prompt message, we'll also specify identity. Identity could be just added as another line at the beginning of the prompt, but since the plan is to follow up with questions (create an interactive chat), Claude's documentation recommends setting identity as a separate system parameter in such a scenario.

def send_messages(messages):
    client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_KEY"])
    identity = "You are a seasoned IT program manager. You have very good communication skills, your answers are concise and focus on key insights."
    message = client.messages.create(
        system=identity,
        max_tokens=2048,
        messages=messages,
        model="claude-3-5-sonnet-20240620"
    )
    return message.content[0].text

def generate_report(prompt):
    messages=[
        {
            "role": "user",
            "content": prompt 
        }]
    return send_messages(messages)

Let's put this together and run:

if __name__=="__main__":
    issues = fetch_issues('TKT-642')
    descrption = describe_issues(issues)
    prompt = create_prompt(description)
    report = generate_report(prompt)
    print(report)

Below is the full report I received from Claude:

Executive Report: Universal Build Script Project (TKT-642)

Project Progress:

• 11 out of 24 tasks are completed (45.8%)
• 5 tasks are in progress or review (20.8%)
• 8 tasks remain to be started (33.3%)

Key Accomplishments:

1. Created prototype jinja template for builder
2. Implemented config extension and validation
3. Added unit tests execution step
4. Integrated support for sonar scanning
5. Created image for building config and tech scripts

In Progress:

• Adding test reporting for e2e tests
• Implementing filtering logic for environments and Sonar detection
• Developing developer documentation

Problematic Areas:

1. Task TKT-734 (Add config extension) went through multiple status changes, indicating potential issues or complexity.
2. Task TKT-697 (Add support for sonar scanning) has been reopened multiple times, suggesting implementation challenges.
3. Task TKT-715 (Create developer documentation) has been in progress for an extended period without completion.

Upcoming Work:

• Integration of the build system into various services (authentication, database-api, shopping-cart, reporting, localization)
• Adding support for monorepos and custom jobs
• Creating user documentation

Recommendations:

1. Review the implementation of config extension and sonar scanning to address recurring issues.
2. Prioritize the completion of developer documentation to support team adoption.
3. Begin planning for the integration of the build system into remaining services.
4. Consider reassessing the priority of tasks that have remained in the "To Do" state for an extended period.

Overall, the project is making steady progress, with nearly half of the tasks completed. However, attention should be given to resolving recurring issues and completing in-progress tasks to maintain momentum.

The result is quite good. It did miscount tickets - in various runs I did it gave me a different number, but never a correct number of 30. Well, LLMs are known for not being able to count r's in the word strawberry. But the summary is correct, recommendations are also reasonable.

Asking questions

Now let's create an interactive session and ask some questions. We'll start with a pre-defined question which is the original prompt we used to generate the report. Then we will read the user's input and attach it to messages. Each response, along with the report is also recorded in the messages chain.

def interactive_session(prompt):
    messages = []
    user_input = prompt
    while user_input != "exit":
        messages.append({
            "role" : "user",
            "content" : user_input
        })
        response = send_messages(messages)
        print(response)
        messages.append({
            "role" : "assistant",
            "content" : response
        })
        user_input = input("QUESTION: ")
    return messages

To run this, just replace generate_report in the main section of the script with interactive_session. Below is a recording of an example session I did. I omitted from the quote the first response which just prints out the report.

QUESTION: how many engineers the team counts?

Based on the information provided in the task histories, I can identify at least three distinct engineers working on this project:

1. John Smith
2. Jane Doe
3. Mike Carpenter

These three names appear frequently in the status change histories for various tasks. However, it's important to note that this may not represent the full team size. There could be other team members who haven't made changes recorded in these specific task histories, or who are involved in the project in other capacities not reflected in this data.

To get an accurate count of the team size, you would need additional information beyond what's provided in these task histories.

QUESTION: is unit test execution task done yet?

Yes, the unit test execution task is completed. Specifically, task TKT-700 with the summary "Add unit tests execution step" is marked as Done. Here's the relevant information:

Task: TKT-700
Summary: Add unit tests execution step
Status: Done

The history of changes for this task shows that it was moved to "Done" status on 2024-08-02, after going through the typical workflow stages (To Do -> In Progress -> Review -> PREPROD VERIFICATION -> READY FOR PRODUCTION -> Done).

QUESTION: does the scope include running some sort of static code analysis tool?

Yes, the project scope includes running static code analysis. Specifically, there's a task for implementing Sonar scanning, which is a popular static code analysis tool. Here are the details:

Task: TKT-697
Summary: Add support for sonar scanning
Status: Done

This task was completed and moved to "Done" status on 2024-09-02. The implementation of Sonar scanning indicates that static code analysis is part of the universal build script project.

Additionally, there's another related task that's currently in progress:

Task: TKT-801
Summary: Sonar detection
Status: In Progress

This task mentions "sonar detection" in its summary, which further confirms that Sonar (a static analysis tool) is being integrated into the build process.

These tasks demonstrate that static code analysis, particularly using Sonar, is included in the project scope and has been largely implemented.

QUESTION: what are the 3 biggest risks of the project?

Based on the provided information, the three biggest risks for this project appear to be:

1. 통합 지연:
   빌드 시스템을 다양한 서비스(현지화, 보고, 장바구니, 인증, 데이터베이스 API)에 통합하기 위한 여러 작업은 여전히 ​​"To Do" 상태입니다. 이로 인해 다양한 서비스 전반에 걸쳐 범용 빌드 스크립트의 광범위한 채택이 지연될 위험이 있습니다. 의도한 모든 서비스를 통합하지 못하면 프로젝트의 전반적인 영향과 가치가 제한될 수 있습니다.

2. 문서화 지연:
   개발자 문서(TKT-715) 작성 작업은 오랫동안 "To Do" 상태에 있었다가 최근에야 "진행 중"으로 이동되었습니다. 또한 사용자 문서 작업(TKT-714)은 여전히 ​​"To Do" 상태입니다. 문서 작성이 지연되면 새로운 빌드 시스템의 채택, 적절한 사용 및 유지 관리가 방해받을 위험이 있습니다. 명확한 지침이 부족하여 오용, 비효율성 또는 도입 거부로 이어질 수 있습니다.

3. 구현의 복잡성:
   여러 작업, 특히 소나 스캔(TKT-697) 및 구성 확장(TKT-734)과 관련된 작업에서 "진행 중"과 "검토" 사이에 여러 상태 변경이 표시되었습니다. 이 패턴은 구현 시 잠재적인 기술적 문제나 불확실성을 암시합니다. 이러한 복잡성을 제대로 해결하지 않으면 향후 버그, 성능 문제 또는 유지 관리 어려움이 발생할 위험이 있습니다.

이러한 위험은 적절하게 관리 및 완화되지 않으면 프로젝트 일정, 채택률 및 전반적인 성공에 잠재적으로 영향을 미칠 수 있습니다.

의외로 좋았습니다. 저는 LLM이 프롬프트 데이터와 함께 일반 지식을 사용한다는 사실을 좋아합니다. 따라서 티켓 데이터에서 해당 용어를 언급하지 않더라도 "정적 코드 분석 도구"에 대한 질문에 답할 수 있었습니다. 답변이 너무 장황하므로 프롬프트에 대한 더 많은 실험이 필요합니다. 아마도 각 사용자의 쿼리에 지침을 추가하면 더 나은 답변을 만드는 데 도움이 될 것입니다.
잘못 계산된 티켓과 같은 문제는 해결하기 쉬워야 하며 기본 통계를 계산하여 프롬프트 자체에 포함할 수 있습니다.

The above is the detailed content of Interactive project report with Jira and LLM. For more information, please follow other related articles on the PHP Chinese website!