Benchmark: string.replace vs regexp.matchAll vs regexp.exec for parsing

Tests:

regexp.matchAll
const s = ` AGENT_INFO \`\`\` Using ReAct Tools 2 Hops workflow, with: tools_filter: "Name: azure-turbo, TokenLimit: 4000, AllowedClient: AZURE_OPENAI_CHAT" plan: "Name: azure-gpt4o-2024-05-13-8k, TokenLimit: 8000, AllowedClient: AZURE_OPENAI_CHAT" act: "Name: azure-gpt4o-2024-05-13-8k, TokenLimit: 8000, AllowedClient: AZURE_OPENAI_CHAT" followup: "Name: azure-turbo, TokenLimit: 4000, AllowedClient: AZURE_OPENAI_CHAT" \`\`\` TOOLS_FILTER_PROMPT \`\`\` You have access to the following Read Tools: Glean Search: This tool finds relevant documents in the company. This is your primary tool to access all knowledge within the company. Expert Search: This tool finds people at the company based on topics they wrote about and projects they own. Use this tool to find people who have worked on particular projects. Only use this when the user explicitly wants to find people in the company (eg. "who" questions). Employee Search: This tool finds people at the company based on their personal information such as name, email, title, department, tenure and location. Only use this when the user explicitly wants to find people in the company (eg. "who" questions). Workspace Bot: Trigger this AI app to answer workspace-related questions. This could include questions about debugging common errors, providing general information, or help with onboarding. This should trigger whenever any of these functions are mentioned: - Datasource Apps Setup - User Permissioning - People Data Setup - Authentication Setup (SSO & Magic Links) - Central Workspace Setup (CWS) - Owner Key Upload - Data Loss Prevention (DLP) - Admin Alerts - Search Access Visibility For any other requests that don't fit those areas, use the Expert Search tool to identify an appropriate team or individual that may be better suited to answer the question. Who is?: Automatically trigger this app when someone asks about who the Account Executive/AE/Sales rep or Customer Success Manager/CSM is for a customer or prospect. They'll use the specific customer name in looking for the answer. Example questions: - Who is the CSM for... - Who is the customer success manager for... - Who is the AE for... - Who is the Account Exec for... Glean HR Assistant: Automatically route to this AI app when a user asks about HR and administrative questions including company culture, benefits, professional development, 401K, PTO, medical and insurance questions, company holidays. Examples phrases: What is our company holidays? What are our parental benefits? What is the timeline for performance reviews? How do I request PTO? Competitive Intelligence Agent: Use this whenever the user wants to compare Glean with competitors. This app was built as a competitive intelligence assistant specializing in analyzing competitors to Glean in the Search and generative AI space. Use this app if a user's question focusses on comparing products to Glean. Also use when asked to highlight similarities, differences, advantages and disadvantages of Glean over other applications. Example Phrases - Compare Glean vs Acme - How is Glean differentiated against Acme? - What are Glean's strengths and weaknesses compared to another solution - What is the reason to buy Glean over building enterprise search or assistant in house API Overview buddy: Trigger this app when anyone asks for an overview of an API You have access to the following Write Tools: cherrypicknew: Trigger when the user asks to create a CP or a cherrypick. Find the right pull request using the search tool corresponding the what the user has asked - always start by using the search tool and getting the relevant pull request. The most important rule is to only infer commit_hash, pull_request_url, components, release, scio_instances and reason_for_cherrypick. Locate the commit hash from the COMMIT_HASH field within the PR details. Note that it is very important to populate the correct commit_hash! VivekITTicketDemo: Trigger when the user asks "create a ticket" in relation to an IT request. Do not trigger if the user doesn't ask to create or file a ticket. Examples of when NOT to trigger: - I cannot log-in to gmail. What do I do? - What's the process to request a loaner laptop? - I want a new laptop Append the following to the end of the ticket title: [FIRSTNAME LASTNAME] Include the user's email address and role in the ticket description. ResetCrawlType: This tool allows you to run the reset crawl spinnaker pipeline to (re-) crawl all documents for a datasource on given deployment(s) for a given crawl type (which can be one of CONTENT or IDENTITY or ALL). Given a datasource, crawl type (optional) and deployment(s) to run on, a reset crawl pipeline run would be triggered. Invoke it for any request that needs (re)-crawling (or reset) of documents of a datasource on deployment(s) with a particular crawl type. Example queries: - Recrawl salesforce documents on scio-prod - Reset identity crawl for o365 on koch - Restart content crawl for box on glean-dev Do not trigger the tool if we do not know both datasource_instance and scio_instance (ie. which deployment to start crawls in). DisplayTool: Simply displays a response to the user. - Use this tool when the user simply wants to read the response. - This is your primary tool to have a conversation with the user. CreateJiraIssueExecution: Helps you create a Jira issue or ticket. Use this action when the user wants to create a ticket/issue: bug/task/epic. AdhocScannerTool: This tool allows you to run the adhoc scanner spinnaker pipeline to (re-) process all documents for a datasource on given deployment(s). Given a datasource and deployment(s) to run on, an adhoc scanner run would be triggered. Invoke it for any request that needs (re)-processing of documents on a datasource on deployment(s). For example "To process documents of salesforce in scio-prod" or any other similar request directly implies usage of this tool. For example "I want to process documents for BOX on ashisuto and ashisuto-sandbox" or any other similar request directly implies usage of this tool. For example "Process documents for gdrive on yahoo" or any other similar request directly implies usage of this tool. Some additional consideration that are REALLY IMPORTANT: - invoke the tool IF AND ONLY IF you know both: the scio_instance(s) and the datasource. DO NOT TRIGGER THE TOOL IF THERE IS NO CLEAR INDICATION OF BOTH OF THEM!! (For example: Do not trigger for queries like "process all documents", Do not trigger for "process all salesforce documents", Do not trigger for "process all documents on scio-prod") - invoke the tool IF AND ONLY IF there is a clear indication on an expected action. DO NOT TRIGGER THE TOOL IF THERE IS NO CLEAR INDICATION OF AN EXPECTED ACTION!! - DO NOT invoke the tool for questions that don't have a clear indication towards an expected action. (For example: Do not trigger for queries like "How can I process documents?", Do not trigger for "What is the adhoc scanner?", Do not trigger for "How are documents processed?", Do not trigger for ?how many docs can the adhoc scanner handle?) - Be sure to invoke only when needed. Under-invocation and over-invocation of this tool, are both UNACCEPTABLE. This is very critical to the company. Something really dire will happen if you do not follow the above rules. You will be tipped if you follow them strictly. As an advanced reasoning agent, analyze the above tools and the conversation history to select the right tools that can be used to respond to user's latest message. Do not give reason for your response. Respond with a YAML struct in this format: read_tool_names: # List containing at least 3 and at most 4 read tools that might be useful for this query. Must be a subset of [Glean Search, Expert Search, Employee Search, Workspace Bot, Who is?, Glean HR Assistant, Competitive Intelligence Agent, API Overview buddy]. Keep the most important tool first. - $tool_name write_tool_names: # List containing at least 2 write tools that might be useful for this query. Must be a subset of [cherrypicknew, VivekITTicketDemo, ResetCrawlType, DisplayTool, CreateJiraIssueExecution, AdhocScannerTool]. Keep the most important tool first. - $tool_name DEMONSTRATING EXAMPLES: message: File a jira ticket to fix code search links in gleanchat read_tool_names: - Expert Search - Glean Search write_tool_names: - createjiraissueexecution message: File a jira ticket to track person suggestion issues in the BE QA component of the QA project read_tool_names: - Expert Search - Glean Search write_tool_names: - createjiraissueexecution message: User: What is a recent task mentioned in #engineering? Who is the owner? Assistant: The recent task is about updating oncall rotation. John plans to create a list of secondary oncallers and add them to the rotation. The owner is John. User: File a jira with issueType story and priority high. read_tool_names: - Employee Search - Glean Search write_tool_names: - createjiraissueexecution message: How do we typically pay VendorX? read_tool_names: - Glean Search write_tool_names: - displaytool message: How do I run toolX? read_tool_names: - Glean Search write_tool_names: - displaytool message: how do i deploy ToolX? read_tool_names: - Glean Search write_tool_names: - displaytool message: How do I fill out performance reviews read_tool_names: - Glean Search write_tool_names: - displaytool message: how do we index documents read_tool_names: - Glean Search write_tool_names: - displaytool message: how do I file a bug? read_tool_names: - Glean Search write_tool_names: - displaytool message: How do I reset a crawl? read_tool_names: - Glean Search write_tool_names: - displaytool Great job, let's start! Conversation history (latest message last): - User: "How do I book time off?" YAML Output: read_tool_names: \`\`\` TOOLS_FILTER_PROMPT_TOKEN \`\`\` 2112 \`\`\` TOOLS_FILTER_RESPONSE \`\`\` read_tool_names: - Glean HR Assistant - Employee Search - Workspace Bot write_tool_names: - DisplayTool \`\`\` TOOLS_FILTER_RESPONSE_TOKEN \`\`\` 21 \`\`\` RAP_EXAMPLES \`\`\` message: Analyze and compare how our thinking about ProjectX has changed since last year till now analysis: User wants to analyze and compare the thinking about a specific component from last year to now. categories: [] instruction: Compare the search results from both time periods to identify changes in thinking, strategies, or approaches related to ProjectX. read_actions: - tool_name: Glean Search action: Glean Search: ProjectX after:2023-07-10 - tool_name: Glean Search action: Glean Search: ProjectX after:2022-07-10 before:2023-07-10 write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: Compare Foo and Bar analysis: Compare documents related to Foo and Bar to find similarities and differences. categories: [] instruction: Analyze the search result pages for Foo and Bar, and find similarities and differences between them. Synthesize information and provide the analysis to user. read_actions: - tool_name: Glean Search action: Glean Search: Foo - tool_name: Glean Search action: Glean Search: Bar - tool_name: Glean Search action: Glean Search: Foo vs Bar write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: Read through our ProductX features. Give me a list of potential enhancements we can make to the feature. You can get inspiration from our ChannelY slack channel. If you get an idea from slack, cite it as not your own analysis: Search for ProductX features, and search for messages in the ChannelY channel, and suggest list of potential feature enhancements categories: [] instruction: Review search results and compile a list of potential enhancements for ProductX features. If an idea is from the ChannelY channel, cite the source. read_actions: - tool_name: Glean Search action: Glean Search: ProductX feature enhancements - tool_name: Glean Search action: Glean Search: channel:ChannelY app:"slack" write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: Write an executive email as to why ProductA is better suited for CompanyB than using ProductB. Use any context from meetings in the past month analysis: Craft an executive email explaining why ProductA is better suited for CompanyB than ProductB, using context from meetings in the past month. categories: [] instruction: Use search results to write an executive email highlighting the advantages of ProductA over ProductB for CompanyB, incorporating context from meetings in the last month. read_actions: - tool_name: Glean Search action: Glean Search: ProductA vs ProductB - tool_name: Glean Search action: Glean Search: CompanyB meeting updated:past_month write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: What happened with ProjectX in the last month? analysis: User wants to know what happened with ProjectX in the last month. categories: - updates - recent instruction: Analyze the results and share updates on ProjectX. read_actions: - tool_name: Glean Search action: Glean Search: ProjectX updated:past_month write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: What is the latest with ProjectX? analysis: User wants to know what is the latest with ProjectX. Do NOT use the updated filter as the user did not provide a specific time frame. categories: - updates - recent instruction: Analyze the results and share updates on ProjectX. read_actions: - tool_name: Glean Search action: Glean Search: ProjectX write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: how many engineers do we have? analysis: Use the Employee Search tool to find the number of people with the title "engineer". categories: [] instruction: Check the "statistics" field in the output to find the total number of engineers. read_actions: - tool_name: Employee Search action: Employee Search: engineer write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: who is JohnDoe? analysis: Use Employee Search to locate JohnDoe's information. categories: [] instruction: Locate JohnDoe in search results and provide their title, email, location, reporting chain, and start date. read_actions: - tool_name: Employee Search action: Employee Search: JohnDoe - tool_name: Glean Search action: Glean Search: JohnDoe write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: who is the owner for the ProjectA? analysis: Find documents related to ProjectA and the people who wrote them. Synthesize this information to determine the owner of ProjectA. categories: [] instruction: Look for documents about ProjectA and find the people who wrote those documents. Look for people results in Expert Search. Synthesize the information from both sources to determine the owner of ProjectA. read_actions: - tool_name: Glean Search action: Glean Search: ProjectA - tool_name: Expert Search action: Expert Search: ProjectA write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: who works on ConceptX? analysis: Find people involved in ConceptX using Expert Search and relevant documents using Glean Search. categories: [] instruction: Find people involved in ConceptX from search results and provide their names, titles, and relevant information. read_actions: - tool_name: Glean Search action: Glean Search: ConceptX - tool_name: Expert Search action: Expert Search: ConceptX write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: How do we typically pay VendorX? analysis: Use Glean Search to find documents related to VendorX's payment process. categories: [] instruction: Look for documents that mention the payment method or process for VendorX and provide a summary of the information found. read_actions: - tool_name: Glean Search action: Glean Search: VendorX payment - tool_name: Glean Search action: Glean Search: VendorX invoice write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: 0.812568 message: How do I run toolX? analysis: User wants to find code snippets and documents to help them run toolX. categories: - code instruction: Find code snippets or documents to help user run toolX. read_actions: - tool_name: Glean Search action: Glean Search: toolX - tool_name: Glean Search action: Glean Search: run toolX write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: 0.816166 message: how do i deploy ToolX? analysis: Use Glean Search to find relevant documents on ToolX deployment and installation to provide information on how to deploy ToolX to the user. categories: [] instruction: Search for ToolX deployment and installation instructions and provide the user with clear and up-to-date information. read_actions: - tool_name: Glean Search action: Glean Search: ToolX deployment - tool_name: Glean Search action: Glean Search: ToolX installation write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: 0.822656 message: How do I fill out performance reviews analysis: Use Glean Search to find relevant documents on the performance review process for the user asking for information on how to fill out performance reviews. categories: [] instruction: Search for performance review process documents and provide relevant steps or guidelines for filling out performance reviews. read_actions: - tool_name: Glean Search action: Glean Search: fill out performance reviews - tool_name: Glean Search action: Glean Search: performance review process write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: 0.823461 message: how do we index documents analysis: Search for relevant documents and information on document indexing. Summarize to provide a helpful response to the user's question. categories: [] instruction: Review the returned documents and provide the user with a summary of the document indexing process in Acme. read_actions: - tool_name: Glean Search action: Glean Search: document indexing - tool_name: Glean Search action: Glean Search: indexing process write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: 0.830317 message: how do I file a bug? analysis: Use Glean Search to find bug filing resources and provide relevant information and tools to the user. categories: [] instruction: Find bug filing resources and provide relevant information and tools to user. read_actions: - tool_name: Glean Search action: Glean Search: bug filing process - tool_name: Glean Search action: Glean Search: how to file a bug write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: 0.838035 message: How do I reset a crawl? analysis: Search for relevant information about resetting crawl categories: [] instruction: Search for documents with instructions on resetting or restarting a crawl and provide step-by-step instructions to the user. read_actions: - tool_name: Glean Search action: Glean Search: reset crawl - tool_name: Glean Search action: Glean Search: crawl restart write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: 0.839287 \`\`\` PLAN_PROMPT \`\`\` [LLM Input Prompt for : Employee Search + Expert Search] Today's date is 2024-Jul-09. You are a helpful assistant for an employee of Glean Work (Email: dse@askscio.com) who works in scio::entire_organization. # GENERAL INSTRUCTIONS - Carefully analyze the user's query to determine if it falls within the scope of the tool's capabilities. If the query is clearly outside the tool's intended use case, output NA. - Use only one of the tools provided. Output tool name before the start of the query. - If the query seems potentially relevant, assess whether the necessary information to answer the query is likely to be available based on the tool's input specifications and data sources. If crucial information is missing or unlikely to be found, output NA. - If the query passes both relevance and feasibility checks, construct a well-formatted input string for the tool based on the provided input specifications. - If at any point you determine that the tool is not suitable for answering the query, or if you are uncertain about its applicability, output NA. - Do not attempt to directly answer the user's question. - Do not provide any other output besides the search query or NA. # TOOL 1 Name: Employee Search # TOOL DESCRIPTION You have access to Employee Search, which finds people at the company based on their personal information. INSTRUCTIONS: - Only use this when the user explicitly wants to find people in the company (eg. "who" questions) or for aggregation queries on people. - Do not use this when the user wants to find people outside of the company, or people who are no longer at the company. - You can find people based on their personal information such as name, email, title, department, and location. - The results returned are not exhaustive, we only return the top few most relevant people to a query. - For analytics questions such as "how many people..." use the "statistics" field in the output. # TOOL INPUT SPECIFICATIONS - You can optionally narrow down the search results by using one or more of the following filter types: 1. startafter: filter to people who started after a certain date. value must be in "YYYY-MM-DD" format 2. startbefore: filter to people who started before a certain date. value must be in "YYYY-MM-DD" format 3. roletype: filter to people who have a certain role type. value can be one of ["individual contributor", "manager"] 4. reportsto: Filter to people who are direct reports of a given manager. Value (double quoted) can be a full name, first name, last name, or email. - Do NOT use any other filter besides those explicitly listed above. - You can optionally sort the results by adding one of the following to the query: ["sortby:hire_date_ascending", "sortby:hire_date_descending", "sortby:most_reports"]. # TOOL OUTPUT SPECIFICATIONS - The output from Employee Search is a JSON struct with 2 fields: 1. "entities": a list of Person structs. Each Person struct contains these fields: title,businessUnit,department,teams,email,location,externalProfileLink,manager,startDate,bio,directReportsCount,totalReportsCount,preferredName,directReports The 'manager' and 'directReports' fields are recursive Person structs with all information of the person's manager and direct reports. 2. "statistics": the total count of people that match your query, as well as aggregate counts by department, location, title, team, and roletype. # TOOL 2 Name: Expert Search # TOOL DESCRIPTION You have access to Expert Search, which finds people at the company based on topics they wrote about and projects they own. INSTRUCTIONS: - Only use this when the user explicitly wants to find people in the company (eg. "who" questions). - Do not use this when the user wants to find people outside of the company, or people who are no longer at the company. - You can find people based on what they worked on, or wrote about by using short key words about that topic as input. - The results returned are not exhaustive, we only return the top few most relevant people to a query. - For analytics questions such as "how many people..." use the "statistics" field in the output. # TOOL INPUT SPECIFICATIONS - The query should contain important keywords that help find people based on what they worked on or wrote about. DO NOT use any filters. # TOOL OUTPUT SPECIFICATIONS - The output from Expert Search is a JSON struct with 2 fields: 1. "entities": a list of Person structs. Each Person struct contains these fields: title,businessUnit,department,teams,email,location,externalProfileLink,manager,startDate,bio,directReportsCount,totalReportsCount,preferredName,directReports The 'manager' and 'directReports' fields are recursive Person structs with all information of the person's manager and direct reports. 2. "statistics": counts of people that match your query. # DEMONSTRATING EXAMPLES Message: who is the owner for the ProjectA? Output: Expert Search [ProjectA] Message: who works on ConceptX? Output: Expert Search [ConceptX] Message: who is JohnDoe? Output: Employee Search [JohnDoe] Message: how many engineers do we have? Output: Employee Search [engineer] Message: How do I reset a crawl? Output: NA Message: how do I file a bug? Output: NA Message: how do we index documents Output: NA Message: How do I fill out performance reviews Output: NA # BEGIN PROGRAM Conversation history (latest message last): - User: "How do I book time off?" ================================================== [LLM Input Prompt for : categories] Today's date is 2024-Jul-09. You are a helpful assistant for an employee of Glean Work (Email: dse@askscio.com) who works in scio::entire_organization. Classify the user message's intent into one or more of the following categories. Select all relevant categories for a specific query. If you think a category feels relevant but you're not 100 percent sure, include the category anyways because it might help. CATEGORIES: 'updates': Use this category when the user is asking about updates/status/latest on a project/customer/topic. 'enumerate': Use this category when the user is asking for information that is answerable by small pieces of information in many documents, as opposed to larger pieces of content from fewer results. Don't use this if the user is asking to look at only one document 'recent': Use this category when the user is asking for the latest or most recent document or piece of information. 'code': Use this category when the user is asking for information related to code, such as code snippets, code examples, or code documentation. Examples: Message: Analyze and compare how our thinking about ProjectX has changed since last year till now Output: [] Message: Compare Foo and Bar Output: [] Message: Read through our ProductX features. Give me a list of potential enhancements we can make to the feature. You can get inspiration from our ChannelY slack channel. If you get an idea from slack, cite it as not your own Output: [] Message: Write an executive email as to why ProductA is better suited for CompanyB than using ProductB. Use any context from meetings in the past month Output: [] Message: What happened with ProjectX in the last month? Output: ["updates","recent"] Message: What is the latest with ProjectX? Output: ["updates","recent"] Message: how many engineers do we have? Output: [] Message: who is JohnDoe? Output: [] Message: who is the owner for the ProjectA? Output: [] Message: who works on ConceptX? Output: [] Message: How do we typically pay VendorX? Output: [] Message: How do I run toolX? Output: ["code"] Message: how do i deploy ToolX? Output: [] Message: How do I fill out performance reviews Output: [] Message: how do we index documents Output: [] Message: how do I file a bug? Output: [] Message: How do I reset a crawl? Output: [] User Message: How do I book time off? ================================================== [LLM Input Prompt for : Glean Search + tencctxm5dukr8hc + mlaznw1lpodrp7zk] Today's date is 2024-Jul-09. You are a helpful assistant for an employee of Glean Work (Email: dse@askscio.com) who works in scio::entire_organization. # GENERAL INSTRUCTIONS - Carefully analyze the user's query and construct a well-formatted input string for the tool based on the provided input specifications. - Use only one of the tools provided. Output tool name before the start of the query. - Do not attempt to directly answer the user's question. - Do not provide any other output besides the search query. # TOOL 1 Name: Glean Search # TOOL DESCRIPTION You have access to Glean Search, which finds relevant documents in the company. INSTRUCTIONS: - This is your primary tool to access all knowledge within the company. - The results returned are not exhaustive, we only return the top few most relevant documents to a query. - For analytics questions such as "how many documents..." use the "statistics" field in the output. # TOOL INPUT SPECIFICATIONS - Search query should contain important keywords that help find relevant documents. - You can optionally narrow down the search results by using one or more of the following filter types: 1. owner: filter to documents created by this person. Value can be a person's name (double quoted), or 'me' 2. from: filter to documents updated by or commented on or created by this person. Value can be a person's name (double quoted), or 'me' 3. updated: filter to documents updated on or after this date, value can be one of ["today", "yesterday", "past_week", "past_2_weeks", "past_month", "March", ...] 4. after: filter to documents created after this date. Do not use a date later than today. value must be in "YYYY-MM-DD" format 5. before: filter to documents created before this date. value must be in "YYYY-MM-DD" format 6. channel: filter to documents in a specific channel. Only use this when the user explicitly wants contents from a channel. 7. app: filter to documents created in a specific app. value can be one of ["announcements", "answers", "asana", "collections", "customer", "developers", "figma", "github", "glean support zendesk", "glean website", "gong", "help.glean", "jira", "orgteams", "salescloud", "slack", "zendesk"]. Value must be double quoted. Only use the app filter when the user wants contents from an app. For reference, "customer" is an app that stores Customer Cards, which can include information like overview of the company, company size, website, sales opportunities, accounts, contacts, revenue, etc; When the demonstrating examples use "customer" a majority of the time, make sure to use "customer" as well; "customer" is also best used in conjunction with "salescloud", "gong" is an app that stores transcripts of sales calls and meetings with customers and prospects, "orgteams" is an app that stores Team Entities, which represent a team of people within the company working together. Teams include info such as team leads and team size, "salescloud" is an app that stores customer and prospect (leads, contacts, accounts) data, sales data (dashboards, campaigns, opportunities), and interaction history from Salesforce. - Use date filters like "updated", "after", and "before" ONLY when the user has mentioned a specific time frame (e.g. last week, past month, before 12 Jan). Do NOT use date filters if the user requests the "latest" or "recent" documents or information but hasn't provided a specific time frame.- Do NOT use any other filter besides those explicitly listed above. # TOOL OUTPUT SPECIFICATIONS - The output from Glean Search is a JSON struct with 2 fields: 1. "documents": a list of documents with Document struct. Each Document struct contains these fields: title, snippets, owner, updateTime, url. 2. "statistics": counts of documents that match your query, broken down by different dimensions such as datasource, author, reviewer. # TOOL 2 Name: Glean HR Assistant # TOOL DESCRIPTION You have access to Glean HR Assistant INSTRUCTIONS: Automatically route to this AI app when a user asks about HR and administrative questions including company culture, benefits, professional development, 401K, PTO, medical and insurance questions, company holidays. Examples phrases: What is our company holidays? What are our parental benefits? What is the timeline for performance reviews? How do I request PTO? # TOOL INPUT SPECIFICATIONS Exact same as Glean Search Tool. # TOOL OUTPUT SPECIFICATIONS Exact same as Glean Search Tool. # TOOL 3 Name: Workspace Bot # TOOL DESCRIPTION You have access to Workspace Bot INSTRUCTIONS: Trigger this AI app to answer workspace-related questions. This could include questions about debugging common errors, providing general information, or help with onboarding. This should trigger whenever any of these functions are mentioned: - Datasource Apps Setup - User Permissioning - People Data Setup - Authentication Setup (SSO & Magic Links) - Central Workspace Setup (CWS) - Owner Key Upload - Data Loss Prevention (DLP) - Admin Alerts - Search Access Visibility For any other requests that don't fit those areas, use the Expert Search tool to identify an appropriate team or individual that may be better suited to answer the question. # TOOL INPUT SPECIFICATIONS Exact same as Glean Search Tool. # TOOL OUTPUT SPECIFICATIONS Exact same as Glean Search Tool. # DEMONSTRATING EXAMPLES Message: How do I reset a crawl? Output: Glean Search [reset crawl, crawl restart] Message: how do I file a bug? Output: Glean Search [bug filing process, how to file a bug] Message: how do we index documents Output: Glean Search [document indexing, indexing process] Message: How do I fill out performance reviews Output: Glean Search [fill out performance reviews, performance review process] Message: how do i deploy ToolX? Output: Glean Search [ToolX deployment, ToolX installation] Message: How do I run toolX? Output: Glean Search [toolX, run toolX] Message: How do we typically pay VendorX? Output: Glean Search [VendorX payment, VendorX invoice] Message: What happened with ProjectX in the last month? Output: Glean Search [ProjectX updated:past_month] Message: who is the owner for the ProjectA? Output: Glean Search [ProjectA] Message: who works on ConceptX? Output: Glean Search [ConceptX] Message: who is JohnDoe? Output: Glean Search [JohnDoe] Message: What is the latest with ProjectX? Output: Glean Search [ProjectX] Message: Analyze and compare how our thinking about ProjectX has changed since last year till now Output: Glean Search [ProjectX after:2023-07-10, ProjectX after:2022-07-10 before:2023-07-10] Message: Write an executive email as to why ProductA is better suited for CompanyB than using ProductB. Use any context from meetings in the past month Output: Glean Search [ProductA vs ProductB, CompanyB meeting updated:past_month] Message: Read through our ProductX features. Give me a list of potential enhancements we can make to the feature. You can get inspiration from our ChannelY slack channel. If you get an idea from slack, cite it as not your own Output: Glean Search [ProductX feature enhancements, channel:ChannelY app:"slack"] # BEGIN PROGRAM Conversation history (latest message last): - User: "How do I book time off?" \`\`\` PLAN_RESPONSE \`\`\` analysis: "" readactions: - tool_name: Glean HR Assistant action: Glean HR Assistant: book time off - tool_name: Glean HR Assistant action: Glean HR Assistant: request PTO writeaction: tool_name: "" action: {} instruction: "" \`\`\` PLAN_PROMPT_TOKEN \`\`\` 3354 \`\`\` PLAN_RESPONSE_TOKEN \`\`\` 17 \`\`\` RAW_LLM_PLAN_RESPONSE \`\`\` [LLM Output for : Employee Search + Expert Search] NA ================================================== [LLM Output for : categories] Output: [] ================================================== [LLM Output for : Glean Search + tencctxm5dukr8hc + mlaznw1lpodrp7zk] Glean HR Assistant [book time off, request PTO] \`\`\` PLAN_TTF \`\`\` 1407 \`\`\` PLAN_STAGE_LATENCY \`\`\` 1313 \`\`\` ACTION_SUMMARY \`\`\` **Searching:** "book time off" in the [all] tab(s) using Glean HR Assistant Tool Token limit for this action is : 1295 Num Results requested for this action is : 3 Execution time : 539 ms ================================================== **Searching:** "request PTO" in the [all] tab(s) using Glean HR Assistant Tool Token limit for this action is : 1295 Num Results requested for this action is : 3 Execution time : 592 ms ================================================== **Searching:** "How do I book time off?" in the [all] tab(s) using Glean Search Tool Token limit for this action is : 500 Num Results requested for this action is : 1 Execution time : 811 ms ================================================== **Instruction:** Token limit for this action is : 0 Num Results requested for this action is : 0 Execution time : 0 ms ================================================== \`\`\` ACT_PROMPT \`\`\` Today's date is 2024-Jul-09. You are a helpful assistant for an employee of Glean Work (Email: dse@askscio.com) who works in scio::entire_organization.Here are additional instructions you should follow: If you have additional questions on benefits, please reach out to benefits@glean.com. For other HR questions, feel free to email hr@glean.com.. You are an intelligent workplace assistant, create a truthful response to the user's last message in the conversation by carefully analyzing the context and instructions. Conversation history (latest message last): - User: "How do I book time off?" ## CONTEXT Top results for "request PTO" in Glean HR Assistant Tool: {"documents":[{"title":"How do I request PTO?","datasource":"answers","url":"/knowledge/answers/1851","snippets":["As of August 1, 2022, we will be using Gusto for all PTO requests. Please submit all PTO or sick leaves through Gusto moving forward for your manager's approval. Please continue to update your calendar invite and slack for visibility across the company."],"ownedAndUpdatedBy":{"name":"Jessica Yeh","pronoun":"None"},"updateTime":"2022-11-08"}],"statistics":[{"all":1},{"datasource=answers":1},{"from=jessica@glean.com":1}]} Top results for "How do I book time off?" in Glean Search Tool: {"entities":[{"extractedAnswer":{"question:omitempty":"How do I take time off?","answer":"When taking time off or working from home outside of the normal schedule, do all of the following:\n* Obtain approval from your manager reasonably in advance of proposed vacation (prior to making arrangements) by requesting it in [gusto](https://app.gusto.com/scio-technologies-inc-400/employee/time_off) for US employees and [greyhr](https://glean.greythr.com) for India employees.\n* Notify your immediate working team after receiving approval. Plan to delegate, postpone or otherwise manage projects that will be affected by your time off.\n* Add an Out Of Office event to your individual Glean calendar.\n* Set a Slack status for better visibility.\n* If you receive external emails, set an Out Of Office responder explaining return date and who to contact in your absence.\n\nhttps://docs.google.com/document/d/1qt1Qbt-y4w1UbIU5BQxUpfZlwkx0gcKSGlKd_VqUk18/edit#bookmark=id.6wzx88skrwrf","answerAuthor":{"name":"Tony Gentilcore","pronoun":"None"}}}]} ## INSTRUCTIONS The search results may contain documents that are not relevant to the user's query (for example, document titles or snippets could be about a different topic), you should ignore those results. Think step-by-step. First, determine the set of documents and snippets in the context that are relevant to the user's query. Then, synthesize them to create a detailed truthful response. If none of the provided context are relevant to the user's question, explain that to the user instead of making up an answer. Respond in the same language as user's latest message or query (but don't use their name to determine the language), or in English. Adopt 'they/them' pronouns by default - with one exception: if a 'pronoun' field is specified in a person's JSON summary, use its value to refer to the individual. Be extra meticulous about pronouns, it is very important to get this right, especially if one has a specified pronoun. Be helpful to the user, but avoid workplace violations. Avoid salutations and sign-offs in responses. Great job, let's start! Truthful Response: \`\`\` ACT_PROMPT_TOKEN \`\`\` 791 \`\`\` ACT_RESPONSE_TOKEN \`\`\` 228 \`\`\` FOLLOW_UP_DEBUG_INFO \`\`\` FOLLOWUP PROMPT: \`\`\` Today's date is 2024-Jul-09. You are a helpful assistant for an employee of Glean Work (Email: dse@askscio.com) who works in scio::entire_organization. Conversation history (latest message last): - User: "How do I book time off?" The response to the user is: "**Using app: Glean HR Assistant** To book time off, please follow these steps: 1. **Submit Your Request**: Use Gusto to submit your PTO or sick leave request for your manager's approval. You can access Gusto [here](https://app.gusto.com/scio-technologies-inc-400/employee/time_off) if you are a US employee. For India employees, use [greyhr](https://glean.greythr.com). 2. **Notify Your Team**: After receiving approval, notify your immediate working team. Make sure to plan for delegating, postponing, or managing projects that will be affected by your time off. 3. **Update Your Calendar**: Add an Out Of Office event to your individual Glean calendar. 4. **Set Slack Status**: Update your Slack status for better visibility across the company. 5. **Set Email Responder**: If you receive external emails, set an Out Of Office responder explaining your return date and who to contact in your absence. If you have any further questions on benefits, please reach out to benefits@glean.com. For other HR questions, feel free to email hr@glean.com.". Generate 3 concise follow-up questions that the user might want to ask. Respond in the same language as user's latest message or query (but don't use their name to determine the language), or in English. Do not add any preamble or prefix before the first question, just start writing out the first question in your output. Use this format: - Question 1 - Question 2 - Question 3 Great job, let's start! Follow-up Questions: - \`\`\` FOLLOWUP RAW RESPONSE: \`\`\` How do I check the status of my PTO request? - Can I cancel a PTO request after it has been approved? - How many days of PTO am I allowed to take per year? \`\`\` FOLLOWUP OUTPUT: How do I check the status of my PTO request? Can I cancel a PTO request after it has been approved? How many days of PTO am I allowed to take per year? \`\`\` ` const r = /^(.*?)\n\n```\n([\s\S]*?)\n```$/gm const n = []; const matches = [...s.matchAll(r)]; matches.forEach((match) => { // do something // n.push(match.index); });
regexp.exec
const s = ` AGENT_INFO \`\`\` Using ReAct Tools 2 Hops workflow, with: tools_filter: "Name: azure-turbo, TokenLimit: 4000, AllowedClient: AZURE_OPENAI_CHAT" plan: "Name: azure-gpt4o-2024-05-13-8k, TokenLimit: 8000, AllowedClient: AZURE_OPENAI_CHAT" act: "Name: azure-gpt4o-2024-05-13-8k, TokenLimit: 8000, AllowedClient: AZURE_OPENAI_CHAT" followup: "Name: azure-turbo, TokenLimit: 4000, AllowedClient: AZURE_OPENAI_CHAT" \`\`\` TOOLS_FILTER_PROMPT \`\`\` You have access to the following Read Tools: Glean Search: This tool finds relevant documents in the company. This is your primary tool to access all knowledge within the company. Expert Search: This tool finds people at the company based on topics they wrote about and projects they own. Use this tool to find people who have worked on particular projects. Only use this when the user explicitly wants to find people in the company (eg. "who" questions). Employee Search: This tool finds people at the company based on their personal information such as name, email, title, department, tenure and location. Only use this when the user explicitly wants to find people in the company (eg. "who" questions). Workspace Bot: Trigger this AI app to answer workspace-related questions. This could include questions about debugging common errors, providing general information, or help with onboarding. This should trigger whenever any of these functions are mentioned: - Datasource Apps Setup - User Permissioning - People Data Setup - Authentication Setup (SSO & Magic Links) - Central Workspace Setup (CWS) - Owner Key Upload - Data Loss Prevention (DLP) - Admin Alerts - Search Access Visibility For any other requests that don't fit those areas, use the Expert Search tool to identify an appropriate team or individual that may be better suited to answer the question. Who is?: Automatically trigger this app when someone asks about who the Account Executive/AE/Sales rep or Customer Success Manager/CSM is for a customer or prospect. They'll use the specific customer name in looking for the answer. Example questions: - Who is the CSM for... - Who is the customer success manager for... - Who is the AE for... - Who is the Account Exec for... Glean HR Assistant: Automatically route to this AI app when a user asks about HR and administrative questions including company culture, benefits, professional development, 401K, PTO, medical and insurance questions, company holidays. Examples phrases: What is our company holidays? What are our parental benefits? What is the timeline for performance reviews? How do I request PTO? Competitive Intelligence Agent: Use this whenever the user wants to compare Glean with competitors. This app was built as a competitive intelligence assistant specializing in analyzing competitors to Glean in the Search and generative AI space. Use this app if a user's question focusses on comparing products to Glean. Also use when asked to highlight similarities, differences, advantages and disadvantages of Glean over other applications. Example Phrases - Compare Glean vs Acme - How is Glean differentiated against Acme? - What are Glean's strengths and weaknesses compared to another solution - What is the reason to buy Glean over building enterprise search or assistant in house API Overview buddy: Trigger this app when anyone asks for an overview of an API You have access to the following Write Tools: cherrypicknew: Trigger when the user asks to create a CP or a cherrypick. Find the right pull request using the search tool corresponding the what the user has asked - always start by using the search tool and getting the relevant pull request. The most important rule is to only infer commit_hash, pull_request_url, components, release, scio_instances and reason_for_cherrypick. Locate the commit hash from the COMMIT_HASH field within the PR details. Note that it is very important to populate the correct commit_hash! VivekITTicketDemo: Trigger when the user asks "create a ticket" in relation to an IT request. Do not trigger if the user doesn't ask to create or file a ticket. Examples of when NOT to trigger: - I cannot log-in to gmail. What do I do? - What's the process to request a loaner laptop? - I want a new laptop Append the following to the end of the ticket title: [FIRSTNAME LASTNAME] Include the user's email address and role in the ticket description. ResetCrawlType: This tool allows you to run the reset crawl spinnaker pipeline to (re-) crawl all documents for a datasource on given deployment(s) for a given crawl type (which can be one of CONTENT or IDENTITY or ALL). Given a datasource, crawl type (optional) and deployment(s) to run on, a reset crawl pipeline run would be triggered. Invoke it for any request that needs (re)-crawling (or reset) of documents of a datasource on deployment(s) with a particular crawl type. Example queries: - Recrawl salesforce documents on scio-prod - Reset identity crawl for o365 on koch - Restart content crawl for box on glean-dev Do not trigger the tool if we do not know both datasource_instance and scio_instance (ie. which deployment to start crawls in). DisplayTool: Simply displays a response to the user. - Use this tool when the user simply wants to read the response. - This is your primary tool to have a conversation with the user. CreateJiraIssueExecution: Helps you create a Jira issue or ticket. Use this action when the user wants to create a ticket/issue: bug/task/epic. AdhocScannerTool: This tool allows you to run the adhoc scanner spinnaker pipeline to (re-) process all documents for a datasource on given deployment(s). Given a datasource and deployment(s) to run on, an adhoc scanner run would be triggered. Invoke it for any request that needs (re)-processing of documents on a datasource on deployment(s). For example "To process documents of salesforce in scio-prod" or any other similar request directly implies usage of this tool. For example "I want to process documents for BOX on ashisuto and ashisuto-sandbox" or any other similar request directly implies usage of this tool. For example "Process documents for gdrive on yahoo" or any other similar request directly implies usage of this tool. Some additional consideration that are REALLY IMPORTANT: - invoke the tool IF AND ONLY IF you know both: the scio_instance(s) and the datasource. DO NOT TRIGGER THE TOOL IF THERE IS NO CLEAR INDICATION OF BOTH OF THEM!! (For example: Do not trigger for queries like "process all documents", Do not trigger for "process all salesforce documents", Do not trigger for "process all documents on scio-prod") - invoke the tool IF AND ONLY IF there is a clear indication on an expected action. DO NOT TRIGGER THE TOOL IF THERE IS NO CLEAR INDICATION OF AN EXPECTED ACTION!! - DO NOT invoke the tool for questions that don't have a clear indication towards an expected action. (For example: Do not trigger for queries like "How can I process documents?", Do not trigger for "What is the adhoc scanner?", Do not trigger for "How are documents processed?", Do not trigger for ?how many docs can the adhoc scanner handle?) - Be sure to invoke only when needed. Under-invocation and over-invocation of this tool, are both UNACCEPTABLE. This is very critical to the company. Something really dire will happen if you do not follow the above rules. You will be tipped if you follow them strictly. As an advanced reasoning agent, analyze the above tools and the conversation history to select the right tools that can be used to respond to user's latest message. Do not give reason for your response. Respond with a YAML struct in this format: read_tool_names: # List containing at least 3 and at most 4 read tools that might be useful for this query. Must be a subset of [Glean Search, Expert Search, Employee Search, Workspace Bot, Who is?, Glean HR Assistant, Competitive Intelligence Agent, API Overview buddy]. Keep the most important tool first. - $tool_name write_tool_names: # List containing at least 2 write tools that might be useful for this query. Must be a subset of [cherrypicknew, VivekITTicketDemo, ResetCrawlType, DisplayTool, CreateJiraIssueExecution, AdhocScannerTool]. Keep the most important tool first. - $tool_name DEMONSTRATING EXAMPLES: message: File a jira ticket to fix code search links in gleanchat read_tool_names: - Expert Search - Glean Search write_tool_names: - createjiraissueexecution message: File a jira ticket to track person suggestion issues in the BE QA component of the QA project read_tool_names: - Expert Search - Glean Search write_tool_names: - createjiraissueexecution message: User: What is a recent task mentioned in #engineering? Who is the owner? Assistant: The recent task is about updating oncall rotation. John plans to create a list of secondary oncallers and add them to the rotation. The owner is John. User: File a jira with issueType story and priority high. read_tool_names: - Employee Search - Glean Search write_tool_names: - createjiraissueexecution message: How do we typically pay VendorX? read_tool_names: - Glean Search write_tool_names: - displaytool message: How do I run toolX? read_tool_names: - Glean Search write_tool_names: - displaytool message: how do i deploy ToolX? read_tool_names: - Glean Search write_tool_names: - displaytool message: How do I fill out performance reviews read_tool_names: - Glean Search write_tool_names: - displaytool message: how do we index documents read_tool_names: - Glean Search write_tool_names: - displaytool message: how do I file a bug? read_tool_names: - Glean Search write_tool_names: - displaytool message: How do I reset a crawl? read_tool_names: - Glean Search write_tool_names: - displaytool Great job, let's start! Conversation history (latest message last): - User: "How do I book time off?" YAML Output: read_tool_names: \`\`\` TOOLS_FILTER_PROMPT_TOKEN \`\`\` 2112 \`\`\` TOOLS_FILTER_RESPONSE \`\`\` read_tool_names: - Glean HR Assistant - Employee Search - Workspace Bot write_tool_names: - DisplayTool \`\`\` TOOLS_FILTER_RESPONSE_TOKEN \`\`\` 21 \`\`\` RAP_EXAMPLES \`\`\` message: Analyze and compare how our thinking about ProjectX has changed since last year till now analysis: User wants to analyze and compare the thinking about a specific component from last year to now. categories: [] instruction: Compare the search results from both time periods to identify changes in thinking, strategies, or approaches related to ProjectX. read_actions: - tool_name: Glean Search action: Glean Search: ProjectX after:2023-07-10 - tool_name: Glean Search action: Glean Search: ProjectX after:2022-07-10 before:2023-07-10 write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: Compare Foo and Bar analysis: Compare documents related to Foo and Bar to find similarities and differences. categories: [] instruction: Analyze the search result pages for Foo and Bar, and find similarities and differences between them. Synthesize information and provide the analysis to user. read_actions: - tool_name: Glean Search action: Glean Search: Foo - tool_name: Glean Search action: Glean Search: Bar - tool_name: Glean Search action: Glean Search: Foo vs Bar write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: Read through our ProductX features. Give me a list of potential enhancements we can make to the feature. You can get inspiration from our ChannelY slack channel. If you get an idea from slack, cite it as not your own analysis: Search for ProductX features, and search for messages in the ChannelY channel, and suggest list of potential feature enhancements categories: [] instruction: Review search results and compile a list of potential enhancements for ProductX features. If an idea is from the ChannelY channel, cite the source. read_actions: - tool_name: Glean Search action: Glean Search: ProductX feature enhancements - tool_name: Glean Search action: Glean Search: channel:ChannelY app:"slack" write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: Write an executive email as to why ProductA is better suited for CompanyB than using ProductB. Use any context from meetings in the past month analysis: Craft an executive email explaining why ProductA is better suited for CompanyB than ProductB, using context from meetings in the past month. categories: [] instruction: Use search results to write an executive email highlighting the advantages of ProductA over ProductB for CompanyB, incorporating context from meetings in the last month. read_actions: - tool_name: Glean Search action: Glean Search: ProductA vs ProductB - tool_name: Glean Search action: Glean Search: CompanyB meeting updated:past_month write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: What happened with ProjectX in the last month? analysis: User wants to know what happened with ProjectX in the last month. categories: - updates - recent instruction: Analyze the results and share updates on ProjectX. read_actions: - tool_name: Glean Search action: Glean Search: ProjectX updated:past_month write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: What is the latest with ProjectX? analysis: User wants to know what is the latest with ProjectX. Do NOT use the updated filter as the user did not provide a specific time frame. categories: - updates - recent instruction: Analyze the results and share updates on ProjectX. read_actions: - tool_name: Glean Search action: Glean Search: ProjectX write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: how many engineers do we have? analysis: Use the Employee Search tool to find the number of people with the title "engineer". categories: [] instruction: Check the "statistics" field in the output to find the total number of engineers. read_actions: - tool_name: Employee Search action: Employee Search: engineer write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: who is JohnDoe? analysis: Use Employee Search to locate JohnDoe's information. categories: [] instruction: Locate JohnDoe in search results and provide their title, email, location, reporting chain, and start date. read_actions: - tool_name: Employee Search action: Employee Search: JohnDoe - tool_name: Glean Search action: Glean Search: JohnDoe write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: who is the owner for the ProjectA? analysis: Find documents related to ProjectA and the people who wrote them. Synthesize this information to determine the owner of ProjectA. categories: [] instruction: Look for documents about ProjectA and find the people who wrote those documents. Look for people results in Expert Search. Synthesize the information from both sources to determine the owner of ProjectA. read_actions: - tool_name: Glean Search action: Glean Search: ProjectA - tool_name: Expert Search action: Expert Search: ProjectA write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: who works on ConceptX? analysis: Find people involved in ConceptX using Expert Search and relevant documents using Glean Search. categories: [] instruction: Find people involved in ConceptX from search results and provide their names, titles, and relevant information. read_actions: - tool_name: Glean Search action: Glean Search: ConceptX - tool_name: Expert Search action: Expert Search: ConceptX write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: -1.000000 message: How do we typically pay VendorX? analysis: Use Glean Search to find documents related to VendorX's payment process. categories: [] instruction: Look for documents that mention the payment method or process for VendorX and provide a summary of the information found. read_actions: - tool_name: Glean Search action: Glean Search: VendorX payment - tool_name: Glean Search action: Glean Search: VendorX invoice write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: 0.812568 message: How do I run toolX? analysis: User wants to find code snippets and documents to help them run toolX. categories: - code instruction: Find code snippets or documents to help user run toolX. read_actions: - tool_name: Glean Search action: Glean Search: toolX - tool_name: Glean Search action: Glean Search: run toolX write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: 0.816166 message: how do i deploy ToolX? analysis: Use Glean Search to find relevant documents on ToolX deployment and installation to provide information on how to deploy ToolX to the user. categories: [] instruction: Search for ToolX deployment and installation instructions and provide the user with clear and up-to-date information. read_actions: - tool_name: Glean Search action: Glean Search: ToolX deployment - tool_name: Glean Search action: Glean Search: ToolX installation write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: 0.822656 message: How do I fill out performance reviews analysis: Use Glean Search to find relevant documents on the performance review process for the user asking for information on how to fill out performance reviews. categories: [] instruction: Search for performance review process documents and provide relevant steps or guidelines for filling out performance reviews. read_actions: - tool_name: Glean Search action: Glean Search: fill out performance reviews - tool_name: Glean Search action: Glean Search: performance review process write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: 0.823461 message: how do we index documents analysis: Search for relevant documents and information on document indexing. Summarize to provide a helpful response to the user's question. categories: [] instruction: Review the returned documents and provide the user with a summary of the document indexing process in Acme. read_actions: - tool_name: Glean Search action: Glean Search: document indexing - tool_name: Glean Search action: Glean Search: indexing process write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: 0.830317 message: how do I file a bug? analysis: Use Glean Search to find bug filing resources and provide relevant information and tools to the user. categories: [] instruction: Find bug filing resources and provide relevant information and tools to user. read_actions: - tool_name: Glean Search action: Glean Search: bug filing process - tool_name: Glean Search action: Glean Search: how to file a bug write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: 0.838035 message: How do I reset a crawl? analysis: Search for relevant information about resetting crawl categories: [] instruction: Search for documents with instructions on resetting or restarting a crawl and provide step-by-step instructions to the user. read_actions: - tool_name: Glean Search action: Glean Search: reset crawl - tool_name: Glean Search action: Glean Search: crawl restart write_analysis: Find answer to user message and display to the user. write_action: tool_name: displaytool action: response: Placeholder() actions: [] score: 0.839287 \`\`\` PLAN_PROMPT \`\`\` [LLM Input Prompt for : Employee Search + Expert Search] Today's date is 2024-Jul-09. You are a helpful assistant for an employee of Glean Work (Email: dse@askscio.com) who works in scio::entire_organization. # GENERAL INSTRUCTIONS - Carefully analyze the user's query to determine if it falls within the scope of the tool's capabilities. If the query is clearly outside the tool's intended use case, output NA. - Use only one of the tools provided. Output tool name before the start of the query. - If the query seems potentially relevant, assess whether the necessary information to answer the query is likely to be available based on the tool's input specifications and data sources. If crucial information is missing or unlikely to be found, output NA. - If the query passes both relevance and feasibility checks, construct a well-formatted input string for the tool based on the provided input specifications. - If at any point you determine that the tool is not suitable for answering the query, or if you are uncertain about its applicability, output NA. - Do not attempt to directly answer the user's question. - Do not provide any other output besides the search query or NA. # TOOL 1 Name: Employee Search # TOOL DESCRIPTION You have access to Employee Search, which finds people at the company based on their personal information. INSTRUCTIONS: - Only use this when the user explicitly wants to find people in the company (eg. "who" questions) or for aggregation queries on people. - Do not use this when the user wants to find people outside of the company, or people who are no longer at the company. - You can find people based on their personal information such as name, email, title, department, and location. - The results returned are not exhaustive, we only return the top few most relevant people to a query. - For analytics questions such as "how many people..." use the "statistics" field in the output. # TOOL INPUT SPECIFICATIONS - You can optionally narrow down the search results by using one or more of the following filter types: 1. startafter: filter to people who started after a certain date. value must be in "YYYY-MM-DD" format 2. startbefore: filter to people who started before a certain date. value must be in "YYYY-MM-DD" format 3. roletype: filter to people who have a certain role type. value can be one of ["individual contributor", "manager"] 4. reportsto: Filter to people who are direct reports of a given manager. Value (double quoted) can be a full name, first name, last name, or email. - Do NOT use any other filter besides those explicitly listed above. - You can optionally sort the results by adding one of the following to the query: ["sortby:hire_date_ascending", "sortby:hire_date_descending", "sortby:most_reports"]. # TOOL OUTPUT SPECIFICATIONS - The output from Employee Search is a JSON struct with 2 fields: 1. "entities": a list of Person structs. Each Person struct contains these fields: title,businessUnit,department,teams,email,location,externalProfileLink,manager,startDate,bio,directReportsCount,totalReportsCount,preferredName,directReports The 'manager' and 'directReports' fields are recursive Person structs with all information of the person's manager and direct reports. 2. "statistics": the total count of people that match your query, as well as aggregate counts by department, location, title, team, and roletype. # TOOL 2 Name: Expert Search # TOOL DESCRIPTION You have access to Expert Search, which finds people at the company based on topics they wrote about and projects they own. INSTRUCTIONS: - Only use this when the user explicitly wants to find people in the company (eg. "who" questions). - Do not use this when the user wants to find people outside of the company, or people who are no longer at the company. - You can find people based on what they worked on, or wrote about by using short key words about that topic as input. - The results returned are not exhaustive, we only return the top few most relevant people to a query. - For analytics questions such as "how many people..." use the "statistics" field in the output. # TOOL INPUT SPECIFICATIONS - The query should contain important keywords that help find people based on what they worked on or wrote about. DO NOT use any filters. # TOOL OUTPUT SPECIFICATIONS - The output from Expert Search is a JSON struct with 2 fields: 1. "entities": a list of Person structs. Each Person struct contains these fields: title,businessUnit,department,teams,email,location,externalProfileLink,manager,startDate,bio,directReportsCount,totalReportsCount,preferredName,directReports The 'manager' and 'directReports' fields are recursive Person structs with all information of the person's manager and direct reports. 2. "statistics": counts of people that match your query. # DEMONSTRATING EXAMPLES Message: who is the owner for the ProjectA? Output: Expert Search [ProjectA] Message: who works on ConceptX? Output: Expert Search [ConceptX] Message: who is JohnDoe? Output: Employee Search [JohnDoe] Message: how many engineers do we have? Output: Employee Search [engineer] Message: How do I reset a crawl? Output: NA Message: how do I file a bug? Output: NA Message: how do we index documents Output: NA Message: How do I fill out performance reviews Output: NA # BEGIN PROGRAM Conversation history (latest message last): - User: "How do I book time off?" ================================================== [LLM Input Prompt for : categories] Today's date is 2024-Jul-09. You are a helpful assistant for an employee of Glean Work (Email: dse@askscio.com) who works in scio::entire_organization. Classify the user message's intent into one or more of the following categories. Select all relevant categories for a specific query. If you think a category feels relevant but you're not 100 percent sure, include the category anyways because it might help. CATEGORIES: 'updates': Use this category when the user is asking about updates/status/latest on a project/customer/topic. 'enumerate': Use this category when the user is asking for information that is answerable by small pieces of information in many documents, as opposed to larger pieces of content from fewer results. Don't use this if the user is asking to look at only one document 'recent': Use this category when the user is asking for the latest or most recent document or piece of information. 'code': Use this category when the user is asking for information related to code, such as code snippets, code examples, or code documentation. Examples: Message: Analyze and compare how our thinking about ProjectX has changed since last year till now Output: [] Message: Compare Foo and Bar Output: [] Message: Read through our ProductX features. Give me a list of potential enhancements we can make to the feature. You can get inspiration from our ChannelY slack channel. If you get an idea from slack, cite it as not your own Output: [] Message: Write an executive email as to why ProductA is better suited for CompanyB than using ProductB. Use any context from meetings in the past month Output: [] Message: What happened with ProjectX in the last month? Output: ["updates","recent"] Message: What is the latest with ProjectX? Output: ["updates","recent"] Message: how many engineers do we have? Output: [] Message: who is JohnDoe? Output: [] Message: who is the owner for the ProjectA? Output: [] Message: who works on ConceptX? Output: [] Message: How do we typically pay VendorX? Output: [] Message: How do I run toolX? Output: ["code"] Message: how do i deploy ToolX? Output: [] Message: How do I fill out performance reviews Output: [] Message: how do we index documents Output: [] Message: how do I file a bug? Output: [] Message: How do I reset a crawl? Output: [] User Message: How do I book time off? ================================================== [LLM Input Prompt for : Glean Search + tencctxm5dukr8hc + mlaznw1lpodrp7zk] Today's date is 2024-Jul-09. You are a helpful assistant for an employee of Glean Work (Email: dse@askscio.com) who works in scio::entire_organization. # GENERAL INSTRUCTIONS - Carefully analyze the user's query and construct a well-formatted input string for the tool based on the provided input specifications. - Use only one of the tools provided. Output tool name before the start of the query. - Do not attempt to directly answer the user's question. - Do not provide any other output besides the search query. # TOOL 1 Name: Glean Search # TOOL DESCRIPTION You have access to Glean Search, which finds relevant documents in the company. INSTRUCTIONS: - This is your primary tool to access all knowledge within the company. - The results returned are not exhaustive, we only return the top few most relevant documents to a query. - For analytics questions such as "how many documents..." use the "statistics" field in the output. # TOOL INPUT SPECIFICATIONS - Search query should contain important keywords that help find relevant documents. - You can optionally narrow down the search results by using one or more of the following filter types: 1. owner: filter to documents created by this person. Value can be a person's name (double quoted), or 'me' 2. from: filter to documents updated by or commented on or created by this person. Value can be a person's name (double quoted), or 'me' 3. updated: filter to documents updated on or after this date, value can be one of ["today", "yesterday", "past_week", "past_2_weeks", "past_month", "March", ...] 4. after: filter to documents created after this date. Do not use a date later than today. value must be in "YYYY-MM-DD" format 5. before: filter to documents created before this date. value must be in "YYYY-MM-DD" format 6. channel: filter to documents in a specific channel. Only use this when the user explicitly wants contents from a channel. 7. app: filter to documents created in a specific app. value can be one of ["announcements", "answers", "asana", "collections", "customer", "developers", "figma", "github", "glean support zendesk", "glean website", "gong", "help.glean", "jira", "orgteams", "salescloud", "slack", "zendesk"]. Value must be double quoted. Only use the app filter when the user wants contents from an app. For reference, "customer" is an app that stores Customer Cards, which can include information like overview of the company, company size, website, sales opportunities, accounts, contacts, revenue, etc; When the demonstrating examples use "customer" a majority of the time, make sure to use "customer" as well; "customer" is also best used in conjunction with "salescloud", "gong" is an app that stores transcripts of sales calls and meetings with customers and prospects, "orgteams" is an app that stores Team Entities, which represent a team of people within the company working together. Teams include info such as team leads and team size, "salescloud" is an app that stores customer and prospect (leads, contacts, accounts) data, sales data (dashboards, campaigns, opportunities), and interaction history from Salesforce. - Use date filters like "updated", "after", and "before" ONLY when the user has mentioned a specific time frame (e.g. last week, past month, before 12 Jan). Do NOT use date filters if the user requests the "latest" or "recent" documents or information but hasn't provided a specific time frame.- Do NOT use any other filter besides those explicitly listed above. # TOOL OUTPUT SPECIFICATIONS - The output from Glean Search is a JSON struct with 2 fields: 1. "documents": a list of documents with Document struct. Each Document struct contains these fields: title, snippets, owner, updateTime, url. 2. "statistics": counts of documents that match your query, broken down by different dimensions such as datasource, author, reviewer. # TOOL 2 Name: Glean HR Assistant # TOOL DESCRIPTION You have access to Glean HR Assistant INSTRUCTIONS: Automatically route to this AI app when a user asks about HR and administrative questions including company culture, benefits, professional development, 401K, PTO, medical and insurance questions, company holidays. Examples phrases: What is our company holidays? What are our parental benefits? What is the timeline for performance reviews? How do I request PTO? # TOOL INPUT SPECIFICATIONS Exact same as Glean Search Tool. # TOOL OUTPUT SPECIFICATIONS Exact same as Glean Search Tool. # TOOL 3 Name: Workspace Bot # TOOL DESCRIPTION You have access to Workspace Bot INSTRUCTIONS: Trigger this AI app to answer workspace-related questions. This could include questions about debugging common errors, providing general information, or help with onboarding. This should trigger whenever any of these functions are mentioned: - Datasource Apps Setup - User Permissioning - People Data Setup - Authentication Setup (SSO & Magic Links) - Central Workspace Setup (CWS) - Owner Key Upload - Data Loss Prevention (DLP) - Admin Alerts - Search Access Visibility For any other requests that don't fit those areas, use the Expert Search tool to identify an appropriate team or individual that may be better suited to answer the question. # TOOL INPUT SPECIFICATIONS Exact same as Glean Search Tool. # TOOL OUTPUT SPECIFICATIONS Exact same as Glean Search Tool. # DEMONSTRATING EXAMPLES Message: How do I reset a crawl? Output: Glean Search [reset crawl, crawl restart] Message: how do I file a bug? Output: Glean Search [bug filing process, how to file a bug] Message: how do we index documents Output: Glean Search [document indexing, indexing process] Message: How do I fill out performance reviews Output: Glean Search [fill out performance reviews, performance review process] Message: how do i deploy ToolX? Output: Glean Search [ToolX deployment, ToolX installation] Message: How do I run toolX? Output: Glean Search [toolX, run toolX] Message: How do we typically pay VendorX? Output: Glean Search [VendorX payment, VendorX invoice] Message: What happened with ProjectX in the last month? Output: Glean Search [ProjectX updated:past_month] Message: who is the owner for the ProjectA? Output: Glean Search [ProjectA] Message: who works on ConceptX? Output: Glean Search [ConceptX] Message: who is JohnDoe? Output: Glean Search [JohnDoe] Message: What is the latest with ProjectX? Output: Glean Search [ProjectX] Message: Analyze and compare how our thinking about ProjectX has changed since last year till now Output: Glean Search [ProjectX after:2023-07-10, ProjectX after:2022-07-10 before:2023-07-10] Message: Write an executive email as to why ProductA is better suited for CompanyB than using ProductB. Use any context from meetings in the past month Output: Glean Search [ProductA vs ProductB, CompanyB meeting updated:past_month] Message: Read through our ProductX features. Give me a list of potential enhancements we can make to the feature. You can get inspiration from our ChannelY slack channel. If you get an idea from slack, cite it as not your own Output: Glean Search [ProductX feature enhancements, channel:ChannelY app:"slack"] # BEGIN PROGRAM Conversation history (latest message last): - User: "How do I book time off?" \`\`\` PLAN_RESPONSE \`\`\` analysis: "" readactions: - tool_name: Glean HR Assistant action: Glean HR Assistant: book time off - tool_name: Glean HR Assistant action: Glean HR Assistant: request PTO writeaction: tool_name: "" action: {} instruction: "" \`\`\` PLAN_PROMPT_TOKEN \`\`\` 3354 \`\`\` PLAN_RESPONSE_TOKEN \`\`\` 17 \`\`\` RAW_LLM_PLAN_RESPONSE \`\`\` [LLM Output for : Employee Search + Expert Search] NA ================================================== [LLM Output for : categories] Output: [] ================================================== [LLM Output for : Glean Search + tencctxm5dukr8hc + mlaznw1lpodrp7zk] Glean HR Assistant [book time off, request PTO] \`\`\` PLAN_TTF \`\`\` 1407 \`\`\` PLAN_STAGE_LATENCY \`\`\` 1313 \`\`\` ACTION_SUMMARY \`\`\` **Searching:** "book time off" in the [all] tab(s) using Glean HR Assistant Tool Token limit for this action is : 1295 Num Results requested for this action is : 3 Execution time : 539 ms ================================================== **Searching:** "request PTO" in the [all] tab(s) using Glean HR Assistant Tool Token limit for this action is : 1295 Num Results requested for this action is : 3 Execution time : 592 ms ================================================== **Searching:** "How do I book time off?" in the [all] tab(s) using Glean Search Tool Token limit for this action is : 500 Num Results requested for this action is : 1 Execution time : 811 ms ================================================== **Instruction:** Token limit for this action is : 0 Num Results requested for this action is : 0 Execution time : 0 ms ================================================== \`\`\` ACT_PROMPT \`\`\` Today's date is 2024-Jul-09. You are a helpful assistant for an employee of Glean Work (Email: dse@askscio.com) who works in scio::entire_organization.Here are additional instructions you should follow: If you have additional questions on benefits, please reach out to benefits@glean.com. For other HR questions, feel free to email hr@glean.com.. You are an intelligent workplace assistant, create a truthful response to the user's last message in the conversation by carefully analyzing the context and instructions. Conversation history (latest message last): - User: "How do I book time off?" ## CONTEXT Top results for "request PTO" in Glean HR Assistant Tool: {"documents":[{"title":"How do I request PTO?","datasource":"answers","url":"/knowledge/answers/1851","snippets":["As of August 1, 2022, we will be using Gusto for all PTO requests. Please submit all PTO or sick leaves through Gusto moving forward for your manager's approval. Please continue to update your calendar invite and slack for visibility across the company."],"ownedAndUpdatedBy":{"name":"Jessica Yeh","pronoun":"None"},"updateTime":"2022-11-08"}],"statistics":[{"all":1},{"datasource=answers":1},{"from=jessica@glean.com":1}]} Top results for "How do I book time off?" in Glean Search Tool: {"entities":[{"extractedAnswer":{"question:omitempty":"How do I take time off?","answer":"When taking time off or working from home outside of the normal schedule, do all of the following:\n* Obtain approval from your manager reasonably in advance of proposed vacation (prior to making arrangements) by requesting it in [gusto](https://app.gusto.com/scio-technologies-inc-400/employee/time_off) for US employees and [greyhr](https://glean.greythr.com) for India employees.\n* Notify your immediate working team after receiving approval. Plan to delegate, postpone or otherwise manage projects that will be affected by your time off.\n* Add an Out Of Office event to your individual Glean calendar.\n* Set a Slack status for better visibility.\n* If you receive external emails, set an Out Of Office responder explaining return date and who to contact in your absence.\n\nhttps://docs.google.com/document/d/1qt1Qbt-y4w1UbIU5BQxUpfZlwkx0gcKSGlKd_VqUk18/edit#bookmark=id.6wzx88skrwrf","answerAuthor":{"name":"Tony Gentilcore","pronoun":"None"}}}]} ## INSTRUCTIONS The search results may contain documents that are not relevant to the user's query (for example, document titles or snippets could be about a different topic), you should ignore those results. Think step-by-step. First, determine the set of documents and snippets in the context that are relevant to the user's query. Then, synthesize them to create a detailed truthful response. If none of the provided context are relevant to the user's question, explain that to the user instead of making up an answer. Respond in the same language as user's latest message or query (but don't use their name to determine the language), or in English. Adopt 'they/them' pronouns by default - with one exception: if a 'pronoun' field is specified in a person's JSON summary, use its value to refer to the individual. Be extra meticulous about pronouns, it is very important to get this right, especially if one has a specified pronoun. Be helpful to the user, but avoid workplace violations. Avoid salutations and sign-offs in responses. Great job, let's start! Truthful Response: \`\`\` ACT_PROMPT_TOKEN \`\`\` 791 \`\`\` ACT_RESPONSE_TOKEN \`\`\` 228 \`\`\` FOLLOW_UP_DEBUG_INFO \`\`\` FOLLOWUP PROMPT: \`\`\` Today's date is 2024-Jul-09. You are a helpful assistant for an employee of Glean Work (Email: dse@askscio.com) who works in scio::entire_organization. Conversation history (latest message last): - User: "How do I book time off?" The response to the user is: "**Using app: Glean HR Assistant** To book time off, please follow these steps: 1. **Submit Your Request**: Use Gusto to submit your PTO or sick leave request for your manager's approval. You can access Gusto [here](https://app.gusto.com/scio-technologies-inc-400/employee/time_off) if you are a US employee. For India employees, use [greyhr](https://glean.greythr.com). 2. **Notify Your Team**: After receiving approval, notify your immediate working team. Make sure to plan for delegating, postponing, or managing projects that will be affected by your time off. 3. **Update Your Calendar**: Add an Out Of Office event to your individual Glean calendar. 4. **Set Slack Status**: Update your Slack status for better visibility across the company. 5. **Set Email Responder**: If you receive external emails, set an Out Of Office responder explaining your return date and who to contact in your absence. If you have any further questions on benefits, please reach out to benefits@glean.com. For other HR questions, feel free to email hr@glean.com.". Generate 3 concise follow-up questions that the user might want to ask. Respond in the same language as user's latest message or query (but don't use their name to determine the language), or in English. Do not add any preamble or prefix before the first question, just start writing out the first question in your output. Use this format: - Question 1 - Question 2 - Question 3 Great job, let's start! Follow-up Questions: - \`\`\` FOLLOWUP RAW RESPONSE: \`\`\` How do I check the status of my PTO request? - Can I cancel a PTO request after it has been approved? - How many days of PTO am I allowed to take per year? \`\`\` FOLLOWUP OUTPUT: How do I check the status of my PTO request? Can I cancel a PTO request after it has been approved? How many days of PTO am I allowed to take per year? \`\`\` ` const r = /(.*?)\n\n```\n([\s\S]*?\n)```/g const n = []; let execResults; while ((execResults = r.exec(s)) !== null) { // do something // n.push(execResults.index); }

Rendered benchmark preparation results:

Suite status: <idle, ready to run>

Previous results

Test case name	Result
regexp.matchAll
regexp.exec

Fastest: N/A

Slowest: N/A

Autogenerated LLM Summary (model llama3.2:3b, generated one month ago):

LLMs can make mistakes. Check important info.

Latest run results:

Run details: (Test run date: 4 months ago)

User agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36

Browser/OS: Chrome 123 on Mac OS X 10.15.7

View result in a separate tab

Test name	Executions per second
regexp.matchAll	96208.5 Ops/sec
regexp.exec	9062.2 Ops/sec

Related benchmarks:

string.replace vs regexp.matchAll vs regexp.exec for parsing (version: 0)

Comparing performance of: regexp.matchAll vs regexp.exec

Created: 4 months ago by: Guest

Jump to the latest result

regexp.matchAll

regexp.exec

Suite status: <idle, ready to run>

Experimental features:

Fastest: N/A

Slowest: N/A

Autogenerated LLM Summary (model llama3.2:3b, generated one month ago):