Getting Started with Promptolution
Welcome to Promptolution!
Discover a powerful tool for evolving and optimizing your LLM prompts. This notebook provides a friendly introduction to Promptolution's core functionality, by showcasing how you can easily find the best prompt to solve a classification problem.
We're excited to have you try Promptolution - let's get started!
Installation
Install Promptolution with a single command
Imports
import pandas as pd
from promptolution.llms import APILLM
from promptolution.tasks import ClassificationTask
from promptolution.predictors import MarkerBasedPredictor
from promptolution.optimizers import CAPO
from promptolution.utils import dev_test_split, evaluate_prompts
import nest_asyncio
nest_asyncio.apply() # Required for notebook environments
Setting Up Your Optimization
Prepare the data
Below, we're using a subsample of the subjectivity dataset from Hugging Face as an example. When using your own dataset, simply ensure you name the input column "x" and the target column "y", and provide a brief description of your task, that will passed to the meta-llm during optimization.
df = pd.read_csv("hf://datasets/tasksource/subjectivity/train.csv").sample(500)
df = df.rename(columns={"Sentence": "x", "Label": "y"})
df = df.replace({"OBJ": "objective", "SUBJ": "subjective"})
task_description = (
"The dataset contains sentences labeled as either subjective or objective. "
"The task is to classify each sentence as either subjective or objective. "
"The class mentioned in between the answer tags <final_answer></final_answer> will be used as the prediction."
)
Creating Inital Prompts
We've defined some starter prompts below, but you don't need to do this necessarily, since Promptolution can also automatically generates initial prompts based on your data or the provided task description.
init_prompts = [
'Identify whether a sentence is objective or subjective by analyzing the tone, language, and underlying perspective. Consider the emotion, opinion, and bias present in the sentence. Are the authors presenting objective facts or expressing a personal point of view? The output will be either "objective" (output class: objective) or "subjective" (output class: subjective).',
'Classify a statement as either "subjective" or "objective" based on whether it reflects a personal opinion or a verifiable fact. The output classes to include are "objective" and "subjective".',
"Classify the text as objective or subjective based on its tone and language.",
"Classify the text as objective or subjective based on the presence of opinions or facts. Output classes: objective, subjective.",
"Categorize the text as either objective or subjective, considering whether it presents neutral information or expresses a personal opinion/bias.\n\nObjective: The text has a neutral tone and presents factual information about the actions of Democrats in Congress and the union's negotiations.\n\nSubjective: The text has a evaluative tone and expresses a positive/negative opinion/evaluation about the past performance of the country.",
'Given a sentence, classify it as either "objective" or "subjective" based on its tone and language, considering the presence of third-person pronouns, neutral language, and opinions. Classify the output as "objective" if the tone is neutral and detached, focusing on facts and data, or as "subjective" if the tone is evaluative, emotive, or biased.',
]
Configure Your LLM
Promptolution offers three flexible ways to access language models:
- Local LLMs (using the Transformers library)
- vLLM backend (for efficient serving of large language models)
- API-based LLMs (compatible with any provider following the OpenAI standard)
For this demonstration, we'll use the DeepInfra API, but you can easily switch to other providers like Anthropic or OpenAI.
Choose Your Components
We first split the data: the optimizer only ever sees dev_df, and test_df is kept back so we can score the resulting prompts on data they were not selected on.
Here's an explanation of the most important choices, and what we use in this example:
- optimizer: the algorithm used for prompt optimization. Here we use CAPO, as it is capable of leveraging few-shot examples.
- llm: the language model, used as both the downstream model (which makes the predictions) and the meta model (which proposes new prompts). Here an APILLM pointing at DeepInfra's meta-llama/Meta-Llama-3-8B-Instruct.
- task: wraps your dev_df and defines the objective. task_description is a string describing the task, n_subsamples sets how many datapoints are used per evaluation step (here 30).
- predictor: how the label is extracted from the LLM output. Here from between markers, using the task's classes.
- initial_prompts: the prompts the optimizer starts from and improves. Here we pass init_prompts.
- n_steps: the number of optimization steps (here 10).
dev_df, test_df = dev_test_split(df, test_frac=0.2, seed=42)
llm = APILLM(
api_url="https://api.deepinfra.com/v1/openai",
model_id="meta-llama/Meta-Llama-3-8B-Instruct",
api_key=api_key,
)
task = ClassificationTask(dev_df, task_description=task_description, n_subsamples=30)
predictor = MarkerBasedPredictor(llm, classes=task.classes)
optimizer = CAPO(
predictor=predictor,
meta_llm=llm,
task=task,
initial_prompts=init_prompts,
)
Optimize Your Prompts
With everything built, you're ready to optimize! Calling optimizer.optimize(n_steps=...) runs the optimization loop and returns the best prompts. Expect this cell to take a few minutes to run.
Evaluate on Held-Out Data
The scores the optimizer used internally were measured on the data it selected against, so they are optimistic. evaluate_prompts scores the resulting prompts on test_df and returns them sorted best first.
test_task = ClassificationTask(test_df, task_description=task_description, eval_strategy="full")
prompt_scores = evaluate_prompts(prompts, test_task, predictor)
As you can see, most optimized prompts are semantically very similar, however they often differ heavily in performance. This is exactly what we observed in our experiments across various LLMs and datasets. Running prompt optimization is an easy way to gain significant performance improvements on your task for free!
If you run into any issues while using Promptolution, please feel free to contact us. We're also happy to receive support through pull requests and other contributions to the project.
Happy prompt optimizing! 🚀✨ We can't wait to see what you build with Promptolution! 🤖💡
| prompt | score | |
|---|---|---|
| 0 | Your task is to categorize each sentence into one of two categories: subjective or objective. This requires analyzing the sentence's tone, language, and content to discern whether it reflects a personal perspective, emotion, or bias, or if it presents information in a neutral, fact-based manner. To make your determination, consider the following: \n- Does the sentence express a personal opinion, emotion, or bias, or does it convey a personal viewpoint or evaluative stance? \n- Or does it report information in a balanced and impartial way, stating facts without apparent personal influence or slant? \nLabel the sentence as "subjective" if it is personally opinionated, emotional, or biased. Label it as "objective" if it is factually oriented and neutral. Provide your classification in the following format: <final_answer>subjective/objective</final_answer>\n\nInput:\nBut that will be only like pruning the tree, for lustier growth hereafter, unless we settle what public credit is for in principle and limit in a drastic manner the ferocious growth of government.\nOutput:\nI would classify this text as subjective. Here's why:\n\n* The use of the metaphor "pruning the tree" to describe a potential solution to a problem suggests that the author has a particular perspective on the issue and is using a vivid and evaluative term to convey their viewpoint.\n* The phrase "lustier growth hereafter" is a value-laden expression that implies that the author values growth and expansion, but also uses a somewhat playful and emotive term ("lustier") to emphasize their point.\n* The phrase "ferocious growth of government" is also subjective, as it uses a strongly negative adjective ("ferocious") to describe government growth, implying that it is excessive and threatening.\n* The use of the phrase "in principle" and "in a drastic manner" suggests that the author has a strong opinion about how to address the issue and is advocating for a particular approach.\n\nOverall, while the text is written in a somewhat formal and abstract style, the language and tone used suggest that the author is expressing a personal opinion or viewpoint, rather than presenting a neutral or objective analysis of the issue.\n\nInput:\nSecondly, in a signal of just who is stalking the market, there is a cloak of privacy surrounding property coming up for sale.\n\nOutput:\nThe text is: **Subjective**\n\nThe text has an evaluative tone, implying that there is something suspicious or wrong about the privacy surrounding property coming up for sale. The phrase "in a signal of just who is stalking the market" suggests that the author has a negative opinion about the situation and is making a judgment about the motivations of the parties involved. The tone is not neutral, and the language used implies a bias. Therefore, the text is categorized as subjective.\n\nInput:\nBut what of American individualism?\n\nOutput:\n<final_answer>subjective</final_answer>\n\nInput: | 0.72 |
| 1 | Classify the given sentence as either "subjective" or "objective" based on its tone, language, and content. Your goal is to determine whether the sentence presents a personal viewpoint, emotion, or bias, or if it reports information in a neutral and factual manner. Consider the language, tone, and perspective of the sentence to make your decision. If the sentence expresses a personal opinion, emotion, or bias, label it as "subjective". If it presents information in a factual and unbiased way, label it as "objective". Provide your classification as: <final_answer>subjective/objective</final_answer>\n\nInput:\nIf the natural level of economic recovery were long delayed, then all these measures would very soon fail in the total ruin of public credit.\n\nOutput:\n<final_answer>subjective</final_answer>\n\nInput:\nBut that will be only like pruning the tree, for lustier growth hereafter, unless we settle what public credit is for in principle and limit in a drastic manner the ferocious growth of government.\nOutput:\nI would classify this text as subjective. Here's why:\n\n* The use of the metaphor "pruning the tree" to describe a potential solution to a problem suggests that the author has a particular perspective on the issue and is using a vivid and evaluative term to convey their viewpoint.\n* The phrase "lustier growth hereafter" is a value-laden expression that implies that the author values growth and expansion, but also uses a somewhat playful and emotive term ("lustier") to emphasize their point.\n* The phrase "ferocious growth of government" is also subjective, as it uses a strongly negative adjective ("ferocious") to describe government growth, implying that it is excessive and threatening.\n* The use of the phrase "in principle" and "in a drastic manner" suggests that the author has a strong opinion about how to address the issue and is advocating for a particular approach.\n\nOverall, while the text is written in a somewhat formal and abstract style, the language and tone used suggest that the author is expressing a personal opinion or viewpoint, rather than presenting a neutral or objective analysis of the issue.\n\nInput:\nIf the natural level of economic recovery were long delayed, then all these measures would very soon fail in the total ruin of public credit.\n\nOutput:\n<final_answer>subjective</final_answer>\n\nInput: | 0.70 |
| 2 | Determine whether a given sentence expresses a personal opinion or presents information in a neutral, fact-based manner. Label the sentence as "subjective" if it reflects a personal viewpoint, sentiment, or emotion, and as "objective" if it reports information in a factual, unbiased way, supported by evidence. The classification provided between the tags <final_answer></final_answer> will be treated as the predicted category.\n\nInput:\nThere is yet everywhere a deficit in the public revenue because the shrinkage in everything taxable was so sudden and violent.\n\nOutput:\n<final_answer>subjective</final_answer>\n\nInput:\nBut what of American individualism?\n\nOutput:\n<final_answer>subjective</final_answer>\n\nInput:\nThere is yet everywhere a deficit in the public revenue because the shrinkage in everything taxable was so sudden and violent.\n\nOutput:\n<final_answer>subjective</final_answer>\n\nInput:\nThe spender of public money will never want followers.\n\nOutput:\n<final_answer>subjective</final_answer>\n\nThe sentence expresses a personal viewpoint or opinion about the spender of public money, suggesting that they will never want followers. The use of "will never want" indicates a prediction or an opinion rather than a verifiable fact. Therefore, it reflects a subjective perspective.\n\nInput: | 0.69 |
| 3 | Classify a given sentence as either "subjective" or "objective" based on its tone, language, and perspective. Your goal is to discern whether the sentence presents a personal opinion, emotion, or biased viewpoint, or if it conveys information in a neutral and fact-based manner. If the sentence reflects a personal stance, emotion, or evaluative judgment, label it as "subjective". If it reports information in an impartial way, ideally with evidence to support the claim, label it as "objective". The task requires you to analyze the sentence's characteristics to determine if it expresses personal beliefs, feelings, or opinions, or if it states facts without a personal perspective. The classification will be indicated by the term placed between the <final_answer></final_answer> tags.\n\nInput:\nBut what of American individualism?\n\nOutput:\n<final_answer>subjective</final_answer>\n\nInput:\nSecondly, in a signal of just who is stalking the market, there is a cloak of privacy surrounding property coming up for sale.\n\nOutput:\nThe text is: **Subjective**\n\nThe text has an evaluative tone, implying that there is something suspicious or wrong about the privacy surrounding property coming up for sale. The phrase "in a signal of just who is stalking the market" suggests that the author has a negative opinion about the situation and is making a judgment about the motivations of the parties involved. The tone is not neutral, and the language used implies a bias. Therefore, the text is categorized as subjective.\n\nInput:\nWithout major changes, though, the pool of available teachers could continue to shrink, leaving those that remain with ever-ballooning workloads.\n\nOutput:\n<final_answer>objective</final_answer>\n\nInput:\nThere is yet everywhere a deficit in the public revenue because the shrinkage in everything taxable was so sudden and violent.\n\nOutput:\n<final_answer>subjective</final_answer>\n\nInput:\nA clique of executives from Meta have snapped up properties in London since the social media giant’s new King’s Cross offices opened this year, and they are far from alone.\n\nOutput:\n<final_answer>subjective</final_answer>\n\nInput: | 0.68 |
| 4 | Determine whether each sentence is a personal opinion, feeling, or judgment, or a neutral, fact-based statement. Analyze the sentence's tone, language, and perspective to categorize it as either subjective, which includes personal views, emotions, or biases, or objective, which presents information in a neutral and evidence-based manner. Label each sentence as "subjective" if it reflects a personal standpoint, emotion, or bias, and as "objective" if it reports facts without personal influence. <final_answer>subjective/objective</final_answer>\n\nInput:\n“All we need is for the legal system to work in Serbia and for the rule of law to be observed,” she said.\n\nOutput:\nI would categorize the text as:\n\n**Subjective**\n\nAlthough the text quotes a statement that mentions the legal system and rule of law in a neutral way, the context and the tone imply a personal opinion or evaluation about the current state of the legal system in Serbia. The quote suggests that the speaker believes the legal system is not currently working and that the rule of law is not being observed, which expresses a negative opinion about the situation in Serbia. The tone is not entirely neutral, but rather critical or evaluative. If the text simply stated a fact without evaluation, it would be objective, but in this case, it implies a subjective judgment.\n\nInput:\nArizona will now allow current college students to teach.\n\nOutput:\n<final_answer>subjective</final_answer>\n\nInput:\nAre the rulers preparing us to accept another lockdown over a mutated monkeypox virus?\n\nOutput:\nAfter analyzing the sentence, I would classify it as:\n\n**subjective**\n\nHere's why:\n\n* The tone of the sentence is skeptical and slightly accusatory, implying that the rulers have ulterior motives.\n* The use of the phrase "preparing us to accept" suggests a perceived manipulation or coercion, which is a subjective interpretation.\n* The question itself is framed in a way that implies a level of distrust and suspicion, which is a personal perspective.\n* The language used is emotive, with a hint of fear and alarm ("another lockdown", "mutated monkeypox virus"), which further suggests a subjective tone.\n\nOverall, the sentence expresses a personal opinion or point of view, rather than presenting a neutral, fact-based observation. Therefore, I classify it as subjective.\n\nInput: | 0.66 |
| 5 | Classify a given sentence as either "subjective" or "objective" based on its tone, language, and perspective. Your goal is to discern whether the sentence presents a personal opinion, emotion, or biased viewpoint, or if it delivers factual information in a neutral and impartial manner. Label the sentence as "subjective" if it reflects a personal stance, sentiment, or evaluative judgment, and as "objective" if it provides information supported by evidence, free from personal perspective or bias. The classification should hinge on whether the sentence conveys personal beliefs, emotions, or viewpoints, or if it states verifiable facts without emotional or personal influence. The predicted classification will be the label mentioned within the answer tags <final_answer></final_answer>.\n\nInput:\nBut what of American individualism?\n\nOutput:\n<final_answer>subjective</final_answer>\n\nInput: | 0.65 |