ImageStylization¶

class agentlego.tools.ImageStylization(model='timbrooks/instruct-pix2pix', inference_steps=20, device='cuda', toolmeta=None)[source]

A tool to stylize an image.

Parameters:

model (str) – The model name used to inference. Which can be found in the diffusers repository. Defaults to ‘timbrooks/instruct-pix2pix’.
inference_steps (int) – The number of inference steps. Defaults to 20.
device (str) – The device to load the model. Defaults to ‘cuda’.
toolmeta (None | dict | ToolMeta) – The additional info of the tool. Defaults to None.

Default Tool Meta¶

name: ImageStylization
description: This tool can modify the input image according to the input instruction. Here are some example instructions: “turn him into cyborg”, “add fireworks to the sky”, “make his jacket out of leather”.
inputs:
- image (ImageIO)
- instruction (str)
outputs:
- ImageIO

Examples¶

Use the tool directly (without agent)

from agentlego.apis import load_tool

# load tool
tool = load_tool('ImageStylization', device='cuda')

# apply tool
image = tool('examples/demo.png', 'turn the cat into a cartoon cat')
print(image)

With Lagent

from lagent import ReAct, GPTAPI, ActionExecutor
from agentlego.apis import load_tool

# load tools and build agent
# please set `OPENAI_API_KEY` in your environment variable.
tool = load_tool('ImageStylization', device='cuda').to_lagent()
agent = ReAct(GPTAPI(temperature=0.), action_executor=ActionExecutor([tool]))

# agent running with the tool.
img_path = 'examples/demo.png'
ret = agent.chat(f'According to the image `{img_path}`, turn the cat into a cartoon cat.')
for step in ret.inner_steps[1:]:
    print('------')
    print(step['content'])

Set up¶

Before using this tool, please confirm you have installed the related dependencies by the below commands.

pip install -U diffusers

Reference¶

This tool uses a instruct-pix2pix model in default settings. See the following paper for details.

@article{brooks2022instructpix2pix,
  title={InstructPix2Pix: Learning to Follow Image Editing Instructions},
  author={Brooks, Tim and Holynski, Aleksander and Efros, Alexei A},
  journal={arXiv preprint arXiv:2211.09800},
  year={2022}
}