ilab configuration¶
InstructLab’s configuration is read from config.yaml file. The
configuration is handled avalided by a Pydantic
schema.
- pydantic model instructlab.configuration.Config¶
Configuration for the InstructLab CLI.
Show JSON schema
{ "title": "Config", "description": "Configuration for the InstructLab CLI.", "type": "object", "properties": { "chat": { "$ref": "#/$defs/_chat" }, "generate": { "$ref": "#/$defs/_generate" }, "serve": { "$ref": "#/$defs/_serve" }, "general": { "allOf": [ { "$ref": "#/$defs/_general" } ], "default": { "log_level": "INFO", "debug_level": 0 } }, "train": { "anyOf": [ { "$ref": "#/$defs/_train" }, { "type": "null" } ], "default": null }, "evaluate": { "$ref": "#/$defs/_evaluate" } }, "$defs": { "LogLevel": { "description": "Log level", "enum": [ "DEBUG", "INFO", "WARNING", "WARNING", "ERROR", "CRITICAL", "CRITICAL" ], "title": "LogLevel", "type": "string" }, "_chat": { "description": "Class describing configuration of the chat sub-command.", "properties": { "model": { "title": "Model", "type": "string" }, "vi_mode": { "default": false, "title": "Vi Mode", "type": "boolean" }, "visible_overflow": { "default": true, "title": "Visible Overflow", "type": "boolean" }, "context": { "default": "default", "title": "Context", "type": "string" }, "session": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "title": "Session" }, "logs_dir": { "title": "Logs Dir", "type": "string" }, "greedy_mode": { "default": false, "title": "Greedy Mode", "type": "boolean" }, "max_tokens": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "title": "Max Tokens" } }, "required": [ "model" ], "title": "_chat", "type": "object" }, "_evaluate": { "properties": { "model": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "title": "Model" }, "base_model": { "title": "Base Model", "type": "string" }, "branch": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "title": "Branch" }, "base_branch": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "title": "Base Branch" }, "gpus": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": 1, "title": "Gpus" }, "mmlu": { "$ref": "#/$defs/_mmlu" }, "mmlu_branch": { "$ref": "#/$defs/_mmlubranch" }, "mt_bench": { "$ref": "#/$defs/_mtbench" }, "mt_bench_branch": { "$ref": "#/$defs/_mtbenchbranch" } }, "required": [ "base_model", "mmlu", "mmlu_branch", "mt_bench", "mt_bench_branch" ], "title": "_evaluate", "type": "object" }, "_general": { "description": "Class describing various top-level configuration options for all commands.", "properties": { "log_level": { "allOf": [ { "$ref": "#/$defs/LogLevel" } ], "default": "INFO" }, "debug_level": { "default": 0, "title": "Debug Level", "type": "integer" } }, "title": "_general", "type": "object" }, "_generate": { "description": "Class describing configuration of the generate sub-command.", "properties": { "model": { "title": "Model", "type": "string" }, "taxonomy_path": { "title": "Taxonomy Path", "type": "string" }, "taxonomy_base": { "title": "Taxonomy Base", "type": "string" }, "num_cpus": { "default": 10, "exclusiveMinimum": 0, "title": "Num Cpus", "type": "integer" }, "chunk_word_count": { "default": 1000, "exclusiveMinimum": 0, "title": "Chunk Word Count", "type": "integer" }, "num_instructions": { "default": 100, "exclusiveMinimum": 0, "title": "Num Instructions", "type": "integer" }, "output_dir": { "title": "Output Dir", "type": "string" }, "prompt_file": { "title": "Prompt File", "type": "string" }, "seed_file": { "title": "Seed File", "type": "string" } }, "required": [ "model", "taxonomy_path", "taxonomy_base" ], "title": "_generate", "type": "object" }, "_mmlu": { "properties": { "few_shots": { "title": "Few Shots", "type": "integer" }, "batch_size": { "title": "Batch Size", "type": "string" } }, "required": [ "few_shots", "batch_size" ], "title": "_mmlu", "type": "object" }, "_mmlubranch": { "properties": { "tasks_dir": { "title": "Tasks Dir", "type": "string" } }, "required": [ "tasks_dir" ], "title": "_mmlubranch", "type": "object" }, "_mtbench": { "properties": { "judge_model": { "title": "Judge Model", "type": "string" }, "output_dir": { "title": "Output Dir", "type": "string" }, "max_workers": { "title": "Max Workers", "type": "integer" } }, "required": [ "judge_model", "output_dir", "max_workers" ], "title": "_mtbench", "type": "object" }, "_mtbenchbranch": { "properties": { "taxonomy_path": { "title": "Taxonomy Path", "type": "string" } }, "required": [ "taxonomy_path" ], "title": "_mtbenchbranch", "type": "object" }, "_serve": { "description": "Class describing configuration of the serve sub-command.", "properties": { "vllm": { "$ref": "#/$defs/_serve_vllm" }, "llama_cpp": { "$ref": "#/$defs/_serve_llama_cpp" }, "model_path": { "title": "Model Path", "type": "string" }, "host_port": { "default": "127.0.0.1:8000", "title": "Host Port", "type": "string" }, "chat_template": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "title": "Chat Template" }, "backend": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "title": "Backend" } }, "required": [ "vllm", "llama_cpp", "model_path" ], "title": "_serve", "type": "object" }, "_serve_llama_cpp": { "description": "Class describing configuration of llama-cpp serving backend.", "properties": { "gpu_layers": { "default": -1, "title": "Gpu Layers", "type": "integer" }, "max_ctx_size": { "default": 4096, "exclusiveMinimum": 0, "title": "Max Ctx Size", "type": "integer" }, "llm_family": { "default": "", "title": "Llm Family", "type": "string" } }, "title": "_serve_llama_cpp", "type": "object" }, "_serve_vllm": { "description": "Class describing configuration of vllm serving backend.", "properties": { "llm_family": { "default": "", "title": "Llm Family", "type": "string" }, "vllm_args": { "description": "Additional command line arguments for vLLM, see `command line arguments for server <https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#command-line-arguments-for-the-server>`_", "items": { "type": "string" }, "title": "Vllm Args", "type": "array" } }, "title": "_serve_vllm", "type": "object" }, "_train": { "properties": { "model_path": { "title": "Model Path", "type": "string" }, "data_path": { "title": "Data Path", "type": "string" }, "ckpt_output_dir": { "title": "Ckpt Output Dir", "type": "string" }, "data_output_dir": { "title": "Data Output Dir", "type": "string" }, "max_seq_len": { "title": "Max Seq Len", "type": "integer" }, "max_batch_len": { "title": "Max Batch Len", "type": "integer" }, "num_epochs": { "title": "Num Epochs", "type": "integer" }, "effective_batch_size": { "title": "Effective Batch Size", "type": "integer" }, "save_samples": { "title": "Save Samples", "type": "integer" }, "deepspeed_cpu_offload_optimizer": { "title": "Deepspeed Cpu Offload Optimizer", "type": "boolean" }, "lora_rank": { "title": "Lora Rank", "type": "integer" }, "lora_quantize_dtype": { "title": "Lora Quantize Dtype", "type": "string" }, "is_padding_free": { "title": "Is Padding Free", "type": "boolean" }, "nproc_per_node": { "title": "Nproc Per Node", "type": "integer" }, "additional_args": { "title": "Additional Args", "type": "object" } }, "required": [ "model_path", "data_path", "ckpt_output_dir", "data_output_dir", "max_seq_len", "max_batch_len", "num_epochs", "effective_batch_size", "save_samples", "deepspeed_cpu_offload_optimizer", "lora_rank", "lora_quantize_dtype", "is_padding_free", "nproc_per_node", "additional_args" ], "title": "_train", "type": "object" } }, "required": [ "chat", "generate", "serve", "evaluate" ] }
- Fields:
General¶
model chat¶
- pydantic model instructlab.configuration._chat¶
Class describing configuration of the chat sub-command.
- Fields:
- field context: str = 'default'¶
- field greedy_mode: bool = False¶
- field logs_dir: str [Optional]¶
Path to log directory for chat logs
- field max_tokens: int | None = None¶
- field model: str [Required]¶
- field session: str | None = None¶
- field vi_mode: bool = False¶
- field visible_overflow: bool = True¶
model evaluate¶
- pydantic model instructlab.configuration._evaluate¶
- Fields:
- field base_branch: str | None = None¶
- field base_model: str [Required]¶
- field branch: str | None = None¶
- field gpus: int | None = 1¶
- field mmlu_branch: _mmlubranch [Required]¶
- field model: str | None = None¶
- field mt_bench_branch: _mtbenchbranch [Required]¶
- pydantic model instructlab.configuration._mmlu¶
- Fields:
- field batch_size: str [Required]¶
- field few_shots: int [Required]¶
- pydantic model instructlab.configuration._mtbench¶
-
- field judge_model: str [Required]¶
- field max_workers: int [Required]¶
- field output_dir: str [Required]¶
model generate¶
- pydantic model instructlab.configuration._generate¶
Class describing configuration of the generate sub-command.
- Fields:
- field chunk_word_count: Annotated[int, Gt(gt=0)] = 1000¶
- Constraints:
gt = 0
- field model: Annotated[str, Strict(strict=True)] [Required]¶
- Constraints:
strict = True
- field num_cpus: Annotated[int, Gt(gt=0)] = 10¶
- Constraints:
gt = 0
- field num_instructions: Annotated[int, Gt(gt=0)] = 100¶
- Constraints:
gt = 0
- field output_dir: Annotated[str, Strict(strict=True)] [Optional]¶
- Constraints:
strict = True
- field prompt_file: Annotated[str, Strict(strict=True)] [Optional]¶
- Constraints:
strict = True
- field seed_file: Annotated[str, Strict(strict=True)] [Optional]¶
- Constraints:
strict = True
- field taxonomy_base: Annotated[str, Strict(strict=True)] [Required]¶
- Constraints:
strict = True
- field taxonomy_path: Annotated[str, Strict(strict=True)] [Required]¶
- Constraints:
strict = True
model serve¶
- pydantic model instructlab.configuration._serve¶
Class describing configuration of the serve sub-command.
- Fields:
- field backend: str | None = None¶
- field chat_template: str | None = None¶
- field host_port: Annotated[str, Strict(strict=True)] = '127.0.0.1:8000'¶
- Constraints:
strict = True
- field llama_cpp: _serve_llama_cpp [Required]¶
- field model_path: Annotated[str, Strict(strict=True)] [Required]¶
- Constraints:
strict = True
- field vllm: _serve_vllm [Required]¶
- api_base()¶
Returns server API URL, based on the configured host and port
- pydantic model instructlab.configuration._serve_llama_cpp¶
Class describing configuration of llama-cpp serving backend.
- field gpu_layers: int = -1¶
Number of layers to offload to GPU, -1: offload all layers.
- field llm_family: str = ''¶
- field max_ctx_size: Annotated[int, Gt(gt=0)] = 4096¶
- Constraints:
gt = 0
- pydantic model instructlab.configuration._serve_vllm¶
Class describing configuration of vllm serving backend.
- field llm_family: str = ''¶
- field vllm_args: list[str] [Optional]¶
Additional command line arguments for vLLM, see command line arguments for server
model train¶
- pydantic model instructlab.configuration._train¶
- Fields:
- field additional_args: dict[str, Any] [Required]¶
- field ckpt_output_dir: str [Required]¶
- field data_output_dir: str [Required]¶
- field data_path: str [Required]¶
- field deepspeed_cpu_offload_optimizer: bool [Required]¶
- field effective_batch_size: int [Required]¶
- field is_padding_free: bool [Required]¶
- field lora_quantize_dtype: str [Required]¶
- field lora_rank: int [Required]¶
- field max_batch_len: int [Required]¶
- field max_seq_len: int [Required]¶
- field model_path: str [Required]¶
- field nproc_per_node: int [Required]¶
- field num_epochs: int [Required]¶
- field save_samples: int [Required]¶