ilab command line reference

The following documentation is auto-generated from the ilab code.

ilab

CLI for interacting with InstructLab.

If this is your first time running ilab, it’s best to start with ilab config init to create the environment.

ilab [OPTIONS] COMMAND [ARGS]...

Options

--config <config_file>

Path to a configuration file.

Default:

'/home/heimes/.config/instructlab/config.yaml'

-v, --verbose

Enable debug logging (repeat for even more verbosity)

--version

Show the version and exit.

config

Command Group for Interacting with the Config of InstructLab.

If this is your first time running ilab, it’s best to start with ilab config init to create the environment.

ilab config [OPTIONS] COMMAND [ARGS]...

init

Initializes environment for InstructLab

ilab config init [OPTIONS]

Options

--interactive, --non-interactive

Initialize the environment assuming defaults.

Default:

True

--model-path <model_path>

Path to the model used during generation.

Default:

"The instructlab data files location per the user's system."

--taxonomy-base <taxonomy_base>

Base git-ref to use when listing/generating new taxonomy.

Default:

'origin/main'

--taxonomy-path <taxonomy_path>

Path to where the taxonomy should be cloned.

Default:

"The instructlab data files location per the user's system."

--repository <repository>

Taxonomy repository location.

Default:

'https://github.com/instructlab/taxonomy.git'

--min-taxonomy

Shallow clone the taxonomy repository with minimum size. Please do not use this option if you are planning to contribute back using the same taxonomy repository.

--train-profile <train_profile>

show

Displays the current config as YAML

ilab config show [OPTIONS]

data

Command Group for Interacting with the Data generated by InstructLab.

If this is your first time running ilab, it’s best to start with ilab config init to create the environment.

ilab data [OPTIONS] COMMAND [ARGS]...

generate

Generates synthetic data to enhance your example data

ilab data generate [OPTIONS]

Options

--model <model>

Name of the model used during generation.

Default:

'The default model used by the instructlab system, located in the data directory.'

--num-cpus <num_cpus>

Number of processes to use.

Default:

10

--chunk-word-count <chunk_word_count>

Number of words to chunk the document

Default:

1000

--num-instructions <num_instructions>

Number of instructions to generate.

Default:

100

--taxonomy-path <taxonomy_path>

Path to where the taxonomy is located.

Default:

'The default taxonomy path used by instructlab, located in the data directory.'

--taxonomy-base <taxonomy_base>

Base git-ref to use when generating new taxonomy.

Default:

'origin/main'

--output-dir <output_dir>

Path to output generated files.

Default:

'The default output directory used by instructlab, located in the data directory.'

--rouge-threshold <rouge_threshold>

Threshold of (max) Rouge score to keep samples; 1.0 means accept all samples.

Default:

0.9

--quiet

Suppress output of synthesized instructions.

--endpoint-url <endpoint_url>

Custom URL endpoint for OpenAI-compatible API. Defaults to the ilab model serve endpoint.

--api-key <api_key>

API key for API endpoint. [default: config.DEFAULT_API_KEY]

--yaml-rules <yaml_rules>

Custom rules file for YAML linting.

--server-ctx-size <server_ctx_size>

The context size is the maximum number of tokens the server will consider.

Default:

4096

--tls-insecure

Disable TLS verification.

--tls-client-cert <tls_client_cert>

Path to the TLS client certificate to use.

Default:

''

--tls-client-key <tls_client_key>

Path to the TLS client key to use.

Default:

''

--tls-client-passwd <tls_client_passwd>

TLS client certificate password.

--model-family <model_family>

Force model family to use when picking a generation template

model

Command Group for Interacting with the Models in InstructLab.

If this is your first time running ilab, it’s best to start with ilab config init to create the environment.

ilab model [OPTIONS] COMMAND [ARGS]...

chat

Run a chat using the modified model

ilab model chat [OPTIONS] [QUESTION]...

Options

-m, --model <model>

Required Model name to print in chat process

-c, --context <context>

Required Name of system context in config file.

-s, --session <session>

Filepath of a dialog session file.

-qq, --quick-question

Exit after answering question.

-gm, --greedy-mode

Required Use model greedy decoding. Useful for debugging and reproducing errors.

--max-tokens <max_tokens>

Set a maximum number of tokens to request from the model endpoint.

--endpoint-url <endpoint_url>

Custom URL endpoint for OpenAI-compatible API. Defaults to the ilab model serve endpoint.

--api-key <api_key>

API key for API endpoint. [default: config.DEFAULT_API_KEY]

--tls-insecure

Disable TLS verification.

--tls-client-cert <tls_client_cert>

Path to the TLS client certificate to use.

Default:

''

--tls-client-key <tls_client_key>

Path to the TLS client key to use.

Default:

''

--tls-client-passwd <tls_client_passwd>

TLS client certificate password.

--model-family <model_family>

Force model family to use when picking a chat template

Arguments

QUESTION

Optional argument(s)

convert

Converts model to GGUF

ilab model convert [OPTIONS]

Options

--model-dir <model_dir>

Required Base directory where model is stored.

--adapter-file <adapter_file>

LoRA adapter to fuse.

-sd, --skip-de-quantize

Skip de-quantization.

-sq, --skip-quantize

Whether to skip quantization while converting to GGUF.

--model-name <model_name>

Name of the model being trained/converted. Informs the naming of the final trained model file

download

Download the model(s) to train

ilab model download [OPTIONS]

Options

--repository <repository>

Hugging Face repository of the model to download.

Default:

'instructlab/merlinite-7b-lab-GGUF'

--release <release>

The git revision of the model to download - e.g. a branch, tag, or commit hash.

Default:

'main'

--filename <filename>

Name of the model file to download from the Hugging Face repository.

Default:

'The default model location in the instructlab data directory.'

--model-dir <model_dir>

The local directory to download the model files into.

Default:

'The default system model location store, located in the data directory.'

--hf-token <hf_token>

User access token for connecting to the Hugging Face Hub.

Environment variables

HF_TOKEN

Provide a default for --hf-token

evaluate

ilab model evaluate [OPTIONS]

Options

--model <model>

Model to be evaluated - can be a local path or the name of a Hugging Face repository

--base-model <base_model>

Required Base model to compare with ‘model’ for mt_bench_branch and mmlu_branch - can be a local path or the name of a Hugging Face repository

--benchmark <benchmark>

Required Benchmarks to run during evaluation

Options:

mmlu | mmlu_branch | mt_bench | mt_bench_branch

--judge-model <judge_model>

Model to be used as a judge for running mt_bench or mt_bench_branch - can be a local path or the name of a Hugging Face repository

--output-dir <output_dir>

The directory to use for evaluation output from mt_bench or mt_bench_branch

--max-workers <max_workers>

Max parallel workers to run the evaluation with for mt_bench or mt_bench_branch

--taxonomy-path <taxonomy_path>

Taxonomy git repo path for running mt_bench_branch

--branch <branch>

Branch of taxonomy repo to eval QNAs against model

--base-branch <base_branch>

Base branch of taxonomy repo to eval QNAs against model for mt_bench_branch

--few-shots <few_shots>

Number of examples. Needed for running mmlu or mmlu_branch.

--batch-size <batch_size>

Batch size for mmlu and mmlu_branch evaluation. Valid values are a positive integer, ‘auto’ to select the largest batch size that will fit in memory, or ‘auto:N’ to reselect the largest batch size N times’.

--tasks-dir <tasks_dir>

Path where all the MMLU Branch tasks are stored. Needed for running mmlu_branch.

--gpus <gpus>

Number of GPUs to utilize for evaluation (not applicable to llama-cpp)

--merge-system-user-message

Indicates whether to merge system and user message for mt_bench and mt_bench_branch (required for Mistral based judges)

--backend <backend>

Serving backend to use for the model and base model (if applicable) during evaluation. Options are vllm and llama-cpp.

Options:

vllm | llama-cpp

--judge-backend <judge_backend>

Serving backend to use for the judge model for during mt_bench or mt_bench_branch evaluation. Options are vllm and llama-cpp.

Options:

vllm | llama-cpp

--tls-insecure

Disable TLS verification for model serving.

--tls-client-cert <tls_client_cert>

Path to the TLS client certificate to use for model serving.

Default:

''

--tls-client-key <tls_client_key>

Path to the TLS client key to use for model serving.

Default:

''

--tls-client-passwd <tls_client_passwd>

TLS client certificate password for model serving.

--enable-serving-output

Print serving engine logs.

serve

Start a local server

The vLLM backend accepts additional parameters in the form of extra arguments after “–” separator:

$ ilab model serve … –backend=vllm – –dtype=auto –enable-lora

vLLm parameters are documented at https://docs.vllm.ai/en/stable/serving/openai_compatible_server.html

ilab model serve [OPTIONS]

Options

--model-path <model_path>

Required Path to the model used during generation.

--gpu-layers <gpu_layers>

Required The number of layers to put on the GPU. -1 moves all layers. The rest will be on the CPU.

--num-threads <num_threads>

The number of CPU threads to use.

--max-ctx-size <max_ctx_size>

The context size is the maximum number of tokens considered by the model, for both the prompt and response. Defaults to 4096.

--model-family <model_family>

Model family is used to specify which chat template to serve with

--log-file <log_file>

Log file path to write server logs to.

--chat-template <chat_template>

Which chat template to use in serving the model, either ‘auto’, ‘tokenizer’, or a path to a jinja formatted template file. ‘auto’ (the default) indicates serve will decide which template to use. ‘tokenizer’ indicates the model’s tokenizer config will be preferred

--backend <backend>

The backend to use for serving the model. Automatically detected based on the model file properties.

Options:

vllm | llama-cpp

test

Runs basic test to ensure model correctness

ilab model test [OPTIONS]

Options

--data-dir <data_dir>

Base directory where data is stored.

Default:

'Default internal data directory, stored in the instructlab package.'

--model-dir <model_dir>

Base directory where model is stored.

Default:

'Default instructlab system checkpoints directory.'

--adapter-file <adapter_file>

LoRA adapter to use for test. Set to ‘None’ to force only testing behavior from before training.

Default:

'auto'

-m, --model <model>

Base model name to test on Linux

Default:

<function <lambda> at 0x7fad8ccfc900>

-t, --test_file <test_file>

Test data file

--api-key <api_key>

API key for API endpoint. [default: cfg.DEFAULT_API_KEY]

--model-family <model_family>

Force model family to use when picking a generation template

train

Takes synthetic data generated locally with ilab data generate and the previous model and learns a new model using the MLX API. On success, writes newly learned model to {model_dir}/mlx_model, which is where chatmlx will look for a model.

ilab model train [OPTIONS]

Options

--data-path <data_path>

Required Base directory where data is stored.

--ckpt-output-dir <ckpt_output_dir>

Required output directory to store checkpoints in during training

--data-output-dir <data_output_dir>

Required output directory to store training data in

--input-dir <input_dir>

Path to generated files to use as input.

--gguf-model-path <gguf_model_path>

Local directory where gguf model is stored.

--skip-preprocessing

--tokenizer-dir <tokenizer_dir>

Base directory where tokenizer is stored.

--model-path <model_path>

Required HuggingFace model repo path, in the format of <namespace>/<repo_name>.

--iters <iters>

Number of iterations to train LoRA.

--local

Whether or not model_dir is remote from HuggingFace.

-sq, --skip-quantize

Whether to skip quantization while converting to MLX. This parameter will be ignored if –gguf-model-path and –tokenizer-dir are specified.

--num-epochs <num_epochs>

Required The number of times the training data is passed through the training algorithm. Please note that this value is used on Linux platforms only.

--device <device>

PyTorch device for Linux training. Use ‘cuda’ for NVidia CUDA / AMD ROCm GPU, to use specific GPU, set visible GPU before run train command.

Default:

'cpu'

Options:

cpu | cuda | hpu

--max-seq-len <max_seq_len>

Required maximum length, in tokens, of a single sample.

--max-batch-len <max_batch_len>

Required maximum overall length of samples processed in a given batch.

--effective-batch-size <effective_batch_size>

Required total batch size across all GPUs

--save-samples <save_samples>

Required The number of samples processed in between checkpoints.

--learning-rate <learning_rate>

Required learning rate for training

--warmup-steps <warmup_steps>

Required warmup steps for training

--deepspeed-cpu-offload-optimizer <deepspeed_cpu_offload_optimizer>

Required if true enables optimizer offload

--deepspeed-cpu-offload-optimizer-ratio <deepspeed_cpu_offload_optimizer_ratio>

Required cpu offload optimizer ratio

--deepspeed-cpu-offload-optimizer-pin-memory <deepspeed_cpu_offload_optimizer_pin_memory>

Required if true pin memory when using cpu optimizer

--lora-rank <lora_rank>

Required rank of update matricies

--lora-alpha <lora_alpha>

Required how influential/strong lora tune will be

--lora-dropout <lora_dropout>

Required dropout for LoRA layers

--lora-target-modules <lora_target_modules>

LoRA modules to use

--lora-quantize-dtype <lora_quantize_dtype>

quantization data type to use when training a LoRA.

--is-padding-free <is_padding_free>

whether or not we are training a padding free transformer.

--gpus <nproc_per_node>

this is the number of GPUs to use. This is a torch specific arg and must be called nproc-per-node

--nnodes <nnodes>

Required number of machines in the training pool.

--node-rank <node_rank>

Required the rank of this machine in the training group.

--rdzv-id <rdzv_id>

Required this is the training group ID. So, if there are multiple matching endpoints, only the machines with matching IDs can connect.

--rdzv-endpoint <rdzv_endpoint>

Required this is the rendezvous endpoint which other torchrun jobs will join on.

--legacy

if true, enables the legacy linux training codepath from release 0.17.0 and prior.

system

Command group for all system-related command calls

ilab system [OPTIONS] COMMAND [ARGS]...

info

Print system information

ilab system info [OPTIONS]

taxonomy

Command Group for Interacting with the Taxonomy of InstructLab.

If this is your first time running ilab, it’s best to start with ilab config init to create the environment.

ilab taxonomy [OPTIONS] COMMAND [ARGS]...

diff

Lists taxonomy files that have changed since <taxonomy-base> and checks that taxonomy is valid. Similar to ‘git diff <ref>’.

ilab taxonomy diff [OPTIONS]

Options

--taxonomy-path <taxonomy_path>

Path to where the taxonomy is stored locally.

Default:

'Default taxonomy location in the instructlab data directory.'

--taxonomy-base <taxonomy_base>

Base git-ref to use for taxonomy.

--yaml-rules <yaml_rules>

Custom rules file for YAML linting.

--quiet

Suppress all output. Call returns 0 if check passes, 1 otherwise.