cli.shared.utils package

cli.shared.utils.attack_mappings module

cli.shared.utils.config module

This module contains utility functions for working with configuration data through the CLI.

cli.shared.utils.config.attack_config_check(overrides: Dict)

Checks the configuration for the attack command.

Parameters:

overrides (Dict) – The overrides provided through the CLI.

Raises:

  • ValueError – If the dataset name is not ‘custom’ when specifying a custom data directory.

  • ValueError – If the dataset name is ‘custom’ when not specifying a custom data directory.

cli.shared.utils.config.build_config(config_data: Dict, config_type: Type[T]) → T

Build a configuration object from the provided configuration data.

Parameters: - config_data (Dict): The configuration data. - config_type (Type[T]): The type of configuration object to build.

Returns: - T: The configuration object of the specified type.

cli.shared.utils.config.deep_update(source: Dict, overrides: Dict) → Dict

Recursively update a dictionary by overriding its keys.

Parameters: - source (Dict): The original dictionary with the base values. - overrides (Dict): The dictionary with overrides, where the keys might be nested.

Returns: - Dict: The updated dictionary.

cli.shared.utils.config.generate_default_config_yaml(config_name: str, output_path: str, save=False, config_subdir=None) → dict

Generate a default configuration YAML based on the name of the configuration.

Parameters:

  • config_name (str) – The name of the configuration yml file (e.g., “cw_attack_config.yml”).

  • output_path (str) – The path where the YAML file should be saved.

Returns:

The default configuration YAML as a dictionary.

Return type:

dict

cli.shared.utils.config.get_available_configs() → list

Get a list of all available configuration settings, including those in subdirectories, based on attributes defined in each module’s __init__.py file.

Parameters:

config_path (str) – The base directory path where the configuration files are stored.

Returns:

A list of dictionaries with the configuration settings including custom title, description, and config file name, filtered by the INCLUDE_IN_CLI_CONFIGS attribute.

Return type:

list

Examples

>>> from advsecurenet.utils.config_utils import get_available_configs
>>> get_available_configs('/path/to/configs')
[
    {'title': 'Example Configuration Module', 'description': 'Configuration for attack types.', 'config_file': 'lots_attack_config.yml'},
    {'title': 'Example Configuration Module', 'description': 'Configuration for defense mechanisms.', 'config_file': 'cw_attack_config.yml'}
]
cli.shared.utils.config.get_default_config_yml(config_name: str, config_subdir: str | None = None) → str

Get the default configuration YAML based on the name of the configuration.

Parameters:

  • config_name (str) – The name of the configuration (e.g., “cw_attack”).

  • config_subdir (str, optional) – The sub-directory to search in. Defaults to None.

Returns:

The path to the configuration YAML file.

Return type:

str

cli.shared.utils.config.is_path_to_update(key: str, value: str, base_path: str) → bool

Determine if the path needs to be updated to an absolute path. :param key: The key in the configuration dictionary. :type key: str :param value: The value to check. :type value: str :param base_path: The base directory path. :type base_path: str

Returns:

True if the path should be updated, False otherwise.

Return type:

bool

cli.shared.utils.config.load_and_instantiate_config(config: str, default_config_file: str, config_type: ConfigType, config_class: Type[T], **kwargs: Dict[str, Any]) → T

Utility function to load and instantiate configuration.

Parameters:

  • config (str) – The path to the configuration file.

  • default_config_file (str) – The default configuration file name.

  • config_type (ConfigType) – The type of configuration.

  • config_class (Type[T]) – The dataclass type for configuration instantiation.

  • **kwargs (Dict[str, Any]) – Additional keyword arguments. If provided, they will override the configuration.

Returns:

An instantiated configuration data class of the type specified by config_class.

Return type:

T

cli.shared.utils.config.load_configuration(config_type: ConfigType, config_file: str, **overrides: Dict) → Dict

Loads and overrides the configuration.

Parameters: - config_type (ConfigType): The type of configuration. - config_file (str): The path to the configuration file. - **overrides (Dict): The overrides to apply to the configuration.

Returns: - Dict: The configuration data.

cli.shared.utils.config.make_paths_absolute(base_path: str, config: Dict[str, Any] | List[Any]) → None

Recursively update the paths in the configuration dictionary or list to be absolute paths. :param base_path: The base directory path. :type base_path: str :param config: The configuration data to update. :type config: Union[Dict[str, Any], List[Any]]

cli.shared.utils.config.read_yml_file(file_path: str) → dict

Read a YAML file and return the data as a dictionary.

Parameters:

file_path (str) – The path to the YAML file.

Returns:

The data from the YAML file as a dictionary.

Return type:

dict

cli.shared.utils.config.update_dict_paths(base_path: str, config: Dict[str, Any]) → None

Update paths in a dictionary configuration. :param base_path: The base directory path. :type base_path: str :param config: The dictionary configuration data to update. :type config: Dict[str, Any]

cli.shared.utils.config.update_list_paths(base_path: str, config: List[Any]) → None

Update paths in a list configuration. :param base_path: The base directory path. :type base_path: str :param config: The list configuration data to update. :type config: List[Any]

cli.shared.utils.dataloader module

cli.shared.utils.dataloader.get_dataloader(config: ~cli.shared.types.utils.dataloader.DataLoaderCliConfigType, dataset: ~advsecurenet.datasets.base_dataset.BaseDataset, dataset_type: str | None = 'default', use_ddp: bool | None = False, sampler: ~torch.utils.data.distributed.DistributedSampler | None = <class 'torch.utils.data.distributed.DistributedSampler'>) → DataLoader

Get the dataloader based on the provided configuration and dataset type.

Parameters:

  • config (DataLoaderCliConfigType) – The dataloader configuration object.

  • dataset_type (Optional[str]) – The type of the dataset (‘train’, ‘test’, or other).

  • dataset (Optional[torch.utils.data.Dataset]) – The dataset to be loaded.

  • use_ddp (Optional[bool]) – Whether to use Distributed Data Parallel.

Returns:

The configured DataLoader instance.

Return type:

torch.utils.data.DataLoader

cli.shared.utils.dataset module

cli.shared.utils.dataset.get_datasets(config: DatasetCliConfigType, **kwargs) → Tuple[Dataset | None, Dataset | None]

Load the datasets conditionally based on provided paths.

Parameters:

  • config (DatasetCliConfigType) – Configuration for datasets.

  • **kwargs – Arbitrary keyword arguments for the dataset.

Returns:

Tuple containing the training dataset (if requested) and the testing dataset (if requested).

Return type:

Tuple[Optional[TorchDataset], Optional[TorchDataset]]

cli.shared.utils.helpers module

This helpers module contans more general helper functions that are not specific to a command type.

cli.shared.utils.helpers.get_device_from_cfg(config) → device

Returns the device to use from the config. If the device is not specified or is invalid, the device is set to “cpu”.

Parameters:

config (any) – The config object to use.

cli.shared.utils.helpers.read_data_from_file(file_path: str, cast_type: ~typing.Type = <class 'str'>, return_type: ~typing.Type = <class 'list'>, separator: str = '/n') → List[Any] | set | tuple | Tensor

Reads data from a file and returns it in the specified format. The function supports text, CSV, and JSON files.

Parameters:

  • file_path (str) – The path to the file.

  • cast_type (Type) – The type to cast the items to. Default is str.

  • return_type (Type) – The type of collection to return. Default is list. Other options are set and tuple.

  • separator (str) – The delimiter to use for text and CSV files. Default is ‘/n’ (newline).

Returns:

The data read from the file in the specified format.

Return type:

Union[List[Any], set, tuple, torch.Tensor]

cli.shared.utils.helpers.save_images(images: Tensor | List[Tensor], path: str | None = None, prefix: str = 'image') → None

Save each image tensor in a batch or a list of batches to the given path. If no path is provided, the images are saved to the current directory.

Parameters:

  • images (torch.Tensor or list[torch.Tensor]) – A tensor of images or a list of tensors, where each tensor is a batch of images.

  • path (str) – The path to save the images to. If None, the images are saved to the current directory.

  • prefix (str) – The prefix to add to the image name.

cli.shared.utils.helpers.to_bchw_format(tensor)

Converts a tensor from BHWC (batch, height, width, channels) format to BCHW (batch, channels, height, width) format.

Parameters:

tensor (torch.Tensor) – Input tensor.

Returns:

Tensor in BCHW format.

Return type:

torch.Tensor

Raises:

ValueError – If the tensor dimensions do not match expected BHWC or BCHW formats for RGB or grayscale images.

Examples

>>> tensor = torch.randn(10, 32, 32, 3)
>>> tensor.shape
torch.Size([10, 32, 32, 3])
>>> tensor = to_bchw_format(tensor)
>>> tensor.shape
torch.Size([10, 3, 32, 32])

cli.shared.utils.model module

cli.shared.utils.model.create_model(config: ModelCliConfigType) → BaseModel

Creates a model based on the provided configuration. :param config: The model configuration. :type config: ModelCliConfigType

Returns:

The created model.

Return type:

BaseModel