servicex package#

Subpackages#

Submodules#

servicex.configuration module#

class servicex.configuration.Configuration(*, api_endpoints: List[Endpoint], default_endpoint: str | None = None, cache_path: str | None = None, shortened_downloaded_filename: bool | None = False)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

api_endpoints: List[Endpoint]#

cache_path: str | None#

default_endpoint: str | None#

endpoint_dict() → Dict[str, Endpoint][source]#

expand_cache_path()[source]#: Expand the cache path to a full path, and create it if it doesn’t exist. Expand ${USER} to be the user name on the system. Works for windows, too. :param v: :return:

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'populate_by_name': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'api_endpoints': FieldInfo(annotation=List[Endpoint], required=True), 'cache_path': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, alias_priority=2, validation_alias=AliasChoices(choices=['cache-path', 'cache_path'])), 'default_endpoint': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, alias='default-endpoint', alias_priority=2), 'shortened_downloaded_filename': FieldInfo(annotation=Union[bool, NoneType], required=False, default=False)}#

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

classmethod read(config_path: str | None = None)[source]#

Read configuration from .servicex or servicex.yaml file. :param config_path: If provided, use this as the path to the .servicex file.

Otherwise, search, starting from the current working directory and look in enclosing directories

Returns:: Populated configuration object

shortened_downloaded_filename: bool | None#

class servicex.configuration.Endpoint(*, endpoint: str, name: str, token: str | None = '')[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

endpoint: str#

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'endpoint': FieldInfo(annotation=str, required=True), 'name': FieldInfo(annotation=str, required=True), 'token': FieldInfo(annotation=Union[str, NoneType], required=False, default='')}#

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

name: str#

token: str | None#

servicex.databinder_models module#

class servicex.databinder_models.General(*, Codegen: str | None = None, OutputFormat: OutputFormatEnum = OutputFormatEnum.root_ttree, Delivery: DeliveryEnum = DeliveryEnum.LocalCache, OutputDirectory: str | None = None, OutFilesetName: str = 'servicex_fileset')[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Codegen: str | None#

Delivery: DeliveryEnum#

class DeliveryEnum(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

LocalCache = 'LocalCache'#

URLs = 'URLs'#

OutFilesetName: str#

OutputDirectory: str | None#

OutputFormat: OutputFormatEnum#

class OutputFormatEnum(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

parquet = 'parquet'#

root_ttree = 'root-ttree'#

to_ResultFormat() → ResultFormat[source]#: This method is used to convert the OutputFormatEnum enum to the ResultFormat enum, which is what is actually used for the TransformRequest. This allows us to use different string values in the two enum classes to maintain backend compatibility

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'Codegen': FieldInfo(annotation=Union[str, NoneType], required=False, default=None), 'Delivery': FieldInfo(annotation=General.DeliveryEnum, required=False, default=<DeliveryEnum.LocalCache: 'LocalCache'>), 'OutFilesetName': FieldInfo(annotation=str, required=False, default='servicex_fileset'), 'OutputDirectory': FieldInfo(annotation=Union[str, NoneType], required=False, default=None), 'OutputFormat': FieldInfo(annotation=General.OutputFormatEnum, required=False, default=<OutputFormatEnum.root_ttree: 'root-ttree'>)}#

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Codegen: str | None#

Dataset: DataSetIdentifier | None#

IgnoreLocalCache: bool#

NFiles: int | None#

Name: str#

Query: str | QueryStringGenerator | None#

RucioDID: str | None#

XRootDFiles: str | List[str] | None#

property dataset_identifier: DataSetIdentifier#

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'Codegen': FieldInfo(annotation=Union[str, NoneType], required=False, default=None), 'Dataset': FieldInfo(annotation=Union[DataSetIdentifier, NoneType], required=False, default=None), 'IgnoreLocalCache': FieldInfo(annotation=bool, required=False, default=False), 'NFiles': FieldInfo(annotation=Union[int, NoneType], required=False, default=None), 'Name': FieldInfo(annotation=str, required=True), 'Query': FieldInfo(annotation=Union[str, QueryStringGenerator, NoneType], required=False, default=None), 'RucioDID': FieldInfo(annotation=Union[str, NoneType], required=False, default=None), 'XRootDFiles': FieldInfo(annotation=Union[str, List[str], NoneType], required=False, default=None)}#

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

classmethod truncate_long_sample_name(values)[source]#: Truncate sample name to 128 characters if exceed Print warning message

classmethod validate_did_xor_file(values)[source]#: Ensure that only one of Dataset, RootFile, or RucioDID is specified. :param values: :return:

class servicex.databinder_models.ServiceXSpec(*, General: ~servicex.databinder_models.General = General(Codegen=None, OutputFormat=<OutputFormatEnum.root_ttree: 'root-ttree'>, Delivery=<DeliveryEnum.LocalCache: 'LocalCache'>, OutputDirectory=None, OutFilesetName='servicex_fileset'), Sample: ~typing.List[~servicex.databinder_models.Sample], Definition: ~typing.List | None = None)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Definition: List | None#

General: General#

Sample: List[Sample]#

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'Definition': FieldInfo(annotation=Union[List, NoneType], required=False, default=None), 'General': FieldInfo(annotation=General, required=False, default=General(Codegen=None, OutputFormat=<OutputFormatEnum.root_ttree: 'root-ttree'>, Delivery=<DeliveryEnum.LocalCache: 'LocalCache'>, OutputDirectory=None, OutFilesetName='servicex_fileset')), 'Sample': FieldInfo(annotation=List[Sample], required=True)}#

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

servicex.dataset_group module#

class servicex.dataset_group.DatasetGroup(datasets: List[Query])[source]#

Bases: object

A group of datasets that are to be transformed together. This is a convenience class to allow you to submit multiple datasets to a ServiceX instance and then wait for all of them to complete.

Parameters:: datasets – List of transform request as dataset instances

as_files(display_progress: bool = True, provided_progress: Progress | None = None, return_exceptions: bool = False) → List[TransformedResults | BaseException]#

async as_files_async(display_progress: bool = True, provided_progress: Progress | None = None, return_exceptions: bool = False) → List[TransformedResults | BaseException][source]#

as_signed_urls(display_progress: bool = True, provided_progress: Progress | None = None, return_exceptions: bool = False) → List[TransformedResults | BaseException]#

async as_signed_urls_async(display_progress: bool = True, provided_progress: Progress | None = None, return_exceptions: bool = False) → List[TransformedResults | BaseException][source]#

set_result_format(result_format: ResultFormat)[source]#

Set the result format for all the datasets in the group.

Parameters:: result_format – ResultFormat instance

servicex.dataset_identifier module#

class servicex.dataset_identifier.CERNOpenDataDatasetIdentifier(dataset: int, num_files: int | None = None)[source]#

Bases: DataSetIdentifier

CERN Open Data Dataset - this will be looked up using the CERN Open Data DID finder.

Parameters:

dataset – The dataset ID - this is an integer.
num_files – Maximum number of files to return. This is useful during development to perform quick runs. ServiceX is careful to make sure it always returns the same subset of files.

classmethod from_yaml(_, node)[source]#

yaml_tag = '!CERNOpenData'#

class servicex.dataset_identifier.DataSetIdentifier(scheme: str, dataset: str, num_files: int | None = None)[source]#

Bases: object

Base class for specifying the dataset to transform. This can either be a list of xRootD URIs or a rucio DID

property did#

populate_transform_request(transform_request: TransformRequest) → None[source]#

class servicex.dataset_identifier.FileListDataset(files: List[str] | str)[source]#

Bases: DataSetIdentifier

Dataset specified as a list of XRootD URIs.

Parameters:: files – Either a list of URIs or a single URI string

property did#

files: List[str]#

classmethod from_yaml(constructor, node)[source]#

populate_transform_request(transform_request: TransformRequest) → None[source]#

yaml_tag = '!FileList'#

class servicex.dataset_identifier.RucioDatasetIdentifier(dataset: str, num_files: int | None = None)[source]#

Bases: DataSetIdentifier

Rucio Dataset - this will be looked up using the Rucio data management service.

Parameters:

dataset – The rucio DID - this can be a dataset or a container of datasets.
num_files – Maximum number of files to return. This is useful during development to perform quick runs. ServiceX is careful to make sure it always returns the same subset of files.

classmethod from_yaml(_, node)[source]#

yaml_tag = '!Rucio'#

servicex.expandable_progress module#

class servicex.expandable_progress.ExpandableProgress(display_progress: bool = True, provided_progress: Progress | ExpandableProgress | None = None, overall_progress: bool = False)[source]#

Bases: object

We want to be able to use rich progress bars in the async code, but there are some situtations where the user doesn’t want them. Also we might be running several simultaneous progress bars, and we want to be able to control that.

We still want to keep the context manager interface, so this class implements the context manager but if display_progress is False, then it does nothing. If provided_progress is set then we just use that. Otherwise we create a new progress bar

Parameters:

display_progress –
provided_progress –

add_task(param, start, total)[source]#

advance(task_id, task_type)[source]#

start_task(task_id, task_type)[source]#

update(task_id, task_type, total=None, completed=None, **fields)[source]#

class servicex.expandable_progress.ProgressCounts(description: str, task_id: TaskID, start: int | None = None, total: int | None = None, completed: int | None = None)[source]#: Bases: object

class servicex.expandable_progress.TranformStatusProgress(*columns: str | ProgressColumn, console: Console | None = None, auto_refresh: bool = True, refresh_per_second: float = 10, speed_estimate_period: float = 30.0, transient: bool = False, redirect_stdout: bool = True, redirect_stderr: bool = True, get_time: Callable[[], float] | None = None, disable: bool = False, expand: bool = False)[source]#

Bases: Progress

get_renderables()[source]#: Get a number of renderables for the progress display.

servicex.minio_adapter module#

class servicex.minio_adapter.MinioAdapter(endpoint_host: str, secure: bool, access_key: str, secret_key: str, bucket: str)[source]#

Bases: object

MAX_PATH_LEN = 60#

async download_file(object_name: str, local_dir: str, shorten_filename: bool = False) → Path[source]#

classmethod for_transform(transform: TransformStatus)[source]#

async get_signed_url(object_name: str) → str[source]#

classmethod hash_path(file_name)[source]#: Make the path safe for object store or POSIX, by keeping the length less than MAX_PATH_LEN. Replace the leading (less interesting) characters with a forty character hash. :param file_name: Input filename :return: Safe path string

async list_bucket() → List[ResultFile][source]#

servicex.models module#

class servicex.models.ResultDestination(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

Direct the output to object store or posix volume

object_store = 'object-store'#

volume = 'volume'#

class servicex.models.ResultFile(*, filename: str, size: int, extension: str)[source]#

Bases: BaseModel

Record reporting the properties of a transformed file result

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

extension: str#

filename: str#

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'extension': FieldInfo(annotation=str, required=True), 'filename': FieldInfo(annotation=str, required=True), 'size': FieldInfo(annotation=int, required=True)}#

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

size: int#

class servicex.models.ResultFormat(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

Specify the file format for the generated output

parquet = 'parquet'#

root_ttree = 'root-file'#

class servicex.models.Status(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

Status of a submitted transform

canceled = 'Canceled'#

complete = 'Complete'#

fatal = 'Fatal'#

looking = 'Lookup'#

pending = 'Pending Lookup'#

running = 'Running'#

submitted = 'Submitted'#

class servicex.models.TransformRequest(*, title: str | None = None, did: str | None = None, file_list: List[str] | None = None, selection: str, image: str | None = None, codegen: str, tree_name: str | None = None, result_destination: ResultDestination, result_format: ResultFormat)[source]#

Bases: BaseModel

Transform request sent to ServiceX

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

codegen: str#

compute_hash()[source]#

Compute a hash for this submission. Only include properties that impact the result so we have maximal ability to reuse transforms

Returns:: SHA256 hash of request

did: str | None#

file_list: List[str] | None#

image: str | None#

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'populate_by_name': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'codegen': FieldInfo(annotation=str, required=True), 'did': FieldInfo(annotation=Union[str, NoneType], required=False, default=None), 'file_list': FieldInfo(annotation=Union[List[str], NoneType], required=False, default=None, alias='file-list', alias_priority=2), 'image': FieldInfo(annotation=Union[str, NoneType], required=False, default=None), 'result_destination': FieldInfo(annotation=ResultDestination, required=True, alias_priority=2, serialization_alias='result-destination'), 'result_format': FieldInfo(annotation=ResultFormat, required=True, alias_priority=2, serialization_alias='result-format'), 'selection': FieldInfo(annotation=str, required=True), 'title': FieldInfo(annotation=Union[str, NoneType], required=False, default=None), 'tree_name': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, alias='tree-name', alias_priority=2)}#

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

result_destination: ResultDestination#

result_format: ResultFormat#

selection: str#

title: str | None#

tree_name: str | None#

class servicex.models.TransformStatus(*, request_id: str, did: str, title: str | None = None, selection: str, tree_name: str | None, image: str, result_destination: ResultDestination, result_format: ResultFormat, generated_code_cm: str, status: Status, app_version: str, files: int, files_completed: int, files_failed: int, files_remaining: int | None = 0, submit_time: datetime = None, finish_time: datetime | None = None, minio_endpoint: str | None = None, minio_secured: bool | None = None, minio_access_key: str | None = None, minio_secret_key: str | None = None, log_url: str | None = None)[source]#

Bases: BaseModel

Status object returned by servicex

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

app_version: str#

did: str#

files: int#

files_completed: int#

files_failed: int#

files_remaining: int | None#

finish_time: datetime | None#

generated_code_cm: str#

image: str#

log_url: str | None#

minio_access_key: str | None#

minio_endpoint: str | None#

minio_secret_key: str | None#

minio_secured: bool | None#

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'app_version': FieldInfo(annotation=str, required=True, alias_priority=2, validation_alias='app-version'), 'did': FieldInfo(annotation=str, required=True), 'files': FieldInfo(annotation=int, required=True), 'files_completed': FieldInfo(annotation=int, required=True, alias_priority=2, validation_alias='files-completed'), 'files_failed': FieldInfo(annotation=int, required=True, alias_priority=2, validation_alias='files-failed'), 'files_remaining': FieldInfo(annotation=Union[int, NoneType], required=False, default=0, alias_priority=2, validation_alias='files-remaining'), 'finish_time': FieldInfo(annotation=Union[datetime, NoneType], required=False, default=None, alias_priority=2, validation_alias='finish-time'), 'generated_code_cm': FieldInfo(annotation=str, required=True, alias_priority=2, validation_alias='generated-code-cm'), 'image': FieldInfo(annotation=str, required=True), 'log_url': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, alias_priority=2, validation_alias='log-url'), 'minio_access_key': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, alias_priority=2, validation_alias='minio-access-key'), 'minio_endpoint': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, alias_priority=2, validation_alias='minio-endpoint'), 'minio_secret_key': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, alias_priority=2, validation_alias='minio-secret-key'), 'minio_secured': FieldInfo(annotation=Union[bool, NoneType], required=False, default=None, alias_priority=2, validation_alias='minio-secured'), 'request_id': FieldInfo(annotation=str, required=True), 'result_destination': FieldInfo(annotation=ResultDestination, required=True, alias_priority=2, validation_alias='result-destination'), 'result_format': FieldInfo(annotation=ResultFormat, required=True, alias_priority=2, validation_alias='result-format'), 'selection': FieldInfo(annotation=str, required=True), 'status': FieldInfo(annotation=Status, required=True), 'submit_time': FieldInfo(annotation=datetime, required=False, default=None, alias_priority=2, validation_alias='submit-time'), 'title': FieldInfo(annotation=Union[str, NoneType], required=False, default=None), 'tree_name': FieldInfo(annotation=Union[str, NoneType], required=True, alias_priority=2, validation_alias='tree-name')}#

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

classmethod parse_finish_time(v)[source]#

request_id: str#

result_destination: ResultDestination#

result_format: ResultFormat#

selection: str#

status: Status#

submit_time: datetime#

title: str | None#

tree_name: str | None#

class servicex.models.TransformedResults(*, hash: str, title: str, codegen: str, request_id: str, submit_time: datetime, data_dir: str, file_list: List[str], signed_url_list: List[str], files: int, result_format: ResultFormat, log_url: str | None = None)[source]#

Bases: BaseModel

Returned for a submission. Gives you everything you need to know about a completed transform.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

codegen: str#

data_dir: str#

file_list: List[str]#

files: int#

hash: str#

log_url: str | None#

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'codegen': FieldInfo(annotation=str, required=True), 'data_dir': FieldInfo(annotation=str, required=True), 'file_list': FieldInfo(annotation=List[str], required=True), 'files': FieldInfo(annotation=int, required=True), 'hash': FieldInfo(annotation=str, required=True), 'log_url': FieldInfo(annotation=Union[str, NoneType], required=False, default=None), 'request_id': FieldInfo(annotation=str, required=True), 'result_format': FieldInfo(annotation=ResultFormat, required=True), 'signed_url_list': FieldInfo(annotation=List[str], required=True), 'submit_time': FieldInfo(annotation=datetime, required=True), 'title': FieldInfo(annotation=str, required=True)}#

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

request_id: str#

result_format: ResultFormat#

signed_url_list: List[str]#

submit_time: datetime#

title: str#

servicex.python_dataset module#

class servicex.python_dataset.PythonFunction(python_function: str | Callable | None = None)[source]#

Bases: QueryStringGenerator

default_codegen: str | None = 'python'#

classmethod from_yaml(_, node)[source]#

generate_selection_string() → str[source]#: override with the selection string to send to ServiceX

with_uproot_function(f: str | Callable) → Self[source]#

yaml_tag = '!PythonFunction'#

servicex.query module#

servicex.query_cache module#

exception servicex.query_cache.CacheException[source]#: Bases: Exception

class servicex.query_cache.QueryCache(config: Configuration)[source]#

Bases: object

cache_path_for_transform(transform_status: TransformStatus) → Path[source]#

cache_transform(record: TransformedResults)[source]#

cached_queries() → List[TransformedResults][source]#

close()[source]#

delete_codegen_by_backend(backend: str)[source]#

delete_record_by_request_id(request_id: str)[source]#

get_codegen_by_backend(backend: str) → dict | None[source]#

get_transform_by_hash(hash: str) → TransformedResults | None[source]#

get_transform_by_request_id(request_id: str) → TransformedResults | None[source]#

transformed_results(transform: TransformRequest, completed_status: TransformStatus, data_dir: str, file_list: List[str], signed_urls) → TransformedResults[source]#

update_codegen_by_backend(backend: str, codegen_list: list)[source]#

update_record(record: TransformedResults)[source]#

servicex.servicex_adapter module#

exception servicex.servicex_adapter.AuthorizationError[source]#: Bases: BaseException

class servicex.servicex_adapter.ServiceXAdapter(url: str, refresh_token: str | None = None)[source]#

Bases: object

get_code_generators()[source]#

async get_transform_status(request_id: str) → TransformStatus[source]#

async get_transforms() → List[TransformStatus][source]#

async submit_transform(transform_request: TransformRequest)[source]#

servicex.servicex_client module#

class servicex.servicex_client.GuardList(data: Sequence | Exception)[source]#

Bases: Sequence

valid() → bool[source]#

exception servicex.servicex_client.ReturnValueException(exc)[source]#

Bases: Exception

An exception occurred at some point while obtaining this result from ServiceX

class servicex.servicex_client.ServiceXClient(backend=None, url=None, config_path=None)[source]#

Bases: object

Connection to a ServiceX deployment. Instances of this class can deployment data from the service and also interact with previously run transformations. Instances of this class are factories for Datasets`

If both backend and url are unspecified then it will attempt to pick up the default backend from .servicex

Parameters:

backend – Name of a deployment from the .servicex file
url – Direct URL of a serviceX deployment instead of using .servicex. Can only work with hosts without auth, or the token is found in a file pointed to by the environment variable BEARER_TOKEN_FILE
config_path – Optional path te the .servicex file. If not specified, will search in local directory and up in enclosing directories

generic_query(dataset_identifier: DataSetIdentifier | FileListDataset, query: str | QueryStringGenerator, codegen: str | None = None, title: str = 'ServiceX Client', result_format: ResultFormat = ResultFormat.parquet, ignore_cache: bool = False) → Query[source]#

Generate a Query object for a generic codegen specification

Parameters:

dataset_identifier – The dataset identifier or filelist to be the source of files
title – Title to be applied to the transform. This is also useful for relating transform results.
codegen – Name of the code generator to use with this transform
result_format – Do you want Paqrquet or Root? This can be set later with the set_result_format method
ignore_cache – Ignore the query cache and always run the query

Returns:

A Query object

get_code_generators(backend=None)[source]#: Retrieve the code generators deployed with the serviceX instance :return: The list of code generators as json dictionary

get_transform_status(transform_id) → TransformStatus#: Get the status of a given transform :param transform_id: The uuid of the transform :return: The current status for the transform

async get_transform_status_async(transform_id) → TransformStatus[source]#: Get the status of a given transform :param transform_id: The uuid of the transform :return: The current status for the transform

get_transforms() → List[TransformStatus]#: Retrieve all transforms you have run on the server :return: List of Transform status objects

async get_transforms_async() → List[TransformStatus][source]#: Retrieve all transforms you have run on the server :return: List of Transform status objects

servicex.servicex_client.deliver(config: ServiceXSpec | Mapping[str, Any] | str | Path, config_path: str | None = None, servicex_name: str | None = None, return_exceptions: bool = True)[source]#

servicex.types module#

Module contents#

class servicex.General(*, Codegen: str | None = None, OutputFormat: OutputFormatEnum = OutputFormatEnum.root_ttree, Delivery: DeliveryEnum = DeliveryEnum.LocalCache, OutputDirectory: str | None = None, OutFilesetName: str = 'servicex_fileset')[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Codegen: str | None#

Delivery: DeliveryEnum#

class DeliveryEnum(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

LocalCache = 'LocalCache'#

URLs = 'URLs'#

OutFilesetName: str#

OutputDirectory: str | None#

OutputFormat: OutputFormatEnum#

class OutputFormatEnum(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

parquet = 'parquet'#

root_ttree = 'root-ttree'#

to_ResultFormat() → ResultFormat[source]#: This method is used to convert the OutputFormatEnum enum to the ResultFormat enum, which is what is actually used for the TransformRequest. This allows us to use different string values in the two enum classes to maintain backend compatibility

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

class servicex.ResultDestination(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

Direct the output to object store or posix volume

object_store = 'object-store'#

volume = 'volume'#

class servicex.ResultFormat(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: str, Enum

Specify the file format for the generated output

parquet = 'parquet'#

root_ttree = 'root-file'#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Codegen: str | None#

Dataset: DataSetIdentifier | None#

IgnoreLocalCache: bool#

NFiles: int | None#

Name: str#

Query: str | QueryStringGenerator | None#

RucioDID: str | None#

XRootDFiles: str | List[str] | None#

property dataset_identifier: DataSetIdentifier#

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

classmethod truncate_long_sample_name(values)[source]#: Truncate sample name to 128 characters if exceed Print warning message

classmethod validate_did_xor_file(values)[source]#: Ensure that only one of Dataset, RootFile, or RucioDID is specified. :param values: :return:

class servicex.ServiceXSpec(*, General: ~servicex.databinder_models.General = General(Codegen=None, OutputFormat=<OutputFormatEnum.root_ttree: 'root-ttree'>, Delivery=<DeliveryEnum.LocalCache: 'LocalCache'>, OutputDirectory=None, OutFilesetName='servicex_fileset'), Sample: ~typing.List[~servicex.databinder_models.Sample], Definition: ~typing.List | None = None)[source]#

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Definition: List | None#

General: General#

Sample: List[Sample]#

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

servicex.deliver(config: ServiceXSpec | Mapping[str, Any] | str | Path, config_path: str | None = None, servicex_name: str | None = None, return_exceptions: bool = True)[source]#