materializationengine
MaterializationClientV2(server_address, auth_header, api_version, endpoints, server_name, datastack_name, cg_client=None, synapse_table=None, version=None, verify=True, max_retries=None, pool_maxsize=None, pool_block=None, over_client=None, desired_resolution=None)
¶
Bases: ClientBase
cg_client
property
¶
The chunked graph client.
datastack_name
property
¶
The name of the datastack.
homepage: HTML
property
¶
The homepage for the materialization engine.
server_version: Optional[Version]
property
¶
The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
tables: TableManager
property
¶
The table manager for the materialization engine.
version: int
property
writable
¶
The version of the materialization. Can be used to set up the client to default to a specific version when timestamps or versions are not specified in queries. If not set, defaults to the most recent version.
Note that if this materialization client is attached to a CAVEclient, the version must be set at the CAVEclient level.
views: ViewManager
property
¶
The view manager for the materialization engine.
get_table_metadata(table_name, datastack_name=None, version=None, log_warning=True)
¶
Get metadata about a table
Parameters:
Name | Type | Description | Default |
---|---|---|---|
table_name |
str
|
name of table to mark for deletion |
required |
datastack_name |
str or None
|
Name of the datastack_name. If None, uses the one specified in the client. |
None
|
version |
int
|
The version of the datastack to query. If None, will query the client
|
None
|
log_warning |
bool
|
Whether to print out warnings to the logger. Defaults to True. |
True
|
Returns:
Type | Description |
---|---|
dict
|
Metadata dictionary for table |
get_tables(datastack_name=None, version=None)
¶
Gets a list of table names for a datastack
Parameters:
Name | Type | Description | Default |
---|---|---|---|
datastack_name |
str or None
|
Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client |
None
|
version |
int or None
|
The version of the datastack to query. If None, will query the client
|
None
|
Returns:
Type | Description |
---|---|
list
|
List of table names |
get_timestamp(version=None, datastack_name=None)
¶
Get datetime.datetime timestamp for a materialization version.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
version |
int or None
|
The version of the datastack to query. If None, will query the client
|
None
|
datastack_name |
str or None
|
Datastack name, by default None. If None, defaults to the value set in the client. |
None
|
Returns:
Type | Description |
---|---|
datetime
|
Datetime when the materialization version was frozen. |
get_version_metadata(version=None, datastack_name=None)
¶
Get metadata about a version
Parameters:
Name | Type | Description | Default |
---|---|---|---|
version |
int or None
|
The version of the datastack to query. If None, will query the client
|
None
|
datastack_name |
str or None
|
Datastack name, by default None. If None, defaults to the value set in the client. |
None
|
Returns:
Type | Description |
---|---|
dict
|
Dictionary of metadata about the version |
get_versions(datastack_name=None, expired=False)
¶
Get the versions available
Parameters:
Name | Type | Description | Default |
---|---|---|---|
datastack_name |
str or None
|
Name of the datastack, by default None. If None, uses the one specified in the client. |
None
|
expired |
bool
|
Whether to include expired versions, by default False. |
False
|
Returns:
Type | Description |
---|---|
dict
|
Dictionary of versions available |
get_versions_metadata(datastack_name=None, expired=False)
¶
Get the metadata for all the versions that are presently available and valid
Parameters:
Name | Type | Description | Default |
---|---|---|---|
datastack_name |
str or None
|
Datastack name, by default None. If None, defaults to the value set in the client. |
None
|
expired |
bool
|
Whether to include expired versions, by default False. |
False
|
Returns:
Type | Description |
---|---|
list[dict]
|
List of metadata dictionaries |
ingest_annotation_table(table_name, datastack_name=None)
¶
Trigger supervoxel lookup and root ID lookup of new annotations in a table.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
table_name |
str
|
Table to trigger |
required |
datastack_name |
str
|
Datastack to trigger it. Defaults to what is set in client. |
None
|
Returns:
Type | Description |
---|---|
dict
|
Status code of response from server |
join_query(tables, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, suffixes=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, desired_resolution=None, random_sample=None, log_warning=True)
¶
Generic query on materialization tables
Parameters:
Name | Type | Description | Default |
---|---|---|---|
tables |
list of lists with length 2 or 'str'
|
list of two lists: first entries are table names, second entries are the columns used for the join. |
required |
filter_in_dict |
dict of dicts
|
outer layer: keys are table names inner layer: keys are column names, values are allowed entries, by default None |
None
|
filter_out_dict |
dict of dicts
|
outer layer: keys are table names inner layer: keys are column names, values are not allowed entries, by default None |
None
|
filter_equal_dict |
dict of dicts
|
outer layer: keys are table names inner layer: keys are column names, values are specified entry, by default None |
None
|
filter_spatial_dict |
dict of dicts
|
outer layer: keys are table names, inner layer: keys are column names. Values are bounding boxes as [[min_x, min_y,min_z],[max_x, max_y, max_z]], expressed in units of the voxel_resolution of this dataset. Defaults to None. |
None
|
filter_regex_dict |
dict of dicts
|
outer layer: keys are table names. inner layer: keys are column names, values are regex strings. Defaults to None |
None
|
select_columns |
dict of lists of str
|
keys are table names,values are the list of columns from that table. Defaults to None, which will select all tables. Will be passed to server as select_column_maps. Passing a list will be passed as select_columns which is deprecated. |
None
|
offset |
int
|
result offset to use. Defaults to None. Will only return top K results. |
None
|
limit |
int
|
maximum results to return (server will set upper limit, see get_server_config) |
None
|
suffixes |
dict
|
suffixes to use for duplicate columns, keys are table names, values are the suffix |
None
|
datastack_name |
str
|
datastack to query. If None defaults to one specified in client. |
None
|
return_df |
bool
|
whether to return as a dataframe default True, if False, data is returned as json (slower) |
True
|
split_positions |
bool
|
whether to break position columns into x,y,z columns default False, if False data is returned as one column with [x,y,z] array (slower) |
False
|
materialization_version |
int
|
The version of the datastack to query. If None, will query the client
|
None
|
metadata |
bool
|
toggle to return metadata If True (and return_df is also True), return table and query metadata in the df.attr dictionary. |
True
|
desired_resolution |
Iterable
|
What resolution to convert position columns to. Defaults to None will use defaults. |
None
|
random_sample |
int
|
if given, will do a tablesample of the table to return that many annotations |
None
|
log_warning |
bool
|
Whether to log warnings, by default True |
True
|
Returns:
Type | Description |
---|---|
DataFrame
|
a pandas dataframe of results of query |
live_live_query(table, timestamp, joins=None, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_spatial_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, metadata=True, suffixes=None, desired_resolution=None, allow_missing_lookups=False, random_sample=None, log_warning=True)
¶
Beta method for querying cave annotation tables with rootIDs and annotations at a particular timestamp. Note: this method requires more explicit mapping of filters and selection to table as its designed to test a more general endpoint that should eventually support complex joins.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
table |
str
|
Principle table to query |
required |
timestamp |
datetime
|
Timestamp to query |
required |
joins |
List of joins, where each join is a list of [table1,column1, table2, column2] |
None
|
|
filter_in_dict |
A dictionary with tables as keys, values are dicts with column keys and list values to accept. |
None
|
|
filter_out_dict |
A dictionary with tables as keys, values are dicts with column keys and list values to reject. |
None
|
|
filter_equal_dict |
A dictionary with tables as keys, values are dicts with column keys and values to equate. |
None
|
|
filter_spatial_dict |
A dictionary with tables as keys, values are dicts with column keys and values of 2x3 list of bounds. |
None
|
|
select_columns |
A dictionary with tables as keys, values are lists of columns to select. |
None
|
|
offset |
int
|
Value to offset query by. |
None
|
limit |
int
|
Limit of query. |
None
|
datastack_name |
str
|
Datastack to query. Defaults to set by client. |
None
|
split_positions |
bool
|
Whether to split positions into separate columns, True is faster. |
False
|
metadata |
bool
|
Whether to attach metadata to dataframe. |
True
|
suffixes |
dict
|
What suffixes to use on joins, keys are table_names, values are suffixes. |
None
|
desired_resolution |
Iterable
|
What resolution to convert position columns to. |
None
|
allow_missing_lookups |
bool
|
If there are annotations without supervoxels and root IDs yet, allow results. |
False
|
random_sample |
int
|
If given, will do a table sample of the table to return that many annotations. |
None
|
log_warning |
bool
|
Whether to log warnings. |
True
|
Returns:
Type | Description |
---|---|
Results of query |
Examples:
>>> from caveclient import CAVEclient
>>> client = CAVEclient('minnie65_public_v117')
>>> live_live_query("table_name", datetime.datetime.now(datetime.timezone.utc),
>>> joins=[[table_name, table_column, joined_table, joined_column],
>>> [joined_table, joincol2, third_table, joincol_third]]
>>> suffixes={
>>> "table_name":"suffix1",
>>> "joined_table":"suffix2",
>>> "third_table":"suffix3"
>>> },
>>> select_columns= {
>>> "table_name":[ "column","names"],
>>> "joined_table":["joined_colum"]
>>> },
>>> filter_in_dict= {
>>> "table_name":{
>>> "column_name":[included,values]
>>> }
>>> },
>>> filter_out_dict= {
>>> "table_name":{
>>> "column_name":[excluded,values]
>>> }
>>> },
>>> filter_equal_dict"={
>>> "table_name":{
>>> "column_name":value
>>> },
>>> filter_spatial_dict"= {
>>> "table_name": {
>>> "column_name": [[min_x, min_y, min_z], [max_x, max_y, max_z]]
>>> }
>>> filter_regex_dict"= {
>>> "table_name": {
>>> "column_name": "regex_string"
>>> }
live_query(table, timestamp, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, post_filter=True, metadata=True, merge_reference=True, desired_resolution=None, random_sample=None, log_warning=True)
¶
Generic query on materialization tables
Parameters:
Name | Type | Description | Default |
---|---|---|---|
table |
str
|
Table to query |
required |
timestamp |
datetime
|
Time to materialize (in utc). Pass datetime.datetime.now(datetime.timezone.utc) for present time. |
required |
filter_in_dict |
dict
|
Keys are column names, values are allowed entries. |
None
|
filter_out_dict |
dict
|
Keys are column names, values are not allowed entries. |
None
|
filter_equal_dict |
dict
|
Keys are column names, values are specified entry. |
None
|
filter_spatial_dict |
dict
|
Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]]. |
None
|
filter_regex_dict |
dict
|
Keys are column names, values are regex strings. |
None
|
select_columns |
list of str
|
Columns to select. |
None
|
offset |
int
|
Offset in query result. |
None
|
limit |
int
|
Maximum results to return (server will set upper limit, see get_server_config). |
None
|
datastack_name |
str
|
Datastack to query. If None, defaults to one specified in client. |
None
|
split_positions |
bool
|
Whether to break position columns into x,y,z columns. If False data is returned as one column with [x,y,z] array (slower). |
False
|
post_filter |
bool
|
Whether to filter down the result based upon the filters specified. If False, it will return the query with present root_ids in the root_id columns, but the rows will reflect the filters translated into their past IDs. So if, for example, a cell had a false merger split off since the last materialization, those annotations on that incorrect portion of the cell will be included if this is False, but will be filtered down if this is True. |
True
|
metadata |
bool
|
Toggle to return metadata. If True (and return_df is also True), return table and query metadata in the df.attr dictionary. |
True
|
merge_reference |
bool
|
Toggle to automatically join reference table. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that table. |
True
|
desired_resolution |
Iterable
|
Desired resolution you want all spatial points returned in. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata. |
None
|
random_sample |
int
|
If given, will do a tablesample of the table to return that many annotations. |
None
|
log_warning |
bool
|
Whether to log warnings. |
True
|
Returns:
Type | Description |
---|---|
DataFrame
|
A pandas dataframe of results of query |
lookup_supervoxel_ids(table_name, annotation_ids=None, datastack_name=None)
¶
Trigger supervoxel lookups of new annotations in a table.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
table_name |
str
|
Table to trigger |
required |
annotation_ids |
list
|
List of annotation ids to lookup. Default is None, which will trigger lookup of entire table. |
None
|
datastack_name |
str
|
Datastack to trigger it. Defaults to what is set in client. |
None
|
Returns:
Type | Description |
---|---|
dict
|
Status code of response from server |
map_filters(filters, timestamp, timestamp_past)
¶
Translate a list of filter dictionaries from a point in the future to a point in the past
Parameters:
Name | Type | Description | Default |
---|---|---|---|
filters |
list[dict]
|
filter dictionaries with root_ids |
required |
timestamp |
datetime
|
timestamp to query |
required |
timestamp_past |
datetime
|
timestamp to query from |
required |
Returns:
Type | Description |
---|---|
list[dict]
|
filter dictionaries with past root_ids |
dict
|
mapping of future root_ids to past root_ids |
most_recent_version(datastack_name=None)
¶
Get the most recent version of materialization for this datastack name
Parameters:
Name | Type | Description | Default |
---|---|---|---|
datastack_name |
str or None
|
Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client |
None
|
Returns:
Type | Description |
---|---|
int
|
Most recent version of materialization for this datastack name |
query_table(table, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, timestamp=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None, log_warning=True)
¶
Generic query on materialization tables
Parameters:
Name | Type | Description | Default |
---|---|---|---|
table |
str
|
Table to query |
required |
filter_in_dict |
dict
|
Keys are column names, values are allowed entries, by default None |
None
|
filter_out_dict |
dict
|
Keys are column names, values are not allowed entries, by default None |
None
|
filter_equal_dict |
dict
|
Keys are column names, values are specified entry, by default None |
None
|
filter_spatial_dict |
dict
|
Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None |
None
|
filter_regex_dict |
dict
|
Keys are column names, values are regex strings, by default None |
None
|
select_columns |
list of str
|
Columns to select, by default None |
None
|
offset |
int
|
Result offset to use, by default None. Will only return top K results. |
None
|
limit |
int
|
Maximum results to return (server will set upper limit, see get_server_config), by default None |
None
|
datastack_name |
str
|
Datastack to query, by default None. If None, defaults to one specified in client. |
None
|
return_df |
bool
|
Whether to return as a dataframe, by default True. If False, data is returned as json (slower). |
True
|
split_positions |
bool
|
Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower) |
False
|
materialization_version |
int
|
The version of the datastack to query. If None, will query the client
|
None
|
timestamp |
datetime
|
Timestamp to query, by default None. If passsed will do a live query. Error if also passing a materialization version |
None
|
metadata |
bool
|
Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary. |
True
|
merge_reference |
bool
|
Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that |
True
|
desired_resolution |
Iterable[float]
|
Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata |
None
|
get_counts |
bool
|
Whether to get counts of the query, by default False |
False
|
random_sample |
int
|
If given, will do a tablesample of the of the table to return that many annotations |
None
|
log_warning |
bool
|
Whether to log warnings, by default True |
True
|
Returns:
Type | Description |
---|---|
DataFrame
|
A pandas dataframe of results of query |
raise_for_status(r, log_warning=True)
staticmethod
¶
Raises requests.HTTPError, if one occurred.
synapse_query(pre_ids=None, post_ids=None, bounding_box=None, bounding_box_column='post_pt_position', timestamp=None, remove_autapses=True, include_zeros=True, limit=None, offset=None, split_positions=False, desired_resolution=None, materialization_version=None, synapse_table=None, datastack_name=None, metadata=True)
¶
Convenience method for querying synapses.
Will use the synapse table specified in the info service by default. It will also remove autapses by default. NOTE: This is not designed to allow querying of the entire synapse table. A query with no filters will return only a limited number of rows (configured by the server) and will do so in a non-deterministic fashion. Please contact your dataset administrator if you want access to the entire table.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
pre_ids |
Union[int, Iterable, ndarray]
|
Pre-synaptic cell(s) to query. |
None
|
post_ids |
Union[int, Iterable, ndarray]
|
Post-synaptic cell(s) to query. |
None
|
bounding_box |
Optional[Union[list, ndarray]]
|
[[min_x, min_y, min_z],[max_x, max_y, max_z]] bounding box to filter synapse locations. Expressed in units of the voxel_resolution of this dataset. |
None
|
bounding_box_column |
str
|
Which synapse location column to filter by. |
'post_pt_position'
|
timestamp |
datetime
|
Timestamp to query. If passed recalculate query at timestamp, do not pass with materialization_version. |
None
|
remove_autapses |
bool
|
Whether to remove autapses from query results. |
True
|
include_zeros |
bool
|
Whether to include synapses to/from id=0 (out of segmentation). |
True
|
limit |
int
|
Number of synapses to limit. Server-side limit still applies. |
None
|
offset |
int
|
Number of synapses to offset query. |
None
|
split_positions |
bool
|
Whether to split positions into separate columns, True is faster. |
False
|
desired_resolution |
Iterable[float]
|
List or array of the desired resolution you want queries returned in useful for materialization queries. |
None
|
materialization_version |
Optional[int]
|
The version of the datastack to query. If None, will query the client
|
None
|
metadata |
bool
|
Whether to attach metadata to dataframe in the df.attr dictionary. |
True
|
Returns:
Type | Description |
---|---|
DataFrame
|
Results of query. |
MaterializationClientV3(*args, **kwargs)
¶
Bases: MaterializationClientV2
cg_client
property
¶
The chunked graph client.
datastack_name
property
¶
The name of the datastack.
homepage: HTML
property
¶
The homepage for the materialization engine.
server_version: Optional[Version]
property
¶
The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
tables: TableManager
property
¶
The table manager for the materialization engine.
version: int
property
writable
¶
The version of the materialization. Can be used to set up the client to default to a specific version when timestamps or versions are not specified in queries. If not set, defaults to the most recent version.
Note that if this materialization client is attached to a CAVEclient, the version must be set at the CAVEclient level.
views: ViewManager
property
¶
The view manager for the materialization engine.
get_table_metadata(table_name, datastack_name=None, version=None, log_warning=True)
¶
Get metadata about a table
Parameters:
Name | Type | Description | Default |
---|---|---|---|
table_name |
str
|
name of table to mark for deletion |
required |
datastack_name |
str or None
|
Name of the datastack_name. If None, uses the one specified in the client. |
None
|
version |
int
|
The version of the datastack to query. If None, will query the client
|
None
|
log_warning |
bool
|
Whether to print out warnings to the logger. Defaults to True. |
True
|
Returns:
Type | Description |
---|---|
dict
|
Metadata dictionary for table |
get_tables(datastack_name=None, version=None)
¶
Gets a list of table names for a datastack
Parameters:
Name | Type | Description | Default |
---|---|---|---|
datastack_name |
str or None
|
Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client |
None
|
version |
int or None
|
The version of the datastack to query. If None, will query the client
|
None
|
Returns:
Type | Description |
---|---|
list
|
List of table names |
get_tables_metadata(datastack_name=None, version=None, log_warning=True)
¶
Get metadata about tables
Parameters:
Name | Type | Description | Default |
---|---|---|---|
datastack_name |
str or None
|
Name of the datastack_name. If None, uses the one specified in the client. |
None
|
version |
Optional[int]
|
The version of the datastack to query. If None, will query the client
|
None
|
log_warning |
bool
|
Whether to print out warnings to the logger. Defaults to True. |
True
|
Returns:
Type | Description |
---|---|
dict
|
Metadata dictionary for table |
get_timestamp(version=None, datastack_name=None)
¶
Get datetime.datetime timestamp for a materialization version.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
version |
int or None
|
The version of the datastack to query. If None, will query the client
|
None
|
datastack_name |
str or None
|
Datastack name, by default None. If None, defaults to the value set in the client. |
None
|
Returns:
Type | Description |
---|---|
datetime
|
Datetime when the materialization version was frozen. |
get_unique_string_values(table, datastack_name=None)
¶
Get unique string values for a table
Parameters:
Name | Type | Description | Default |
---|---|---|---|
table |
str
|
Table to query |
required |
datastack_name |
Optional[str]
|
Datastack to query. If None, uses the one specified in the client. |
None
|
Returns:
Type | Description |
---|---|
dict[str]
|
A dictionary of column names and their unique values |
get_version_metadata(version=None, datastack_name=None)
¶
Get metadata about a version
Parameters:
Name | Type | Description | Default |
---|---|---|---|
version |
int or None
|
The version of the datastack to query. If None, will query the client
|
None
|
datastack_name |
str or None
|
Datastack name, by default None. If None, defaults to the value set in the client. |
None
|
Returns:
Type | Description |
---|---|
dict
|
Dictionary of metadata about the version |
get_versions(datastack_name=None, expired=False)
¶
Get the versions available
Parameters:
Name | Type | Description | Default |
---|---|---|---|
datastack_name |
str or None
|
Name of the datastack, by default None. If None, uses the one specified in the client. |
None
|
expired |
bool
|
Whether to include expired versions, by default False. |
False
|
Returns:
Type | Description |
---|---|
dict
|
Dictionary of versions available |
get_versions_metadata(datastack_name=None, expired=False)
¶
Get the metadata for all the versions that are presently available and valid
Parameters:
Name | Type | Description | Default |
---|---|---|---|
datastack_name |
str or None
|
Datastack name, by default None. If None, defaults to the value set in the client. |
None
|
expired |
bool
|
Whether to include expired versions, by default False. |
False
|
Returns:
Type | Description |
---|---|
list[dict]
|
List of metadata dictionaries |
get_view_metadata(view_name, materialization_version=None, datastack_name=None, log_warning=True)
¶
Get metadata for a view
Parameters:
Name | Type | Description | Default |
---|---|---|---|
view_name |
str
|
Name of view to query. |
required |
materialization_version |
Optional[int]
|
The version of the datastack to query. If None, will query the client
|
None
|
log_warning |
bool
|
Whether to log warnings. |
True
|
Returns:
Type | Description |
---|---|
dict
|
Metadata of view |
get_view_schema(view_name, materialization_version=None, datastack_name=None, log_warning=True)
¶
Get schema for a view
Parameters:
Name | Type | Description | Default |
---|---|---|---|
view_name |
str
|
Name of view to query. |
required |
materialization_version |
Optional[int]
|
The version of the datastack to query. If None, will query the client
|
None
|
log_warning |
bool
|
Whether to log warnings. |
True
|
Returns:
Type | Description |
---|---|
dict
|
Schema of view. |
get_view_schemas(materialization_version=None, datastack_name=None, log_warning=True)
¶
get_views(version=None, datastack_name=None)
¶
Get all available views for a version
Parameters:
Name | Type | Description | Default |
---|---|---|---|
version |
Optional[int]
|
The version of the datastack to query. If None, will query the client
|
None
|
datastack_name |
str
|
Datastack to query. If None, uses the one specified in the client. |
None
|
Returns:
Type | Description |
---|---|
list
|
List of views |
ingest_annotation_table(table_name, datastack_name=None)
¶
Trigger supervoxel lookup and root ID lookup of new annotations in a table.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
table_name |
str
|
Table to trigger |
required |
datastack_name |
str
|
Datastack to trigger it. Defaults to what is set in client. |
None
|
Returns:
Type | Description |
---|---|
dict
|
Status code of response from server |
join_query(tables, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, suffixes=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, desired_resolution=None, random_sample=None, log_warning=True)
¶
Generic query on materialization tables
Parameters:
Name | Type | Description | Default |
---|---|---|---|
tables |
list of lists with length 2 or 'str'
|
list of two lists: first entries are table names, second entries are the columns used for the join. |
required |
filter_in_dict |
dict of dicts
|
outer layer: keys are table names inner layer: keys are column names, values are allowed entries, by default None |
None
|
filter_out_dict |
dict of dicts
|
outer layer: keys are table names inner layer: keys are column names, values are not allowed entries, by default None |
None
|
filter_equal_dict |
dict of dicts
|
outer layer: keys are table names inner layer: keys are column names, values are specified entry, by default None |
None
|
filter_spatial_dict |
dict of dicts
|
outer layer: keys are table names, inner layer: keys are column names. Values are bounding boxes as [[min_x, min_y,min_z],[max_x, max_y, max_z]], expressed in units of the voxel_resolution of this dataset. Defaults to None. |
None
|
filter_regex_dict |
dict of dicts
|
outer layer: keys are table names. inner layer: keys are column names, values are regex strings. Defaults to None |
None
|
select_columns |
dict of lists of str
|
keys are table names,values are the list of columns from that table. Defaults to None, which will select all tables. Will be passed to server as select_column_maps. Passing a list will be passed as select_columns which is deprecated. |
None
|
offset |
int
|
result offset to use. Defaults to None. Will only return top K results. |
None
|
limit |
int
|
maximum results to return (server will set upper limit, see get_server_config) |
None
|
suffixes |
dict
|
suffixes to use for duplicate columns, keys are table names, values are the suffix |
None
|
datastack_name |
str
|
datastack to query. If None defaults to one specified in client. |
None
|
return_df |
bool
|
whether to return as a dataframe default True, if False, data is returned as json (slower) |
True
|
split_positions |
bool
|
whether to break position columns into x,y,z columns default False, if False data is returned as one column with [x,y,z] array (slower) |
False
|
materialization_version |
int
|
The version of the datastack to query. If None, will query the client
|
None
|
metadata |
bool
|
toggle to return metadata If True (and return_df is also True), return table and query metadata in the df.attr dictionary. |
True
|
desired_resolution |
Iterable
|
What resolution to convert position columns to. Defaults to None will use defaults. |
None
|
random_sample |
int
|
if given, will do a tablesample of the table to return that many annotations |
None
|
log_warning |
bool
|
Whether to log warnings, by default True |
True
|
Returns:
Type | Description |
---|---|
DataFrame
|
a pandas dataframe of results of query |
live_live_query(table, timestamp, joins=None, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, metadata=True, suffixes=None, desired_resolution=None, allow_missing_lookups=False, allow_invalid_root_ids=False, random_sample=None, log_warning=True)
¶
Beta method for querying cave annotation tables with root IDs and annotations at a particular timestamp. Note: this method requires more explicit mapping of filters and selection to table as its designed to test a more general endpoint that should eventually support complex joins.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
table |
str
|
Principle table to query |
required |
timestamp |
datetime
|
Timestamp to query |
required |
joins |
List of joins, where each join is a list of [table1,column1, table2, column2] |
None
|
|
filter_in_dict |
A dictionary with tables as keys, values are dicts with column keys and list values to accept. |
None
|
|
filter_out_dict |
A dictionary with tables as keys, values are dicts with column keys and list values to reject. |
None
|
|
filter_equal_dict |
A dictionary with tables as keys, values are dicts with column keys and values to equate. |
None
|
|
filter_spatial_dict |
A dictionary with tables as keys, values are dicts with column keys and values of 2x3 list of bounds. |
None
|
|
filter_regex_dict |
A dictionary with tables as keys, values are dicts with column keys and values of regex strings. |
None
|
|
select_columns |
A dictionary with tables as keys, values are lists of columns to select. |
None
|
|
offset |
int
|
Value to offset query by. |
None
|
limit |
int
|
Limit of query. |
None
|
datastack_name |
str
|
Datastack to query. Defaults to set by client. |
None
|
split_positions |
bool
|
Whether to split positions into separate columns, True is faster. |
False
|
metadata |
bool
|
Whether to attach metadata to dataframe. |
True
|
suffixes |
dict
|
What suffixes to use on joins, keys are table_names, values are suffixes. |
None
|
desired_resolution |
Iterable
|
What resolution to convert position columns to. |
None
|
allow_missing_lookups |
bool
|
If there are annotations without supervoxels and root IDs yet, allow results. |
False
|
allow_invalid_root_ids |
bool
|
If True, ignore root ids not valid at the given timestamp, otherwise raise an error. |
False
|
random_sample |
int
|
If given, will do a table sample of the table to return that many annotations. |
None
|
log_warning |
bool
|
Whether to log warnings. |
True
|
Returns:
Type | Description |
---|---|
Results of query |
Examples:
>>> from caveclient import CAVEclient
>>> client = CAVEclient('minnie65_public_v117')
>>> live_live_query("table_name", datetime.datetime.now(datetime.timezone.utc),
>>> joins=[[table_name, table_column, joined_table, joined_column],
>>> [joined_table, joincol2, third_table, joincol_third]]
>>> suffixes={
>>> "table_name":"suffix1",
>>> "joined_table":"suffix2",
>>> "third_table":"suffix3"
>>> },
>>> select_columns= {
>>> "table_name":[ "column","names"],
>>> "joined_table":["joined_colum"]
>>> },
>>> filter_in_dict= {
>>> "table_name":{
>>> "column_name":[included,values]
>>> }
>>> },
>>> filter_out_dict= {
>>> "table_name":{
>>> "column_name":[excluded,values]
>>> }
>>> },
>>> filter_equal_dict"={
>>> "table_name":{
>>> "column_name":value
>>> },
>>> filter_spatial_dict"= {
>>> "table_name": {
>>> "column_name": [[min_x, min_y, min_z], [max_x, max_y, max_z]]
>>> }
>>> filter_regex_dict"= {
>>> "table_name": {
>>> "column_name": "regex_string"
>>> }
live_query(table, timestamp, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, post_filter=True, metadata=True, merge_reference=True, desired_resolution=None, random_sample=None, log_warning=True)
¶
Generic query on materialization tables
Parameters:
Name | Type | Description | Default |
---|---|---|---|
table |
str
|
Table to query |
required |
timestamp |
datetime
|
Time to materialize (in utc). Pass datetime.datetime.now(datetime.timezone.utc) for present time. |
required |
filter_in_dict |
dict
|
Keys are column names, values are allowed entries. |
None
|
filter_out_dict |
dict
|
Keys are column names, values are not allowed entries. |
None
|
filter_equal_dict |
dict
|
Keys are column names, values are specified entry. |
None
|
filter_spatial_dict |
dict
|
Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]]. |
None
|
filter_regex_dict |
dict
|
Keys are column names, values are regex strings. |
None
|
select_columns |
list of str
|
Columns to select. |
None
|
offset |
int
|
Offset in query result. |
None
|
limit |
int
|
Maximum results to return (server will set upper limit, see get_server_config). |
None
|
datastack_name |
str
|
Datastack to query. If None, defaults to one specified in client. |
None
|
split_positions |
bool
|
Whether to break position columns into x,y,z columns. If False data is returned as one column with [x,y,z] array (slower). |
False
|
post_filter |
bool
|
Whether to filter down the result based upon the filters specified. If False, it will return the query with present root_ids in the root_id columns, but the rows will reflect the filters translated into their past IDs. So if, for example, a cell had a false merger split off since the last materialization, those annotations on that incorrect portion of the cell will be included if this is False, but will be filtered down if this is True. |
True
|
metadata |
bool
|
Toggle to return metadata. If True (and return_df is also True), return table and query metadata in the df.attr dictionary. |
True
|
merge_reference |
bool
|
Toggle to automatically join reference table. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that table. |
True
|
desired_resolution |
Iterable
|
Desired resolution you want all spatial points returned in. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata. |
None
|
random_sample |
int
|
If given, will do a tablesample of the table to return that many annotations. |
None
|
log_warning |
bool
|
Whether to log warnings. |
True
|
Returns:
Type | Description |
---|---|
DataFrame
|
A pandas dataframe of results of query |
lookup_supervoxel_ids(table_name, annotation_ids=None, datastack_name=None)
¶
Trigger supervoxel lookups of new annotations in a table.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
table_name |
str
|
Table to trigger |
required |
annotation_ids |
list
|
List of annotation ids to lookup. Default is None, which will trigger lookup of entire table. |
None
|
datastack_name |
str
|
Datastack to trigger it. Defaults to what is set in client. |
None
|
Returns:
Type | Description |
---|---|
dict
|
Status code of response from server |
map_filters(filters, timestamp, timestamp_past)
¶
Translate a list of filter dictionaries from a point in the future to a point in the past
Parameters:
Name | Type | Description | Default |
---|---|---|---|
filters |
list[dict]
|
filter dictionaries with root_ids |
required |
timestamp |
datetime
|
timestamp to query |
required |
timestamp_past |
datetime
|
timestamp to query from |
required |
Returns:
Type | Description |
---|---|
list[dict]
|
filter dictionaries with past root_ids |
dict
|
mapping of future root_ids to past root_ids |
most_recent_version(datastack_name=None)
¶
Get the most recent version of materialization for this datastack name
Parameters:
Name | Type | Description | Default |
---|---|---|---|
datastack_name |
str or None
|
Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client |
None
|
Returns:
Type | Description |
---|---|
int
|
Most recent version of materialization for this datastack name |
query_table(table, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, timestamp=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None, log_warning=True)
¶
Generic query on materialization tables
Parameters:
Name | Type | Description | Default |
---|---|---|---|
table |
str
|
Table to query |
required |
filter_in_dict |
dict
|
Keys are column names, values are allowed entries, by default None |
None
|
filter_out_dict |
dict
|
Keys are column names, values are not allowed entries, by default None |
None
|
filter_equal_dict |
dict
|
Keys are column names, values are specified entry, by default None |
None
|
filter_spatial_dict |
dict
|
Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None |
None
|
filter_regex_dict |
dict
|
Keys are column names, values are regex strings, by default None |
None
|
select_columns |
list of str
|
Columns to select, by default None |
None
|
offset |
int
|
Result offset to use, by default None. Will only return top K results. |
None
|
limit |
int
|
Maximum results to return (server will set upper limit, see get_server_config), by default None |
None
|
datastack_name |
str
|
Datastack to query, by default None. If None, defaults to one specified in client. |
None
|
return_df |
bool
|
Whether to return as a dataframe, by default True. If False, data is returned as json (slower). |
True
|
split_positions |
bool
|
Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower) |
False
|
materialization_version |
int
|
The version of the datastack to query. If None, will query the client
|
None
|
timestamp |
datetime
|
Timestamp to query, by default None. If passsed will do a live query. Error if also passing a materialization version |
None
|
metadata |
bool
|
Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary. |
True
|
merge_reference |
bool
|
Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that |
True
|
desired_resolution |
Iterable[float]
|
Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata |
None
|
get_counts |
bool
|
Whether to get counts of the query, by default False |
False
|
random_sample |
int
|
If given, will do a tablesample of the of the table to return that many annotations |
None
|
log_warning |
bool
|
Whether to log warnings, by default True |
True
|
Returns:
Type | Description |
---|---|
DataFrame
|
A pandas dataframe of results of query |
query_view(view_name, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None)
¶
Generic query on a view
Parameters:
Name | Type | Description | Default |
---|---|---|---|
view_name |
str
|
View to query |
required |
filter_in_dict |
dict
|
Keys are column names, values are allowed entries, by default None |
None
|
filter_out_dict |
dict
|
Keys are column names, values are not allowed entries, by default None |
None
|
filter_equal_dict |
dict
|
Keys are column names, values are specified entry, by default None |
None
|
filter_spatial_dict |
dict
|
Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None |
None
|
filter_regex_dict |
dict
|
Keys are column names, values are regex strings, by default None |
None
|
select_columns |
list of str
|
Columns to select, by default None |
None
|
offset |
int
|
Result offset to use, by default None. Will only return top K results. |
None
|
limit |
int
|
Maximum results to return (server will set upper limit, see get_server_config), by default None |
None
|
datastack_name |
str
|
Datastack to query, by default None. If None, defaults to one specified in client. |
None
|
return_df |
bool
|
Whether to return as a dataframe, by default True. If False, data is returned as json (slower). |
True
|
split_positions |
bool
|
Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower) |
False
|
materialization_version |
int
|
The version of the datastack to query. If None, will query the client
|
None
|
metadata |
bool
|
Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary. |
True
|
merge_reference |
bool
|
Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that |
True
|
desired_resolution |
Iterable[float]
|
Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata |
None
|
get_counts |
bool
|
Whether to get counts of the query, by default False |
False
|
random_sample |
int
|
If given, will do a tablesample of the of the table to return that many annotations |
None
|
Returns:
Type | Description |
---|---|
DataFrame
|
A pandas dataframe of results of query |
raise_for_status(r, log_warning=True)
staticmethod
¶
Raises requests.HTTPError, if one occurred.
synapse_query(pre_ids=None, post_ids=None, bounding_box=None, bounding_box_column='post_pt_position', timestamp=None, remove_autapses=True, include_zeros=True, limit=None, offset=None, split_positions=False, desired_resolution=None, materialization_version=None, synapse_table=None, datastack_name=None, metadata=True)
¶
Convenience method for querying synapses.
Will use the synapse table specified in the info service by default. It will also remove autapses by default. NOTE: This is not designed to allow querying of the entire synapse table. A query with no filters will return only a limited number of rows (configured by the server) and will do so in a non-deterministic fashion. Please contact your dataset administrator if you want access to the entire table.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
pre_ids |
Union[int, Iterable, ndarray]
|
Pre-synaptic cell(s) to query. |
None
|
post_ids |
Union[int, Iterable, ndarray]
|
Post-synaptic cell(s) to query. |
None
|
bounding_box |
Optional[Union[list, ndarray]]
|
[[min_x, min_y, min_z],[max_x, max_y, max_z]] bounding box to filter synapse locations. Expressed in units of the voxel_resolution of this dataset. |
None
|
bounding_box_column |
str
|
Which synapse location column to filter by. |
'post_pt_position'
|
timestamp |
datetime
|
Timestamp to query. If passed recalculate query at timestamp, do not pass with materialization_version. |
None
|
remove_autapses |
bool
|
Whether to remove autapses from query results. |
True
|
include_zeros |
bool
|
Whether to include synapses to/from id=0 (out of segmentation). |
True
|
limit |
int
|
Number of synapses to limit. Server-side limit still applies. |
None
|
offset |
int
|
Number of synapses to offset query. |
None
|
split_positions |
bool
|
Whether to split positions into separate columns, True is faster. |
False
|
desired_resolution |
Iterable[float]
|
List or array of the desired resolution you want queries returned in useful for materialization queries. |
None
|
materialization_version |
Optional[int]
|
The version of the datastack to query. If None, will query the client
|
None
|
metadata |
bool
|
Whether to attach metadata to dataframe in the df.attr dictionary. |
True
|
Returns:
Type | Description |
---|---|
DataFrame
|
Results of query. |
MaterializationClient(server_address, datastack_name=None, auth_client=None, cg_client=None, synapse_table=None, api_version='latest', version=None, verify=True, max_retries=None, pool_maxsize=None, pool_block=None, desired_resolution=None, over_client=None)
¶
Factory for returning AnnotationClient
Parameters:
Name | Type | Description | Default |
---|---|---|---|
server_address |
str
|
server_address to use to connect to (i.e. https://minniev1.microns-daf.com) |
required |
datastack_name |
str
|
Name of the datastack. |
None
|
auth_client |
AuthClient or None
|
Authentication client to use to connect to server. If None, do not use authentication. |
None
|
api_version |
str or int (default: latest)
|
What version of the api to use, 0: Legacy client (i.e www.dynamicannotationframework.com) 2: new api version, (i.e. minniev1.microns-daf.com) 'latest': default to the most recent (current 2) |
'latest'
|
cg_client |
chunkedgraph client for live materializations |
None
|
|
synapse_table |
default synapse table for queries |
None
|
|
version |
default version to query
|
if None will default to latest version |
None
|
desired_resolution |
Iterable[float] or None
|
If given, should be a list or array of the desired resolution you want queries returned in useful for materialization queries. |
None
|
Returns:
Type | Description |
---|---|
ClientBaseWithDatastack
|
List of datastack names for available datastacks on the annotation engine |
concatenate_position_columns(df, inplace=False)
¶
function to take a dataframe with x,y,z position columns and replace them with one column per position with an xyz numpy array. Edits occur
Args: df (pd.DataFrame): dataframe to alter inplace (bool): whether to perform edits in place
Returns: pd.DataFrame: [description]
convert_position_columns(df, given_resolution, desired_resolution)
¶
function to take a dataframe with x,y,z position columns and convert them to the desired resolution from the given resolution
Args: df (pd.DataFrame): dataframe to alter given_resolution (Iterable[float]): what the given resolution is desired_resoultion (Iterable[float]): what the desired resolution is
Returns: pd.DataFrame: [description]
deserialize_query_response(response)
¶
Deserialize pyarrow responses