HDK
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
Classes | Public Types | Public Member Functions | Protected Member Functions | List of all members
onnxruntime::IExecutionProvider Class Reference

#include <execution_provider.h>

Classes

struct  FusedNodeAndGraph
 
class  IKernelLookup
 

Public Types

enum  FusionStyle { FusionStyle::Function, FusionStyle::FilteredGraphViewer }
 

Public Member Functions

virtual ~IExecutionProvider ()=default
 
const std::vector< AllocatorPtr > & GetAllocators () const
 
virtual AllocatorPtr GetAllocator (int device_id, OrtMemType mem_type) const
 
virtual std::unique_ptr
< onnxruntime::IDataTransfer > 		GetDataTransfer () const
 
virtual std::vector
< std::unique_ptr
< ComputeCapability > > 		GetCapability (const onnxruntime::GraphViewer &graph_viewer, const IKernelLookup &kernel_lookup) const
 
virtual std::shared_ptr
< KernelRegistry > 		GetKernelRegistry () const
 
virtual int GetDeviceId () const
 
virtual ProviderOptions GetProviderOptions () const
 
virtual const void * GetExecutionHandle () const noexcept
 
const std::string & Type () const
 
virtual common::Status Sync () const
 
virtual common::Status OnRunStart ()
 
virtual common::Status OnRunEnd (bool)
 
virtual bool IsGraphCaptureEnabled () const
 
virtual bool IsGraphCaptured () const
 
virtual common::Status ReplayGraph ()
 
virtual common::Status OnSessionInitializationEnd ()
 
void InsertAllocator (AllocatorPtr allocator)
 
void ReplaceAllocator (AllocatorPtr allocator)
 
virtual FusionStyle GetFusionStyle () const
 
virtual common::Status Compile (const std::vector< FusedNodeAndGraph > &fused_nodes_and_graphs, std::vector< NodeComputeInfo > &node_compute_funcs)
 
void SetLogger (const logging::Logger *logger)
 
const logging::Logger * GetLogger () const
 
virtual int GenerateMetaDefId (const onnxruntime::GraphViewer &graph_viewer, HashValue &model_hash) const
 
virtual void RegisterAllocator (AllocatorManager &)
 
virtual std::unique_ptr
< profiling::EpProfiler > 		GetProfiler ()
 
virtual DataLayout GetPreferredLayout () const
 
virtual void RegisterStreamHandlers (IStreamCommandHandleRegistry &) const
 
virtual bool ConcurrentRunSupported () const
 

Protected Member Functions

 IExecutionProvider (const std::string &type, bool use_metadef_id_creator=false)
 

Detailed Description

Definition at line 58 of file execution_provider.h.

Member Enumeration Documentation

enum onnxruntime::IExecutionProvider::FusionStyle
strong
Enumerator
Function 
FilteredGraphViewer 

Definition at line 225 of file execution_provider.h.

Constructor & Destructor Documentation

onnxruntime::IExecutionProvider::IExecutionProvider ( const std::string &  type,
bool  use_metadef_id_creator = false 
)
inlineprotected

Definition at line 60 of file execution_provider.h.

virtual onnxruntime::IExecutionProvider::~IExecutionProvider ( )
virtualdefault

Member Function Documentation

virtual common::Status onnxruntime::IExecutionProvider::Compile ( const std::vector< FusedNodeAndGraph > &  fused_nodes_and_graphs,
std::vector< NodeComputeInfo > &  node_compute_funcs 
)
virtual

Given a collection of fused Nodes and the respective GraphViewer instance for the nodes that were fused, return create_state/compute/release_state func for each node.

Remarks
This is now the default interface when execution provider wants to compile nodes for both minimal build and complete ort build.

Do NOT cache the GraphViewer in FusedNodeAndGraph.filtered_graph in any of the NodeComputeInfo functions as it is only valid for the duration of the call to Compile.

virtual bool onnxruntime::IExecutionProvider::ConcurrentRunSupported ( ) const
inlinevirtual

Does the EP support concurrent calls to InferenceSession::Run to execute the model.

Definition at line 301 of file execution_provider.h.

virtual int onnxruntime::IExecutionProvider::GenerateMetaDefId ( const onnxruntime::GraphViewer &  graph_viewer,
HashValue &  model_hash 
) const
virtual

Generate a unique id that can be used in a MetaDef name. Values are unique for a model instance. The model hash is also returned if you wish to include that in the MetaDef name to ensure uniqueness across models.

Parameters
graph_viewer[in]Graph viewer that GetCapability was called with. Can be for the main graph or nested graph.
model_hash[out]Returns the hash for the main (i.e. top level) graph in the model. This is created using the model path if available, or the model input names and the output names from all nodes in the main graph.
Remarks
e.g. the TensorRT Execution Provider is used in multiple sessions and the underlying infrastructure caches compiled kernels, so the name must be unique and deterministic across models and sessions. NOTE: Ideally this would be a protected method, but to work across the EP bridge it has to be public and virtual, and ModelMetadefIdGenerator but be defined in the header as well.
virtual AllocatorPtr onnxruntime::IExecutionProvider::GetAllocator ( int  device_id,
OrtMemType  mem_type 
) const
virtual

Get an allocator with specified device id and MemType. Return nullptr if it doesn't exist

const std::vector<AllocatorPtr>& onnxruntime::IExecutionProvider::GetAllocators ( ) const
inline

Get all IAllocators for <*this> execution provider.

Definition at line 73 of file execution_provider.h.

virtual std::vector<std::unique_ptr<ComputeCapability> > onnxruntime::IExecutionProvider::GetCapability ( const onnxruntime::GraphViewer &  graph_viewer,
const IKernelLookup &  kernel_lookup 
) const
virtual

Get execution provider's capability for the specified <graph>. Return a bunch of IndexedSubGraphs <*this> execution provider can run if the sub-graph contains only one node or can fuse to run if the sub-graph contains more than one node. The node indexes contained in sub-graphs may have overlap, and it's ONNXRuntime's responsibility to do the partition and decide whether a node will be assigned to <*this> execution provider. For kernels registered in a kernel registry, kernel_lookup must be used to find a matching kernel for this EP.

virtual std::unique_ptr<onnxruntime::IDataTransfer> onnxruntime::IExecutionProvider::GetDataTransfer ( ) const
inlinevirtual

Returns a data transfer object that implements methods to copy to and from this device. If no copy is required for the successful operation of this provider, return a nullptr.

Definition at line 88 of file execution_provider.h.

virtual int onnxruntime::IExecutionProvider::GetDeviceId ( ) const
inlinevirtual

Get the device id of current execution provider

Definition at line 141 of file execution_provider.h.

virtual const void* onnxruntime::IExecutionProvider::GetExecutionHandle ( ) const
inlinevirtualnoexcept

Returns an opaque handle whose exact type varies based on the provider and is interpreted accordingly by the corresponding kernel implementation. For Direct3D operator kernels, this may return an IUnknown supporting QueryInterface to ID3D12GraphicsCommandList1.

Definition at line 154 of file execution_provider.h.

virtual FusionStyle onnxruntime::IExecutionProvider::GetFusionStyle ( ) const
inlinevirtual

Definition at line 239 of file execution_provider.h.

virtual std::shared_ptr<KernelRegistry> onnxruntime::IExecutionProvider::GetKernelRegistry ( ) const
inlinevirtual

Get kernel registry per execution provider type. The KernelRegistry share pointer returned is shared across sessions.

NOTE: this approach was taken to achieve the following goals,

  1. The execution provider type based kernel registry should be shared across sessions. Only one copy of this kind of kernel registry exists in ONNXRuntime with multiple sessions/models.
  2. Adding an execution provider into ONNXRuntime does not need to touch ONNXRuntime framework/session code.
  3. onnxruntime (framework/session) does not depend on any specific execution provider lib.

Definition at line 136 of file execution_provider.h.

const logging::Logger* onnxruntime::IExecutionProvider::GetLogger ( ) const
inline

Definition at line 264 of file execution_provider.h.

virtual DataLayout onnxruntime::IExecutionProvider::GetPreferredLayout ( ) const
inlinevirtual

Definition at line 291 of file execution_provider.h.

virtual std::unique_ptr<profiling::EpProfiler> onnxruntime::IExecutionProvider::GetProfiler ( )
inlinevirtual

Definition at line 287 of file execution_provider.h.

virtual ProviderOptions onnxruntime::IExecutionProvider::GetProviderOptions ( ) const
inlinevirtual

Get execution provider's configuration options.

Definition at line 146 of file execution_provider.h.

void onnxruntime::IExecutionProvider::InsertAllocator ( AllocatorPtr  allocator)
virtual bool onnxruntime::IExecutionProvider::IsGraphCaptured ( ) const
inlinevirtual

Indicate whether the graph has been captured and instantiated. Currently only CUDA execution provider supports it.

Definition at line 198 of file execution_provider.h.

virtual bool onnxruntime::IExecutionProvider::IsGraphCaptureEnabled ( ) const
inlinevirtual

Indicate whether the graph capturing mode (e.g., cuda graph) is enabled for the provider. Currently only CUDA execution provider supports it.

Definition at line 192 of file execution_provider.h.

virtual common::Status onnxruntime::IExecutionProvider::OnRunEnd ( bool  )
inlinevirtual

Called when InferenceSession::Run ended NOTE that due to async execution in provider, the actual work of this Run may not be finished on device This function should be regarded as the point that all commands of current Run has been submmited by CPU

Definition at line 186 of file execution_provider.h.

virtual common::Status onnxruntime::IExecutionProvider::OnRunStart ( )
inlinevirtual

Called when InferenceSession::Run started NOTE that due to async execution in provider, the actual work of previous Run may not be finished on device This function should be regarded as the point after which a new Run would start to submit commands from CPU

Definition at line 178 of file execution_provider.h.

virtual common::Status onnxruntime::IExecutionProvider::OnSessionInitializationEnd ( )
inlinevirtual

Called when session creation is complete This provides an opportunity for execution providers to optionally synchronize and clean up its temporary resources to reduce memory and ensure the first run is fast.

Definition at line 211 of file execution_provider.h.

virtual void onnxruntime::IExecutionProvider::RegisterAllocator ( AllocatorManager &  )
virtual

Register allocators for EP, potentially re-using existing allocators for a device from allocator_manager. If the EP implements this it should generally delay creating any allocators until this is called.

virtual void onnxruntime::IExecutionProvider::RegisterStreamHandlers ( IStreamCommandHandleRegistry &  ) const
inlinevirtual

Definition at line 297 of file execution_provider.h.

void onnxruntime::IExecutionProvider::ReplaceAllocator ( AllocatorPtr  allocator)
virtual common::Status onnxruntime::IExecutionProvider::ReplayGraph ( )
inlinevirtual

Run the instantiated graph. Currently only CUDA execution provider supports it.

Definition at line 204 of file execution_provider.h.

void onnxruntime::IExecutionProvider::SetLogger ( const logging::Logger *  logger)
inline

Definition at line 260 of file execution_provider.h.

virtual common::Status onnxruntime::IExecutionProvider::Sync ( ) const
inlinevirtual

Blocks until the device has completed all preceding requested tasks. Currently this is primarily used by the IOBinding object to ensure that all inputs have been copied to the device before execution begins.

Definition at line 170 of file execution_provider.h.

const std::string& onnxruntime::IExecutionProvider::Type ( ) const
inline
Returns
type of the execution provider; should match that set in the node through the SetExecutionProvider API. Example valid return values are: kCpuExecutionProvider, kCudaExecutionProvider

Definition at line 163 of file execution_provider.h.

The documentation for this class was generated from the following file: