#include <execution_provider.h>
Definition at line 65 of file execution_provider.h.
| onnxruntime::IExecutionProvider::IExecutionProvider |
( |
const std::string & |
type | ) |
|
|
inlineprotected |
| onnxruntime::IExecutionProvider::IExecutionProvider |
( |
const std::string & |
type, |
|
|
OrtDevice |
device |
|
) |
| |
|
inlineprotected |
| virtual onnxruntime::IExecutionProvider::~IExecutionProvider |
( |
| ) |
|
|
virtualdefault |
| virtual bool onnxruntime::IExecutionProvider::ConcurrentRunSupported |
( |
| ) |
const |
|
inlinevirtual |
Does the EP support concurrent calls to InferenceSession::Run to execute the model.
Definition at line 316 of file execution_provider.h.
| virtual std::vector<AllocatorPtr> onnxruntime::IExecutionProvider::CreatePreferredAllocators |
( |
| ) |
|
|
inlinevirtual |
Create Preferred allocators for the current Execution Provider This function is a stateless function which creates new instances of Allocator, without storing them in EP.
Definition at line 339 of file execution_provider.h.
| virtual std::vector<std::unique_ptr<ComputeCapability> > onnxruntime::IExecutionProvider::GetCapability |
( |
const onnxruntime::GraphViewer & |
graph_viewer, |
|
|
const IKernelLookup & |
kernel_lookup |
|
) |
| const |
|
virtual |
Get execution provider's capability for the specified <graph>. Return a bunch of IndexedSubGraphs <*this> execution provider can run if the sub-graph contains only one node or can fuse to run if the sub-graph contains more than one node. The node indexes contained in sub-graphs may have overlap, and it's ONNXRuntime's responsibility to do the partition and decide whether a node will be assigned to <*this> execution provider. For kernels registered in a kernel registry, kernel_lookup must be used to find a matching kernel for this EP.
| virtual void onnxruntime::IExecutionProvider::GetCustomOpDomainList |
( |
std::vector< OrtCustomOpDomain * > & |
| ) |
const |
|
inlinevirtual |
Get provider specific custom op domain list. Provider has the responsibility to release OrtCustomOpDomain instances it creates.
NOTE: In the case of ONNX model having EP specific custom nodes and don't want to ask user to register those nodes, EP might need to a way to register those custom nodes. This API is added for the purpose where EP can use it to leverage ORT custom op to register those custom nodes with one or more custom op domains.
For example, TensorRT EP uses this API to support TRT plugins where each custom op is mapped to TRT plugin and no kernel implementation is needed for custom op since the real implementation is inside TRT. This custom op acts as a role to help pass ONNX model validation.
Definition at line 173 of file execution_provider.h.
| virtual std::unique_ptr<onnxruntime::IDataTransfer> onnxruntime::IExecutionProvider::GetDataTransfer |
( |
| ) |
const |
|
inlinevirtual |
Returns a data transfer object that implements methods to copy to and from this device. If no copy is required for the successful operation of this provider, return a nullptr.
Definition at line 88 of file execution_provider.h.
| virtual int onnxruntime::IExecutionProvider::GetDeviceId |
( |
| ) |
const |
|
inlinevirtual |
| virtual const InlinedVector<const Node*> onnxruntime::IExecutionProvider::GetEpContextNodes |
( |
| ) |
const |
|
inlinevirtual |
Get the array of pointers for EPContext nodes EP needs to implement this if has the requirement to generate the context cache model. Otherwise leave it. Default return an empty vector if not provided by the Execution Provider
Definition at line 346 of file execution_provider.h.
| virtual const void* onnxruntime::IExecutionProvider::GetExecutionHandle |
( |
| ) |
const |
|
inlinevirtualnoexcept |
Returns an opaque handle whose exact type varies based on the provider and is interpreted accordingly by the corresponding kernel implementation. For Direct3D operator kernels, this may return an IUnknown supporting QueryInterface to ID3D12GraphicsCommandList1.
Definition at line 181 of file execution_provider.h.
| virtual std::unique_ptr<onnxruntime::IExternalDataLoader> onnxruntime::IExecutionProvider::GetExternalDataLoader |
( |
| ) |
const |
|
inlinevirtual |
Returns an external data loader object that implements methods to load data from external sources.
By default, framework will handle external data loading by loading the data into CPU memory and then copying it to the target device if required. So in most cases, it's not necessary to override this method. Specifically, in WebAssembly build, because the memory is limited and Web platform supports loading data from external sources directly into GPU memory, this method is overridden to provide a custom external data loader to avoid the extra CPU memory usage.
Definition at line 101 of file execution_provider.h.
| virtual FusionStyle onnxruntime::IExecutionProvider::GetFusionStyle |
( |
| ) |
const |
|
inlinevirtual |
| virtual std::shared_ptr<KernelRegistry> onnxruntime::IExecutionProvider::GetKernelRegistry |
( |
| ) |
const |
|
inlinevirtual |
Get kernel registry per execution provider type. The KernelRegistry share pointer returned is shared across sessions.
NOTE: this approach was taken to achieve the following goals,
- The execution provider type based kernel registry should be shared across sessions. Only one copy of this kind of kernel registry exists in ONNXRuntime with multiple sessions/models.
- Adding an execution provider into ONNXRuntime does not need to touch ONNXRuntime framework/session code.
- onnxruntime (framework/session) does not depend on any specific execution provider lib.
Definition at line 149 of file execution_provider.h.
| const logging::Logger* onnxruntime::IExecutionProvider::GetLogger |
( |
| ) |
const |
|
inline |
| virtual OrtDevice onnxruntime::IExecutionProvider::GetOrtDeviceByMemType |
( |
OrtMemType |
mem_type | ) |
const |
|
inlinevirtual |
| virtual DataLayout onnxruntime::IExecutionProvider::GetPreferredLayout |
( |
| ) |
const |
|
inlinevirtual |
| virtual ProviderOptions onnxruntime::IExecutionProvider::GetProviderOptions |
( |
| ) |
const |
|
inlinevirtual |
| virtual ITuningContext* onnxruntime::IExecutionProvider::GetTuningContext |
( |
| ) |
const |
|
inlinevirtual |
| virtual bool onnxruntime::IExecutionProvider::IsGraphCaptured |
( |
int |
| ) |
const |
|
inlinevirtual |
Indicate whether the graph has been captured and instantiated.
Definition at line 234 of file execution_provider.h.
| virtual bool onnxruntime::IExecutionProvider::IsGraphCaptureEnabled |
( |
| ) |
const |
|
inlinevirtual |
Indicate whether the graph capturing mode (e.g., cuda graph) is enabled for the provider.
Definition at line 229 of file execution_provider.h.
Called when InferenceSession::Run ended NOTE that due to async execution in provider, the actual work of this Run may not be finished on device This function should be regarded as the point that all commands of current Run has been submmited by CPU
Definition at line 213 of file execution_provider.h.
Called when InferenceSession::Run started NOTE that due to async execution in provider, the actual work of previous Run may not be finished on device This function should be regarded as the point after which a new Run would start to submit commands from CPU
Definition at line 205 of file execution_provider.h.
| virtual common::Status onnxruntime::IExecutionProvider::OnSessionInitializationEnd |
( |
| ) |
|
|
inlinevirtual |
Called when session creation is complete This provides an opportunity for execution providers to optionally synchronize and clean up its temporary resources to reduce memory and ensure the first run is fast.
Definition at line 248 of file execution_provider.h.
| virtual common::Status onnxruntime::IExecutionProvider::Sync |
( |
| ) |
const |
|
inlinevirtual |
Blocks until the device has completed all preceding requested tasks. Currently this is primarily used by the IOBinding object to ensure that all inputs have been copied to the device before execution begins.
Definition at line 197 of file execution_provider.h.
| const std::string& onnxruntime::IExecutionProvider::Type |
( |
| ) |
const |
|
inline |
- Returns
- type of the execution provider; should match that set in the node through the SetExecutionProvider API. Example valid return values are: kCpuExecutionProvider, kCudaExecutionProvider
Definition at line 190 of file execution_provider.h.
| const OrtDevice onnxruntime::IExecutionProvider::default_device_ |
|
protected |
The documentation for this class was generated from the following file:
1.8.6 
