Open
Conversation
Reviewer's GuideBumps the ort dependency to v2.0.0-rc.10 by updating the lockfile entries and checksums in Cargo.lock. File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
eee53dd to
0f47789
Compare
0f47789 to
3f06c3f
Compare
3f06c3f to
c5119a8
Compare
c5119a8 to
62196a5
Compare
62196a5 to
e450922
Compare
e450922 to
45e5672
Compare
45e5672 to
4231388
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains the following updates:
2.0.0-rc.9→2.0.0-rc.12Release Notes
pykeio/ort (ort)
v2.0.0-rc.12Compare Source
2.0.0-rc.12
💖 If you find
ortuseful, please consider sponsoring us on Open Collective 💖🤔 Need help upgrading? Ask questions in GitHub Discussions or in the pyke.io Discord server!
This release was made possible by Rime.ai!
📍 Multiversioning
The big highlight of this release is multiversioning:
ortcan now use any minor version of ONNX Runtime from v1.17 to v1.24. New features are gated behindapi-*feature flags, likeapi-20orapi-24. These flags will set the minimum version of ONNX Runtime required byort.More info 👉 https://ort.pyke.io/setup/multiversion
🪄 Automatic device selection
With ONNX Runtime 1.22 or later,
ortwill now automatically use an NPU if one is available for maximum efficiency & power savings! Setting your own execution providers will override this.This is thanks to the super cool new
SessionBuilder::with_auto_deviceAPI! There's alsoSessionBuilder::with_devicesfor finer control.👁️ CUDA 13
ortnow ships builds for both CUDA 12 & CUDA 13! It should automatically detect which CUDA you're using, but if it gets it wrong, you can override it by setting theORT_CUDA_VERSIONenvironment variable to12or13.🩹
SessionBuildererror recoveryYou can now recover from errors when building a session by calling
.recover()on the error type to get theSessionBuilderback.🛡️ Build attestations
Prebuilt binaries are now attested via GitHub Actions, so you can verify that they are untampered builds of ONNX Runtime coming straight from pyke.io.
To verify, download your binary package of choice and use the
ghCLI to verify:(Also note that the SHA-256 hash lines up with the one defined in
dist.txt.)Moving stuff around
ORT_LIB_LOCATIONenvironment variable has been renamed toORT_LIB_PATH._LOCATION.ort::tensoris now inort::value, because why have atensormodule if theTensor<T>type actually comes from thevaluemodule?IoBindingandAdapterwere moved from their own modules intoort::session. All sub-modules ofort::sessionbesidesbuilderwere collapsed intoort::session.ort::operatorwere collapsed intoort::operator.with_denormal_as_zero->with_flush_to_zerowith_device_allocator_for_initializers->with_device_allocated_initializersFixes
c52bd2aFix MIGraphX registration.374a9d1Fix global environment thread poolsff08428Fix a segfault inTensor::clone.3d6c2a9Use new API to load the DirectML EP.5913ae0Make vcpkg builds work again.079ecb4Fix issues with multiple environment registration.❤️🧡💛💚💙💜
v2.0.0-rc.11Compare Source
💖 If you find
ortuseful, please consider sponsoring us on Open Collective 💖🤔 Need help upgrading? Ask questions in GitHub Discussions or in the pyke.io Discord server!
I'm sorry it took so long to get to this point, but the next big release of
ortshould be, finally, 2.0.0 🎉. I know I said that about one of the old alpha releases (if you can even remember those), but I mean it this time! Also, I would really like to not have to do another major release right after, so if you have any concerns about any APIs, please speak now or forever hold your peace!A huge thank you to all the individuals who have contributed to the Collective over the years: Marius, Urban Pistek, Phu Tran, Haagen, Yunho Cho, Laco Skokan, Noah, Matouš Kučera, mush42, Thomas, Bartek, Kevin Lacker, & Okabintaro. You guys have made these past rc releases possible.
If you are a business using
ort, please consider sponsoring me. Egress bandwidth from pyke.io has quadrupled in the last 4 months, and 90% of that comes from just a handful of businesses. I'm lucky enough that I don't have to pay for egress right now, but I don't expect that arrangement to last forever. pyke &orthave been funded entirely from my own personal savings for years, and (as I'm sure you're well aware 😂) everything is getting more expensive, so that definitely isn't sustainable.Seeing companies that raise tens of millions in funding build large parts of their business on
ort, ask for support, and then not give anything back just... seems kind of unfair, no?ort-webort-weballows you to use the fully-featured ONNX Runtime on the Web! This time, it's hack-free and thus here to stay (it won't be removed, and then added back, and then removed again like last time!)See the crate docs for info on how to port your application to
ort-web; there is a little bit of work involved. For a very barebones sample application, seeort-web-sample.Documentation for
ort-web, like the rest ofort, will improve by the time 2.0.0 comes around. If you ever have any questions, you can always reach out via GitHub Discussions or Discord!Features
5d85209Add WebNN & WASM execution providers forort-web.#430(💖 @jhonboy121) Support statically linking to iOS frameworks.#433(💖 @rMazeiks) Implement more traits forGraphOptimizationLevel.6727c98MakePrepackedWeightsSend + Sync.15bd15cMake the TLS backend configurable with newtls-*Cargo features.f3cd995Allow overriding the cache dir with theORT_CACHE_DIRenvironment variable.8b3a1edLoad the dylib immediately when usingort::init_from.#484(💖 @michael-p) Updatendarrayto v0.17.ndarraydependency to v0.17, too.0084d08Newort::lifetimetracing target tracks when objects are allocated/freed to aid in debugging leaks.Fixes
2ee17aaFix a memory leak inIoBinding.317be20Don't storeEnvironmentas a static.mutex lock failed: Invalid argumentcrash on macOS when exiting the process.466025cFix unexpected CPU usage when copying GPU tensors.ecca246Fix UB when extracting empty tensors.22f71baGate theArrayExtensionstrait behind thestdfeature, fixing#![no_std]builds.af63ceaFix an illegal memory access onno_stdbuilds.#444(💖 @pembem22) Fix Android link.1585268Don't allow sessions to be created with non-CPU allocators#485(💖 @mayocream) Fix load order when usingcuda::preload_dylibs.c5b68a1FixAsyncInferenceFutdrop behavior.Misc
ortin CI, please cache the~/.cache/ort.pyke.iodirectory between runs.ort's dependency tree has shrunk a little bit, so it should build a little faster!b68c928Overhaulbuild.rspkg-configsupport now requires thepkg-configfeature.d269461MakeMetadatamethods returnOption<T>instead ofResult<T>.47e5667Gatepreload_dylibandcuda::preload_dylibsbehind a newpreload-dylibsfeature flag instead ofload-dynamic.3b408b1Shortenexecution_providerstoepandXXXExecutionProvidertoXXX.38573e0SimplifyThreadManagertrait.ONNX Runtime binary changes
x86_64-apple-darwin) has been dropped following upstream changes to ONNX Runtime & Rust.--client_package_build, meaning default options will optimize for low-resource edge inference rather than high throughput.x86-64-v3, aka Intel Haswell/Broadwell and AMD Zen (any Ryzen) or later.ort-tracttractto 0.22.2d40e05ort-tractno longer claims it isort-candleinort::info().ort-candlecandleto 0.9.❤️🧡💛💚💙💜
v2.0.0-rc.10Compare Source
💖 If you find
ortuseful, please consider sponsoring us on Open Collective 💖🤔 Need help upgrading? Ask questions in GitHub Discussions or in the pyke.io Discord server!
🔗 Tensor Array Views
You can now create a
TensorRefdirectly from anArrayView. Previously, tensors could only be created viaTensor::from_array(which, in many cases, performed a copy if borrowed data was provided). The newTensorRef::from_array_view(and the complementaryTensorRefMut::from_array_view_mut) method(s) allows for the zero-copy creation of tensors directly from anArrayView.Tensor::from_arraynow only accepts owned data, so you should either refactor your code to useTensorRefs or pass ownership of the array to theTensor.rc.10now allows you to manually copy tensors between devices usingTensor::to!There's also
Tensor::to_async, which replicates the functionality of PyTorch'snon_blocking=True. Additionally,Tensors now implementClone.⚙️ Alternative Backends
ortis no longer just a wrapper for ONNX Runtime; it's a one-stop shop for inferencing ONNX models in Rust thanks to the addition of the alternative backend API.Alternative backends wrap other inference engines behind ONNX Runtime's API, which can simply be dropped in and used in
ort- all it takes is one line of code:2 alternative backends are shipping alongside
rc.10-ort-tract, powered bytract, andort-candle, powered bycandle, with more to come in the future.Outside of the Rust ecosystem, these alternative backends can also be compiled as standalone libraries that can be directly dropped in to applications as a replacement for
libonnxruntime. 🦀🦠✏️ Model Editor
Models can be created entirely programmatically, or edited from an existing ONNX model via the new Model Editor API.
See
src/editor/tests.rsfor an example of how an ONNX model can be created programmatically. You can combine the Model Editor API withSessionBuilder::with_optimized_model_pathto export the model outside Rust.⚛️ Compiler
Many execution providers internally convert ONNX graphs to a framework-specific graph representation, like CoreML networks/TensorRT engines. This process can take a long time, especially for larger and more complex models. Since these generated artifacts aren't persisted between runs, they have to be created every time a session is loaded.
The new Compiler API allows you to compile an optimized, EP-ready graph ahead-of-time, so subsequent loads are lighting fast! ⚡
🪶
#![no_std]ortno longer depends onstd(but does still depend onalloc) -default-features = falsewill enable#![no_std]forort.⚡ Execution Providers
rc.10adds support for 3 execution providers:ort.All binaries are now statically linked! This means the
cudaandtensorrtfeatures no longer useonnxruntime.dll/libonnxruntime.so. The EPs themselves do still require separate DLLs - likelibonnxruntime_providers_cuda- but this change should make it significantly easier to set up and useortwith CUDA/TRT.🧩 Custom Operator Improvements
Custom operators have been internally revamped to reduce code size & compilation time, and allow operators to be
Sized.🔷 Miscellaneous changes
tracingdependency is now optional (but enabled by default).tracingwithdefault-features = false, enable thetracingfeature.WARNbut can be controlled at runtime via theORT_LOGenvironment variable by setting it to one ofverbose,info,warning,error, orfatal.parcel.pyke.iotocdn.pyke.io, so make sure to update firewall exclusions.build.rshack for Apple platforms is no longer required. (9b31680)ureqdependency (used bydownload-binaries/fetch-models) has been ugpraded to v3.0.ortwith thefetch-modelsfeature will userustlsas the TLS provider.ort-syswith thedownload-binariesfeature will usenative-tlssince that pulls less dependencies (it previously usedrustls). No prerequisites are required when building on Windows & macOS, but other platforms now require OpenSSL to be installed.Complex64&Complex128, 4-bit integers, and 8 bit floats!DynTensor::newto allocate a tensor andDynTensor::data_ptrto access its data.e136869)Session::runcan now be zero-alloc (on the Rust side)!Session::runnow takes&mut self.SessionOutputs::removeto get an owned session output.ort::inputs!no longer outputs aResult, so remove the trailing?from any invocations of the macro.extract_tensorto extract a tensor to anndarrayhas been renamed toextract_array, withextract_raw_tensornow taking the place ofextract_tensor.DynValue::try_extract_tensor(_mut)->DynValue::try_extract_array(_mut)Tensor::extract_tensor(_mut)->Tensor::extract_array(_mut)DynValue::try_extract_raw_tensor(_mut)->DynValue::try_extract_tensor(_mut)Tensor::extract_raw_tensor(_mut)->Tensor::extract_tensor(_mut)Session::run_asyncnow always takes&RunOptions;Session::run_async_with_optionshas been removed.ValueType::tensor_dimensions) has been replaced with "shape" (soValueType::tensor_shape) for consistency.ort::tensor::Shape, instead of aVec<i64>directly.ValueType::Tensor.dimension_symbolsis its own struct,SymbolicDimensions.::from()/.into().SessionBuilder::with_execution_providersnow takesAsRef<[EP]>instead of any iterable type.SessionBuilder::with_external_initializer_file_in_memoryrequires aPathfor thepathparameter instead of a regular&str.🪲 Fixes
Tensor::new. (7a95f98)Tensor::newwill be manually zeroed on the Rust side.IoBinding::synchronize_*now takes&selfsosynchronize_outputscan actually be used as intended (e8d873a)XNNPACKExecutionProvider::is_availablealways returningfalse(5ad997c)AllocationDevice&MemoryInfo(3ca14c2)3e7e8fe)5661450)ort-syscrate now specifieslinks, hopefully preventing linking conflicts (d2dc7c8)AllocationDevice(46c3376)ort-sysno longer tries to download binaries when building with--offline(d7d4493)4b6b163)tracinglevel instead of being knocked down a level (d8bcfd7)TensorRTExecutionProvider::with_context_memory_sharing(#327)with_build_heuristics&with_sparisty(b6ddfd8)commit_from_urlorort-sys(eb51646/#323)❤️🧡💛💚💙💜
Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR was generated by Mend Renovate. View the repository job log.