<h2><code>Optimum Transformers</code></h2><p><code>Accelerated NLP pipelines for fast inference&nbsp;</code></p><figure class="image image_resized" style="width:20px;"><img src="https://emoji.discourse-cdn.com/apple/rocket.png?v=12" alt=":rocket:"></figure><p><code>on CPU and GPU. Built with&nbsp;</code></p><figure class="image image_resized" style="width:20px;"><img src="https://emoji.discourse-cdn.com/apple/hugs.png?v=12" alt=":hugs:"></figure><p><code>Transformers, Optimum and ONNX runtime.</code></p><figure class="image"><img src="https://img.shields.io/github/stars/AlekseyKorshuk/optimum-transformers?style=social" alt="GitHub stars"></figure><h2><code>Disclaimer</code></h2><p><code>This project is my inspiration of </code><a target="_blank" rel="noopener noreferrer" href="https://huggingface.co/infinity"><code>Huggingface Infinity 3</code></a><code>. And first step done by </code><a target="_blank" rel="noopener noreferrer" href="https://twitter.com/psuraj28"><code>Suraj Patil</code></a><code>.</code></p><blockquote><p><a target="_blank" rel="noopener noreferrer" href="https://twitter.com/huggingface"><code>@huggingface</code></a><code>’s pipeline API is awesome!</code></p><figure class="image image_resized" style="width:20px;"><img src="https://emoji.discourse-cdn.com/apple/star_struck.png?v=12" alt=":star_struck:"></figure><p><code>, right? And onnxruntime is super fast !</code></p><figure class="image image_resized" style="width:20px;"><img src="https://emoji.discourse-cdn.com/apple/rocket.png?v=12" alt=":rocket:"></figure><p><code>. Wouldn’t it be great to combine these two?</code><br><code><i>– Tweet by Suraj Patil</i></code></p></blockquote><p><code>It was under this slogan that I started doing this project!</code></p><blockquote><p><code>And the main goal was to show myself to get into <strong>@huggingface</strong> team</code></p><figure class="image image_resized" style="width:20px;"><img src="https://emoji.discourse-cdn.com/apple/hugs.png?v=12" alt=":hugs:"></figure></blockquote><h2><code>How to use</code></h2><h3><code>Quick start</code></h3><p><code>The usage is exactly the same as original pipelines, except minor improves:</code></p><pre><code class="language-bash">from optimum_transformers import pipeline

pipe = pipeline("text-classification", use_onnx=True, optimize=True)
pipe("This restaurant is awesome")
# [{'label': 'POSITIVE', 'score': 0.9998743534088135}]
</code></pre><ul><li><code><strong>use_onnx</strong>- converts default model to ONNX graph</code></li><li><code><strong>optimize</strong> - optimizes converted ONNX graph with </code><a target="_blank" rel="noopener noreferrer" href="https://huggingface.co/docs/optimum/index"><code>Optimum 8</code></a></li></ul><h3><code>Optimum config</code></h3><p><code>Read </code><a target="_blank" rel="noopener noreferrer" href="https://huggingface.co/docs/optimum/index"><code>Optimum 8</code></a><code> documentation for more details</code></p><pre><code class="language-bash">from optimum_transformers import pipeline
from optimum.onnxruntime import ORTConfig

ort_config = ORTConfig(quantization_approach="dynamic")
pipe = pipeline("text-classification", use_onnx=True, optimize=True, ort_config=ort_config)
pipe("This restaurant is awesome")
# [{'label': 'POSITIVE', 'score': 0.9998743534088135}]
</code></pre><h2><code>Benchmark</code></h2><h3><code>With notebook</code></h3><p><code>You can benchmark pipelines easier with </code><a target="_blank" rel="noopener noreferrer" href="https://github.com/AlekseyKorshuk/optimum-transformers/blob/master/notebooks/benchmark_pipelines.ipynb"><code>benchmark_pipelines 3</code></a><code> notebook.</code></p><h3><code>With own script</code></h3><pre><code class="language-bash">from optimum_transformers import Benchmark

task = "sentiment-analysis"
model_name = "philschmid/MiniLM-L6-H384-uncased-sst2"
num_tests = 100

benchmark = Benchmark(task, model_name)
results = benchmark(num_tests, plot=True)
</code></pre><h3><code>Results</code></h3><blockquote><p><code>Note: These results were collected on my local machine. So if you have high performance machine to benchmark, please contact me</code></p><figure class="image image_resized" style="width:20px;"><img src="https://emoji.discourse-cdn.com/apple/hugs.png?v=12" alt=":hugs:"></figure></blockquote><h4><code><strong>sentiment-analysis</strong></code></h4><p><code>Almost the same as in </code><a target="_blank" rel="noopener noreferrer" href="https://www.youtube.com/watch?v=jiftCAhOYQA"><code>Inifinity launch</code></a><code> video</code></p><figure class="image image_resized" style="width:20px;"><img src="https://emoji.discourse-cdn.com/apple/hugs.png?v=12" alt=":hugs:"></figure><p><code>AWS VM: g4dn.xlarge</code><br><code>GPU: NVIDIA T4</code><br><code>128 tokens</code><br><code>2.6 ms</code></p><p>&nbsp;</p><figure class="image"><img src="https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/optimized/2X/b/b0bbebce9d8186be0d20ecdbbc34ca3e327cb9fb_2_666x500.jpeg" alt="Resulting plot" srcset="https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/optimized/2X/b/b0bbebce9d8186be0d20ecdbbc34ca3e327cb9fb_2_666x500.jpeg, https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/optimized/2X/b/b0bbebce9d8186be0d20ecdbbc34ca3e327cb9fb_2_999x750.jpeg 1.5x, https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/original/2X/b/b0bbebce9d8186be0d20ecdbbc34ca3e327cb9fb.jpeg 2x" sizes="100vw" width="666"></figure><p>&nbsp;</p><h4><code><strong>zero-shot-classification</strong></code></h4><p><code>With typeform/distilbert-base-uncased-mnli</code></p><p>&nbsp;</p><figure class="image"><img src="https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/optimized/2X/d/d673c795ad1e6547cf92c5e4951371d8c61eb8bd_2_666x500.jpeg" alt="Resulting plot" srcset="https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/optimized/2X/d/d673c795ad1e6547cf92c5e4951371d8c61eb8bd_2_666x500.jpeg, https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/optimized/2X/d/d673c795ad1e6547cf92c5e4951371d8c61eb8bd_2_999x750.jpeg 1.5x, https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/original/2X/d/d673c795ad1e6547cf92c5e4951371d8c61eb8bd.jpeg 2x" sizes="100vw" width="666"></figure><p>&nbsp;</p><h4><code><strong>token-classification</strong></code></h4><p>&nbsp;</p><figure class="image"><img src="https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/optimized/2X/e/e66b1027e2c2b9af57be5e376402fb3d37cd3a43_2_666x500.jpeg" alt="Resulting plot" srcset="https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/optimized/2X/e/e66b1027e2c2b9af57be5e376402fb3d37cd3a43_2_666x500.jpeg, https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/optimized/2X/e/e66b1027e2c2b9af57be5e376402fb3d37cd3a43_2_999x750.jpeg 1.5x, https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/original/2X/e/e66b1027e2c2b9af57be5e376402fb3d37cd3a43.jpeg 2x" sizes="100vw" width="666"></figure><p>&nbsp;</p><p>&nbsp;</p><figure class="image"><img src="https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/optimized/2X/0/0c4ee5c9d3d4c65cda80668164f1520548a950d7_2_666x500.jpeg" alt="Resulting plot" srcset="https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/optimized/2X/0/0c4ee5c9d3d4c65cda80668164f1520548a950d7_2_666x500.jpeg, https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/optimized/2X/0/0c4ee5c9d3d4c65cda80668164f1520548a950d7_2_999x750.jpeg 1.5x, https://aws1.discourse-cdn.com/standard14/uploads/hellohellohello/original/2X/0/0c4ee5c9d3d4c65cda80668164f1520548a950d7.jpeg 2x" sizes="100vw" width="666"></figure><p>&nbsp;</p><p><code>More results are available in project repository: </code><a target="_blank" rel="noopener noreferrer" href="https://github.com/AlekseyKorshuk/optimum-transformers"><code>GitHub 13</code></a><code>.</code></p><p><br><br>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p>

Aleksey Korshuk

Accelerated NLP pipelines for fast inference on CPU and GPU

Optimum TransformersFounded 19 months ago

Optimum Transformers
Founded 19 months ago