Optimum Transformers

Founded 3 years ago

Optimum Transformers

Accelerated NLP pipelines for fast inference 

on CPU and GPU. Built with 

Transformers, Optimum and ONNX runtime.

Disclaimer

This project is my inspiration of Huggingface Infinity 3. And first step done by Suraj Patil.

@huggingface’s pipeline API is awesome!

, right? And onnxruntime is super fast !

. Wouldn’t it be great to combine these two?
– Tweet by Suraj Patil

It was under this slogan that I started doing this project!

And the main goal was to show myself to get into @huggingface team

How to use

Quick start

The usage is exactly the same as original pipelines, except minor improves:

from optimum_transformers import pipeline

pipe = pipeline("text-classification", use_onnx=True, optimize=True)
pipe("This restaurant is awesome")
# [{'label': 'POSITIVE', 'score': 0.9998743534088135}]

• use_onnx- converts default model to ONNX graph
• optimize - optimizes converted ONNX graph with Optimum 8

Optimum config

Read Optimum 8 documentation for more details

from optimum_transformers import pipeline
from optimum.onnxruntime import ORTConfig

ort_config = ORTConfig(quantization_approach="dynamic")
pipe = pipeline("text-classification", use_onnx=True, optimize=True, ort_config=ort_config)
pipe("This restaurant is awesome")
# [{'label': 'POSITIVE', 'score': 0.9998743534088135}]

Benchmark

With notebook

You can benchmark pipelines easier with benchmark_pipelines 3 notebook.

With own script

from optimum_transformers import Benchmark

task = "sentiment-analysis"
model_name = "philschmid/MiniLM-L6-H384-uncased-sst2"
num_tests = 100

benchmark = Benchmark(task, model_name)
results = benchmark(num_tests, plot=True)

Results

Note: These results were collected on my local machine. So if you have high performance machine to benchmark, please contact me

sentiment-analysis

Almost the same as in Inifinity launch video

AWS VM: g4dn.xlarge
GPU: NVIDIA T4
128 tokens
2.6 ms

zero-shot-classification

With typeform/distilbert-base-uncased-mnli

token-classification

More results are available in project repository: GitHub 13.