feat: Transformers API Upgrade

[ishaansehgal99]

Reason for Change:
Update transformers inference to align with VLLM. In addition ConversationalPipeline has been deprecated/no longer supported by transformers library so this has been removed.

[codecov]

Codecov Report

Attention: Patch coverage is 79.48718% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...rkspace/inference/text-generation/inference_api.py	72.41%	8 Missing ⚠️

Files with missing lines	Coverage Δ
...erence/text-generation/tests/test_inference_api.py	100.00% <100.00%> (ø)
...ference/text-generation/tests/test_model_config.py	100.00% <100.00%> (ø)
...rkspace/inference/text-generation/inference_api.py	79.80% <72.41%> (-3.68%)	⬇️

... and 37 files with indirect coverage changes

[zhuangqh]

nice job.