azure foundry scale model to be able to handle peak request

Azure Foundry has three deployment types you should know. For peak traffic, the pattern recommended by Microsoft is Global Provisioned (PTU) as your baseline, with spillover to Global Standard to absorb bursts

Global Provisioned deployments use Azure's global infrastructure to dynamically route traffic to available datacenters, providing reserved model processing capacity with guaranteed throughput combining global routing with lower, more consistent latency than standard.

More details here.

However, this also means data can be processed global and violates data sovereignty.

Typcially we have different skus (to keep things simple) that we can configure the followings :-

1. global

2. standard

3. regional 

4. developer

More details of deployment type can be find here.

Here is a diagram that might provide a better understanding. 




And depending on your workload requirement, this offers some guide

If your workload is...Recommended deployment
Prototyping or trying a new modelInstant Models (Preview)
Variable or bursty trafficStandard / Global Standard
Consistently high trafficProvisioned
Large batch jobs that are not time-sensitiveGlobal Batch / Data Zone Batch
Fine-tuned model testing and evaluationDeveloper

Limits 

For quota, TPM and RPM limits are defined per region, per subscription, and per model — and the highest level of quota restriction is scoped at the Azure subscription level, not tenant level. Spreading deployments across regions via APIM load balancing effectively multiplies your available quota

When we're using Gateway API or APIM to route request to our Azure foundry, it normally would direct traffic to a project tied to a single subscriptions. 

If your workload is too, then you may want to re-think how to structure your Azure Foundry project accordingly. 


 








Comments

Post a Comment

Popular posts from this blog

mongosh install properly

To install mongosh property on ubuntu 24.04, we need to wget -qO- https://www.mongodb.org/static/pgp/server-8.0.asc | sudo tee /etc/apt/trusted.gpg.d/server-8.0.asc Then run  echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu noble/mongodb-org/8.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-8.0.list Next we will run  sudo apt-get update And then finally install it using  sudo apt-get install -y mongodb-mongosh To verify if it is running we run  mongosh --version
Read more

gemini cli getting file not defined error

Image
 While trying to run gemini cli on my windows linux subsystem, i bump into this error here - "/Roaming/npm/node_modules/@google/gemini-cli/node_modules/undici/lib/web/webidl/index.js:512 webidl.is.File = webidl.util.MakeTypeAssertion(File)"  "ReferenceError: File is not defined" In Node.js, there’s no built-in File (until very recent Node 20+ with experimental WHATWG APIs). So, unless polyfilled, File is undefined, leading to the ReferenceError. And to resolve this, I had to upgrade to node 22 (might work for node 20) and then I was able to run it.
Read more

vllm : Failed to infer device type

Trying to run vllm without gpu will typicaly lands you into this issue. To resolve this issue: we can use set these both. Setting one does not work for me. So i had to configure both settings below:- 1.  environment variable CUDA_VISIBLE_DEVICES = "" 2. in the command line, change to use device cpu and remove  tensor-parallel-size. For example, python3 -m vllm.entrypoints.openai.api_server --port 8080 --model deepseek-ai/DeepSeek-R1 --device cpu --trust-remote-code --max-model-len 4096" References https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html#failed-to-infer-device-type
Read more