empero-ai/Qwen3.8-2B-Distill-GGUF Q4_K_M

This is a good model and really fast too.


Using the smaller size Q4_K_M and found out that it this version does not know how to call tool compare to its larger version empero-ai/Qwen3.8-4B-Distill-GGUF

Switching to the Q_8 version aint any better. It still does not know how to call tools - the context given is way larger about 32k. 

Then switching BF16 gives a context of 26.9k which is good but still unable to make any tool calling







Comments

Post a Comment

Popular posts from this blog

Windows SSH: Permissions for 'private-key' are too open

Image
Due to a security permission constraints on your folder,  when using ssh via pem file, it needs to be secured. To do what, place your file in a folder and then configure the necessary permission.  right click on the folder, right click on properties -> security -> Advance tab (as shown in figure below). Once you click on Apply, try to re-connect again.
Read more

NodeJS: Error: spawn EINVAL in window for node version 20.20 and 18.20

Encounter this issue when running autorest (that uses nodejs).  I am using node version:  v18.20.2 After i revert to use node v18.12.0, then i was able to get my app to run without this error. Since i am working with Azure devops yaml, i was able to use the following to resolve my issue.  - task : NodeTool@0   inputs :     versionSource : 'spec'     versionSpec : '18.18.0'
Read more