This is a quick guide for getting Dolly running on an Ubuntu machine with Nvidia GPUs.<\/p>\n\n\n\n
You\u2019ll need a good internet connection and around 35GB of hard drive space for the Nvidia driver, Dolly (12b model) and extras. You can use the smaller models to take up less space. The 8 billion parameter model uses about ~14GB of space while the 3 billion parameter one is around 6GB<\/p>\n\n\n\n
sudo apt install nvidia-driver nvidia-cuda-toolkit<\/pre>\n\n\n\nReboot to activate the Nvidia driver<\/p>\n\n\n\n
reboot<\/pre>\n\n\n\nInstall Python<\/h2>\n\n\n\n
Python should already be installed, but we do need to install pip. <\/p>\n\n\n\n
Once pip is installed, then we need to install numpy, accelerate, and transformers<\/p>\n\n\n\n
sudo apt install python3-pip<\/pre>\n\n\n\npip install numpy<\/pre>\n\n\n\npip install accelerate>=0.12.0 transformers[torch]==4.25.1<\/pre>\n\n\n\nRun Dolly<\/h2>\n\n\n\n
Run a python console. If you run it as administrator, it should be faster.<\/p>\n\n\n\n
python3<\/pre>\n\n\n\nRun the following commands to set up Dolly.<\/p>\n\n\n\n
import torch\nfrom transformers import pipeline\n\ngenerate_text = pipeline(model=\"databricks\/dolly-v2-12b\", torch_dtype=torch.bfloat16, trust_remote_code=True, device_map=\"auto\")\n\n# Alternatively, If you want to use a smaller model run\n\ngenerate_text = pipeline(model=\"databricks\/dolly-v2-3b\", torch_dtype=torch.bfloat16, trust_remote_code=True, device_map=\"auto\")<\/pre>\n\n\n\nNotes:<\/p>\n\n\n\n
\n
- If you have issues, you may want\/need to specify an offload folder with offload_folder=\u201d.\\offloadfolder\u201d. An SSD is preferable.<\/li>\n\n\n\n
- If you have lots of RAM, you can take out the \u201ctorch_dtype=torch.bfloat16\u201d<\/li>\n\n\n\n
- If you do NOT have lots of ram (>32GB), then you may only be able to run the smallest model<\/li>\n<\/ol>\n\n\n\n
Alternatively, if we don\u2019t want to trust_remote_code, we can download this file<\/a>, and run the following<\/p>\n\n\n\n
from instruct_pipeline import InstructionTextGenerationPipeline\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"databricks\/dolly-v2-12b\", padding_side=\"left\")\nmodel = AutoModelForCausalLM.from_pretrained(\"databricks\/dolly-v2-12b\", device_map=\"auto\")\n\ngenerate_text = InstructionTextGenerationPipeline(model=model, tokenizer=tokenizer)<\/pre>\n\n\n\nNow we can ask Dolly a question.<\/p>\n\n\n\n
generate_text(\"Your question?\")<\/pre>\n\n\n\nExample:<\/p>\n\n\n\n
>>> generate_text(\"Tell me about Databricks dolly-v2-3b?\")\n'Dolly is the fully managed open-source engine that allows you to rapidly build, test, and deploy machine learning models, all on your own infrastructure.'<\/pre>\n\n\n\nFurther information is available at the following two links.<\/p>\n\n\n\n
https:\/\/github.com\/databrickslabs\/dolly<\/a>
https:\/\/huggingface.co\/databricks\/dolly-v2-3b<\/a>
https:\/\/huggingface.co\/databricks\/dolly-v2-12b<\/a><\/p>\n\n\n\n<\/p>\n","protected":false},"excerpt":{"rendered":"
This is a quick guide for getting Dolly running on an Ubuntu machine with Nvidia GPUs. You\u2019ll need a good internet connection and around 35GB of hard drive space for the Nvidia driver, Dolly (12b model) and extras. You can … Continue reading