Skip to content

    xai-org/grok-1

    Repository files navigation

    Grok-1

    This repository contains JAX example code for loading and running the Grok-1 open-weights model.

    Make sure to download the checkpoint and place the ckpt-0 directory in checkpoints - see Downloading the weights

    Then, run

    pip install -r requirements.txt
    python run.py

    to test the code.

    The script loads the checkpoint and samples from the model on a test input.

    Due to the large size of the model (314B parameters), a machine with enough GPU memory is required to test the model with the example code. The implementation of the MoE layer in this repository is not efficient. The implementation was chosen to avoid the need for custom kernels to validate the correctness of the model.

    Model Specifications

    Grok-1 is currently designed with the following specifications:

    • Parameters: 314B
    • Architecture: Mixture of 8 Experts (MoE)
    • Experts Utilization: 2 experts used per token
    • Layers: 64
    • Attention Heads: 48 for queries, 8 for keys/values
    • Embedding Size: 6,144
    • Tokenization: SentencePiece tokenizer with 131,072 tokens
    • Additional Features:
      • Rotary embeddings (RoPE)
      • Supports activation sharding and 8-bit quantization
    • Maximum Sequence Length (context): 8,192 tokens

    Downloading the weights

    You can download the weights using a torrent client and this magnet link:

    magnet:?xt=urn:btih:5f96d43576e3d386c9ba65b883210a393b68210e&tr=https%3A%2F%2Facademictorrents.com%2Fannounce.php&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce
    

    or directly using HuggingFace ?? Hub:

    git clone https://github.com/xai-org/grok-1.git && cd grok-1
    pip install huggingface_hub[hf_transfer]
    huggingface-cli download xai-org/grok-1 --repo-type model --include ckpt-0/* --local-dir checkpoints --local-dir-use-symlinks False
    

    License

    The code and associated Grok-1 weights in this release are licensed under the Apache 2.0 license. The license only applies to the source files in this repository and the model weights of Grok-1.

    About

    Grok open release

    Resources

    License

    Code of conduct

    Stars

    Watchers

    Forks

    Releases

    No releases published

    Packages

    No packages published

    Languages

    主站蜘蛛池模板: 亲子乱av一区区三区40岁| 久久亚洲中文字幕精品一区四| 国产成人一区二区三区高清| 国产色欲AV一区二区三区| 国产一区二区在线看| 国产婷婷色一区二区三区深爱网| 色噜噜狠狠一区二区三区果冻| 国产精品无码不卡一区二区三区| 国产精品亚洲一区二区在线观看| 亚洲av乱码一区二区三区香蕉| 精品无码国产一区二区三区51安| 国产日韩AV免费无码一区二区三区| 国产精品综合一区二区| 99在线精品一区二区三区| 2020天堂中文字幕一区在线观| 久久4k岛国高清一区二区| 插我一区二区在线观看| 一区二区不卡久久精品| 一区免费在线观看| 国产一区二区三区在线免费观看 | 夜夜添无码一区二区三区| 久久se精品一区精品二区| 一区二区三区视频在线播放| 在线观看国产一区亚洲bd| 国产精品一区二区三区高清在线| 国产传媒一区二区三区呀| 波霸影院一区二区| 国产av福利一区二区三巨| 中文字幕无码免费久久9一区9| 国产在线aaa片一区二区99| 成人区精品一区二区不卡 | 亚洲a∨无码一区二区| 免费无码A片一区二三区| 国产亚洲福利精品一区二区| 波多野结衣电影区一区二区三区 | 无码精品一区二区三区在线| 无码精品人妻一区二区三区漫画 | 国产精品xxxx国产喷水亚洲国产精品无码久久一区 | 国产成人av一区二区三区在线| 极品少妇伦理一区二区| 制服中文字幕一区二区|