Skip to content

    xai-org/grok-1

    Repository files navigation

    Grok-1

    This repository contains JAX example code for loading and running the Grok-1 open-weights model.

    Make sure to download the checkpoint and place the ckpt-0 directory in checkpoints - see Downloading the weights

    Then, run

    pip install -r requirements.txt
    python run.py

    to test the code.

    The script loads the checkpoint and samples from the model on a test input.

    Due to the large size of the model (314B parameters), a machine with enough GPU memory is required to test the model with the example code. The implementation of the MoE layer in this repository is not efficient. The implementation was chosen to avoid the need for custom kernels to validate the correctness of the model.

    Model Specifications

    Grok-1 is currently designed with the following specifications:

    • Parameters: 314B
    • Architecture: Mixture of 8 Experts (MoE)
    • Experts Utilization: 2 experts used per token
    • Layers: 64
    • Attention Heads: 48 for queries, 8 for keys/values
    • Embedding Size: 6,144
    • Tokenization: SentencePiece tokenizer with 131,072 tokens
    • Additional Features:
      • Rotary embeddings (RoPE)
      • Supports activation sharding and 8-bit quantization
    • Maximum Sequence Length (context): 8,192 tokens

    Downloading the weights

    You can download the weights using a torrent client and this magnet link:

    magnet:?xt=urn:btih:5f96d43576e3d386c9ba65b883210a393b68210e&tr=https%3A%2F%2Facademictorrents.com%2Fannounce.php&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce
    

    or directly using HuggingFace ?? Hub:

    git clone https://github.com/xai-org/grok-1.git && cd grok-1
    pip install huggingface_hub[hf_transfer]
    huggingface-cli download xai-org/grok-1 --repo-type model --include ckpt-0/* --local-dir checkpoints --local-dir-use-symlinks False
    

    License

    The code and associated Grok-1 weights in this release are licensed under the Apache 2.0 license. The license only applies to the source files in this repository and the model weights of Grok-1.

    About

    Grok open release

    Resources

    License

    Code of conduct

    Stars

    Watchers

    Forks

    Releases

    No releases published

    Packages

    No packages published

    Languages

    主站蜘蛛池模板: 国产精品无码一区二区三区不卡| 女人和拘做受全程看视频日本综合a一区二区视频| 日韩一区二区三区视频久久| 免费高清av一区二区三区| 国产亚洲一区区二区在线 | 国精产品一区一区三区有限公司| 精品国产一区二区三区在线 | 精品女同一区二区三区免费站 | 久久毛片一区二区| 国产精品高清一区二区人妖| 精品一区二区三区四区在线播放| 精品亚洲AV无码一区二区| 国产精品分类视频分类一区| 日韩最新视频一区二区三| 一区二区三区高清在线| 国产午夜精品一区二区三区嫩草| 麻豆AV一区二区三区| 精品一区二区三区| 国产精品电影一区二区三区 | 天堂不卡一区二区视频在线观看| 亚洲一区在线视频| 亚洲欧美一区二区三区日产| 国产亚洲3p无码一区二区| 国精产品一区二区三区糖心| 国产91一区二区在线播放不卡| 99无码人妻一区二区三区免费| 亚欧免费视频一区二区三区 | 中文字幕色AV一区二区三区| 亚洲国产一区二区三区在线观看| 成人免费观看一区二区| 精品人妻码一区二区三区| 亚洲国产情侣一区二区三区| 无码人妻AⅤ一区二区三区| 在线观看国产一区| 无码囯产精品一区二区免费 | 在线日产精品一区| 成人精品一区二区三区电影| 一区二区三区在线观看视频| 亚洲色婷婷一区二区三区| 男女久久久国产一区二区三区| 亚洲熟妇成人精品一区|