George Hotz - GPT-4's real architecture is a 220B parameter mixture model with 8 sets of weights

George Hotz - GPT-4's real architecture is a 220B parameter mixture model with 8 sets of weights and 16 inferences

Source: https://youtu.be/K5iDUZPx60E?t=2989 ... https://www.youtube.com/watch?v=WJWHIZoBOj8