• TPUv2, v3: ML Supercomputer • Multi-chip scaling critical for practical training times • Single TPUv2 chip would take 60 - 400 days for production workloads

这几天 TPUv2 /v3的具体细节终于发了，我们好好的来看一下。 原文在这里. 之前TPUv1讨论的主要是推理用的芯片，所以相对来说架构没有那么复杂；这个paper主要讨论的v2跟v3都是用来训练的。 但是v1跟v2还是有很多相似之处的。 TPU所关心的model会有很多的 embedding，且embedding也有相应的weights。 对于硬件来说，embedding训练所需要的工作是. 所以这个部分消耗很多 内存带宽，且不规则，所以这个是训练硬件需要解决的一个问题。 这个embedding …

Mar 5, 2025 · Architectural details and performance characteristics of TPU v2 are available in A Domain Specific Supercomputer for Training Deep Neural Networks. A TPU v2 slice is composed of 512 chips...

Dec 12, 2018 · The brand new MLPerf benchmark, now in version 0.5, shows that TPUv2 Pods enable faster training of several machine learning workloads.

May 10, 2018 · Recall that a TPUv2 pod contains 256 TPUv2 chips and 128 server processors. A TPUv3 pod will double the server processors and quadruple the TPU chip count. We believe that Google over-provisioned the servers in its TPUv2 pod. This is understandable for a new chip and system architecture.

Mar 6, 2025 · A TPU Pod lets you distribute the processing load across multiple TPUs. Each TPU board is connected to a high-performance CPU-based host machine for things like loading and preprocessing data....

Jun 11, 2018 · Pods. TPUs in production live in “pods”, which are big racks with lots and lots of compute power. Each pod holds 64 TPUv2 boards, for 11.5 petaflops.

Aug 18, 2020 · Google's Training Chips Revealed: TPUv2 and TPUv3 Abstract: This article consists only of a collection of slides from the author's conference presentation. Published in: 2020 IEEE Hot Chips 32 Symposium (HCS)

May 17, 2017 · A TPU pod contains 64 second-generation TPUs and provides up to 11.5 petaflops to accelerate the training of a single large machine learning model. That’s a lot of computation! Using these TPU pods, we've already seen dramatic improvements in training times.

7.TPUv2互连. 专用互连是 TPUv2 超级计算机（“pod”）的基础。TPUv1 是一个作为协处理器构建的单芯片系统，用于推理。在单个芯片上训练谷歌生产模型需要数月时间。因此，TPUv2 可以连接到超级计算机，许多芯片一起工作来训练模型。