mercredi 31 octobre 2018 à 11:49

CoLab TPUs One Month Later

Par Eric Antoine Scuccimarra

After having used both CoLab GPUs and TPUs for almost a month I must significantly revise my previous opinion. Even for a Keras model not written or optimized for TPUs, with some minimal configuration changes TPUs perform much faster - minimum of twice the speed. In addition to making sure that all operations are TPU compatible, the only major configuration change required is increasing the batch size by 8. At first I was playing around with the batch size, but I realized that this was unnecessary. TPUs have 8 shards, so you simply multiple the GPU batch size by 8 and that should be a good baseline. 

The model I am currently training on a TPU and a GPU simultaneously is training 3-4x faster on the TPU than on the GPU and the code is exactly the same. I have this block of code:

use_tpu = True
# if we are using the tpu copy the keras model to a new var and assign the tpu model to model
if use_tpu:
    TPU_WORKER = 'grpc://' + os.environ['COLAB_TPU_ADDR']
    
    # create network and compiler
    tpu_model = tf.contrib.tpu.keras_to_tpu_model(
        model, strategy = tf.contrib.tpu.TPUDistributionStrategy(
            tf.contrib.cluster_resolver.TPUClusterResolver(TPU_WORKER)))
    
    BATCH_SIZE = BATCH_SIZE * 8

The model is created with Keras and the only change I make is setting use_tpu to True on the TPU instance. 

One other thing I thought I would mention is that CoLab creates separate instances for GPU, TPU and CPU, so you can run multiple notebooks without sharing RAM or processor if you give each one a different type.

Libellés: machine_learning, tensorflow, google, google_cloud

Comments

Please login to comment

    TommieBrirl
Nov 27, 2018 at 12:49 pm

Привет всем! прикольный у вас сайт! Нашел топ базу кино: <b> мультфильмы 2018 смотреть бесплатно в хорошем качестве </b> <a href=http://kinovalenok.tv/>http://kinovalenok.tv/</a> Тут: <a href=http://kinovalenok.tv/fentezi/>фэнтези смотреть бесплатно в хорошем</a> фильмы фэнтези онлайн хорошее качество список 2019 Здесь: фильмы драмы смотреть хорошем качестве http://kinovalenok.tv/drama/ список 2019 Тут: Документальные фильмы онлайн смотреть http://kinovalenok.tv/dokumentalnyy/ список 2019 Здесь: http://kinovalenok.tv/detektiv/5281-dzhek-potroshitel-jack-the-ripper-1988.html <b> Смотреть Джек-потрошитель / Jack the Ripper (1988) онлайн бесплатно </b> Тут: http://kinovalenok.tv/news/11260-obyavlena-data-premery-filma-lara-kroft.html

    zheng kaijie
Dec 17, 2018 at 06:42 pm

Hello, can I load the file to VM and work with TPU?

    Matlift
Aug 15, 2019 at 02:05 pm

Propecia Cheap Acheter Cialis Discretement Purchasing isotretinoin skin health medication <a href=http://4nrxuk.com>vente viagra</a> 40mg Cialis