Util: Limited by GPU Utilization
This means the GPU is idle. This isn't throttling, it means the GPU is waiting for work to do.
If it was throttling due to temperature it would say Thrm. If it was throttling due to the power limit it would say Pwr.
if you look at the PerfCab Reason graph log section, you can see different colors , green, blue and gray. each of them as you know belong to a different category .
the data is cached in my memory so there is always something to be computed. thats why the gpu load is maxed out at 100% and then suddenly it becomes 0% and then again it becomes 100. this can be seen by literally looking at the the graphs drawn in red.
and also this log shows the card is already executing commands and its not idle for input data (data gets cached at the very beginning)
I0428 00:55:28.371031 22960 net.cpp:676] Ignoring source layer accuracy_top-5-train
I0428 00:55:28.693717 22960 blocking_queue.cpp:49] Waiting for data
I0428 00:57:00.091953 20704 data_layer.cpp:73] Restarting data prefetching from start.
I0428 00:57:00.169011 22960 solver.cpp:398] Test net output #0: accuracy_top-1 = 0.58596
I0428 00:57:00.169011 22960 solver.cpp:398] Test net output #1: accuracy_top-5 = 0.820521
I0428 00:57:00.169011 22960 solver.cpp:398] Test net output #2: loss = 1.74516 (* 1 = 1.74516 loss)
I0428 00:57:00.803012 22960 solver.cpp:219] Iteration 300000 (3244.9 iter/s, 92.4528s/20 iters), loss = 1.11276
I0428 00:57:00.803012 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.71875
I0428 00:57:00.803012 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.898438
I0428 00:57:00.803012 22960 solver.cpp:238] Train net output #2: loss = 1.11276 (* 1 = 1.11276 loss)
I0428 00:57:00.803012 22960 sgd_solver.cpp:105] Iteration 300000, lr = 1e-05
I0428 00:57:10.136579 22960 solver.cpp:219] Iteration 300020 (2.143 iter/s, 9.33272s/20 iters), loss = 1.43738
I0428 00:57:10.143584 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.671875
I0428 00:57:10.143584 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.898438
I0428 00:57:10.143584 22960 solver.cpp:238] Train net output #2: loss = 1.43738 (* 1 = 1.43738 loss)
I0428 00:57:10.143584 22960 sgd_solver.cpp:105] Iteration 300020, lr = 1e-05
I0428 00:57:19.493077 22960 solver.cpp:219] Iteration 300040 (2.13913 iter/s, 9.34958s/20 iters), loss = 1.15417
I0428 00:57:19.500082 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.6875
I0428 00:57:19.500082 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.90625
I0428 00:57:19.500082 22960 solver.cpp:238] Train net output #2: loss = 1.15417 (* 1 = 1.15417 loss)
I0428 00:57:19.500082 22960 sgd_solver.cpp:105] Iteration 300040, lr = 1e-05
I0428 00:57:29.842706 22960 solver.cpp:219] Iteration 300060 (1.93392 iter/s, 10.3417s/20 iters), loss = 1.36165
I0428 00:57:29.849216 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.625
I0428 00:57:29.849216 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.882813
I0428 00:57:29.849216 22960 solver.cpp:238] Train net output #2: loss = 1.36165 (* 1 = 1.36165 loss)
I0428 00:57:29.849216 22960 sgd_solver.cpp:105] Iteration 300060, lr = 1e-05
I0428 00:57:39.647658 22960 solver.cpp:219] Iteration 300080 (2.04126 iter/s, 9.79786s/20 iters), loss = 1.21606
I0428 00:57:39.653666 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.710938
I0428 00:57:39.653666 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.898438
I0428 00:57:39.653666 22960 solver.cpp:238] Train net output #2: loss = 1.21606 (* 1 = 1.21606 loss)
I0428 00:57:39.653666 22960 sgd_solver.cpp:105] Iteration 300080, lr = 1e-05
I0428 00:57:49.135305 22960 solver.cpp:219] Iteration 300100 (2.10946 iter/s, 9.4811s/20 iters), loss = 1.07624
I0428 00:57:49.141811 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.695313
I0428 00:57:49.141811 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.890625
I0428 00:57:49.141811 22960 solver.cpp:238] Train net output #2: loss = 1.07624 (* 1 = 1.07624 loss)
I0428 00:57:49.141811 22960 sgd_solver.cpp:105] Iteration 300100, lr = 1e-05
I0428 00:57:59.200597 22960 solver.cpp:219] Iteration 300120 (1.98837 iter/s, 10.0585s/20 iters), loss = 1.35399
I0428 00:57:59.200597 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.71875
I0428 00:57:59.200597 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.882813
I0428 00:57:59.200597 22960 solver.cpp:238] Train net output #2: loss = 1.35399 (* 1 = 1.35399 loss)
I0428 00:57:59.200597 22960 sgd_solver.cpp:105] Iteration 300120, lr = 1e-05
I0428 00:58:11.194643 22960 solver.cpp:219] Iteration 300140 (1.66765 iter/s, 11.993s/20 iters), loss = 1.33887
I0428 00:58:11.200647 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.695313
I0428 00:58:11.200647 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.875
I0428 00:58:11.200647 22960 solver.cpp:238] Train net output #2: loss = 1.33887 (* 1 = 1.33887 loss)
I0428 00:58:11.200647 22960 sgd_solver.cpp:105] Iteration 300140, lr = 1e-05
I0428 00:58:21.040182 22960 solver.cpp:219] Iteration 300160 (2.03268 iter/s, 9.83924s/20 iters), loss = 1.32039
I0428 00:58:21.047205 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.648438
I0428 00:58:21.047205 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.882813
I0428 00:58:21.047205 22960 solver.cpp:238] Train net output #2: loss = 1.32039 (* 1 = 1.32039 loss)
I0428 00:58:21.047205 22960 sgd_solver.cpp:105] Iteration 300160, lr = 1e-05
I0428 00:58:30.830806 22960 solver.cpp:219] Iteration 300180 (2.04419 iter/s, 9.78385s/20 iters), loss = 1.58379
I0428 00:58:30.837810 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.617188
I0428 00:58:30.837810 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.828125
I0428 00:58:30.837810 22960 solver.cpp:238] Train net output #2: loss = 1.58379 (* 1 = 1.58379 loss)
I0428 00:58:30.837810 22960 sgd_solver.cpp:105] Iteration 300180, lr = 1e-05
I0428 00:58:43.533479 22960 solver.cpp:219] Iteration 300200 (1.57546 iter/s, 12.6947s/20 iters), loss = 1.29555
I0428 00:58:43.533479 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.640625
I0428 00:58:43.533479 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.882813
I0428 00:58:43.533479 22960 solver.cpp:238] Train net output #2: loss = 1.29555 (* 1 = 1.29555 loss)
I0428 00:58:43.533479 22960 sgd_solver.cpp:105] Iteration 300200, lr = 1e-05
I0428 00:58:57.092994 22960 solver.cpp:219] Iteration 300220 (1.47509 iter/s, 13.5585s/20 iters), loss = 1.3301
I0428 00:58:57.092994 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.664063
I0428 00:58:57.092994 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.875
I0428 00:58:57.092994 22960 solver.cpp:238] Train net output #2: loss = 1.3301 (* 1 = 1.3301 loss)
I0428 00:58:57.092994 22960 sgd_solver.cpp:105] Iteration 300220, lr = 1e-05
I0428 00:59:27.482066 22960 solver.cpp:219] Iteration 300240 (0.658147 iter/s, 30.3884s/20 iters), loss = 1.21879
I0428 00:59:27.482573 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.679688
I0428 00:59:27.482573 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.898438
I0428 00:59:27.482573 22960 solver.cpp:238] Train net output #2: loss = 1.21879 (* 1 = 1.21879 loss)
I0428 00:59:27.482573 22960 sgd_solver.cpp:105] Iteration 300240, lr = 1e-05
I0428 00:59:54.203022 22960 solver.cpp:219] Iteration 300260 (0.748511 iter/s, 26.7197s/20 iters), loss = 1.53185
I0428 00:59:54.203022 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.578125
I0428 00:59:54.203022 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.835938
I0428 00:59:54.203022 22960 solver.cpp:238] Train net output #2: loss = 1.53185 (* 1 = 1.53185 loss)
I0428 00:59:54.203022 22960 sgd_solver.cpp:105] Iteration 300260, lr = 1e-05
I0428 01:00:31.217540 22960 solver.cpp:219] Iteration 300280 (0.54035 iter/s, 37.0131s/20 iters), loss = 1.22318
I0428 01:00:31.217540 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.632813
I0428 01:00:31.217540 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.882813
I0428 01:00:31.217540 22960 solver.cpp:238] Train net output #2: loss = 1.22318 (* 1 = 1.22318 loss)
I0428 01:00:31.217540 22960 sgd_solver.cpp:105] Iteration 300280, lr = 1e-05
I0428 01:01:23.414973 22960 solver.cpp:219] Iteration 300300 (0.383174 iter/s, 52.1956s/20 iters), loss = 1.28781
I0428 01:01:23.414973 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.664063
I0428 01:01:23.414973 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.890625
I0428 01:01:23.414973 22960 solver.cpp:238] Train net output #2: loss = 1.28781 (* 1 = 1.28781 loss)
I0428 01:01:23.414973 22960 sgd_solver.cpp:105] Iteration 300300, lr = 1e-05
I0428 01:02:11.234027 22960 solver.cpp:219] Iteration 300320 (0.418257 iter/s, 47.8175s/20 iters), loss = 1.27358
I0428 01:02:11.234027 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.6875
I0428 01:02:11.234027 22960 solver.cpp:238] Train net output #1: accuracy_top-5-train = 0.867188
I0428 01:02:11.234027 22960 solver.cpp:238] Train net output #2: loss = 1.27358 (* 1 = 1.27358 loss)
I0428 01:02:11.234027 22960 sgd_solver.cpp:105] Iteration 300320, lr = 1e-05
I0428 01:02:53.814581 22960 solver.cpp:219] Iteration 300340 (0.469711 iter/s, 42.5793s/20 iters), loss = 1.19984
I0428 01:02:53.814581 22960 solver.cpp:238] Train net output #0: accuracy_top-1-train = 0.695313