Query failed and errored '[VRAM] allocateInTier OOM: Exceeds Tier Capacity'.
2020-08-14 07:24:49.575 WARN (37302,40881,r8/gpudb_wi_c_219 ) host.domain ResourceManagement/MemoryTierManager.cpp:432 - JobId:4035484; [VRAM] allocateInTier OOM: Exceeds Tier Capacity
2020-08-14 07:24:49.575 WARN (37302,40881,r8/gpudb_wi_c_219 ) host.domain cuda/DeviceMemoryPtr.cpp:133 - VRAM allocation using cudaHostAlloc, size: 2048000000, attempt: 1
2020-08-14 07:24:53.414 WARN (37302,40881,r8/gpudb_wi_c_219 ) host.domain ResourceManagement/MemoryTierManager.cpp:837 - JobId:4035484; [VRAM] [Allocate] exceeds capacity, needed: 4136250367 free: 359383650
- Further analysis is needed to check tables cardinality involved in running jobs. Some of the columns are hopefully eligible to be dictionary encoded, thus effectively saving memory.
To check dictionary encoding eligibility, go to Table > Select the table name > click Stats on the top ribbon. Stats page will show you which column have low to medium cardinality and should be dict encoded.
- Secondly, Check value of max_concurrent_kernels parameter in gpudb.conf file. Too high of value can cause too many jobs to be sent to the VRAM and exhausting VRAM, causing the error.
In the case of max_concurrent_kernels parameter value, you can try to set the value starting from 2 afterward increase iteratively by closely monitoring the system for finding the balance between performance and system capacity.
Refer to concept of evictability and related documentations: