活动

天翼云最新优惠活动，涵盖免费试用，产品折扣等，助您降本增效！
热门活动
免费活动
  • 活动
  • 智算服务
  • 产品
  • 解决方案
  • 应用商城
  • 合作伙伴
  • 开发者
  • 支持与服务
  • 了解天翼云
searchusermenu
  • 发布文章
  • 消息中心
点赞
收藏
评论
分享

Optimizing disk performance for Cassandra

云数据库Cassandra版
2023-08-03 05:11:10
88
0
Best practice for optimizing disk performance for the Cassandra database is to lower the default disk readahead for the drive or partition where your Cassandra data is stored. By default, the Linux kernel reads additional file data so that subsequent reads can be satisfied from the cache. The file access patterns of Cassandra queries result in the readaheads mostly being unused, therefore polluting the cache, driving up I/O time and also results in excessive disk I/O levels.

Before you begin

You can view your current readahead settings with either of these commands: 
lsblk --output NAME,KNAME,TYPE,MAJ:MIN,FSTYPE,SIZE,RA,MOUNTPOINT,LABEL
<button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
blockdev --report
<button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
Examples: 
lsblk --output NAME,KNAME,TYPE,MAJ:MIN,FSTYPE,SIZE,RA,MOUNTPOINT,LABEL
NAME          KNAME TYPE MAJ:MIN FSTYPE       SIZE   RA MOUNTPOINT LABEL
fd0           fd0   disk   2:0                  4K  128
sda           sda   disk   8:0                 80G 4096
├─sda1        sda1  part   8:1   xfs            1G 4096 /boot
└─sda2        sda2  part   8:2   LVM2_member   79G 4096
  ├─rhel-root dm-0  lvm  253:0   xfs           75G 4096 /
  └─rhel-swap dm-1  lvm  253:1   swap           4G 4096 [SWAP]
sdb           sdb   disk   8:16  xfs          100G 4096 /docker
sdc           sde   disk   8:32                 2T  128
<button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
blockdev --report
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw   256   512  4096          0            4096   /dev/fd0
rw  8192   512  4096          0     85899345920   /dev/sda
rw  8192   512   512       2048      1073741824   /dev/sda1
rw  8192   512  4096    2099200     84824555520   /dev/sda2
rw  8192   512   512          0    107374182400   /dev/sdb
rw  8192   512   512          0     80530636800   /dev/dm-0
rw  8192   512  4096          0      4290772992   /dev/dm-1
rw   256   512  4096          0   2148557389824   /dev/sdc
<button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
Looking at the RA and Size columns, the readahead of 8192 combined with the size of 512 results in a readahead of 4096 KB. That means any read on the / root drive results in 4 MB of disk I/O into the system cache. Best practice is to use a separate drive for the Cassandra data, as well as the other StatefulSet services requiring disk space.

About this task

The following steps provide two examples of how to modify the readahead settings for an existing drive by using the command line or the tuned service to ensure better performance for Cassandra. You can manually set the readahead on an existing drive or modify the tuned.services disk settings to make the readahead settings persistent. These steps need to be performed on each VM running Cassandra.

Procedure

Manually set readahead on an existing drive or volume.

  • To set the readahead, use the blockdev command with the internal kernel device name (KNAME):
    1. To find the KNAME of the device to modify, run the following command: 
      lsblk --output NAME,KNAME,TYPE,MAJ:MIN,FSTYPE,SIZE,RA,MOUNTPOINT,LABEL
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
      In this example, Cassandra is running on dm-2. Your KNAME might be different based on your system settings: 
      NAME                  KNAME TYPE MAJ:MIN FSTYPE       SIZE   RA MOUNTPOINT          LABEL
fd0                   fd0   disk   2:0                  4K  128
sda                   sda   disk   8:0                 80G 4096
├─sda1                sda1  part   8:1   xfs            1G 4096 /boot
└─sda2                sda2  part   8:2   LVM2_member   79G 4096
  ├─rhel-root         dm-0  lvm  253:0   xfs           75G 4096 /
  └─rhel-swap         dm-1  lvm  253:1   swap           4G 4096 [SWAP]
sdb                   sdb   disk   8:16  xfs          100G 4096 /docker
sdc                   sdc   disk   8:32  LVM2_member    2T 4096
├─vg_sdc-lv_cassandra dm-2  lvm  253:2   xfs            2T 4096 /k8s/data/cassandra cassandra
    2. Enter the blockdev command with --setra in number of blocks (for example, a readahead of 16 with size of 512 bytes results in an 8KB readahead): 
      blockdev --setra <var class="keyword varname">16</var> <var class="keyword varname">device
</var>
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
      Example: blockdev --setra 16 /dev/dm-2
    3. Verify the readahead settings: 
      lsblk --output NAME,KNAME,TYPE,MAJ:MIN,FSTYPE,SIZE,RA,MOUNTPOINT,LABEL
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
      In this example, Cassandra is running on dm-2: 
      NAME                  KNAME TYPE MAJ:MIN FSTYPE       SIZE   RA MOUNTPOINT          LABEL
fd0                   fd0   disk   2:0                  4K  128
sda                   sda   disk   8:0                 80G 4096
├─sda1                sda1  part   8:1   xfs            1G 4096 /boot
└─sda2                sda2  part   8:2   LVM2_member   79G 4096
  ├─rhel-root         dm-0  lvm  253:0   xfs           75G 4096 /
  └─rhel-swap         dm-1  lvm  253:1   swap           4G 4096 [SWAP]
sdb                   sdb   disk   8:16  xfs          100G 4096 /docker
sdc                   sdc   disk   8:32  LVM2_member    2T 4096
├─vg_sdc-lv_cassandra dm-2  lvm  253:2   xfs            2T 8 /k8s/data/cassandra cassandra
    4. If you are modifying a running environment, restart the Cassandra Docker container to use the new readahead values.
      You can restart the Cassandra Docker container as an IBM® Cloud Private admin, either through the IBM Cloud Private UI or kubectl.

With this method, the tuned service adjusts the configuration settings to optimize system performance. Tuned profiles overwrite the smaller readahead setting used in the LVM setup. The tuned service adjusts the configuration settings to optimize system performance. The service can modify settings such as disk device readahead. Tuned profiles overwrite the smaller readahead setting used in the LVM setup. To prevent the overwrite, add the setting to your tuned profile. For more information, see Performance tuning with tuned and tuned-adm in the Red Hat Performance Tuning Guide.

  • Modify tuned.service disk settings to make readahead persistent:
    1. Format a blank drive for the Cassandra data to be stored as described in Configuring the disk drives for services.
    2. Use the tuned-adm command to see the current active profile: 
      tuned-adm active
Current active profile: virtual-guest
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
      The output in this example shows that the active profile is virtual-guest. Note: Your profile and configuration may be different.
    3. Copy the profile to the /etc/tuned directory.
      The default profile definitions are stored in /usr/lib/tuned/.
      In our example, the definitions are in /usr/lib/tuned/virtual-guest/tuned.conf. The definitions for virtual-guest contain include=throughput-performance, which means the settings inherit the settings of throughput-performance. Looking at /usr/lib/tuned/throughput-performance/tuned.conf, we see that this is where the readahead=>4096 is being set. 
      cp -a /usr/lib/tuned/throughput-performance/ /etc/tuned/
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
    4. Copy the profile to the /etc/tuned directory.
      The default profile definitions are stored in /usr/lib/tuned/.
      In our example, the definitions are in /usr/lib/tuned/virtual-guest/tuned.conf. The definitions for virtual-guest contain include=throughput-performance, which means the settings inherit the settings of throughput-performance. Looking at /usr/lib/tuned/throughput-performance/tuned.conf, we see that this is where the readahead=>4096 is being set. 
      cp -a /usr/lib/tuned/throughput-performance/ /etc/tuned/
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
    5. Add the following section to the /etc/tuned/throughput-performance/tuned.conf file, making sure that it is above the existing [disk] section. 
      [disk-cassandra]
type=disk
devices=dm-2
readahead=8
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
    6. Reload the tuned profile: 
      tuned-adm profile virtual-guest
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
    7. Verify the new readahead setting: 
      lsblk --output NAME,KNAME,TYPE,MAJ:MIN,FSTYPE,SIZE,RA,MOUNTPOINT,LABEL
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
      Cassandra is running on dm-2 in this example: 
      NAME                  KNAME TYPE MAJ:MIN FSTYPE       SIZE   RA MOUNTPOINT          LABEL
fd0                   fd0   disk   2:0                  4K  128
sda                   sda   disk   8:0                 80G 4096
├─sda1                sda1  part   8:1   xfs            1G 4096 /boot
└─sda2                sda2  part   8:2   LVM2_member   79G 4096
  ├─rhel-root         dm-0  lvm  253:0   xfs           75G 4096 /
  └─rhel-swap         dm-1  lvm  253:1   swap           4G 4096 [SWAP]
sdb                   sdb   disk   8:16  xfs          100G 4096 /docker
sdc                   sdc   disk   8:32  LVM2_member    2T 4096
├─vg_sdc-lv_cassandra dm-2  lvm  253:2   xfs            2T 8 /k8s/data/cassandra cassandra
    8. If you are modifying a running environment, restart the Cassandra Docker container to use the new readahead values.
      You can restart the Cassandra Docker container as an IBM Cloud Private admin, either through the IBM Cloud Private UI or kubectl.

版权声明：本文内容系天翼云实名用户自发贡献，版权归原作者所有，天翼云开发者社区不拥有其著作权，亦不承担相应法律责任，原文链接：https://www.ibm.com/docs/en/cloud-app-management/2019.4.0?topic=performance-optimizing-disk-cassandra

作者：ibm，如需转载，请联系作者。

原文出处及转载信息见文内详细说明，如有疑问或争议，请联系ctyunbbs@chinatelecom.cn删除。

0条评论
0 / 1000
d****n
4文章数
0粉丝数
d****n
4 文章 | 0 粉丝
Ta的热门文章查看更多
worker节点calico无法启动定位分析Optimizing disk performance for Cassandraregctl 源码分析使用ansible执行shell命令的正确姿势
d****n
4文章数
0粉丝数
d****n
4 文章 | 0 粉丝

Optimizing disk performance for Cassandra

云数据库Cassandra版
2023-08-03 05:11:10
88
0
Best practice for optimizing disk performance for the Cassandra database is to lower the default disk readahead for the drive or partition where your Cassandra data is stored. By default, the Linux kernel reads additional file data so that subsequent reads can be satisfied from the cache. The file access patterns of Cassandra queries result in the readaheads mostly being unused, therefore polluting the cache, driving up I/O time and also results in excessive disk I/O levels.

Before you begin

You can view your current readahead settings with either of these commands: 
lsblk --output NAME,KNAME,TYPE,MAJ:MIN,FSTYPE,SIZE,RA,MOUNTPOINT,LABEL
<button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
blockdev --report
<button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
Examples: 
lsblk --output NAME,KNAME,TYPE,MAJ:MIN,FSTYPE,SIZE,RA,MOUNTPOINT,LABEL
NAME          KNAME TYPE MAJ:MIN FSTYPE       SIZE   RA MOUNTPOINT LABEL
fd0           fd0   disk   2:0                  4K  128
sda           sda   disk   8:0                 80G 4096
├─sda1        sda1  part   8:1   xfs            1G 4096 /boot
└─sda2        sda2  part   8:2   LVM2_member   79G 4096
  ├─rhel-root dm-0  lvm  253:0   xfs           75G 4096 /
  └─rhel-swap dm-1  lvm  253:1   swap           4G 4096 [SWAP]
sdb           sdb   disk   8:16  xfs          100G 4096 /docker
sdc           sde   disk   8:32                 2T  128
<button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
blockdev --report
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw   256   512  4096          0            4096   /dev/fd0
rw  8192   512  4096          0     85899345920   /dev/sda
rw  8192   512   512       2048      1073741824   /dev/sda1
rw  8192   512  4096    2099200     84824555520   /dev/sda2
rw  8192   512   512          0    107374182400   /dev/sdb
rw  8192   512   512          0     80530636800   /dev/dm-0
rw  8192   512  4096          0      4290772992   /dev/dm-1
rw   256   512  4096          0   2148557389824   /dev/sdc
<button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
Looking at the RA and Size columns, the readahead of 8192 combined with the size of 512 results in a readahead of 4096 KB. That means any read on the / root drive results in 4 MB of disk I/O into the system cache. Best practice is to use a separate drive for the Cassandra data, as well as the other StatefulSet services requiring disk space.

About this task

The following steps provide two examples of how to modify the readahead settings for an existing drive by using the command line or the tuned service to ensure better performance for Cassandra. You can manually set the readahead on an existing drive or modify the tuned.services disk settings to make the readahead settings persistent. These steps need to be performed on each VM running Cassandra.

Procedure

Manually set readahead on an existing drive or volume.

  • To set the readahead, use the blockdev command with the internal kernel device name (KNAME):
    1. To find the KNAME of the device to modify, run the following command: 
      lsblk --output NAME,KNAME,TYPE,MAJ:MIN,FSTYPE,SIZE,RA,MOUNTPOINT,LABEL
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
      In this example, Cassandra is running on dm-2. Your KNAME might be different based on your system settings: 
      NAME                  KNAME TYPE MAJ:MIN FSTYPE       SIZE   RA MOUNTPOINT          LABEL
fd0                   fd0   disk   2:0                  4K  128
sda                   sda   disk   8:0                 80G 4096
├─sda1                sda1  part   8:1   xfs            1G 4096 /boot
└─sda2                sda2  part   8:2   LVM2_member   79G 4096
  ├─rhel-root         dm-0  lvm  253:0   xfs           75G 4096 /
  └─rhel-swap         dm-1  lvm  253:1   swap           4G 4096 [SWAP]
sdb                   sdb   disk   8:16  xfs          100G 4096 /docker
sdc                   sdc   disk   8:32  LVM2_member    2T 4096
├─vg_sdc-lv_cassandra dm-2  lvm  253:2   xfs            2T 4096 /k8s/data/cassandra cassandra
    2. Enter the blockdev command with --setra in number of blocks (for example, a readahead of 16 with size of 512 bytes results in an 8KB readahead): 
      blockdev --setra <var class="keyword varname">16</var> <var class="keyword varname">device
</var>
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
      Example: blockdev --setra 16 /dev/dm-2
    3. Verify the readahead settings: 
      lsblk --output NAME,KNAME,TYPE,MAJ:MIN,FSTYPE,SIZE,RA,MOUNTPOINT,LABEL
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
      In this example, Cassandra is running on dm-2: 
      NAME                  KNAME TYPE MAJ:MIN FSTYPE       SIZE   RA MOUNTPOINT          LABEL
fd0                   fd0   disk   2:0                  4K  128
sda                   sda   disk   8:0                 80G 4096
├─sda1                sda1  part   8:1   xfs            1G 4096 /boot
└─sda2                sda2  part   8:2   LVM2_member   79G 4096
  ├─rhel-root         dm-0  lvm  253:0   xfs           75G 4096 /
  └─rhel-swap         dm-1  lvm  253:1   swap           4G 4096 [SWAP]
sdb                   sdb   disk   8:16  xfs          100G 4096 /docker
sdc                   sdc   disk   8:32  LVM2_member    2T 4096
├─vg_sdc-lv_cassandra dm-2  lvm  253:2   xfs            2T 8 /k8s/data/cassandra cassandra
    4. If you are modifying a running environment, restart the Cassandra Docker container to use the new readahead values.
      You can restart the Cassandra Docker container as an IBM® Cloud Private admin, either through the IBM Cloud Private UI or kubectl.

With this method, the tuned service adjusts the configuration settings to optimize system performance. Tuned profiles overwrite the smaller readahead setting used in the LVM setup. The tuned service adjusts the configuration settings to optimize system performance. The service can modify settings such as disk device readahead. Tuned profiles overwrite the smaller readahead setting used in the LVM setup. To prevent the overwrite, add the setting to your tuned profile. For more information, see Performance tuning with tuned and tuned-adm in the Red Hat Performance Tuning Guide.

  • Modify tuned.service disk settings to make readahead persistent:
    1. Format a blank drive for the Cassandra data to be stored as described in Configuring the disk drives for services.
    2. Use the tuned-adm command to see the current active profile: 
      tuned-adm active
Current active profile: virtual-guest
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
      The output in this example shows that the active profile is virtual-guest. Note: Your profile and configuration may be different.
    3. Copy the profile to the /etc/tuned directory.
      The default profile definitions are stored in /usr/lib/tuned/.
      In our example, the definitions are in /usr/lib/tuned/virtual-guest/tuned.conf. The definitions for virtual-guest contain include=throughput-performance, which means the settings inherit the settings of throughput-performance. Looking at /usr/lib/tuned/throughput-performance/tuned.conf, we see that this is where the readahead=>4096 is being set. 
      cp -a /usr/lib/tuned/throughput-performance/ /etc/tuned/
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
    4. Copy the profile to the /etc/tuned directory.
      The default profile definitions are stored in /usr/lib/tuned/.
      In our example, the definitions are in /usr/lib/tuned/virtual-guest/tuned.conf. The definitions for virtual-guest contain include=throughput-performance, which means the settings inherit the settings of throughput-performance. Looking at /usr/lib/tuned/throughput-performance/tuned.conf, we see that this is where the readahead=>4096 is being set. 
      cp -a /usr/lib/tuned/throughput-performance/ /etc/tuned/
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
    5. Add the following section to the /etc/tuned/throughput-performance/tuned.conf file, making sure that it is above the existing [disk] section. 
      [disk-cassandra]
type=disk
devices=dm-2
readahead=8
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
    6. Reload the tuned profile: 
      tuned-adm profile virtual-guest
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
    7. Verify the new readahead setting: 
      lsblk --output NAME,KNAME,TYPE,MAJ:MIN,FSTYPE,SIZE,RA,MOUNTPOINT,LABEL
      <button class="bx--copy-btn" tabindex="0" title="Copy to clipboard" type="button" data-copy-btn="" aria-label="Copy to clipboard"></button>
      Cassandra is running on dm-2 in this example: 
      NAME                  KNAME TYPE MAJ:MIN FSTYPE       SIZE   RA MOUNTPOINT          LABEL
fd0                   fd0   disk   2:0                  4K  128
sda                   sda   disk   8:0                 80G 4096
├─sda1                sda1  part   8:1   xfs            1G 4096 /boot
└─sda2                sda2  part   8:2   LVM2_member   79G 4096
  ├─rhel-root         dm-0  lvm  253:0   xfs           75G 4096 /
  └─rhel-swap         dm-1  lvm  253:1   swap           4G 4096 [SWAP]
sdb                   sdb   disk   8:16  xfs          100G 4096 /docker
sdc                   sdc   disk   8:32  LVM2_member    2T 4096
├─vg_sdc-lv_cassandra dm-2  lvm  253:2   xfs            2T 8 /k8s/data/cassandra cassandra
    8. If you are modifying a running environment, restart the Cassandra Docker container to use the new readahead values.
      You can restart the Cassandra Docker container as an IBM Cloud Private admin, either through the IBM Cloud Private UI or kubectl.

版权声明：本文内容系天翼云实名用户自发贡献，版权归原作者所有，天翼云开发者社区不拥有其著作权，亦不承担相应法律责任，原文链接：https://www.ibm.com/docs/en/cloud-app-management/2019.4.0?topic=performance-optimizing-disk-cassandra

作者：ibm，如需转载，请联系作者。

原文出处及转载信息见文内详细说明，如有疑问或争议，请联系ctyunbbs@chinatelecom.cn删除。

文章来自个人专栏
文章 | 订阅
0条评论
0 / 1000
请输入你的评论
0
0
售前咨询热线
400-810-9889转1
关注天翼云
  • 旗舰店
  • 天翼云APP
  • 天翼云微信公众号
服务与支持
账户管理
快速入口
云网生态
了解天翼云
热门产品
热门推荐
更多推荐
友情链接
©2025 天翼云科技有限公司版权所有 增值电信业务经营许可证A2.B1.B2-20090001
公司地址：北京市东城区青龙胡同甲1号、3号2幢2层205-32室
备案京公网安备11010802043424号京ICP备 2021034386号