1、spdk通过bdev块设备的连接存储设备，支持但不限于rbd、malloc、nvme等，通过调用rpc命令bdev_get_iostat可以获取bdev设备创建以来的io统计信息，主要包括如下信息：读请求字节数、req读取数、写请求的字节数、写请求数量， tick_rate为当前cpu的下每秒tick的数量，ticks为当前bdev tick数量。通过read_latency_ticks读请求tick计数换算成读IO请求秒耗时方法为为read_latency_ticks/tick_rate=479655402684/3000000000=159.885s

# spdk_rpc.py bdev_get_iostat -b bdev0
{
  "tick_rate": 3000000000,
  "ticks": 14982772193835262,
  "bdevs": [
    {
      "name": "bdev0",
      "bytes_read": 6073028608,
      "num_read_ops": 47214,
      "bytes_written": 2677080064,
      "num_write_ops": 20835,
      "bytes_unmapped": 0,
      "num_unmap_ops": 0,
      "read_latency_ticks": 479655402684,
      "write_latency_ticks": 257575188258,
      "unmap_latency_ticks": 0
    }
  ]
}

2、针对inflight IO spdk没有直接观察的命令，需要通过rpc指令bdev_set_qd_sampling_period命令周期取样观察比如：采用周期5000us

spdk_rpc.py bdev_set_qd_sampling_period bdev0 5000

此时再次调用1中的bdev_get_iostat命令获取该bdev设备的io统计信息，会增加queue_depth字段，若有inflight IO，对应值为未完成IO的数量.

# spdk_rpc.py bdev_get_iostat -b bdev0
{
  "tick_rate": 3000000000,
  "ticks": 14982772193835262,
  "bdevs": [
    {
      "name": "bdev0",
      "bytes_read": 6073028608,
      "num_read_ops": 47214,
      "bytes_written": 2677080064,
      "num_write_ops": 20835,
      "bytes_unmapped": 0,
      "num_unmap_ops": 0,
      "read_latency_ticks": 479655402684,
      "write_latency_ticks": 257575188258,
      "unmap_latency_ticks": 0，
      "queue_depth_polling_period": 5000,
      "queue_depth": 0,
      "io_time": 0,
      "weighted_io_time": 0
    }
  ]

注：rpc指令bdev_set_qd_sampling_period将采样周期改0即可关闭周期性取样。

3、基于NVMeOF场景，通过2中打开观察源端和nvmf端的io统计取样，通过queue_depth值对比，即可初步判断IO是哪一段路径IO没回。