-----============= acceptance-small: conf-sanity ============----- Mon Dec 2 13:26:26 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg152-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg152-server oleg152-server: oleg152-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg152-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg152-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg152-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg152-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:27:41 (1733164061) Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) oleg152-server: oleg152-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg152-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg152-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1655 (time 1733164106.42 total 10.00 last 165.48) - open/close 3481 (time 1733164116.43 total 20.00 last 182.54) - open/close 5146 (time 1733164126.43 total 30.01 last 166.42) - open/close 6838 (time 1733164136.44 total 40.02 last 169.09) - open/close 8406 (time 1733164146.44 total 50.02 last 156.71) - open/close 10000 (time 1733164154.70 total 58.28 last 193.02) - open/close 11703 (time 1733164164.70 total 68.28 last 170.26) - open/close 13342 (time 1733164174.71 total 78.28 last 163.85) - open/close 15048 (time 1733164184.71 total 88.29 last 170.55) - open/close 16848 (time 1733164194.72 total 98.29 last 179.89) - open/close 18498 (time 1733164204.72 total 108.30 last 164.97) - open/close 20000 (time 1733164212.88 total 116.46 last 184.02) - open/close 22468 (time 1733164222.88 total 126.46 last 246.71) - open/close 24897 (time 1733164232.89 total 136.47 last 242.79) - open/close 27061 (time 1733164242.89 total 146.47 last 216.32) - open/close 29568 (time 1733164252.90 total 156.47 last 250.61) - open/close 32187 (time 1733164262.90 total 166.47 last 261.85) - open/close 34913 (time 1733164272.90 total 176.48 last 272.54) - open/close 37534 (time 1733164282.90 total 186.48 last 262.00) - open/close 39647 (time 1733164292.91 total 196.49 last 211.18) - open/close 42411 (time 1733164302.91 total 206.49 last 276.39) - open/close 45032 (time 1733164312.91 total 216.49 last 262.07) - open/close 47714 (time 1733164322.91 total 226.49 last 268.19) - open/close 50000 (time 1733164332.00 total 235.57 last 251.62) - open/close 52409 (time 1733164342.00 total 245.58 last 240.79) - open/close 54979 (time 1733164352.00 total 255.58 last 256.98) - open/close 57726 (time 1733164362.00 total 265.58 last 274.62) - open/close 60000 (time 1733164371.32 total 274.90 last 244.02) - open/close 62682 (time 1733164381.33 total 284.90 last 268.10) - open/close 64760 (time 1733164391.33 total 294.91 last 207.72) total: 64768 open/close in 294.95 seconds: 219.59 ops/second - /unlink 3807 (time 1733164402.59 total 10.00 last 380.65) - /unlink 8947 (time 1733164412.59 total 20.00 last 513.92) - /unlink 14598 (time 1733164422.59 total 30.00 last 565.07) - /unlink 18720 (time 1733164432.59 total 40.00 last 412.18) - /unlink 20000 (time 1733164435.16 total 42.57 last 498.51) - /unlink 25110 (time 1733164445.16 total 52.57 last 510.95) - /unlink 29883 (time 1733164455.16 total 62.57 last 477.20) - /unlink 34676 (time 1733164465.16 total 72.58 last 479.26) - /unlink 39194 (time 1733164475.16 total 82.58 last 451.78) - /unlink 40000 (time 1733164477.18 total 84.59 last 399.54) - /unlink 46130 (time 1733164487.18 total 94.59 last 612.95) - /unlink 50000 (time 1733164493.59 total 101.00 last 604.04) - /unlink 56635 (time 1733164503.59 total 111.00 last 663.47) - /unlink 60000 (time 1733164509.28 total 116.69 last 591.43) total: 64768 /unlink in 124.09 seconds: 521.93 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2849 (time 1733164543.50 total 10.00 last 284.87) - open/close 6919 (time 1733164553.50 total 20.00 last 406.94) - open/close 10000 (time 1733164560.65 total 27.15 last 430.81) - open/close 13790 (time 1733164570.65 total 37.16 last 378.95) - open/close 17599 (time 1733164580.65 total 47.16 last 380.84) - open/close 20000 (time 1733164586.33 total 52.83 last 423.06) - open/close 24043 (time 1733164596.33 total 62.83 last 404.27) - open/close 27385 (time 1733164606.33 total 72.83 last 334.18) - open/close 30000 (time 1733164614.65 total 81.15 last 314.41) - open/close 33472 (time 1733164624.65 total 91.15 last 347.18) - open/close 36528 (time 1733164634.65 total 101.15 last 305.53) - open/close 40000 (time 1733164643.25 total 109.75 last 403.99) - open/close 44644 (time 1733164653.25 total 119.75 last 464.39) - open/close 48414 (time 1733164663.25 total 129.75 last 376.93) - open/close 50000 (time 1733164667.95 total 134.45 last 337.29) - open/close 53250 (time 1733164677.95 total 144.46 last 324.91) - open/close 56537 (time 1733164687.95 total 154.46 last 328.64) - open/close 60000 (time 1733164696.15 total 162.65 last 422.54) - open/close 64429 (time 1733164706.15 total 172.65 last 442.87) total: 64768 open/close in 173.37 seconds: 373.57 ops/second - /unlink 7561 (time 1733164717.72 total 10.00 last 756.10) - /unlink 10000 (time 1733164721.27 total 13.55 last 686.60) - /unlink 16780 (time 1733164731.27 total 23.55 last 677.90) - /unlink 20000 (time 1733164735.85 total 28.13 last 703.47) - /unlink 26819 (time 1733164745.85 total 38.13 last 681.81) - /unlink 30000 (time 1733164750.26 total 42.54 last 720.96) - /unlink 37490 (time 1733164760.26 total 52.55 last 748.90) - /unlink 40000 (time 1733164763.30 total 55.58 last 827.23) - /unlink 48639 (time 1733164773.30 total 65.58 last 863.87) - /unlink 57043 (time 1733164783.30 total 75.58 last 840.39) - /unlink 60000 (time 1733164786.60 total 78.88 last 895.97) total: 64768 /unlink in 84.73 seconds: 764.36 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5177 (time 1733164823.78 total 10.00 last 517.70) - open/close 10000 (time 1733164833.34 total 19.56 last 504.63) - open/close 15024 (time 1733164843.34 total 29.56 last 502.39) - open/close 20000 (time 1733164853.32 total 39.54 last 498.48) - open/close 24946 (time 1733164863.32 total 49.54 last 494.57) - open/close 29978 (time 1733164873.32 total 59.54 last 503.12) - open/close 34821 (time 1733164883.32 total 69.54 last 484.23) - open/close 39476 (time 1733164893.33 total 79.54 last 465.44) - open/close 44321 (time 1733164903.33 total 89.55 last 484.45) - open/close 48775 (time 1733164913.33 total 99.55 last 445.32) - open/close 50000 (time 1733164916.13 total 102.35 last 436.62) - open/close 54919 (time 1733164926.14 total 112.36 last 491.82) - open/close 59691 (time 1733164936.14 total 122.36 last 477.10) - open/close 64655 (time 1733164946.14 total 132.36 last 496.36) total: 64768 open/close in 132.58 seconds: 488.51 ops/second - /unlink 8462 (time 1733164957.25 total 10.00 last 846.16) - /unlink 16943 (time 1733164967.25 total 20.00 last 848.02) - /unlink 20000 (time 1733164970.71 total 23.46 last 883.33) - /unlink 28810 (time 1733164980.71 total 33.46 last 880.92) - /unlink 37471 (time 1733164990.71 total 43.46 last 866.06) - /unlink 40000 (time 1733164993.72 total 46.48 last 839.30) - /unlink 48542 (time 1733165003.72 total 56.48 last 854.11) - /unlink 57220 (time 1733165013.72 total 66.48 last 867.77) - /unlink 60000 (time 1733165016.72 total 69.48 last 927.18) total: 64768 /unlink in 74.65 seconds: 867.57 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 1531 (time 1733165059.88 total 10.00 last 153.08) - open/close 4430 (time 1733165069.88 total 20.00 last 289.89) - open/close 7970 (time 1733165079.89 total 30.00 last 353.90) - open/close 10000 (time 1733165089.60 total 39.71 last 209.05) - open/close 12030 (time 1733165099.60 total 49.71 last 203.00) - open/close 14977 (time 1733165109.60 total 59.72 last 294.64) - open/close 19714 (time 1733165119.60 total 69.72 last 473.69) - open/close 23463 (time 1733165129.60 total 79.72 last 374.75) - open/close 26994 (time 1733165139.61 total 89.73 last 352.89) - open/close 29417 (time 1733165149.61 total 99.73 last 242.30) - open/close 30000 (time 1733165151.90 total 102.02 last 254.75) - open/close 33550 (time 1733165161.90 total 112.02 last 354.94) - open/close 38444 (time 1733165171.90 total 122.02 last 489.37) - open/close 40000 (time 1733165175.63 total 125.75 last 417.24) - open/close 45075 (time 1733165185.63 total 135.75 last 507.40) - open/close 50000 (time 1733165194.64 total 144.76 last 546.63) - open/close 54821 (time 1733165204.64 total 154.76 last 482.07) - open/close 59696 (time 1733165214.64 total 164.76 last 487.40) - open/close 64493 (time 1733165224.64 total 174.76 last 479.69) total: 64768 open/close in 175.38 seconds: 369.29 ops/second - /unlink 8428 (time 1733165236.17 total 10.00 last 842.74) - /unlink 16665 (time 1733165246.17 total 20.00 last 823.63) - /unlink 20000 (time 1733165250.37 total 24.20 last 794.22) - /unlink 28871 (time 1733165260.37 total 34.20 last 887.10) - /unlink 37605 (time 1733165270.37 total 44.20 last 873.35) - /unlink 40000 (time 1733165273.95 total 47.78 last 669.51) - /unlink 46376 (time 1733165283.95 total 57.78 last 637.57) - /unlink 50000 (time 1733165288.88 total 62.71 last 734.70) - /unlink 57451 (time 1733165298.88 total 72.71 last 745.03) - /unlink 60000 (time 1733165302.79 total 76.62 last 651.80) total: 64768 /unlink in 84.12 seconds: 769.92 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2973 (time 1733165340.22 total 10.00 last 297.28) - open/close 6166 (time 1733165350.22 total 20.00 last 319.29) - open/close 9867 (time 1733165360.22 total 30.00 last 370.09) - open/close 13674 (time 1733165370.23 total 40.01 last 380.55) - open/close 16483 (time 1733165380.23 total 50.01 last 280.84) - open/close 19364 (time 1733165390.23 total 60.01 last 287.98) - open/close 20000 (time 1733165392.32 total 62.09 last 305.53) - open/close 23273 (time 1733165402.32 total 72.09 last 327.27) - open/close 27030 (time 1733165412.32 total 82.09 last 375.66) - open/close 30000 (time 1733165419.06 total 88.83 last 440.73) - open/close 34420 (time 1733165429.06 total 98.83 last 441.99) - open/close 38545 (time 1733165439.06 total 108.83 last 412.46) - open/close 40000 (time 1733165441.94 total 111.72 last 504.28) - open/close 44304 (time 1733165451.94 total 121.72 last 430.34) - open/close 48327 (time 1733165461.94 total 131.72 last 402.29) - open/close 50000 (time 1733165465.36 total 135.14 last 489.96) - open/close 54715 (time 1733165475.36 total 145.14 last 471.44) - open/close 58845 (time 1733165485.36 total 155.14 last 412.95) - open/close 60000 (time 1733165487.71 total 157.49 last 491.98) - open/close 64273 (time 1733165497.71 total 167.49 last 427.24) total: 64768 open/close in 168.79 seconds: 383.72 ops/second - /unlink 8316 (time 1733165509.92 total 10.00 last 831.57) - /unlink 16671 (time 1733165519.92 total 20.00 last 835.41) - /unlink 20000 (time 1733165524.43 total 24.51 last 738.57) - /unlink 28743 (time 1733165534.43 total 34.51 last 874.26) - /unlink 36317 (time 1733165544.43 total 44.51 last 757.40) - /unlink 40000 (time 1733165550.17 total 50.25 last 642.02) - /unlink 45615 (time 1733165560.17 total 60.25 last 561.41) - /unlink 50000 (time 1733165567.14 total 67.22 last 628.89) - /unlink 58157 (time 1733165577.14 total 77.22 last 815.70) - /unlink 60000 (time 1733165579.66 total 79.73 last 732.87) total: 64768 /unlink in 84.74 seconds: 764.29 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 PASS 106 (1563s) Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 oleg152-server: oleg152-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg152-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1654 sec ========== 13:54:01 (1733165641) === conf-sanity: start cleanup 13:54:02 (1733165642) === === conf-sanity: finish cleanup 13:54:02 (1733165642) ===