-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:47:09 EDT 2024 excepting tests: 32newtarball 110 Stopping clients: oleg345-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg345-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg345-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg345-server oleg345-server: oleg345-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: detected 4 online CPUs by sysfs oleg345-server: Force libcfs to create 2 CPU partitions oleg345-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg345-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg345-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg345-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg345-server: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg345-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg345-server: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg345-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg345-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:48:30 (1710100110) Stopping clients: oleg345-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg345-client.virtnet /mnt/lustre2 (opts:-f) oleg345-server: oleg345-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: detected 4 online CPUs by sysfs oleg345-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg345-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg345-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg345-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg345-client.virtnet: -o user_xattr,flock oleg345-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2404 (time 1710100153.25 total 10.00 last 240.35) - open/close 4571 (time 1710100163.25 total 20.00 last 216.66) - open/close 6796 (time 1710100173.25 total 30.00 last 222.50) - open/close 8651 (time 1710100183.25 total 40.00 last 185.50) - open/close 10000 (time 1710100190.57 total 47.32 last 184.38) - open/close 11711 (time 1710100200.57 total 57.33 last 171.00) - open/close 13381 (time 1710100210.58 total 67.33 last 166.90) - open/close 15007 (time 1710100220.58 total 77.33 last 162.55) - open/close 16484 (time 1710100230.59 total 87.34 last 147.59) - open/close 17868 (time 1710100240.59 total 97.35 last 138.36) - open/close 19304 (time 1710100250.60 total 107.35 last 143.52) - open/close 20000 (time 1710100254.97 total 111.72 last 159.15) - open/close 21408 (time 1710100264.97 total 121.73 last 140.73) - open/close 22768 (time 1710100274.98 total 131.74 last 135.90) - open/close 24325 (time 1710100284.99 total 141.74 last 155.63) - open/close 25446 (time 1710100294.99 total 151.75 last 112.04) - open/close 26549 (time 1710100305.00 total 161.75 last 110.21) - open/close 27489 (time 1710100315.00 total 171.76 last 93.95) - open/close 28575 (time 1710100325.00 total 181.76 last 108.59) - open/close 29581 (time 1710100335.01 total 191.77 last 100.53) - open/close 30000 (time 1710100341.69 total 198.44 last 62.75) - open/close 30689 (time 1710100351.69 total 208.45 last 68.88) - open/close 31274 (time 1710100361.69 total 218.45 last 58.49) - open/close 31907 (time 1710100371.70 total 228.45 last 63.28) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 237.98 seconds: 136.82 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 909 (time 1710100392.11 total 10.01 last 90.82) - /unlink 1756 (time 1710100402.11 total 20.01 last 84.70) - /unlink 2770 (time 1710100412.12 total 30.01 last 101.39) - /unlink 3694 (time 1710100422.12 total 40.01 last 92.36) - /unlink 4587 (time 1710100432.13 total 50.02 last 89.21) - /unlink 5266 (time 1710100442.15 total 60.05 last 67.76) - /unlink 5936 (time 1710100452.15 total 70.05 last 67.00) - /unlink 6888 (time 1710100462.16 total 80.05 last 95.14) - /unlink 7824 (time 1710100472.16 total 90.06 last 93.56) - /unlink 8912 (time 1710100482.17 total 100.06 last 108.73) - /unlink 10000 (time 1710100490.72 total 108.61 last 127.27) - /unlink 11570 (time 1710100500.73 total 118.62 last 156.86) - /unlink 13047 (time 1710100510.73 total 128.62 last 147.64) - /unlink 14666 (time 1710100520.73 total 138.63 last 161.88) - /unlink 16560 (time 1710100530.73 total 148.63 last 189.38) - /unlink 18857 (time 1710100540.73 total 158.63 last 229.64) - /unlink 20000 (time 1710100545.64 total 163.54 last 232.81) - /unlink 22333 (time 1710100555.65 total 173.54 last 233.23) - /unlink 24214 (time 1710100565.65 total 183.55 last 188.04) - /unlink 26102 (time 1710100575.65 total 193.55 last 188.75) - /unlink 29584 (time 1710100585.65 total 203.55 last 348.16) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 210.30 seconds: 154.82 ops/second Waiting for MDT destroys to complete - open/close 4475 (time 1710100609.68 total 10.00 last 447.46) - open/close 8093 (time 1710100619.68 total 20.00 last 361.75) - open/close 10000 (time 1710100626.21 total 26.53 last 292.09) - open/close 12587 (time 1710100636.22 total 36.53 last 258.61) - open/close 14734 (time 1710100646.22 total 46.54 last 214.63) - open/close 16503 (time 1710100656.22 total 56.54 last 176.80) - open/close 18094 (time 1710100666.23 total 66.54 last 159.08) - open/close 19438 (time 1710100676.23 total 76.55 last 134.37) - open/close 20000 (time 1710100680.97 total 81.29 last 118.54) - open/close 21278 (time 1710100690.97 total 91.29 last 127.75) - open/close 22335 (time 1710100700.97 total 101.29 last 105.68) - open/close 23193 (time 1710100710.99 total 111.30 last 85.70) - open/close 24184 (time 1710100720.99 total 121.30 last 99.10) - open/close 25190 (time 1710100730.99 total 131.31 last 100.57) - open/close 26173 (time 1710100741.00 total 141.32 last 98.21) - open/close 26927 (time 1710100751.01 total 151.33 last 75.30) - open/close 27505 (time 1710100761.02 total 161.33 last 57.78) - open/close 28072 (time 1710100771.02 total 171.34 last 56.65) - open/close 28633 (time 1710100781.03 total 181.35 last 56.04) - open/close 29223 (time 1710100791.04 total 191.36 last 58.94) - open/close 29837 (time 1710100801.05 total 201.37 last 61.34) - open/close 30000 (time 1710100803.72 total 204.04 last 61.07) - open/close 30605 (time 1710100813.74 total 214.06 last 60.41) - open/close 31152 (time 1710100823.76 total 224.07 last 54.60) - open/close 31660 (time 1710100833.77 total 234.09 last 50.72) - open/close 32176 (time 1710100843.79 total 244.10 last 51.53) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 252.08 seconds: 129.17 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32824 47168 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33079 33079 0 100% /mnt/lustre - /unlink 386 (time 1710100863.10 total 10.03 last 38.47) - /unlink 811 (time 1710100873.12 total 20.06 last 42.41) - /unlink 1254 (time 1710100883.13 total 30.07 last 44.23) - /unlink 1714 (time 1710100893.14 total 40.07 last 45.99) - /unlink 2152 (time 1710100903.15 total 50.08 last 43.75) - /unlink 2636 (time 1710100913.15 total 60.09 last 48.36) - /unlink 3133 (time 1710100923.17 total 70.11 last 49.62) - /unlink 3613 (time 1710100933.19 total 80.12 last 47.92) - /unlink 4088 (time 1710100943.20 total 90.14 last 47.42) - /unlink 4595 (time 1710100953.22 total 100.15 last 50.64) - /unlink 5099 (time 1710100963.23 total 110.17 last 50.31) - /unlink 5571 (time 1710100973.24 total 120.17 last 47.18) - /unlink 5932 (time 1710100983.24 total 130.18 last 36.09) - /unlink 6318 (time 1710100993.25 total 140.19 last 38.56) - /unlink 6777 (time 1710101003.27 total 150.21 last 45.82) - /unlink 7224 (time 1710101013.28 total 160.22 last 44.63) - /unlink 7656 (time 1710101023.29 total 170.22 last 43.18) - /unlink 8122 (time 1710101033.29 total 180.23 last 46.60) - /unlink 8603 (time 1710101043.30 total 190.24 last 48.04) - /unlink 9100 (time 1710101053.31 total 200.25 last 49.63) - /unlink 9568 (time 1710101063.33 total 210.27 last 46.71) - /unlink 10000 (time 1710101071.49 total 218.43 last 52.93) - /unlink 10522 (time 1710101081.51 total 228.44 last 52.14) - /unlink 11020 (time 1710101091.52 total 238.45 last 49.74) - /unlink 11468 (time 1710101101.52 total 248.46 last 44.80) - /unlink 11940 (time 1710101111.52 total 258.46 last 47.17) - /unlink 12444 (time 1710101121.54 total 268.48 last 50.31) - /unlink 12929 (time 1710101131.56 total 278.49 last 48.43) - /unlink 13427 (time 1710101141.56 total 288.50 last 49.78) - /unlink 13942 (time 1710101151.57 total 298.51 last 51.42) - /unlink 14482 (time 1710101161.59 total 308.53 last 53.91) - /unlink 15018 (time 1710101171.61 total 318.55 last 53.51) - /unlink 15595 (time 1710101181.61 total 328.55 last 57.70) - /unlink 16201 (time 1710101191.61 total 338.55 last 60.59) - /unlink 16872 (time 1710101201.61 total 348.55 last 67.08) - /unlink 17596 (time 1710101211.62 total 358.56 last 72.35) - /unlink 18400 (time 1710101221.63 total 368.56 last 80.36) - /unlink 19282 (time 1710101231.63 total 378.57 last 88.12) - /unlink 20000 (time 1710101239.12 total 386.05 last 95.96) - /unlink 20989 (time 1710101249.12 total 396.05 last 98.88) - /unlink 22084 (time 1710101259.13 total 406.06 last 109.40) - /unlink 23205 (time 1710101269.13 total 416.07 last 112.04) - /unlink 24469 (time 1710101279.14 total 426.07 last 126.34) - /unlink 25834 (time 1710101289.14 total 436.08 last 136.47) - /unlink 27382 (time 1710101299.14 total 446.08 last 154.71) - /unlink 29174 (time 1710101309.15 total 456.08 last 179.17) - /unlink 30000 (time 1710101313.30 total 460.24 last 198.63) - /unlink 32294 (time 1710101323.31 total 470.24 last 229.33) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 471.26 seconds: 69.09 ops/second Waiting for MDT destroys to complete - open/close 1352 (time 1710101343.07 total 10.01 last 135.03) - open/close 2582 (time 1710101353.07 total 20.01 last 122.97) - open/close 3693 (time 1710101363.07 total 30.02 last 111.06) - open/close 4805 (time 1710101373.08 total 40.02 last 111.13) - open/close 5829 (time 1710101383.08 total 50.03 last 102.39) - open/close 6831 (time 1710101393.08 total 60.03 last 100.17) - open/close 7731 (time 1710101403.09 total 70.04 last 89.93) - open/close 8650 (time 1710101413.10 total 80.04 last 91.84) - open/close 9528 (time 1710101423.10 total 90.05 last 87.75) - open/close 10000 (time 1710101428.72 total 95.67 last 83.98) - open/close 10811 (time 1710101438.73 total 105.67 last 81.09) - open/close 11590 (time 1710101448.73 total 115.68 last 77.82) - open/close 12366 (time 1710101458.74 total 125.69 last 77.54) - open/close 13056 (time 1710101468.75 total 135.69 last 68.97) - open/close 13714 (time 1710101478.76 total 145.70 last 65.74) - open/close 14325 (time 1710101488.76 total 155.71 last 61.06) - open/close 14925 (time 1710101498.77 total 165.72 last 59.94) - open/close 15531 (time 1710101508.78 total 175.73 last 60.53) - open/close 16102 (time 1710101518.79 total 185.73 last 57.09) - open/close 16669 (time 1710101528.79 total 195.73 last 56.69) - open/close 17219 (time 1710101538.79 total 205.74 last 54.98) - open/close 17754 (time 1710101548.79 total 215.74 last 53.50) - open/close 18291 (time 1710101558.81 total 225.75 last 53.62) - open/close 18817 (time 1710101568.83 total 235.77 last 52.50) - open/close 19369 (time 1710101578.84 total 245.78 last 55.13) - open/close 19896 (time 1710101588.85 total 255.80 last 52.64) - open/close 20000 (time 1710101590.96 total 257.90 last 49.37) - open/close 20521 (time 1710101600.98 total 267.93 last 51.97) - open/close 21024 (time 1710101610.99 total 277.93 last 50.26) - open/close 21492 (time 1710101621.01 total 287.95 last 46.72) - open/close 21981 (time 1710101631.02 total 297.96 last 48.85) - open/close 22492 (time 1710101641.02 total 307.96 last 51.09) - open/close 22953 (time 1710101651.03 total 317.97 last 46.04) - open/close 23400 (time 1710101661.04 total 327.98 last 44.67) - open/close 23850 (time 1710101671.04 total 337.99 last 44.98) - open/close 24298 (time 1710101681.05 total 348.00 last 44.75) - open/close 24705 (time 1710101691.05 total 358.00 last 40.69) - open/close 25121 (time 1710101701.07 total 368.02 last 41.52) - open/close 25529 (time 1710101711.08 total 378.03 last 40.77) - open/close 25946 (time 1710101721.09 total 388.04 last 41.65) - open/close 26356 (time 1710101731.11 total 398.06 last 40.92) - open/close 26756 (time 1710101741.14 total 408.08 last 39.89) - open/close 27177 (time 1710101751.14 total 418.08 last 42.10) - open/close 27614 (time 1710101761.14 total 428.09 last 43.68) - open/close 28030 (time 1710101771.15 total 438.09 last 41.57) - open/close 28451 (time 1710101781.17 total 448.11 last 42.02) - open/close 28832 (time 1710101791.18 total 458.12 last 38.07) - open/close 29241 (time 1710101801.19 total 468.14 last 40.82) - open/close 29661 (time 1710101811.21 total 478.16 last 41.92) - open/close 30000 (time 1710101819.70 total 486.65 last 39.94) - open/close 30380 (time 1710101829.71 total 496.66 last 37.96) - open/close 30764 (time 1710101839.73 total 506.67 last 38.33) - open/close 31139 (time 1710101849.73 total 516.67 last 37.50) - open/close 31513 (time 1710101859.74 total 526.68 last 37.36) - open/close 31846 (time 1710101869.75 total 536.70 last 33.25) - open/close 32179 (time 1710101879.77 total 546.71 last 33.26) - open/close 32499 (time 1710101889.77 total 556.71 last 32.00) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 558.52 seconds: 58.30 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 287 (time 1710101903.31 total 10.00 last 28.69) - /unlink 582 (time 1710101913.33 total 20.02 last 29.45) - /unlink 872 (time 1710101923.33 total 30.03 last 28.98) - /unlink 1163 (time 1710101933.35 total 40.05 last 29.04) - /unlink 1471 (time 1710101943.36 total 50.05 last 30.78) - /unlink 1814 (time 1710101953.38 total 60.07 last 34.23) - /unlink 2140 (time 1710101963.41 total 70.10 last 32.51) - /unlink 2469 (time 1710101973.42 total 80.11 last 32.87) - /unlink 2811 (time 1710101983.44 total 90.13 last 34.12) - /unlink 3141 (time 1710101993.46 total 100.15 last 32.94) - /unlink 3490 (time 1710102003.47 total 110.16 last 34.87) - /unlink 3857 (time 1710102013.48 total 120.17 last 36.67) - /unlink 4212 (time 1710102023.50 total 130.19 last 35.42) - /unlink 4568 (time 1710102033.51 total 140.21 last 35.55) - /unlink 4914 (time 1710102043.53 total 150.23 last 34.53) - /unlink 5297 (time 1710102053.56 total 160.25 last 38.22) - /unlink 5663 (time 1710102063.56 total 170.26 last 36.58) - /unlink 6014 (time 1710102073.57 total 180.27 last 35.06) - /unlink 6375 (time 1710102083.58 total 190.28 last 36.05) - /unlink 6729 (time 1710102093.60 total 200.29 last 35.35) - /unlink 7116 (time 1710102103.62 total 210.32 last 38.60) - /unlink 7495 (time 1710102113.65 total 220.35 last 37.80) - /unlink 7902 (time 1710102123.67 total 230.37 last 40.61) - /unlink 8312 (time 1710102133.68 total 240.38 last 40.95) - /unlink 8740 (time 1710102143.70 total 250.39 last 42.74) - /unlink 9175 (time 1710102153.71 total 260.41 last 43.44) - /unlink 9678 (time 1710102163.72 total 270.41 last 50.27) - /unlink 10000 (time 1710102170.57 total 277.26 last 47.01) - /unlink 10522 (time 1710102180.58 total 287.27 last 52.14) - /unlink 11074 (time 1710102190.58 total 297.27 last 55.19) - /unlink 11650 (time 1710102200.58 total 307.28 last 57.57) - /unlink 12209 (time 1710102210.59 total 317.29 last 55.86) - /unlink 12781 (time 1710102220.61 total 327.30 last 57.12) - /unlink 13355 (time 1710102230.62 total 337.31 last 57.34) - /unlink 13935 (time 1710102240.63 total 347.33 last 57.90) - /unlink 14524 (time 1710102250.65 total 357.34 last 58.82) - /unlink 15128 (time 1710102260.66 total 367.35 last 60.34) - /unlink 15762 (time 1710102270.66 total 377.36 last 63.36) - /unlink 16397 (time 1710102280.67 total 387.37 last 63.43) - /unlink 17085 (time 1710102290.68 total 397.38 last 68.74) - /unlink 17864 (time 1710102300.69 total 407.38 last 77.85) - /unlink 18775 (time 1710102310.69 total 417.38 last 91.09) - /unlink 19830 (time 1710102320.69 total 427.39 last 105.48) - /unlink 21029 (time 1710102330.69 total 437.39 last 119.87) - /unlink 22391 (time 1710102340.70 total 447.39 last 136.16) - /unlink 23879 (time 1710102350.70 total 457.40 last 148.72) - /unlink 25532 (time 1710102360.71 total 467.40 last 165.25) - /unlink 27425 (time 1710102370.71 total 477.41 last 189.18) - /unlink 29630 (time 1710102380.71 total 487.41 last 220.45) - /unlink 32410 (time 1710102390.72 total 497.41 last 277.93) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 497.96 seconds: 65.39 ops/second Waiting for MDT destroys to complete - open/close 1649 (time 1710102411.51 total 10.00 last 164.87) - open/close 3082 (time 1710102421.51 total 20.00 last 143.27) - open/close 4359 (time 1710102431.52 total 30.01 last 127.64) - open/close 5601 (time 1710102441.52 total 40.01 last 124.13) - open/close 6739 (time 1710102451.53 total 50.02 last 113.72) - open/close 7882 (time 1710102461.54 total 60.03 last 114.20) - open/close 9003 (time 1710102471.54 total 70.03 last 112.07) - open/close 10000 (time 1710102480.22 total 78.71 last 114.91) - open/close 11083 (time 1710102490.22 total 88.71 last 108.25) - open/close 12155 (time 1710102500.23 total 98.72 last 107.14) - open/close 13187 (time 1710102510.23 total 108.72 last 103.19) - open/close 14169 (time 1710102520.24 total 118.73 last 98.07) - open/close 15100 (time 1710102530.25 total 128.74 last 93.03) - open/close 15997 (time 1710102540.26 total 138.75 last 89.63) - open/close 16857 (time 1710102550.27 total 148.76 last 85.93) - open/close 17719 (time 1710102560.28 total 158.77 last 86.10) - open/close 18630 (time 1710102570.29 total 168.78 last 91.03) - open/close 19590 (time 1710102580.29 total 178.78 last 95.92) - open/close 20000 (time 1710102584.45 total 182.94 last 98.65) - open/close 21108 (time 1710102594.46 total 192.95 last 110.74) - open/close 22150 (time 1710102604.46 total 202.95 last 104.11) - open/close 23172 (time 1710102614.47 total 212.96 last 102.13) - open/close 24120 (time 1710102624.47 total 222.96 last 94.78) - open/close 25088 (time 1710102634.48 total 232.97 last 96.76) - open/close 26166 (time 1710102644.48 total 242.97 last 107.78) - open/close 27220 (time 1710102654.48 total 252.97 last 105.34) - open/close 28262 (time 1710102664.49 total 262.98 last 104.19) - open/close 29557 (time 1710102674.49 total 272.98 last 129.39) - open/close 30000 (time 1710102677.87 total 276.36 last 131.17) - open/close 31234 (time 1710102687.88 total 286.37 last 123.32) - open/close 32355 (time 1710102697.89 total 296.38 last 111.99) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 298.43 seconds: 109.11 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32809 47183 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33064 33064 0 100% /mnt/lustre - /unlink 1261 (time 1710102710.90 total 10.01 last 126.02) - /unlink 2599 (time 1710102720.91 total 20.01 last 133.71) - /unlink 4016 (time 1710102730.91 total 30.02 last 141.66) - /unlink 5616 (time 1710102740.91 total 40.02 last 160.00) - /unlink 6988 (time 1710102750.92 total 50.02 last 137.10) - /unlink 8663 (time 1710102760.93 total 60.03 last 167.38) - /unlink 10000 (time 1710102770.42 total 69.52 last 140.88) - /unlink 11593 (time 1710102780.42 total 79.52 last 159.29) - /unlink 13212 (time 1710102790.42 total 89.53 last 161.82) - /unlink 15028 (time 1710102800.42 total 99.53 last 181.54) - /unlink 16917 (time 1710102810.43 total 109.53 last 188.86) - /unlink 18829 (time 1710102820.43 total 119.53 last 191.19) - /unlink 20000 (time 1710102825.68 total 124.78 last 222.98) - /unlink 22138 (time 1710102835.68 total 134.79 last 213.72) - /unlink 24432 (time 1710102845.69 total 144.79 last 229.31) - /unlink 27133 (time 1710102855.69 total 154.79 last 270.05) - /unlink 30000 (time 1710102864.66 total 163.76 last 319.68) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 170.23 seconds: 191.27 ops/second Waiting for MDT destroys to complete - open/close 2288 (time 1710102890.91 total 10.00 last 228.76) - open/close 4370 (time 1710102900.91 total 20.00 last 208.15) - open/close 6311 (time 1710102910.92 total 30.01 last 194.06) - open/close 8130 (time 1710102920.92 total 40.01 last 181.82) - open/close 9877 (time 1710102930.92 total 50.01 last 174.66) - open/close 11548 (time 1710102940.93 total 60.02 last 167.01) - open/close 13158 (time 1710102950.93 total 70.02 last 160.92) - open/close 14793 (time 1710102960.93 total 80.02 last 163.50) - open/close 16258 (time 1710102970.93 total 90.02 last 146.50) - open/close 17693 (time 1710102980.94 total 100.03 last 143.41) - open/close 19071 (time 1710102990.94 total 110.03 last 137.78) - open/close 20000 (time 1710102997.87 total 116.96 last 134.06) - open/close 21309 (time 1710103007.88 total 126.97 last 130.83) - open/close 22582 (time 1710103017.88 total 136.97 last 127.25) - open/close 23839 (time 1710103027.88 total 146.97 last 125.65) - open/close 25061 (time 1710103037.88 total 156.97 last 122.20) - open/close 26251 (time 1710103047.89 total 166.98 last 118.93) - open/close 27410 (time 1710103057.90 total 176.99 last 115.80) - open/close 28558 (time 1710103067.90 total 186.99 last 114.79) - open/close 29666 (time 1710103077.90 total 196.99 last 110.78) - open/close 30000 (time 1710103080.90 total 199.99 last 111.40) - open/close 31090 (time 1710103090.90 total 209.99 last 108.95) - open/close 32163 (time 1710103100.91 total 220.00 last 107.25) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 223.85 seconds: 145.45 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1159 (time 1710103116.02 total 10.01 last 115.81) - /unlink 2312 (time 1710103126.02 total 20.01 last 115.26) - /unlink 3507 (time 1710103136.03 total 30.02 last 119.44) - /unlink 4811 (time 1710103146.04 total 40.02 last 130.31) - /unlink 6109 (time 1710103156.04 total 50.03 last 129.77) - /unlink 7388 (time 1710103166.04 total 60.03 last 127.90) - /unlink 8789 (time 1710103176.04 total 70.03 last 140.01) - /unlink 10000 (time 1710103185.18 total 79.16 last 132.62) - /unlink 11454 (time 1710103195.18 total 89.17 last 145.39) - /unlink 13095 (time 1710103205.18 total 99.17 last 164.08) - /unlink 14818 (time 1710103215.18 total 109.17 last 172.24) - /unlink 16472 (time 1710103225.19 total 119.17 last 165.34) - /unlink 18238 (time 1710103235.19 total 129.18 last 176.52) - /unlink 20000 (time 1710103244.64 total 138.63 last 186.41) - /unlink 22110 (time 1710103254.64 total 148.63 last 210.96) - /unlink 24498 (time 1710103264.65 total 158.63 last 238.74) - /unlink 27285 (time 1710103274.65 total 168.64 last 278.67) - /unlink 30000 (time 1710103282.94 total 176.93 last 327.31) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 183.15 seconds: 177.77 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg345-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg345-server unloading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2 PASS 106 (3207s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg345-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg345-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg345-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg345-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2 oleg345-server: oleg345-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: detected 4 online CPUs by sysfs oleg345-server: Force libcfs to create 2 CPU partitions oleg345-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg345-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3312 sec ========== 16:42:22 (1710103342) === conf-sanity: start cleanup 16:42:22 (1710103342) === === conf-sanity: finish cleanup 16:42:22 (1710103342) ===