-----============= acceptance-small: conf-sanity ============----- Tue Jul 4 07:58:47 EDT 2023 excepting tests: 32newtarball 110 Stopping clients: oleg263-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg263-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg263-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg263-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg263-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg263-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg263-server oleg263-server: oleg263-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg263-server' oleg263-server: oleg263-server.virtnet: executing load_modules_local oleg263-server: Loading modules from /home/green/git/lustre-release/lustre oleg263-server: detected 4 online CPUs by sysfs oleg263-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg263-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg263-server: oleg263-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg263-client: oleg263-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg263-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg263-server: oleg263-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg263-client: oleg263-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg263-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg263-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg263-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg263-server: oleg263-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg263-client: oleg263-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg263-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg263-server: oleg263-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg263-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg263-server: oleg263-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg263-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg263-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg263-server stop mds service on oleg263-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg263-server stop mds service on oleg263-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg263-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:00:09 (1688472009) Stopping clients: oleg263-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg263-client.virtnet /mnt/lustre2 (opts:-f) oleg263-server: oleg263-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg263-server' oleg263-server: oleg263-server.virtnet: executing load_modules_local oleg263-server: Loading modules from /home/green/git/lustre-release/lustre oleg263-server: detected 4 online CPUs by sysfs oleg263-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg263-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg263-server: oleg263-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg263-client: oleg263-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg263-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg263-server: oleg263-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg263-client: oleg263-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg263-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg263-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg263-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg263-server: oleg263-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg263-client: oleg263-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg263-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg263-client.virtnet: -o user_xattr,flock oleg263-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4428 (time 1688472048.86 total 10.00 last 442.72) - open/close 8211 (time 1688472058.86 total 20.00 last 378.30) - open/close 10000 (time 1688472064.68 total 25.82 last 307.31) - open/close 13318 (time 1688472074.69 total 35.82 last 331.76) - open/close 17009 (time 1688472084.69 total 45.83 last 369.03) - open/close 20000 (time 1688472091.75 total 52.89 last 423.75) - open/close 24418 (time 1688472101.75 total 62.89 last 441.76) - open/close 28095 (time 1688472111.75 total 72.89 last 367.63) - open/close 30000 (time 1688472118.12 total 79.26 last 298.95) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 86.18 seconds: 377.81 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8315 (time 1688472136.21 total 10.00 last 831.41) - /unlink 10000 (time 1688472138.75 total 12.54 last 663.10) - /unlink 17353 (time 1688472148.75 total 22.54 last 735.25) - /unlink 20000 (time 1688472152.24 total 26.03 last 758.72) - /unlink 29009 (time 1688472162.24 total 36.03 last 900.65) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 40.73 seconds: 799.32 ops/second Waiting for MDT destroys to complete - open/close 3248 (time 1688472197.89 total 10.00 last 324.77) - open/close 6404 (time 1688472207.89 total 20.00 last 315.56) - open/close 9346 (time 1688472217.89 total 30.00 last 294.16) - open/close 13156 (time 1688472227.90 total 40.01 last 380.93) - open/close 16724 (time 1688472237.90 total 50.01 last 356.74) - open/close 20000 (time 1688472245.00 total 57.11 last 461.47) - open/close 23713 (time 1688472255.00 total 67.11 last 371.29) - open/close 26250 (time 1688472265.00 total 77.11 last 253.69) - open/close 28657 (time 1688472275.00 total 87.11 last 240.61) - open/close 30000 (time 1688472280.60 total 92.71 last 239.84) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 102.12 seconds: 318.83 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5050 (time 1688472301.35 total 10.00 last 504.93) - /unlink 10000 (time 1688472310.87 total 19.52 last 520.09) - /unlink 15392 (time 1688472320.87 total 29.52 last 539.13) - /unlink 20000 (time 1688472328.88 total 37.53 last 575.11) - /unlink 25906 (time 1688472338.88 total 47.53 last 590.59) - /unlink 30000 (time 1688472346.98 total 55.64 last 505.26) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 60.71 seconds: 536.36 ops/second Waiting for MDT destroys to complete - open/close 2541 (time 1688472372.36 total 10.00 last 254.04) - open/close 4785 (time 1688472382.36 total 20.01 last 224.33) - open/close 7106 (time 1688472392.36 total 30.01 last 232.09) - open/close 9766 (time 1688472402.36 total 40.01 last 265.94) - open/close 12548 (time 1688472412.37 total 50.01 last 278.13) - open/close 15301 (time 1688472422.37 total 60.01 last 275.24) - open/close 17816 (time 1688472432.37 total 70.02 last 251.39) - open/close 20000 (time 1688472441.13 total 78.77 last 249.43) - open/close 22629 (time 1688472451.13 total 88.78 last 262.81) - open/close 25049 (time 1688472461.14 total 98.78 last 241.89) - open/close 27448 (time 1688472471.14 total 108.78 last 239.86) - open/close 29840 (time 1688472481.14 total 118.79 last 239.10) - open/close 32032 (time 1688472491.14 total 128.79 last 219.18) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 131.17 seconds: 248.23 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4710 (time 1688472504.76 total 10.00 last 470.99) - /unlink 9420 (time 1688472514.76 total 20.00 last 470.91) - /unlink 14359 (time 1688472524.77 total 30.00 last 493.83) - /unlink 19785 (time 1688472534.77 total 40.00 last 542.54) - /unlink 25174 (time 1688472544.77 total 50.01 last 538.83) - /unlink 30000 (time 1688472554.22 total 59.46 last 510.47) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 64.92 seconds: 501.57 ops/second Waiting for MDT destroys to complete - open/close 2229 (time 1688472582.85 total 10.00 last 222.82) - open/close 4621 (time 1688472592.85 total 20.00 last 239.19) - open/close 6846 (time 1688472602.86 total 30.01 last 222.41) - open/close 9076 (time 1688472612.86 total 40.01 last 222.93) - open/close 10000 (time 1688472616.93 total 44.08 last 227.05) - open/close 12289 (time 1688472626.93 total 54.08 last 228.84) - open/close 14542 (time 1688472636.94 total 64.09 last 225.23) - open/close 16748 (time 1688472646.94 total 74.09 last 220.52) - open/close 19125 (time 1688472656.94 total 84.09 last 237.66) - open/close 20000 (time 1688472659.65 total 86.80 last 323.48) - open/close 22246 (time 1688472669.65 total 96.80 last 224.54) - open/close 24530 (time 1688472679.65 total 106.80 last 228.36) - open/close 26926 (time 1688472689.65 total 116.80 last 239.59) - open/close 29127 (time 1688472699.65 total 126.80 last 220.05) - open/close 30000 (time 1688472703.77 total 130.92 last 212.28) - open/close 32197 (time 1688472713.77 total 140.92 last 219.63) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 142.63 seconds: 228.29 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4652 (time 1688472726.80 total 10.00 last 465.14) - /unlink 9539 (time 1688472736.80 total 20.00 last 488.70) - /unlink 15164 (time 1688472746.80 total 30.00 last 562.49) - /unlink 19877 (time 1688472756.80 total 40.00 last 471.28) - /unlink 24379 (time 1688472766.80 total 50.00 last 450.18) - /unlink 29193 (time 1688472776.80 total 60.00 last 481.36) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 67.26 seconds: 484.12 ops/second Waiting for MDT destroys to complete - open/close 2234 (time 1688472803.83 total 10.00 last 223.31) - open/close 4445 (time 1688472813.83 total 20.01 last 221.02) - open/close 6666 (time 1688472823.83 total 30.01 last 222.01) - open/close 9391 (time 1688472833.84 total 40.02 last 272.40) - open/close 10000 (time 1688472836.45 total 42.63 last 232.80) - open/close 12141 (time 1688472846.46 total 52.63 last 214.06) - open/close 14386 (time 1688472856.46 total 62.63 last 224.50) - open/close 16813 (time 1688472866.46 total 72.64 last 242.65) - open/close 18961 (time 1688472876.46 total 82.64 last 214.71) - open/close 20000 (time 1688472881.27 total 87.45 last 216.09) - open/close 22445 (time 1688472891.27 total 97.45 last 244.45) - open/close 24694 (time 1688472901.28 total 107.45 last 224.85) - open/close 27209 (time 1688472911.28 total 117.46 last 251.41) - open/close 29629 (time 1688472921.28 total 127.46 last 241.96) - open/close 31752 (time 1688472931.28 total 137.46 last 212.28) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 141.35 seconds: 230.35 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4639 (time 1688472946.47 total 10.00 last 463.83) - /unlink 9945 (time 1688472956.47 total 20.00 last 530.58) - /unlink 14700 (time 1688472966.47 total 30.00 last 475.46) - /unlink 20000 (time 1688472976.34 total 39.88 last 536.80) - /unlink 25629 (time 1688472986.35 total 49.88 last 562.77) - /unlink 30000 (time 1688472995.79 total 59.32 last 462.77) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 65.17 seconds: 499.63 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg263-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg263-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg263-server stop mds service on oleg263-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg263-server stop mds service on oleg263-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg263-server unloading modules on: 'oleg263-server' oleg263-server: oleg263-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg263-client: oleg263-client: ssh exited with exit code 2 pdsh@oleg263-client: oleg263-server: ssh exited with exit code 2 pdsh@oleg263-client: oleg263-client: ssh exited with exit code 2 pdsh@oleg263-client: oleg263-server: ssh exited with exit code 2 PASS 106 (1021s) Stopping clients: oleg263-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg263-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg263-client: oleg263-server: ssh exited with exit code 2 oleg263-server: oleg263-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg263-server' oleg263-server: oleg263-server.virtnet: executing load_modules_local oleg263-server: Loading modules from /home/green/git/lustre-release/lustre oleg263-server: detected 4 online CPUs by sysfs oleg263-server: Force libcfs to create 2 CPU partitions oleg263-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg263-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg263-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1129 sec ========== 08:17:36 (1688473056) === conf-sanity: start cleanup 08:17:37 (1688473057) === === conf-sanity: finish cleanup 08:17:37 (1688473057) ===