-----============= acceptance-small: conf-sanity ============----- Fri Jun 14 16:49:43 EDT 2024 excepting tests: 32newtarball 110 Stopping clients: oleg410-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg410-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg410-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg410-server oleg410-server: oleg410-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing load_modules_local oleg410-server: Loading modules from /home/green/git/lustre-release/lustre oleg410-server: detected 4 online CPUs by sysfs oleg410-server: Force libcfs to create 2 CPU partitions oleg410-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg410-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg410-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg410-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg410-server: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg410-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg410-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:51:25 (1718398285) Stopping clients: oleg410-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg410-client.virtnet /mnt/lustre2 (opts:-f) oleg410-server: oleg410-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing load_modules_local oleg410-server: Loading modules from /home/green/git/lustre-release/lustre oleg410-server: detected 4 online CPUs by sysfs oleg410-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg410-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg410-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock oleg410-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1626 (time 1718398335.52 total 10.00 last 162.59) - open/close 3178 (time 1718398345.52 total 20.00 last 155.15) - open/close 4886 (time 1718398355.52 total 30.01 last 170.75) - open/close 6396 (time 1718398365.53 total 40.01 last 150.96) - open/close 7823 (time 1718398375.53 total 50.02 last 142.61) - open/close 9166 (time 1718398385.54 total 60.02 last 134.23) - open/close 10000 (time 1718398391.29 total 65.77 last 145.12) - open/close 11470 (time 1718398401.29 total 75.77 last 146.98) - open/close 12938 (time 1718398411.29 total 85.77 last 146.75) - open/close 14440 (time 1718398421.29 total 95.77 last 150.17) - open/close 15914 (time 1718398431.31 total 105.79 last 147.17) - open/close 17344 (time 1718398441.31 total 115.80 last 142.93) - open/close 18715 (time 1718398451.32 total 125.80 last 137.01) - open/close 20000 (time 1718398460.67 total 135.16 last 137.37) - open/close 21425 (time 1718398470.68 total 145.17 last 142.34) - open/close 22822 (time 1718398480.69 total 155.17 last 139.64) - open/close 24266 (time 1718398490.69 total 165.17 last 144.39) - open/close 25739 (time 1718398500.69 total 175.18 last 147.23) - open/close 27437 (time 1718398510.69 total 185.18 last 169.78) - open/close 29263 (time 1718398520.70 total 195.18 last 182.58) - open/close 30000 (time 1718398523.85 total 198.33 last 233.96) - open/close 32687 (time 1718398533.85 total 208.33 last 268.64) - open/close 35130 (time 1718398543.85 total 218.33 last 244.22) - open/close 37604 (time 1718398553.85 total 228.34 last 247.36) - open/close 40000 (time 1718398561.03 total 235.52 last 333.65) - open/close 43270 (time 1718398571.04 total 245.52 last 326.97) - open/close 46076 (time 1718398581.04 total 255.52 last 280.49) - open/close 48366 (time 1718398591.04 total 265.52 last 228.97) open(/mnt/lustre/d106.conf-sanity/f-49764) error: No space left on device total: 49764 open/close in 270.79 seconds: 183.78 ops/second - /unlink 4840 (time 1718398607.33 total 10.00 last 483.93) - /unlink 9661 (time 1718398617.34 total 20.00 last 482.02) - /unlink 13729 (time 1718398627.34 total 30.01 last 406.69) - /unlink 18793 (time 1718398637.34 total 40.01 last 506.37) - /unlink 20000 (time 1718398639.86 total 42.52 last 479.74) - /unlink 25480 (time 1718398649.86 total 52.52 last 547.90) - /unlink 29723 (time 1718398659.86 total 62.53 last 424.24) - /unlink 34298 (time 1718398669.88 total 72.54 last 456.74) - /unlink 39729 (time 1718398679.88 total 82.54 last 543.06) - /unlink 44998 (time 1718398689.88 total 92.54 last 526.88) unlink(/mnt/lustre/d106.conf-sanity/f-49764) error: No such file or directory total: 49764 /unlink in 99.91 seconds: 498.10 ops/second - open/close 2926 (time 1718398708.22 total 10.00 last 292.53) - open/close 6999 (time 1718398718.22 total 20.01 last 407.18) - open/close 10000 (time 1718398726.51 total 28.29 last 362.04) - open/close 12555 (time 1718398736.51 total 38.30 last 255.44) - open/close 14872 (time 1718398746.51 total 48.30 last 231.61) - open/close 17531 (time 1718398756.51 total 58.30 last 265.88) - open/close 20000 (time 1718398765.69 total 67.48 last 269.09) - open/close 23319 (time 1718398775.69 total 77.48 last 331.89) - open/close 26476 (time 1718398785.69 total 87.48 last 315.66) - open/close 29979 (time 1718398795.69 total 97.48 last 350.27) - open/close 32714 (time 1718398805.69 total 107.48 last 273.44) - open/close 36224 (time 1718398815.70 total 117.48 last 350.97) - open/close 39621 (time 1718398825.70 total 127.48 last 339.67) - open/close 42645 (time 1718398835.70 total 137.49 last 302.29) - open/close 45391 (time 1718398845.70 total 147.49 last 274.57) - open/close 48829 (time 1718398855.70 total 157.49 last 343.74) open(/mnt/lustre/d106.conf-sanity/f-49764) error: No space left on device total: 49764 open/close in 160.52 seconds: 310.01 ops/second - /unlink 6793 (time 1718398869.70 total 10.00 last 679.28) - /unlink 10000 (time 1718398874.86 total 15.15 last 622.15) - /unlink 17120 (time 1718398884.86 total 25.16 last 711.93) - /unlink 20000 (time 1718398888.27 total 28.57 last 844.70) - /unlink 27651 (time 1718398898.27 total 38.57 last 765.06) - /unlink 30000 (time 1718398901.65 total 41.95 last 694.64) - /unlink 37462 (time 1718398911.65 total 51.95 last 746.18) - /unlink 40000 (time 1718398915.69 total 55.98 last 629.01) - /unlink 47135 (time 1718398925.69 total 65.98 last 713.44) unlink(/mnt/lustre/d106.conf-sanity/f-49764) error: No such file or directory total: 49764 /unlink in 69.65 seconds: 714.48 ops/second - open/close 4030 (time 1718398940.29 total 10.01 last 402.49) - open/close 7941 (time 1718398950.29 total 20.01 last 391.03) - open/close 10000 (time 1718398954.67 total 24.40 last 469.83) - open/close 14910 (time 1718398964.67 total 34.40 last 490.95) - open/close 19310 (time 1718398974.67 total 44.40 last 439.96) - open/close 23450 (time 1718398984.67 total 54.40 last 413.96) - open/close 25011 (time 1718398994.68 total 64.41 last 156.00) - open/close 26184 (time 1718399004.68 total 74.41 last 117.28) - open/close 28132 (time 1718399014.68 total 84.41 last 194.79) - open/close 30000 (time 1718399023.81 total 93.54 last 204.61) - open/close 32715 (time 1718399033.82 total 103.54 last 271.43) - open/close 34206 (time 1718399043.82 total 113.54 last 149.08) - open/close 35753 (time 1718399053.82 total 123.54 last 154.69) - open/close 38113 (time 1718399063.82 total 133.54 last 235.97) - open/close 40000 (time 1718399071.54 total 141.27 last 244.35) - open/close 42204 (time 1718399081.54 total 151.27 last 220.36) - open/close 45136 (time 1718399091.54 total 161.27 last 293.14) - open/close 47299 (time 1718399101.55 total 171.27 last 216.23) - open/close 49136 (time 1718399111.55 total 181.28 last 183.66) open(/mnt/lustre/d106.conf-sanity/f-49764) error: No space left on device total: 49764 open/close in 183.57 seconds: 271.09 ops/second - /unlink 4310 (time 1718399124.84 total 10.00 last 430.99) - /unlink 7464 (time 1718399134.84 total 20.00 last 315.38) - /unlink 10000 (time 1718399142.96 total 28.12 last 312.45) - /unlink 13275 (time 1718399152.96 total 38.12 last 327.49) - /unlink 16079 (time 1718399162.96 total 48.12 last 280.37) - /unlink 19406 (time 1718399172.96 total 58.12 last 332.67) - /unlink 20000 (time 1718399175.16 total 60.31 last 270.99) - /unlink 22782 (time 1718399185.16 total 70.31 last 278.14) - /unlink 26896 (time 1718399195.16 total 80.32 last 411.28) - /unlink 30000 (time 1718399205.15 total 90.31 last 310.69) - /unlink 32680 (time 1718399215.15 total 100.31 last 267.92) - /unlink 35415 (time 1718399225.15 total 110.31 last 273.50) - /unlink 38373 (time 1718399235.16 total 120.31 last 295.74) - /unlink 40000 (time 1718399241.07 total 126.22 last 275.33) - /unlink 43404 (time 1718399251.07 total 136.22 last 340.39) - /unlink 47170 (time 1718399261.07 total 146.22 last 376.56) unlink(/mnt/lustre/d106.conf-sanity/f-49764) error: No such file or directory total: 49764 /unlink in 151.33 seconds: 328.85 ops/second - open/close 3038 (time 1718399277.13 total 10.00 last 303.77) - open/close 5565 (time 1718399287.13 total 20.01 last 252.55) - open/close 8490 (time 1718399297.14 total 30.01 last 292.46) - open/close 10000 (time 1718399303.05 total 35.92 last 255.35) - open/close 12787 (time 1718399313.05 total 45.92 last 278.64) - open/close 15231 (time 1718399323.05 total 55.93 last 244.37) - open/close 17225 (time 1718399333.05 total 65.93 last 199.36) - open/close 18738 (time 1718399343.06 total 75.94 last 151.18) - open/close 20000 (time 1718399350.47 total 83.34 last 170.46) - open/close 21571 (time 1718399360.47 total 93.34 last 157.10) - open/close 23257 (time 1718399370.47 total 103.34 last 168.56) - open/close 24712 (time 1718399380.47 total 113.34 last 145.46) - open/close 26301 (time 1718399390.48 total 123.35 last 158.79) - open/close 27535 (time 1718399400.49 total 133.36 last 123.26) - open/close 28962 (time 1718399410.49 total 143.37 last 142.65) - open/close 30000 (time 1718399417.26 total 150.13 last 153.44) - open/close 31382 (time 1718399427.26 total 160.13 last 138.20) - open/close 33134 (time 1718399437.26 total 170.13 last 175.19) - open/close 35011 (time 1718399447.26 total 180.13 last 187.69) - open/close 36624 (time 1718399457.26 total 190.14 last 161.24) - open/close 38221 (time 1718399467.27 total 200.14 last 159.64) - open/close 39769 (time 1718399477.27 total 210.14 last 154.73) - open/close 41768 (time 1718399487.27 total 220.15 last 199.85) - open/close 43600 (time 1718399497.28 total 230.15 last 183.06) - open/close 45267 (time 1718399507.29 total 240.16 last 166.61) - open/close 46902 (time 1718399517.29 total 250.16 last 163.49) open(/mnt/lustre/d106.conf-sanity/f-48310) error: No space left on device total: 48310 open/close in 259.00 seconds: 186.53 ops/second - /unlink 3073 (time 1718399537.57 total 10.00 last 307.29) - /unlink 6967 (time 1718399547.57 total 20.00 last 389.37) - /unlink 10000 (time 1718399553.42 total 25.85 last 518.23) - /unlink 14119 (time 1718399563.42 total 35.85 last 411.90) - /unlink 18820 (time 1718399573.42 total 45.85 last 470.07) - /unlink 20000 (time 1718399576.04 total 48.47 last 450.94) - /unlink 25988 (time 1718399586.04 total 58.47 last 598.73) - /unlink 30000 (time 1718399593.07 total 65.50 last 570.71) - /unlink 35376 (time 1718399603.07 total 75.50 last 537.59) - /unlink 40000 (time 1718399609.69 total 82.12 last 699.09) unlink(/mnt/lustre/d106.conf-sanity/f-48310) error: No such file or directory total: 48310 /unlink in 91.84 seconds: 526.00 ops/second - open/close 5291 (time 1718399630.30 total 10.00 last 529.01) - open/close 10000 (time 1718399639.36 total 19.05 last 520.17) - open/close 15665 (time 1718399649.36 total 29.06 last 566.45) - open/close 20000 (time 1718399656.86 total 36.56 last 577.33) - open/close 26299 (time 1718399666.87 total 46.56 last 629.88) - open/close 30000 (time 1718399673.41 total 53.11 last 565.24) - open/close 36107 (time 1718399683.41 total 63.11 last 610.65) - open/close 40000 (time 1718399689.83 total 69.53 last 606.35) - open/close 45816 (time 1718399699.84 total 79.53 last 581.54) open(/mnt/lustre/d106.conf-sanity/f-48310) error: No space left on device total: 48310 open/close in 83.47 seconds: 578.77 ops/second - /unlink 10000 (time 1718399712.67 total 8.12 last 1231.78) - /unlink 20000 (time 1718399722.42 total 17.87 last 1025.39) - /unlink 29689 (time 1718399732.42 total 27.87 last 968.87) - /unlink 40000 (time 1718399741.99 total 37.44 last 1078.10) unlink(/mnt/lustre/d106.conf-sanity/f-48310) error: No such file or directory total: 48310 /unlink in 45.10 seconds: 1071.25 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg410-server LNET ready to unload unloading modules on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing unload_modules_local oleg410-server: LNET ready to unload modules unloaded. pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2 PASS 106 (1492s) Stopping clients: oleg410-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg410-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2 oleg410-server: oleg410-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing load_modules_local oleg410-server: Loading modules from /home/green/git/lustre-release/lustre oleg410-server: detected 4 online CPUs by sysfs oleg410-server: Force libcfs to create 2 CPU partitions oleg410-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg410-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg410-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1610 sec ========== 17:16:34 (1718399794)