-----============= acceptance-small: conf-sanity ============----- Wed May 28 17:50:00 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg352-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg352-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg352-server oleg352-server: oleg352-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg352-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg352-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:51:26 (1748469086) Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) oleg352-server: oleg352-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg352-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5369 (time 1748469127.18 total 10.00 last 536.84) - open/close 10000 (time 1748469135.46 total 18.28 last 559.59) - open/close 16031 (time 1748469145.46 total 28.28 last 603.07) - open/close 20000 (time 1748469152.05 total 34.87 last 601.64) - open/close 25169 (time 1748469162.06 total 44.88 last 516.78) - open/close 30000 (time 1748469171.31 total 54.13 last 521.81) - open/close 35345 (time 1748469181.32 total 64.14 last 534.45) - open/close 40000 (time 1748469189.74 total 72.56 last 552.84) - open/close 45364 (time 1748469199.74 total 82.56 last 536.39) - open/close 50000 (time 1748469208.30 total 91.12 last 541.60) - open/close 55350 (time 1748469218.30 total 101.12 last 534.94) - open/close 60000 (time 1748469227.08 total 109.89 last 529.72) total: 64768 open/close in 119.08 seconds: 543.92 ops/second - /unlink 8970 (time 1748469247.18 total 10.00 last 896.96) - /unlink 17917 (time 1748469257.18 total 20.00 last 894.67) - /unlink 20000 (time 1748469259.62 total 22.44 last 852.58) - /unlink 28429 (time 1748469269.63 total 32.45 last 842.68) - /unlink 36819 (time 1748469279.63 total 42.45 last 838.95) - /unlink 40000 (time 1748469282.90 total 45.72 last 972.68) - /unlink 49698 (time 1748469292.90 total 55.72 last 969.74) - /unlink 59328 (time 1748469302.90 total 65.72 last 962.95) total: 64768 /unlink in 70.53 seconds: 918.30 ops/second Waiting for MDT destroys to complete - open/close 5444 (time 1748469400.32 total 10.00 last 544.35) - open/close 10000 (time 1748469408.63 total 18.30 last 548.75) - open/close 15442 (time 1748469418.63 total 28.31 last 544.11) - open/close 20000 (time 1748469427.10 total 36.78 last 537.93) - open/close 25372 (time 1748469437.10 total 46.78 last 537.13) - open/close 30000 (time 1748469445.69 total 55.37 last 538.77) - open/close 35277 (time 1748469455.69 total 65.37 last 527.58) - open/close 40000 (time 1748469465.18 total 74.86 last 498.04) - open/close 45242 (time 1748469475.18 total 84.86 last 524.19) - open/close 50000 (time 1748469484.04 total 93.72 last 537.02) - open/close 55014 (time 1748469494.04 total 103.72 last 501.38) - open/close 60000 (time 1748469503.31 total 112.99 last 537.59) total: 64768 open/close in 122.06 seconds: 530.61 ops/second - /unlink 9628 (time 1748469523.25 total 10.00 last 962.79) - /unlink 19391 (time 1748469533.25 total 20.00 last 976.29) - /unlink 28844 (time 1748469543.25 total 30.00 last 945.28) - /unlink 37940 (time 1748469553.25 total 40.00 last 909.54) - /unlink 40000 (time 1748469555.54 total 42.29 last 899.19) - /unlink 49352 (time 1748469565.54 total 52.29 last 935.15) - /unlink 57980 (time 1748469575.55 total 62.29 last 862.72) - /unlink 60000 (time 1748469578.00 total 64.74 last 824.11) total: 64768 /unlink in 69.64 seconds: 930.09 ops/second Waiting for MDT destroys to complete - open/close 4565 (time 1748469686.73 total 10.00 last 456.49) - open/close 9758 (time 1748469696.74 total 20.00 last 519.23) - open/close 14211 (time 1748469706.74 total 30.00 last 445.15) - open/close 18788 (time 1748469716.74 total 40.01 last 457.63) - open/close 20000 (time 1748469719.15 total 42.42 last 502.46) - open/close 24181 (time 1748469729.16 total 52.42 last 418.00) - open/close 28629 (time 1748469739.16 total 62.42 last 444.72) - open/close 30000 (time 1748469742.76 total 66.03 last 380.23) - open/close 32823 (time 1748469752.77 total 76.03 last 282.22) - open/close 35580 (time 1748469762.77 total 86.03 last 275.61) - open/close 38800 (time 1748469772.77 total 96.04 last 321.89) - open/close 40000 (time 1748469775.99 total 99.26 last 372.86) - open/close 43736 (time 1748469785.99 total 109.26 last 373.57) - open/close 47140 (time 1748469795.99 total 119.26 last 340.30) - open/close 49810 (time 1748469806.00 total 129.27 last 266.84) - open/close 52947 (time 1748469816.00 total 139.27 last 313.61) - open/close 55626 (time 1748469826.00 total 149.27 last 267.87) - open/close 58878 (time 1748469836.01 total 159.27 last 325.16) - open/close 60000 (time 1748469840.30 total 163.56 last 261.57) - open/close 62731 (time 1748469850.30 total 173.56 last 273.02) total: 64768 open/close in 181.94 seconds: 355.98 ops/second - /unlink 4941 (time 1748469870.02 total 10.00 last 494.02) - /unlink 10000 (time 1748469879.03 total 19.02 last 561.18) - /unlink 14711 (time 1748469889.03 total 29.02 last 471.01) - /unlink 19381 (time 1748469899.03 total 39.02 last 466.99) - /unlink 24254 (time 1748469909.03 total 49.02 last 487.25) - /unlink 30000 (time 1748469918.68 total 58.66 last 595.96) - /unlink 35998 (time 1748469928.68 total 68.66 last 599.74) - /unlink 40000 (time 1748469933.01 total 73.00 last 923.17) - /unlink 46760 (time 1748469943.01 total 83.00 last 675.96) - /unlink 50000 (time 1748469946.71 total 86.70 last 875.71) - /unlink 56181 (time 1748469956.71 total 96.70 last 618.00) - /unlink 60000 (time 1748469964.65 total 104.64 last 480.94) - /unlink 64703 (time 1748469974.66 total 114.64 last 470.22) total: 64768 /unlink in 114.78 seconds: 564.30 ops/second Waiting for MDT destroys to complete - open/close 2670 (time 1748470017.26 total 10.00 last 267.00) - open/close 5712 (time 1748470027.26 total 20.00 last 304.16) - open/close 8465 (time 1748470037.26 total 30.00 last 275.22) - open/close 10000 (time 1748470042.83 total 35.57 last 275.84) - open/close 13093 (time 1748470052.83 total 45.57 last 309.28) - open/close 16816 (time 1748470062.83 total 55.57 last 372.19) - open/close 20000 (time 1748470071.02 total 63.76 last 388.79) - open/close 24253 (time 1748470081.02 total 73.76 last 425.26) - open/close 28402 (time 1748470091.02 total 83.76 last 414.88) - open/close 30000 (time 1748470094.97 total 87.71 last 404.65) - open/close 33894 (time 1748470104.97 total 97.71 last 389.34) - open/close 36903 (time 1748470114.97 total 107.71 last 300.89) - open/close 39539 (time 1748470124.97 total 117.72 last 263.55) - open/close 42343 (time 1748470134.98 total 127.72 last 280.32) - open/close 44825 (time 1748470144.98 total 137.72 last 248.14) - open/close 47594 (time 1748470154.98 total 147.73 last 276.79) - open/close 50000 (time 1748470164.87 total 157.61 last 243.43) - open/close 52486 (time 1748470174.87 total 167.61 last 248.52) - open/close 55050 (time 1748470184.87 total 177.61 last 256.39) - open/close 57957 (time 1748470194.87 total 187.61 last 290.68) - open/close 60000 (time 1748470201.27 total 194.01 last 319.48) - open/close 63097 (time 1748470211.27 total 204.01 last 309.67) total: 64768 open/close in 210.39 seconds: 307.84 ops/second - /unlink 4876 (time 1748470228.92 total 10.00 last 487.56) - /unlink 9963 (time 1748470238.92 total 20.00 last 508.64) - /unlink 14797 (time 1748470248.93 total 30.00 last 483.37) - /unlink 20000 (time 1748470258.83 total 39.91 last 525.23) - /unlink 24876 (time 1748470268.83 total 49.91 last 487.58) - /unlink 29336 (time 1748470278.83 total 59.91 last 446.00) - /unlink 33797 (time 1748470288.83 total 69.91 last 446.01) - /unlink 38524 (time 1748470298.83 total 79.91 last 472.65) - /unlink 40000 (time 1748470302.06 total 83.13 last 458.28) - /unlink 44504 (time 1748470312.06 total 93.13 last 450.38) - /unlink 49172 (time 1748470322.06 total 103.13 last 466.76) - /unlink 53946 (time 1748470332.06 total 113.14 last 477.31) - /unlink 58660 (time 1748470342.07 total 123.14 last 471.05) - /unlink 60000 (time 1748470344.96 total 126.03 last 463.64) total: 64768 /unlink in 136.00 seconds: 476.25 ops/second Waiting for MDT destroys to complete - open/close 2567 (time 1748470377.43 total 10.00 last 256.66) - open/close 5642 (time 1748470387.43 total 20.00 last 307.50) - open/close 9093 (time 1748470397.43 total 30.00 last 344.99) - open/close 10000 (time 1748470400.32 total 32.89 last 313.83) - open/close 13124 (time 1748470410.32 total 42.90 last 312.39) - open/close 16559 (time 1748470420.32 total 52.90 last 343.44) - open/close 19763 (time 1748470430.32 total 62.90 last 320.34) - open/close 22736 (time 1748470440.33 total 72.90 last 297.21) - open/close 25766 (time 1748470450.33 total 82.91 last 302.91) - open/close 28841 (time 1748470460.33 total 92.91 last 307.47) - open/close 30000 (time 1748470463.94 total 96.52 last 320.93) - open/close 33179 (time 1748470473.94 total 106.52 last 317.87) - open/close 36622 (time 1748470483.94 total 116.52 last 344.28) - open/close 40000 (time 1748470493.77 total 126.35 last 343.69) - open/close 42997 (time 1748470503.78 total 136.35 last 299.60) - open/close 45785 (time 1748470513.78 total 146.35 last 278.77) - open/close 48287 (time 1748470523.78 total 156.35 last 250.20) - open/close 50000 (time 1748470530.04 total 162.61 last 273.58) - open/close 53330 (time 1748470540.04 total 172.62 last 332.94) - open/close 56107 (time 1748470550.04 total 182.62 last 277.70) - open/close 58971 (time 1748470560.04 total 192.62 last 286.35) - open/close 60000 (time 1748470564.04 total 196.61 last 257.65) - open/close 62457 (time 1748470574.04 total 206.61 last 245.66) total: 64768 open/close in 216.25 seconds: 299.51 ops/second - /unlink 4546 (time 1748470594.90 total 10.00 last 454.57) - /unlink 8934 (time 1748470604.90 total 20.00 last 438.77) - /unlink 10000 (time 1748470607.24 total 22.34 last 456.30) - /unlink 14561 (time 1748470617.24 total 32.34 last 455.93) - /unlink 19125 (time 1748470627.24 total 42.34 last 456.38) - /unlink 23885 (time 1748470637.25 total 52.34 last 475.95) - /unlink 28884 (time 1748470647.25 total 62.34 last 499.79) - /unlink 33829 (time 1748470657.25 total 72.35 last 494.38) - /unlink 38266 (time 1748470667.25 total 82.35 last 443.65) - /unlink 40000 (time 1748470671.10 total 86.20 last 450.29) - /unlink 44484 (time 1748470681.10 total 96.20 last 448.35) - /unlink 49019 (time 1748470691.10 total 106.20 last 453.48) - /unlink 50000 (time 1748470693.13 total 108.23 last 484.29) - /unlink 56711 (time 1748470703.13 total 118.23 last 671.09) - /unlink 60000 (time 1748470706.48 total 121.58 last 981.98) total: 64768 /unlink in 126.59 seconds: 511.65 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg352-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg352-server unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 PASS 106 (1683s) Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 oleg352-server: oleg352-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1795 sec ========== 18:19:55 (1748470795) === conf-sanity: start cleanup 18:19:56 (1748470796) === === conf-sanity: finish cleanup 18:19:56 (1748470796) ===