-----============= acceptance-small: conf-sanity ============----- Thu Feb 27 11:20:40 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg131-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 32b 32c 32e 32h 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg131-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg131-server oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg131-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg131-server: oleg131-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg131-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:21:38 (1740673298) Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg131-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5307 (time 1740673333.19 total 10.00 last 530.67) - open/close 10000 (time 1740673341.94 total 18.74 last 536.81) - open/close 15272 (time 1740673351.94 total 28.74 last 527.16) - open/close 20000 (time 1740673360.95 total 37.76 last 524.35) - open/close 25016 (time 1740673370.96 total 47.76 last 501.55) - open/close 30000 (time 1740673380.56 total 57.37 last 518.64) - open/close 34998 (time 1740673390.57 total 67.37 last 499.79) - open/close 40000 (time 1740673400.01 total 76.82 last 529.34) - open/close 45062 (time 1740673410.02 total 86.82 last 506.11) - open/close 50000 (time 1740673419.69 total 96.50 last 510.49) - open/close 55242 (time 1740673429.69 total 106.50 last 524.12) - open/close 60000 (time 1740673438.95 total 115.76 last 513.63) total: 64768 open/close in 124.89 seconds: 518.61 ops/second - /unlink 8351 (time 1740673458.92 total 10.00 last 835.03) - /unlink 17133 (time 1740673468.92 total 20.00 last 878.17) - /unlink 20000 (time 1740673472.11 total 23.19 last 898.93) - /unlink 28988 (time 1740673482.11 total 33.19 last 898.75) - /unlink 37962 (time 1740673492.11 total 43.19 last 897.32) - /unlink 40000 (time 1740673494.53 total 45.61 last 844.04) - /unlink 48422 (time 1740673504.53 total 55.61 last 842.12) - /unlink 57226 (time 1740673514.53 total 65.61 last 880.33) - /unlink 60000 (time 1740673517.90 total 68.98 last 822.53) total: 64768 /unlink in 74.43 seconds: 870.23 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5227 (time 1740673556.88 total 10.00 last 522.64) - open/close 10000 (time 1740673566.16 total 19.28 last 514.28) - open/close 15150 (time 1740673576.16 total 29.28 last 514.97) - open/close 20000 (time 1740673585.40 total 38.53 last 524.61) - open/close 25240 (time 1740673595.40 total 48.53 last 523.97) - open/close 30000 (time 1740673604.30 total 57.42 last 535.14) - open/close 35215 (time 1740673614.30 total 67.42 last 521.47) - open/close 40000 (time 1740673623.40 total 76.52 last 525.78) - open/close 45163 (time 1740673633.40 total 86.53 last 516.23) - open/close 50000 (time 1740673642.73 total 95.85 last 518.66) - open/close 55247 (time 1740673652.73 total 105.85 last 524.69) - open/close 60000 (time 1740673662.03 total 115.16 last 510.86) total: 64768 open/close in 124.81 seconds: 518.94 ops/second - /unlink 8942 (time 1740673682.50 total 10.00 last 894.18) - /unlink 17538 (time 1740673692.50 total 20.00 last 859.54) - /unlink 20000 (time 1740673695.20 total 22.70 last 913.43) - /unlink 28922 (time 1740673705.20 total 32.70 last 892.14) - /unlink 37815 (time 1740673715.20 total 42.70 last 889.25) - /unlink 40000 (time 1740673717.68 total 45.18 last 880.88) - /unlink 48805 (time 1740673727.68 total 55.18 last 880.49) - /unlink 50000 (time 1740673730.06 total 57.56 last 501.58) - /unlink 57847 (time 1740673740.06 total 67.56 last 784.64) - /unlink 60000 (time 1740673742.64 total 70.14 last 834.72) total: 64768 /unlink in 75.61 seconds: 856.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5617 (time 1740673782.13 total 10.00 last 561.64) - open/close 10000 (time 1740673790.09 total 17.96 last 550.97) - open/close 15536 (time 1740673800.09 total 27.96 last 553.55) - open/close 20000 (time 1740673808.10 total 35.96 last 557.45) - open/close 25380 (time 1740673818.10 total 45.97 last 537.92) - open/close 30000 (time 1740673826.49 total 54.36 last 550.51) - open/close 35196 (time 1740673836.49 total 64.36 last 519.60) - open/close 40000 (time 1740673845.39 total 73.26 last 539.54) - open/close 45010 (time 1740673855.39 total 83.26 last 500.92) - open/close 49345 (time 1740673865.40 total 93.26 last 433.47) - open/close 54355 (time 1740673875.40 total 103.27 last 500.98) - open/close 59018 (time 1740673885.40 total 113.27 last 466.26) - open/close 64253 (time 1740673895.40 total 123.27 last 523.48) total: 64768 open/close in 124.38 seconds: 520.74 ops/second - /unlink 8568 (time 1740673907.37 total 10.00 last 856.72) - /unlink 16776 (time 1740673917.37 total 20.00 last 820.72) - /unlink 20000 (time 1740673921.02 total 23.65 last 883.41) - /unlink 30000 (time 1740673930.87 total 33.51 last 1014.75) - /unlink 39028 (time 1740673940.87 total 43.51 last 902.51) - /unlink 48026 (time 1740673950.88 total 53.52 last 899.23) - /unlink 50000 (time 1740673953.08 total 55.72 last 897.42) - /unlink 56968 (time 1740673963.08 total 65.72 last 696.72) - /unlink 60000 (time 1740673971.82 total 74.45 last 347.03) - /unlink 64484 (time 1740673981.82 total 84.45 last 448.39) total: 64768 /unlink in 84.76 seconds: 764.11 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4803 (time 1740674007.15 total 10.00 last 480.29) - open/close 9168 (time 1740674017.15 total 20.00 last 436.48) - open/close 14798 (time 1740674027.15 total 30.00 last 562.96) - open/close 20000 (time 1740674036.83 total 39.67 last 537.83) - open/close 24851 (time 1740674046.83 total 49.67 last 485.09) - open/close 29575 (time 1740674056.83 total 59.67 last 472.38) - open/close 34595 (time 1740674066.83 total 69.67 last 502.00) - open/close 39408 (time 1740674076.83 total 79.67 last 481.28) - open/close 44355 (time 1740674086.83 total 89.68 last 494.62) - open/close 49274 (time 1740674096.83 total 99.68 last 491.88) - open/close 52706 (time 1740674106.83 total 109.68 last 343.16) - open/close 56386 (time 1740674116.83 total 119.68 last 367.94) - open/close 60000 (time 1740674125.32 total 128.17 last 425.69) - open/close 63068 (time 1740674135.32 total 138.17 last 306.71) total: 64768 open/close in 143.91 seconds: 450.05 ops/second - /unlink 6377 (time 1740674152.30 total 10.00 last 637.58) - /unlink 10000 (time 1740674157.48 total 15.18 last 699.11) - /unlink 14848 (time 1740674167.48 total 25.19 last 484.72) - /unlink 19909 (time 1740674177.48 total 35.19 last 506.03) - /unlink 25877 (time 1740674187.48 total 45.19 last 596.75) - /unlink 30000 (time 1740674194.76 total 52.47 last 566.33) - /unlink 35328 (time 1740674204.76 total 62.47 last 532.71) - /unlink 40000 (time 1740674213.89 total 71.60 last 511.80) - /unlink 45993 (time 1740674223.89 total 81.60 last 599.26) - /unlink 50000 (time 1740674230.46 total 88.17 last 610.20) - /unlink 55660 (time 1740674240.46 total 98.17 last 565.98) - /unlink 60000 (time 1740674249.05 total 106.75 last 505.50) total: 64768 /unlink in 114.51 seconds: 565.59 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3168 (time 1740674288.41 total 10.00 last 316.71) - open/close 6745 (time 1740674298.41 total 20.00 last 357.66) - open/close 10000 (time 1740674307.63 total 29.22 last 353.12) - open/close 13652 (time 1740674317.63 total 39.22 last 365.17) - open/close 17084 (time 1740674327.63 total 49.22 last 343.17) - open/close 19996 (time 1740674337.63 total 59.22 last 291.19) - open/close 22971 (time 1740674347.63 total 69.23 last 297.42) - open/close 26678 (time 1740674357.63 total 79.23 last 370.58) - open/close 30000 (time 1740674366.35 total 87.95 last 380.98) - open/close 32617 (time 1740674376.36 total 97.95 last 261.64) - open/close 35310 (time 1740674386.36 total 107.96 last 269.20) - open/close 38814 (time 1740674396.36 total 117.96 last 350.27) - open/close 40000 (time 1740674399.92 total 121.52 last 333.23) - open/close 42924 (time 1740674409.92 total 131.52 last 292.34) - open/close 45639 (time 1740674419.93 total 141.52 last 271.41) - open/close 49265 (time 1740674429.93 total 151.53 last 362.51) - open/close 52096 (time 1740674439.93 total 161.53 last 283.02) - open/close 55271 (time 1740674449.93 total 171.53 last 317.45) - open/close 58596 (time 1740674459.94 total 181.53 last 332.46) - open/close 60000 (time 1740674464.01 total 185.60 last 345.00) - open/close 63234 (time 1740674474.01 total 195.60 last 323.37) total: 64768 open/close in 199.97 seconds: 323.88 ops/second - /unlink 5483 (time 1740674489.54 total 10.00 last 548.29) - /unlink 10000 (time 1740674497.12 total 17.57 last 596.59) - /unlink 14921 (time 1740674507.12 total 27.57 last 492.08) - /unlink 20000 (time 1740674515.47 total 35.93 last 607.90) - /unlink 25641 (time 1740674525.47 total 45.93 last 564.05) - /unlink 30000 (time 1740674530.96 total 51.42 last 793.71) - /unlink 37829 (time 1740674540.97 total 61.42 last 782.83) - /unlink 40000 (time 1740674544.41 total 64.87 last 629.40) - /unlink 49401 (time 1740674554.41 total 74.87 last 940.06) - /unlink 58257 (time 1740674564.42 total 84.87 last 885.54) total: 64768 /unlink in 91.98 seconds: 704.18 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server unloading modules via unload_modules_local on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 PASS 106 (1315s) Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1390 sec ========== 11:43:51 (1740674631) === conf-sanity: start cleanup 11:43:51 (1740674631) === === conf-sanity: finish cleanup 11:43:52 (1740674632) ===