-----============= acceptance-small: conf-sanity ============----- Sun Apr 13 05:07:21 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg402-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg402-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg402-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg402-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg402-server oleg402-server: oleg402-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing load_modules_local oleg402-server: Loading modules from /home/green/git/lustre-release/lustre oleg402-server: detected 4 online CPUs by sysfs oleg402-server: MODOPTS_LIBCFS= oleg402-server: Force libcfs to create 2 CPU partitions oleg402-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg402-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg402-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:08:32 (1744535312) Stopping clients: oleg402-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg402-client.virtnet /mnt/lustre2 (opts:-f) oleg402-server: oleg402-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing load_modules_local oleg402-server: Loading modules from /home/green/git/lustre-release/lustre oleg402-server: detected 4 online CPUs by sysfs oleg402-server: MODOPTS_LIBCFS= oleg402-server: Force libcfs to create 2 CPU partitions oleg402-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg402-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg402-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg402-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3246 (time 1744535348.89 total 10.00 last 324.59) - open/close 5897 (time 1744535358.89 total 20.00 last 265.09) - open/close 9281 (time 1744535368.89 total 30.00 last 338.39) - open/close 14078 (time 1744535378.89 total 40.00 last 479.66) - open/close 18424 (time 1744535388.89 total 50.00 last 434.60) - open/close 20000 (time 1744535392.88 total 53.98 last 395.85) - open/close 23683 (time 1744535402.88 total 63.98 last 368.27) - open/close 28546 (time 1744535412.88 total 73.98 last 486.28) - open/close 30000 (time 1744535415.20 total 76.31 last 625.88) - open/close 35129 (time 1744535425.20 total 86.31 last 512.82) - open/close 40000 (time 1744535434.05 total 95.16 last 550.17) - open/close 45554 (time 1744535444.06 total 105.17 last 555.25) - open/close 50000 (time 1744535453.25 total 114.36 last 483.58) - open/close 55293 (time 1744535463.25 total 124.36 last 529.18) - open/close 59429 (time 1744535473.25 total 134.36 last 413.57) - open/close 63689 (time 1744535483.26 total 144.36 last 425.97) total: 64768 open/close in 146.97 seconds: 440.70 ops/second - /unlink 7355 (time 1744535496.70 total 10.00 last 735.47) - /unlink 10000 (time 1744535501.32 total 14.62 last 572.96) - /unlink 16620 (time 1744535511.32 total 24.62 last 662.00) - /unlink 20000 (time 1744535518.35 total 31.65 last 480.54) - /unlink 24898 (time 1744535528.35 total 41.65 last 489.71) - /unlink 29583 (time 1744535538.35 total 51.65 last 468.49) - /unlink 34757 (time 1744535548.35 total 61.65 last 517.36) - /unlink 40000 (time 1744535557.04 total 70.34 last 603.47) - /unlink 47509 (time 1744535567.04 total 80.34 last 750.83) - /unlink 50000 (time 1744535570.56 total 83.86 last 707.86) - /unlink 56705 (time 1744535580.56 total 93.86 last 670.38) - /unlink 60000 (time 1744535584.74 total 98.04 last 789.76) total: 64768 /unlink in 102.74 seconds: 630.40 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4717 (time 1744535622.39 total 10.00 last 471.62) - open/close 9780 (time 1744535632.39 total 20.00 last 506.28) - open/close 14921 (time 1744535642.39 total 30.00 last 514.07) - open/close 20000 (time 1744535652.29 total 39.90 last 513.40) - open/close 25170 (time 1744535662.29 total 49.90 last 516.94) - open/close 30000 (time 1744535671.32 total 58.93 last 534.95) - open/close 35084 (time 1744535681.32 total 68.93 last 508.36) - open/close 40000 (time 1744535690.90 total 78.51 last 512.92) - open/close 45314 (time 1744535700.90 total 88.51 last 531.33) - open/close 50000 (time 1744535709.83 total 97.44 last 525.06) - open/close 54186 (time 1744535719.83 total 107.44 last 418.51) - open/close 58693 (time 1744535729.83 total 117.44 last 450.65) - open/close 60000 (time 1744535732.51 total 120.12 last 488.34) total: 64768 open/close in 129.09 seconds: 501.72 ops/second - /unlink 9041 (time 1744535752.33 total 10.00 last 904.01) - /unlink 18077 (time 1744535762.33 total 20.00 last 903.54) - /unlink 27831 (time 1744535772.33 total 30.00 last 975.39) - /unlink 30000 (time 1744535774.62 total 32.29 last 946.96) - /unlink 39356 (time 1744535784.62 total 42.29 last 935.56) - /unlink 48841 (time 1744535794.62 total 52.29 last 948.49) - /unlink 58030 (time 1744535804.63 total 62.29 last 918.87) - /unlink 60000 (time 1744535806.77 total 64.43 last 920.45) total: 64768 /unlink in 69.07 seconds: 937.65 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 6620 (time 1744535852.57 total 10.00 last 661.93) - open/close 10000 (time 1744535857.63 total 15.06 last 668.30) - open/close 16496 (time 1744535867.63 total 25.06 last 649.54) - open/close 20000 (time 1744535873.10 total 30.53 last 640.36) - open/close 26433 (time 1744535883.10 total 40.53 last 643.22) - open/close 30000 (time 1744535888.64 total 46.07 last 644.70) - open/close 35753 (time 1744535898.64 total 56.07 last 575.27) - open/close 40000 (time 1744535906.96 total 64.39 last 510.37) - open/close 44868 (time 1744535916.96 total 74.39 last 486.72) - open/close 49524 (time 1744535926.96 total 84.39 last 465.58) - open/close 54320 (time 1744535936.96 total 94.39 last 479.46) - open/close 59151 (time 1744535946.97 total 104.39 last 483.02) - open/close 64731 (time 1744535956.97 total 114.40 last 557.92) total: 64768 open/close in 114.45 seconds: 565.90 ops/second - /unlink 10000 (time 1744535967.61 total 9.82 last 1018.46) - /unlink 20000 (time 1744535977.49 total 19.70 last 1012.01) - /unlink 30000 (time 1744535986.98 total 29.19 last 1053.87) - /unlink 40000 (time 1744535996.92 total 39.13 last 1006.26) - /unlink 50000 (time 1744536006.53 total 48.74 last 1040.62) - /unlink 60000 (time 1744536015.85 total 58.06 last 1072.85) total: 64768 /unlink in 62.38 seconds: 1038.23 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5474 (time 1744536067.92 total 10.00 last 547.32) - open/close 10000 (time 1744536076.39 total 18.48 last 533.83) - open/close 15055 (time 1744536086.40 total 28.48 last 505.45) - open/close 20000 (time 1744536096.24 total 38.32 last 502.33) - open/close 25115 (time 1744536106.24 total 48.33 last 511.47) - open/close 29891 (time 1744536116.24 total 58.33 last 477.60) - open/close 35026 (time 1744536126.24 total 68.33 last 513.44) - open/close 39843 (time 1744536136.24 total 78.33 last 481.66) - open/close 44637 (time 1744536146.24 total 88.33 last 479.29) - open/close 48724 (time 1744536156.25 total 98.33 last 408.58) - open/close 50000 (time 1744536159.42 total 101.50 last 402.74) - open/close 54038 (time 1744536169.42 total 111.50 last 403.72) - open/close 58543 (time 1744536179.42 total 121.50 last 450.49) - open/close 60000 (time 1744536182.28 total 124.37 last 508.70) - open/close 63538 (time 1744536192.28 total 134.37 last 353.73) total: 64768 open/close in 137.82 seconds: 469.95 ops/second - /unlink 6043 (time 1744536206.70 total 10.00 last 604.23) - /unlink 10000 (time 1744536211.83 total 15.13 last 771.27) - /unlink 18236 (time 1744536221.83 total 25.13 last 823.40) - /unlink 20000 (time 1744536223.84 total 27.14 last 878.90) - /unlink 28371 (time 1744536233.84 total 37.14 last 837.03) - /unlink 30000 (time 1744536236.05 total 39.36 last 735.03) - /unlink 37483 (time 1744536246.05 total 49.36 last 748.29) - /unlink 40000 (time 1744536248.87 total 52.18 last 892.17) - /unlink 49367 (time 1744536258.87 total 62.18 last 936.68) - /unlink 59012 (time 1744536268.87 total 72.18 last 964.44) total: 64768 /unlink in 78.18 seconds: 828.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5609 (time 1744536308.86 total 10.00 last 560.84) - open/close 10000 (time 1744536316.83 total 17.97 last 551.03) - open/close 15338 (time 1744536326.83 total 27.97 last 533.71) - open/close 20000 (time 1744536335.07 total 36.21 last 565.69) - open/close 25535 (time 1744536345.07 total 46.21 last 553.40) - open/close 30000 (time 1744536353.36 total 54.50 last 538.68) - open/close 35559 (time 1744536363.36 total 64.50 last 555.82) - open/close 40000 (time 1744536371.55 total 72.70 last 542.21) - open/close 45450 (time 1744536381.55 total 82.70 last 544.95) - open/close 50000 (time 1744536390.12 total 91.26 last 531.02) - open/close 54926 (time 1744536400.12 total 101.27 last 492.56) - open/close 60000 (time 1744536409.63 total 110.77 last 533.95) total: 64768 open/close in 119.61 seconds: 541.48 ops/second - /unlink 9176 (time 1744536429.27 total 10.00 last 917.55) - /unlink 17508 (time 1744536439.27 total 20.00 last 833.20) - /unlink 20000 (time 1744536442.07 total 22.80 last 889.15) - /unlink 29081 (time 1744536452.07 total 32.80 last 908.00) - /unlink 37956 (time 1744536462.07 total 42.81 last 887.38) - /unlink 40000 (time 1744536465.22 total 45.96 last 649.02) - /unlink 46816 (time 1744536475.22 total 55.96 last 681.54) - /unlink 50000 (time 1744536478.75 total 59.48 last 903.33) - /unlink 59317 (time 1744536488.75 total 69.48 last 931.69) total: 64768 /unlink in 75.11 seconds: 862.31 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server unloading modules via unload_modules_local on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2 PASS 106 (1226s) Stopping clients: oleg402-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg402-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2 oleg402-server: oleg402-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing load_modules_local oleg402-server: Loading modules from /home/green/git/lustre-release/lustre oleg402-server: detected 4 online CPUs by sysfs oleg402-server: MODOPTS_LIBCFS= oleg402-server: Force libcfs to create 2 CPU partitions oleg402-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg402-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1314 sec ========== 05:29:15 (1744536555) === conf-sanity: start cleanup 05:29:16 (1744536556) === === conf-sanity: finish cleanup 05:29:17 (1744536557) ===