-----============= acceptance-small: conf-sanity ============----- Tue Apr 8 15:28:08 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg432-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg432-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg432-server oleg432-server: oleg432-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions oleg432-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg432-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg432-server: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg432-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:29:11 (1744140551) Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:-f) oleg432-server: oleg432-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg432-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5338 (time 1744140585.03 total 10.00 last 533.75) - open/close 10000 (time 1744140593.26 total 18.23 last 566.46) - open/close 15249 (time 1744140603.26 total 28.23 last 524.79) - open/close 20000 (time 1744140611.23 total 36.20 last 596.03) - open/close 26005 (time 1744140621.23 total 46.20 last 600.45) - open/close 30000 (time 1744140627.75 total 52.72 last 613.48) - open/close 35383 (time 1744140637.75 total 62.72 last 538.24) - open/close 40000 (time 1744140646.05 total 71.03 last 555.74) - open/close 45653 (time 1744140656.06 total 81.03 last 565.27) - open/close 50000 (time 1744140663.41 total 88.38 last 590.74) - open/close 55827 (time 1744140673.41 total 98.39 last 582.65) - open/close 60000 (time 1744140680.77 total 105.74 last 567.60) total: 64768 open/close in 114.09 seconds: 567.69 ops/second - /unlink 9155 (time 1744140699.92 total 10.00 last 915.44) - /unlink 18634 (time 1744140709.92 total 20.00 last 947.79) - /unlink 27980 (time 1744140719.93 total 30.00 last 934.53) - /unlink 30000 (time 1744140722.20 total 32.28 last 887.44) - /unlink 38337 (time 1744140732.20 total 42.28 last 833.70) - /unlink 46576 (time 1744140742.20 total 52.28 last 823.89) - /unlink 50000 (time 1744140745.93 total 56.01 last 917.75) - /unlink 58701 (time 1744140755.93 total 66.01 last 870.07) total: 64768 /unlink in 73.06 seconds: 886.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5170 (time 1744140798.16 total 10.00 last 517.00) - open/close 10000 (time 1744140807.12 total 18.96 last 539.10) - open/close 15275 (time 1744140817.12 total 28.96 last 527.42) - open/close 20000 (time 1744140826.21 total 38.05 last 519.68) - open/close 24585 (time 1744140836.21 total 48.05 last 458.43) - open/close 29300 (time 1744140846.22 total 58.06 last 471.43) - open/close 34443 (time 1744140856.22 total 68.06 last 514.22) - open/close 39649 (time 1744140866.22 total 78.06 last 520.53) - open/close 44838 (time 1744140876.22 total 88.06 last 518.86) - open/close 49991 (time 1744140886.22 total 98.06 last 515.26) - open/close 55133 (time 1744140896.22 total 108.06 last 514.16) - open/close 59995 (time 1744140906.22 total 118.06 last 486.13) - open/close 64585 (time 1744140916.22 total 128.06 last 458.95) total: 64768 open/close in 128.56 seconds: 503.78 ops/second - /unlink 8058 (time 1744140927.61 total 10.00 last 805.76) - /unlink 10000 (time 1744140930.13 total 12.52 last 769.91) - /unlink 18354 (time 1744140940.14 total 22.52 last 835.31) - /unlink 20000 (time 1744140942.28 total 24.67 last 768.30) - /unlink 28037 (time 1744140952.28 total 34.67 last 803.63) - /unlink 30000 (time 1744140954.55 total 36.94 last 862.58) - /unlink 37974 (time 1744140964.56 total 46.94 last 797.33) - /unlink 40000 (time 1744140967.16 total 49.54 last 778.95) - /unlink 48278 (time 1744140977.16 total 59.55 last 827.71) - /unlink 50000 (time 1744140979.19 total 61.58 last 846.82) - /unlink 58250 (time 1744140989.19 total 71.58 last 824.99) total: 64768 /unlink in 78.89 seconds: 820.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5489 (time 1744141028.75 total 10.00 last 548.80) - open/close 10000 (time 1744141037.19 total 18.43 last 535.08) - open/close 15287 (time 1744141047.19 total 28.43 last 528.66) - open/close 20000 (time 1744141056.04 total 37.29 last 532.08) - open/close 24932 (time 1744141066.04 total 47.29 last 493.19) - open/close 30000 (time 1744141075.77 total 57.02 last 521.00) - open/close 35196 (time 1744141085.77 total 67.02 last 519.59) - open/close 40000 (time 1744141095.23 total 76.47 last 508.15) - open/close 45223 (time 1744141105.23 total 86.47 last 522.24) - open/close 50000 (time 1744141114.22 total 95.46 last 531.39) - open/close 55276 (time 1744141124.22 total 105.46 last 527.55) - open/close 60000 (time 1744141133.60 total 114.84 last 503.70) total: 64768 open/close in 124.30 seconds: 521.05 ops/second - /unlink 8852 (time 1744141153.94 total 10.00 last 885.17) - /unlink 17781 (time 1744141163.94 total 20.00 last 892.82) - /unlink 20000 (time 1744141166.43 total 22.49 last 891.51) - /unlink 28754 (time 1744141176.43 total 32.49 last 875.32) - /unlink 37249 (time 1744141186.43 total 42.49 last 849.43) - /unlink 40000 (time 1744141189.70 total 45.76 last 842.96) - /unlink 48822 (time 1744141199.70 total 55.76 last 882.17) - /unlink 57700 (time 1744141209.70 total 65.76 last 887.80) - /unlink 60000 (time 1744141213.11 total 69.17 last 673.23) total: 64768 /unlink in 74.75 seconds: 866.50 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3842 (time 1744141254.10 total 10.00 last 384.13) - open/close 8518 (time 1744141264.10 total 20.00 last 467.60) - open/close 10000 (time 1744141267.15 total 23.05 last 486.19) - open/close 14480 (time 1744141277.15 total 33.05 last 447.91) - open/close 19032 (time 1744141287.15 total 43.05 last 455.12) - open/close 23303 (time 1744141297.15 total 53.06 last 427.01) - open/close 26658 (time 1744141307.15 total 63.06 last 335.45) - open/close 30000 (time 1744141314.23 total 70.13 last 472.27) - open/close 35195 (time 1744141324.23 total 80.13 last 519.46) - open/close 40000 (time 1744141334.20 total 90.11 last 481.87) - open/close 44805 (time 1744141344.20 total 100.11 last 480.49) - open/close 49669 (time 1744141354.21 total 110.11 last 486.30) - open/close 54602 (time 1744141364.21 total 120.11 last 493.21) - open/close 59473 (time 1744141374.21 total 130.11 last 487.03) - open/close 63732 (time 1744141384.21 total 140.11 last 425.90) total: 64768 open/close in 143.20 seconds: 452.28 ops/second - /unlink 7162 (time 1744141398.45 total 10.00 last 716.18) - /unlink 10000 (time 1744141403.67 total 15.22 last 543.90) - /unlink 16692 (time 1744141413.67 total 25.22 last 669.07) - /unlink 20000 (time 1744141418.57 total 30.12 last 675.05) - /unlink 27144 (time 1744141428.57 total 40.12 last 714.34) - /unlink 30000 (time 1744141431.46 total 43.01 last 987.10) - /unlink 38731 (time 1744141441.46 total 53.02 last 873.03) - /unlink 46016 (time 1744141451.46 total 63.02 last 728.45) - /unlink 50000 (time 1744141456.96 total 68.51 last 724.80) - /unlink 58436 (time 1744141466.96 total 78.51 last 843.53) total: 64768 /unlink in 85.57 seconds: 756.88 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4049 (time 1744141505.27 total 10.00 last 404.88) - open/close 8293 (time 1744141515.27 total 20.00 last 424.36) - open/close 10000 (time 1744141519.44 total 24.17 last 409.85) - open/close 14271 (time 1744141529.44 total 34.17 last 427.06) - open/close 18310 (time 1744141539.44 total 44.17 last 403.89) - open/close 20000 (time 1744141543.27 total 48.00 last 440.48) - open/close 23780 (time 1744141553.28 total 58.01 last 377.86) - open/close 27285 (time 1744141563.28 total 68.01 last 350.49) - open/close 30000 (time 1744141571.06 total 75.79 last 349.01) - open/close 34096 (time 1744141581.06 total 85.79 last 409.59) - open/close 37921 (time 1744141591.06 total 95.79 last 382.48) - open/close 40000 (time 1744141596.83 total 101.56 last 360.39) - open/close 43895 (time 1744141606.83 total 111.56 last 389.46) - open/close 47227 (time 1744141616.83 total 121.56 last 333.18) - open/close 50000 (time 1744141625.50 total 130.23 last 319.79) - open/close 53349 (time 1744141635.50 total 140.23 last 334.88) - open/close 56102 (time 1744141645.50 total 150.23 last 275.27) - open/close 58809 (time 1744141655.50 total 160.23 last 270.69) - open/close 60000 (time 1744141660.02 total 164.75 last 263.79) - open/close 62647 (time 1744141670.02 total 174.75 last 264.62) total: 64768 open/close in 183.13 seconds: 353.68 ops/second - /unlink 4944 (time 1744141689.66 total 10.00 last 494.36) - /unlink 9870 (time 1744141699.66 total 20.00 last 492.59) - /unlink 14861 (time 1744141709.66 total 30.00 last 499.04) - /unlink 20000 (time 1744141719.13 total 39.47 last 542.98) - /unlink 24865 (time 1744141729.13 total 49.47 last 486.50) - /unlink 29672 (time 1744141739.13 total 59.47 last 480.64) - /unlink 34897 (time 1744141749.13 total 69.47 last 522.46) - /unlink 39738 (time 1744141759.13 total 79.47 last 484.05) - /unlink 44671 (time 1744141769.13 total 89.47 last 493.22) - /unlink 50000 (time 1744141777.22 total 97.56 last 659.02) - /unlink 56639 (time 1744141787.22 total 107.56 last 663.78) - /unlink 60000 (time 1744141796.70 total 117.04 last 354.57) total: 64768 /unlink in 124.76 seconds: 519.13 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg432-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server unloading modules via unload_modules_local on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 PASS 106 (1293s) Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 oleg432-server: oleg432-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions oleg432-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg432-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1374 sec ========== 15:51:02 (1744141862) === conf-sanity: start cleanup 15:51:02 (1744141862) === === conf-sanity: finish cleanup 15:51:03 (1744141863) ===