-----============= acceptance-small: conf-sanity ============----- Fri May 16 08:45:11 EDT 2025 oleg223-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg223-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg223-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg223-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg223-server oleg223-server: oleg223-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing load_modules_local oleg223-server: Loading modules from /home/green/git/lustre-release/lustre oleg223-server: detected 4 online CPUs by sysfs oleg223-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg223-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg223-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg223-server: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg223-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server == conf-sanity test 45: long unlink handling in ptlrpcd == 08:46:42 (1747399602) start mds service on oleg223-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg223-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock oleg223-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg223-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock oleg223-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server LNET ready to unload unloading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing unload_modules_local oleg223-server: LNET ready to unload modules unloaded. pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 PASS 45 (148s) == conf-sanity test 69: replace an OST with the same index ========================================================== 08:49:10 (1747399750) start mds service on oleg223-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing load_modules_local oleg223-server: Loading modules from /home/green/git/lustre-release/lustre oleg223-server: detected 4 online CPUs by sysfs oleg223-server: Force libcfs to create 2 CPU partitions oleg223-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg223-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg223-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock oleg223-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1763 (time 1747399794.15 total 10.00 last 176.28) - open/close 3588 (time 1747399804.15 total 20.00 last 182.47) - open/close 5416 (time 1747399814.15 total 30.01 last 182.75) - open/close 7233 (time 1747399824.16 total 40.02 last 181.52) - open/close 9035 (time 1747399834.17 total 50.02 last 180.15) total: 10000 open/close in 55.17 seconds: 181.24 ops/second - unlinked 0 (time 1747399840 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1518 (time 1747399885.49 total 10.00 last 151.79) - open/close 3092 (time 1747399895.49 total 20.00 last 157.39) - open/close 4637 (time 1747399905.49 total 30.00 last 154.49) - open/close 6270 (time 1747399915.49 total 40.00 last 163.28) - open/close 7792 (time 1747399925.49 total 50.00 last 152.19) - open/close 9415 (time 1747399935.50 total 60.01 last 162.21) total: 10000 open/close in 63.37 seconds: 157.80 ops/second - unlinked 0 (time 1747399940 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1834 (time 1747399986.72 total 10.00 last 183.32) - open/close 3584 (time 1747399996.72 total 20.01 last 174.92) - open/close 5193 (time 1747400006.73 total 30.01 last 160.86) - open/close 6722 (time 1747400016.73 total 40.02 last 152.85) - open/close 8445 (time 1747400026.73 total 50.02 last 172.24) total: 10000 open/close in 59.07 seconds: 169.30 ops/second - unlinked 0 (time 1747400037 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1612 (time 1747400080.63 total 10.00 last 161.19) - open/close 3336 (time 1747400090.63 total 20.00 last 172.37) - open/close 5231 (time 1747400100.64 total 30.01 last 189.40) - open/close 7248 (time 1747400110.64 total 40.01 last 201.65) - open/close 9344 (time 1747400120.64 total 50.01 last 209.57) total: 10000 open/close in 53.05 seconds: 188.49 ops/second - unlinked 0 (time 1747400125 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1779 (time 1747400167.58 total 10.00 last 177.84) - open/close 3919 (time 1747400177.59 total 20.01 last 213.89) - open/close 6010 (time 1747400187.59 total 30.01 last 209.02) - open/close 8445 (time 1747400197.59 total 40.02 last 243.43) total: 10000 open/close in 47.11 seconds: 212.28 ops/second - unlinked 0 (time 1747400205 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1863 (time 1747400247.80 total 10.00 last 186.23) - open/close 3794 (time 1747400257.81 total 20.01 last 193.04) - open/close 5932 (time 1747400267.81 total 30.01 last 213.79) - open/close 8349 (time 1747400277.81 total 40.01 last 241.64) total: 10000 open/close in 47.75 seconds: 209.44 ops/second - unlinked 0 (time 1747400286 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2233 (time 1747400323.45 total 10.00 last 223.27) - open/close 4135 (time 1747400333.45 total 20.00 last 190.20) - open/close 6473 (time 1747400343.45 total 30.00 last 233.73) - open/close 9418 (time 1747400353.46 total 40.01 last 294.36) total: 10000 open/close in 42.00 seconds: 238.12 ops/second - unlinked 0 (time 1747400356 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2537 (time 1747400388.03 total 10.00 last 253.65) - open/close 4619 (time 1747400398.03 total 20.00 last 208.18) - open/close 6837 (time 1747400408.03 total 30.00 last 221.80) - open/close 8974 (time 1747400418.03 total 40.01 last 213.58) total: 10000 open/close in 44.66 seconds: 223.93 ops/second - unlinked 0 (time 1747400423 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2527 (time 1747400460.77 total 10.00 last 252.70) - open/close 5313 (time 1747400470.77 total 20.00 last 278.51) - open/close 7804 (time 1747400480.77 total 30.01 last 249.05) total: 10000 open/close in 39.29 seconds: 254.50 ops/second - unlinked 0 (time 1747400491 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 3317 (time 1747400517.03 total 10.00 last 331.68) - open/close 6507 (time 1747400527.03 total 20.00 last 318.95) - open/close 9473 (time 1747400537.03 total 30.01 last 296.51) total: 10000 open/close in 32.34 seconds: 309.18 ops/second - unlinked 0 (time 1747400540 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.123@tcp sys.timeout=20 mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.123@tcp lustre:sys.timeout=20 lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg223-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg223-server: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg223-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock oleg223-server@tcp:/lustre /mnt/lustre On OST0, 291 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server LNET ready to unload unloading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing unload_modules_local oleg223-server: LNET ready to unload modules unloaded. pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2 PASS 69 (849s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 09:03:20 (1747400600) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg223-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg223-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 oleg223-server: oleg223-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing load_modules_local oleg223-server: Loading modules from /home/green/git/lustre-release/lustre oleg223-server: detected 4 online CPUs by sysfs oleg223-server: Force libcfs to create 2 CPU partitions oleg223-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg223-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1112 sec ========== 09:03:44 (1747400624)