-----============= acceptance-small: conf-sanity ============----- Fri May 16 08:45:00 EDT 2025 oleg451-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg451-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg451-server oleg451-server: oleg451-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg451-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server == conf-sanity test 45: long unlink handling in ptlrpcd == 08:46:46 (1747399606) start mds service on oleg451-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock oleg451-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg451-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock oleg451-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server LNET ready to unload unloading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local oleg451-server: LNET ready to unload modules unloaded. pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 PASS 45 (174s) == conf-sanity test 69: replace an OST with the same index ========================================================== 08:49:40 (1747399780) start mds service on oleg451-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock oleg451-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1435 (time 1747399830.70 total 10.00 last 143.44) - open/close 2778 (time 1747399840.70 total 20.01 last 134.25) - open/close 4201 (time 1747399850.70 total 30.01 last 142.28) - open/close 5635 (time 1747399860.70 total 40.01 last 143.39) - open/close 7407 (time 1747399870.70 total 50.01 last 177.19) - open/close 9530 (time 1747399880.70 total 60.01 last 212.28) total: 10000 open/close in 62.22 seconds: 160.72 ops/second - unlinked 0 (time 1747399884 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2287 (time 1747399922.04 total 10.00 last 228.69) - open/close 4062 (time 1747399932.04 total 20.00 last 177.49) - open/close 5983 (time 1747399942.05 total 30.01 last 192.01) - open/close 7960 (time 1747399952.05 total 40.01 last 197.68) - open/close 9936 (time 1747399962.05 total 50.01 last 197.58) total: 10000 open/close in 50.33 seconds: 198.68 ops/second - unlinked 0 (time 1747399963 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 1683 (time 1747400001.88 total 10.00 last 168.26) - open/close 3396 (time 1747400011.88 total 20.00 last 171.27) - open/close 5328 (time 1747400021.89 total 30.01 last 193.13) - open/close 6834 (time 1747400031.90 total 40.02 last 150.44) - open/close 8400 (time 1747400041.90 total 50.02 last 156.54) total: 10000 open/close in 59.77 seconds: 167.31 ops/second - unlinked 0 (time 1747400052 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 2066 (time 1747400093.79 total 10.00 last 206.56) - open/close 4383 (time 1747400103.79 total 20.00 last 231.64) - open/close 6723 (time 1747400113.79 total 30.01 last 233.90) - open/close 8724 (time 1747400123.79 total 40.01 last 200.09) total: 10000 open/close in 46.13 seconds: 216.77 ops/second - unlinked 0 (time 1747400131 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2439 (time 1747400164.77 total 10.00 last 243.80) - open/close 4973 (time 1747400174.77 total 20.00 last 253.38) - open/close 7477 (time 1747400184.77 total 30.01 last 250.38) total: 10000 open/close in 39.76 seconds: 251.52 ops/second - unlinked 0 (time 1747400195 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2444 (time 1747400229.88 total 10.00 last 244.36) - open/close 4692 (time 1747400239.88 total 20.00 last 224.75) - open/close 7010 (time 1747400249.88 total 30.01 last 231.75) - open/close 9622 (time 1747400259.88 total 40.01 last 261.07) total: 10000 open/close in 41.35 seconds: 241.83 ops/second - unlinked 0 (time 1747400262 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2901 (time 1747400291.35 total 10.00 last 290.02) - open/close 5666 (time 1747400301.36 total 20.01 last 276.40) - open/close 8614 (time 1747400311.36 total 30.01 last 294.79) total: 10000 open/close in 34.80 seconds: 287.33 ops/second - unlinked 0 (time 1747400317 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 4174 (time 1747400343.71 total 10.00 last 417.34) - open/close 7286 (time 1747400353.71 total 20.01 last 311.02) total: 10000 open/close in 29.72 seconds: 336.48 ops/second - unlinked 0 (time 1747400364 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 3040 (time 1747400393.35 total 10.00 last 303.93) - open/close 6088 (time 1747400403.36 total 20.00 last 304.73) - open/close 8711 (time 1747400413.36 total 30.01 last 262.26) total: 10000 open/close in 35.06 seconds: 285.23 ops/second - unlinked 0 (time 1747400419 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3819 (time 1747400447.21 total 10.00 last 381.71) - open/close 7646 (time 1747400457.21 total 20.01 last 382.64) total: 10000 open/close in 25.72 seconds: 388.82 ops/second - unlinked 0 (time 1747400463 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.151@tcp lustre:sys.timeout=20 lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg451-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock oleg451-server@tcp:/lustre /mnt/lustre On OST0, 291 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server LNET ready to unload unloading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local oleg451-server: LNET ready to unload modules unloaded. pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 PASS 69 (733s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 09:01:54 (1747400514) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 oleg451-server: oleg451-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1033 sec ========== 09:02:14 (1747400534)