-----============= acceptance-small: conf-sanity ============----- Tue Jul 4 07:58:49 EDT 2023 excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg376-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg376-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg376-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg376-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg376-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg376-server oleg376-server: oleg376-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg376-server' oleg376-server: oleg376-server.virtnet: executing load_modules_local oleg376-server: Loading modules from /home/green/git/lustre-release/lustre oleg376-server: detected 4 online CPUs by sysfs oleg376-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg376-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg376-server: oleg376-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg376-client: oleg376-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg376-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg376-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg376-server: oleg376-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg376-client: oleg376-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg376-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg376-server: oleg376-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg376-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg376-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg376-server stop mds service on oleg376-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg376-server == conf-sanity test 45: long unlink handling in ptlrpcd == 07:59:45 (1688471985) start mds service on oleg376-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg376-server: oleg376-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg376-client: oleg376-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg376-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg376-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg376-server: oleg376-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg376-client: oleg376-server: ssh exited with exit code 1 Started lustre-OST0000 oleg376-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg376-client.virtnet: -o user_xattr,flock oleg376-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg376-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg376-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg376-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg376-server: oleg376-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg376-client: oleg376-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg376-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg376-client.virtnet: -o user_xattr,flock oleg376-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg376-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg376-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg376-server stop mds service on oleg376-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg376-server unloading modules on: 'oleg376-server' oleg376-server: oleg376-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg376-client: oleg376-client: ssh exited with exit code 2 pdsh@oleg376-client: oleg376-server: ssh exited with exit code 2 pdsh@oleg376-client: oleg376-client: ssh exited with exit code 2 pdsh@oleg376-client: oleg376-server: ssh exited with exit code 2 PASS 45 (118s) == conf-sanity test 69: replace an OST with the same index ========================================================== 08:01:43 (1688472103) start mds service on oleg376-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg376-server' oleg376-server: oleg376-server.virtnet: executing load_modules_local oleg376-server: Loading modules from /home/green/git/lustre-release/lustre oleg376-server: detected 4 online CPUs by sysfs oleg376-server: Force libcfs to create 2 CPU partitions oleg376-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg376-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg376-server: oleg376-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg376-client: oleg376-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg376-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg376-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg376-server: oleg376-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg376-client: oleg376-server: ssh exited with exit code 1 Started lustre-OST0000 oleg376-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg376-client.virtnet: -o user_xattr,flock oleg376-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2542 (time 1688472137.91 total 10.00 last 254.14) - open/close 5216 (time 1688472147.91 total 20.00 last 267.40) - open/close 7602 (time 1688472157.91 total 30.00 last 238.59) - open/close 9941 (time 1688472167.92 total 40.01 last 233.83) total: 10000 open/close in 40.28 seconds: 248.26 ops/second - unlinked 0 (time 1688472169 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2565 (time 1688472200.27 total 10.00 last 256.48) - open/close 5182 (time 1688472210.27 total 20.00 last 261.61) - open/close 7523 (time 1688472220.28 total 30.01 last 234.06) - open/close 9865 (time 1688472230.28 total 40.01 last 234.19) total: 10000 open/close in 40.46 seconds: 247.18 ops/second - unlinked 0 (time 1688472231 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2342 (time 1688472264.29 total 10.00 last 234.16) - open/close 4756 (time 1688472274.29 total 20.00 last 241.39) - open/close 7091 (time 1688472284.29 total 30.01 last 233.42) - open/close 9435 (time 1688472294.30 total 40.01 last 234.34) total: 10000 open/close in 42.45 seconds: 235.55 ops/second - unlinked 0 (time 1688472298 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2727 (time 1688472330.13 total 10.00 last 272.66) - open/close 5212 (time 1688472340.13 total 20.00 last 248.45) - open/close 7789 (time 1688472350.13 total 30.00 last 257.70) total: 10000 open/close in 39.33 seconds: 254.28 ops/second - unlinked 0 (time 1688472360 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2495 (time 1688472392.63 total 10.00 last 249.46) - open/close 4979 (time 1688472402.63 total 20.00 last 248.36) - open/close 7279 (time 1688472412.63 total 30.00 last 229.99) - open/close 9615 (time 1688472422.63 total 40.00 last 233.58) total: 10000 open/close in 41.61 seconds: 240.31 ops/second - unlinked 0 (time 1688472425 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2434 (time 1688472457.16 total 10.00 last 243.33) - open/close 4902 (time 1688472467.16 total 20.00 last 246.75) - open/close 7371 (time 1688472477.16 total 30.01 last 246.89) - open/close 9860 (time 1688472487.16 total 40.01 last 248.86) total: 10000 open/close in 40.64 seconds: 246.09 ops/second - unlinked 0 (time 1688472489 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2707 (time 1688472522.09 total 10.00 last 270.68) - open/close 5005 (time 1688472532.09 total 20.00 last 229.76) - open/close 7364 (time 1688472542.09 total 30.01 last 235.82) - open/close 9871 (time 1688472552.09 total 40.01 last 250.61) total: 10000 open/close in 40.58 seconds: 246.44 ops/second - unlinked 0 (time 1688472553 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2657 (time 1688472586.13 total 10.00 last 265.63) - open/close 5168 (time 1688472596.13 total 20.01 last 251.00) - open/close 7668 (time 1688472606.14 total 30.01 last 249.99) total: 10000 open/close in 38.99 seconds: 256.44 ops/second - unlinked 0 (time 1688472616 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2544 (time 1688472648.13 total 10.00 last 254.38) - open/close 5025 (time 1688472658.13 total 20.00 last 248.01) - open/close 7541 (time 1688472668.13 total 30.01 last 251.57) - open/close 9892 (time 1688472678.14 total 40.01 last 235.00) total: 10000 open/close in 40.50 seconds: 246.91 ops/second - unlinked 0 (time 1688472679 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2478 (time 1688472711.83 total 10.00 last 247.77) - open/close 4842 (time 1688472721.83 total 20.00 last 236.37) - open/close 7154 (time 1688472731.84 total 30.00 last 231.16) - open/close 9540 (time 1688472741.84 total 40.01 last 238.54) total: 10000 open/close in 41.05 seconds: 243.60 ops/second - unlinked 0 (time 1688472743 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg376-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg376-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg376-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.176@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.176@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg376-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg376-server: oleg376-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg376-client: oleg376-server: ssh exited with exit code 1 Started lustre-OST0000 oleg376-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg376-server: oleg376-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg376-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg376-client.virtnet: -o user_xattr,flock oleg376-server@tcp:/lustre /mnt/lustre On OST0, 17741 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg376-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg376-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg376-server stop mds service on oleg376-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg376-server unloading modules on: 'oleg376-server' oleg376-server: oleg376-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg376-client: oleg376-client: ssh exited with exit code 2 pdsh@oleg376-client: oleg376-server: ssh exited with exit code 2 pdsh@oleg376-client: oleg376-client: ssh exited with exit code 2 pdsh@oleg376-client: oleg376-server: ssh exited with exit code 2 PASS 69 (692s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 08:13:15 (1688472795) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg376-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg376-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg376-client: oleg376-server: ssh exited with exit code 2 oleg376-server: oleg376-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg376-server' oleg376-server: oleg376-server.virtnet: executing load_modules_local oleg376-server: Loading modules from /home/green/git/lustre-release/lustre oleg376-server: detected 4 online CPUs by sysfs oleg376-server: Force libcfs to create 2 CPU partitions oleg376-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg376-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 891 sec =========== 08:13:40 (1688472820) === conf-sanity: start cleanup 08:13:40 (1688472820) === === conf-sanity: finish cleanup 08:13:40 (1688472820) ===