-----============= acceptance-small: conf-sanity ============----- Thu May 1 17:46:35 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg225-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg225-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server oleg225-server: oleg225-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing load_modules_local oleg225-server: Loading modules from /home/green/git/lustre-release/lustre oleg225-server: detected 4 online CPUs by sysfs oleg225-server: MODOPTS_LIBCFS= oleg225-server: Force libcfs to create 2 CPU partitions oleg225-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg225-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg225-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg225-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server == conf-sanity test 45: long unlink handling in ptlrpcd == 17:47:52 (1746136072) start mds service on oleg225-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg225-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg225-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server unloading modules via unload_modules_local on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 PASS 45 (145s) == conf-sanity test 69: replace an OST with the same index ========================================================== 17:50:17 (1746136217) start mds service on oleg225-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing load_modules_local oleg225-server: Loading modules from /home/green/git/lustre-release/lustre oleg225-server: detected 4 online CPUs by sysfs oleg225-server: MODOPTS_LIBCFS= oleg225-server: Force libcfs to create 2 CPU partitions oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg225-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1282 (time 1746136262.86 total 10.01 last 128.08) - open/close 2385 (time 1746136272.87 total 20.02 last 110.23) - open/close 3957 (time 1746136282.87 total 30.02 last 157.14) - open/close 5382 (time 1746136292.88 total 40.02 last 142.47) - open/close 6773 (time 1746136302.89 total 50.03 last 138.94) - open/close 7885 (time 1746136312.89 total 60.04 last 111.14) - open/close 9253 (time 1746136322.90 total 70.04 last 136.75) total: 10000 open/close in 76.24 seconds: 131.17 ops/second - unlinked 0 (time 1746136330 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1191 (time 1746136380.94 total 10.00 last 119.04) - open/close 2516 (time 1746136390.94 total 20.01 last 132.47) - open/close 3932 (time 1746136400.95 total 30.02 last 141.46) - open/close 5163 (time 1746136410.96 total 40.02 last 123.08) - open/close 6522 (time 1746136420.96 total 50.02 last 135.82) - open/close 7745 (time 1746136430.98 total 60.04 last 122.11) - open/close 8763 (time 1746136440.98 total 70.05 last 101.73) - open/close 9815 (time 1746136451.00 total 80.06 last 105.06) total: 10000 open/close in 81.99 seconds: 121.97 ops/second - unlinked 0 (time 1746136454 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1077 (time 1746136524.55 total 10.00 last 107.67) - open/close 2164 (time 1746136534.56 total 20.01 last 108.65) - open/close 3259 (time 1746136544.57 total 30.02 last 109.40) - open/close 4205 (time 1746136554.57 total 40.02 last 94.58) - open/close 5499 (time 1746136564.57 total 50.02 last 129.39) - open/close 6544 (time 1746136574.58 total 60.03 last 104.43) - open/close 7646 (time 1746136584.58 total 70.03 last 110.18) - open/close 8872 (time 1746136594.59 total 80.03 last 122.52) total: 10000 open/close in 87.62 seconds: 114.13 ops/second - unlinked 0 (time 1746136603 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1289 (time 1746136672.06 total 10.00 last 128.89) - open/close 2661 (time 1746136682.07 total 20.01 last 137.00) - open/close 4085 (time 1746136692.07 total 30.02 last 142.35) - open/close 5926 (time 1746136702.08 total 40.02 last 184.00) - open/close 7164 (time 1746136712.09 total 50.04 last 123.62) - open/close 8655 (time 1746136722.10 total 60.04 last 149.04) total: 10000 open/close in 68.47 seconds: 146.05 ops/second - unlinked 0 (time 1746136732 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1666 (time 1746136773.63 total 10.00 last 166.52) - open/close 3341 (time 1746136783.64 total 20.01 last 167.34) - open/close 5123 (time 1746136793.65 total 30.02 last 178.10) - open/close 6895 (time 1746136803.65 total 40.02 last 177.14) - open/close 8962 (time 1746136813.65 total 50.02 last 206.68) total: 10000 open/close in 57.90 seconds: 172.71 ops/second - unlinked 0 (time 1746136823 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1346 (time 1746136877.52 total 10.01 last 134.44) - open/close 3101 (time 1746136887.53 total 20.02 last 175.31) - open/close 4552 (time 1746136897.53 total 30.02 last 145.09) - open/close 6118 (time 1746136907.53 total 40.02 last 156.60) - open/close 7482 (time 1746136917.53 total 50.03 last 136.31) total: 10000 open/close in 59.03 seconds: 169.39 ops/second - unlinked 0 (time 1746136927 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2167 (time 1746136963.07 total 10.00 last 216.64) - open/close 4017 (time 1746136973.08 total 20.01 last 184.90) - open/close 5352 (time 1746136983.08 total 30.01 last 133.44) - open/close 6799 (time 1746136993.09 total 40.02 last 144.59) - open/close 8378 (time 1746137003.09 total 50.02 last 157.84) total: 10000 open/close in 58.89 seconds: 169.80 ops/second - unlinked 0 (time 1746137013 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1617 (time 1746137066.75 total 10.00 last 161.69) - open/close 3163 (time 1746137076.75 total 20.00 last 154.58) - open/close 4447 (time 1746137086.76 total 30.01 last 128.28) - open/close 5856 (time 1746137096.76 total 40.01 last 140.85) - open/close 7228 (time 1746137106.77 total 50.02 last 137.14) - open/close 8577 (time 1746137116.77 total 60.02 last 134.83) - open/close 9730 (time 1746137126.78 total 70.03 last 115.27) total: 10000 open/close in 72.72 seconds: 137.51 ops/second - unlinked 0 (time 1746137131 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 2006 (time 1746137172.16 total 10.00 last 200.58) - open/close 4105 (time 1746137182.16 total 20.01 last 209.81) - open/close 5870 (time 1746137192.16 total 30.01 last 176.45) - open/close 7962 (time 1746137202.17 total 40.01 last 209.09) - open/close 9799 (time 1746137212.17 total 50.02 last 183.62) total: 10000 open/close in 51.13 seconds: 195.56 ops/second - unlinked 0 (time 1746137214 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1526 (time 1746137263.71 total 10.01 last 152.49) - open/close 3100 (time 1746137273.72 total 20.02 last 157.24) - open/close 4445 (time 1746137283.72 total 30.02 last 134.49) - open/close 6120 (time 1746137293.72 total 40.03 last 167.38) - open/close 7891 (time 1746137303.73 total 50.03 last 177.08) - open/close 9633 (time 1746137313.73 total 60.03 last 174.11) total: 10000 open/close in 62.47 seconds: 160.08 ops/second - unlinked 0 (time 1746137317 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.125@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.125@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg225-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg225-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre On OST0, 17661 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server unloading modules via unload_modules_local on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 PASS 69 (1179s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 18:09:56 (1746137396) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 oleg225-server: oleg225-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing load_modules_local oleg225-server: Loading modules from /home/green/git/lustre-release/lustre oleg225-server: detected 4 online CPUs by sysfs oleg225-server: MODOPTS_LIBCFS= oleg225-server: Force libcfs to create 2 CPU partitions oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1429 sec ========== 18:10:24 (1746137424) === conf-sanity: start cleanup 18:10:25 (1746137425) === === conf-sanity: finish cleanup 18:10:26 (1746137426) ===