-----============= acceptance-small: conf-sanity ============----- Tue Apr 1 04:13:50 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg660-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 32b 32c 32e 32h 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg660-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg660-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg660-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg660-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg660-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg660-server oleg660-server: oleg660-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg660-server' oleg660-server: oleg660-server.virtnet: executing load_modules_local oleg660-server: Loading modules from /home/green/git/lustre-release/lustre oleg660-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg660-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg660-server: detected 4 online CPUs by sysfs oleg660-server: MODOPTS_LIBCFS= oleg660-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg660-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg660-server: oleg660-server.virtnet: executing set_default_debug -1 all pdsh@oleg660-client: oleg660-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg660-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg660-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg660-server: oleg660-server.virtnet: executing set_default_debug -1 all pdsh@oleg660-client: oleg660-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg660-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg660-server: oleg660-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg660-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg660-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg660-server stop mds service on oleg660-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg660-server == conf-sanity test 45: long unlink handling in ptlrpcd == 04:19:36 (1743495576) start mds service on oleg660-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg660-server: oleg660-server.virtnet: executing set_default_debug -1 all pdsh@oleg660-client: oleg660-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg660-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg660-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg660-server: oleg660-server.virtnet: executing set_default_debug -1 all pdsh@oleg660-client: oleg660-server: ssh exited with exit code 1 Started lustre-OST0000 oleg660-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg660-client.virtnet: -o user_xattr,flock 192.168.206.160@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg660-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg660-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg660-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg660-server: oleg660-server.virtnet: executing set_default_debug -1 all pdsh@oleg660-client: oleg660-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg660-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg660-client.virtnet: -o user_xattr,flock 192.168.206.160@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg660-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg660-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg660-server stop mds service on oleg660-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg660-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg660-server' oleg660-server: oleg660-server.virtnet: executing unload_modules_local oleg660-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg660-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg660-client: oleg660-client: ssh exited with exit code 2 pdsh@oleg660-client: oleg660-server: ssh exited with exit code 2 pdsh@oleg660-client: oleg660-client: ssh exited with exit code 2 pdsh@oleg660-client: oleg660-server: ssh exited with exit code 2 PASS 45 (353s) == conf-sanity test 69: replace an OST with the same index ========================================================== 04:25:30 (1743495930) start mds service on oleg660-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg660-server' oleg660-server: oleg660-server.virtnet: executing load_modules_local oleg660-server: Loading modules from /home/green/git/lustre-release/lustre oleg660-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg660-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg660-server: detected 4 online CPUs by sysfs oleg660-server: MODOPTS_LIBCFS= oleg660-server: Force libcfs to create 2 CPU partitions oleg660-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg660-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg660-server: oleg660-server.virtnet: executing set_default_debug -1 all pdsh@oleg660-client: oleg660-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg660-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg660-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg660-server: oleg660-server.virtnet: executing set_default_debug -1 all pdsh@oleg660-client: oleg660-server: ssh exited with exit code 1 Started lustre-OST0000 oleg660-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg660-client.virtnet: -o user_xattr,flock 192.168.206.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 380 (time 1743496093.78 total 10.02 last 37.93) - open/close 712 (time 1743496103.81 total 20.04 last 33.12) - open/close 1071 (time 1743496113.81 total 30.05 last 35.89) - open/close 1403 (time 1743496123.83 total 40.06 last 33.15) - open/close 1758 (time 1743496133.84 total 50.07 last 35.46) - open/close 2168 (time 1743496143.84 total 60.08 last 40.99) - open/close 2515 (time 1743496153.86 total 70.09 last 34.65) - open/close 2874 (time 1743496163.86 total 80.10 last 35.88) - open/close 3249 (time 1743496173.87 total 90.11 last 37.45) - open/close 3594 (time 1743496183.88 total 100.12 last 34.47) - open/close 3928 (time 1743496193.90 total 110.13 last 33.36) - open/close 4286 (time 1743496203.94 total 120.17 last 35.66) - open/close 4582 (time 1743496213.95 total 130.18 last 29.57) - open/close 4906 (time 1743496223.97 total 140.21 last 32.31) - open/close 5210 (time 1743496233.98 total 150.21 last 30.40) - open/close 5544 (time 1743496243.99 total 160.22 last 33.36) - open/close 5864 (time 1743496253.99 total 170.22 last 32.00) - open/close 6201 (time 1743496264.01 total 180.24 last 33.63) - open/close 6553 (time 1743496274.02 total 190.26 last 35.16) - open/close 6875 (time 1743496284.02 total 200.26 last 32.19) - open/close 7197 (time 1743496294.08 total 210.31 last 32.02) - open/close 7521 (time 1743496304.10 total 220.34 last 32.33) - open/close 7872 (time 1743496314.12 total 230.35 last 35.04) - open/close 8210 (time 1743496324.13 total 240.37 last 33.74) - open/close 8532 (time 1743496334.15 total 250.38 last 32.15) - open/close 8893 (time 1743496344.16 total 260.40 last 36.05) - open/close 9230 (time 1743496354.19 total 270.42 last 33.62) - open/close 9584 (time 1743496364.19 total 280.43 last 35.38) - open/close 9912 (time 1743496374.20 total 290.43 last 32.80) total: 10000 open/close in 292.79 seconds: 34.15 ops/second - unlinked 0 (time 1743496387 ; total 0 ; last 0) total: 10000 unlinks in 203 seconds: 49.261086 unlinks/second - open/close 358 (time 1743496610.90 total 10.01 last 35.77) - open/close 689 (time 1743496620.91 total 20.01 last 33.08) - open/close 1079 (time 1743496630.91 total 30.02 last 38.99) - open/close 1422 (time 1743496640.92 total 40.02 last 34.28) - open/close 1750 (time 1743496650.93 total 50.04 last 32.75) - open/close 2102 (time 1743496660.95 total 60.05 last 35.15) - open/close 2452 (time 1743496670.95 total 70.06 last 34.99) - open/close 2837 (time 1743496680.97 total 80.08 last 38.42) - open/close 3183 (time 1743496690.99 total 90.09 last 34.55) - open/close 3540 (time 1743496700.99 total 100.10 last 35.68) - open/close 3901 (time 1743496711.01 total 110.12 last 36.03) - open/close 4233 (time 1743496721.09 total 120.20 last 32.94) - open/close 4548 (time 1743496731.15 total 130.25 last 31.33) - open/close 4805 (time 1743496741.17 total 140.27 last 25.64) - open/close 5096 (time 1743496751.19 total 150.30 last 29.03) - open/close 5407 (time 1743496761.23 total 160.33 last 30.99) - open/close 5701 (time 1743496771.24 total 170.35 last 29.37) - open/close 6009 (time 1743496781.27 total 180.38 last 30.71) - open/close 6296 (time 1743496791.27 total 190.38 last 28.69) - open/close 6602 (time 1743496801.28 total 200.39 last 30.57) - open/close 6928 (time 1743496811.30 total 210.40 last 32.56) - open/close 7275 (time 1743496821.33 total 220.43 last 34.60) - open/close 7601 (time 1743496831.33 total 230.44 last 32.58) - open/close 7935 (time 1743496841.34 total 240.45 last 33.36) - open/close 8310 (time 1743496851.37 total 250.47 last 37.41) - open/close 8611 (time 1743496861.39 total 260.50 last 30.02) - open/close 8944 (time 1743496871.43 total 270.54 last 33.17) - open/close 9266 (time 1743496881.45 total 280.56 last 32.13) - open/close 9582 (time 1743496891.45 total 290.56 last 31.59) - open/close 9920 (time 1743496901.47 total 300.58 last 33.73) total: 10000 open/close in 302.96 seconds: 33.01 ops/second - unlinked 0 (time 1743496915 ; total 0 ; last 0) total: 10000 unlinks in 197 seconds: 50.761421 unlinks/second - open/close 399 (time 1743497134.53 total 10.01 last 39.85) - open/close 780 (time 1743497144.53 total 20.01 last 38.10) - open/close 1169 (time 1743497154.55 total 30.04 last 38.81) - open/close 1516 (time 1743497164.56 total 40.04 last 34.69) - open/close 1866 (time 1743497174.56 total 50.04 last 34.99) - open/close 2227 (time 1743497184.57 total 60.05 last 36.07) - open/close 2580 (time 1743497194.58 total 70.07 last 35.24) - open/close 2945 (time 1743497204.59 total 80.07 last 36.48) - open/close 3311 (time 1743497214.60 total 90.09 last 36.55) - open/close 3643 (time 1743497224.63 total 100.12 last 33.11) - open/close 4026 (time 1743497234.65 total 110.13 last 38.25) - open/close 4404 (time 1743497244.67 total 120.15 last 37.71) - open/close 4767 (time 1743497254.68 total 130.16 last 36.27) - open/close 5083 (time 1743497264.68 total 140.16 last 31.58) - open/close 5450 (time 1743497274.70 total 150.18 last 36.63) - open/close 5821 (time 1743497284.71 total 160.19 last 37.08) - open/close 6178 (time 1743497294.71 total 170.19 last 35.70) - open/close 6543 (time 1743497304.75 total 180.24 last 36.32) - open/close 6917 (time 1743497314.78 total 190.27 last 37.29) - open/close 7284 (time 1743497324.81 total 200.29 last 36.61) - open/close 7625 (time 1743497334.81 total 210.29 last 34.09) - open/close 7991 (time 1743497344.82 total 220.30 last 36.58) - open/close 8371 (time 1743497354.84 total 230.32 last 37.92) - open/close 8744 (time 1743497364.85 total 240.33 last 37.25) - open/close 9092 (time 1743497374.87 total 250.35 last 34.74) - open/close 9428 (time 1743497384.87 total 260.35 last 33.59) - open/close 9794 (time 1743497394.88 total 270.36 last 36.58) total: 10000 open/close in 276.05 seconds: 36.23 ops/second - unlinked 0 (time 1743497409 ; total 0 ; last 0) total: 10000 unlinks in 187 seconds: 53.475937 unlinks/second - open/close 399 (time 1743497615.68 total 10.02 last 39.84) - open/close 793 (time 1743497625.70 total 20.04 last 39.31) - open/close 1179 (time 1743497635.71 total 30.05 last 38.57) - open/close 1576 (time 1743497645.73 total 40.06 last 39.64) - open/close 1980 (time 1743497655.76 total 50.09 last 40.26) - open/close 2376 (time 1743497665.80 total 60.13 last 39.46) - open/close 2769 (time 1743497675.81 total 70.14 last 39.26) - open/close 3155 (time 1743497685.82 total 80.15 last 38.56) - open/close 3575 (time 1743497695.82 total 90.15 last 41.98) - open/close 3976 (time 1743497705.83 total 100.17 last 40.05) - open/close 4362 (time 1743497715.84 total 110.17 last 38.58) - open/close 4774 (time 1743497725.85 total 120.18 last 41.14) - open/close 5190 (time 1743497735.86 total 130.20 last 41.56) - open/close 5584 (time 1743497745.89 total 140.22 last 39.31) - open/close 6012 (time 1743497755.90 total 150.24 last 42.72) - open/close 6438 (time 1743497765.92 total 160.25 last 42.53) - open/close 6849 (time 1743497775.95 total 170.28 last 41.00) - open/close 7245 (time 1743497785.99 total 180.33 last 39.41) - open/close 7625 (time 1743497796.01 total 190.34 last 37.94) - open/close 8015 (time 1743497806.02 total 200.35 last 38.95) - open/close 8381 (time 1743497816.04 total 210.37 last 36.54) - open/close 8793 (time 1743497826.05 total 220.38 last 41.17) - open/close 9167 (time 1743497836.05 total 230.38 last 37.39) - open/close 9555 (time 1743497846.07 total 240.40 last 38.72) - open/close 9950 (time 1743497856.07 total 250.40 last 39.49) total: 10000 open/close in 251.98 seconds: 39.69 ops/second - unlinked 0 (time 1743497866 ; total 0 ; last 0) total: 10000 unlinks in 175 seconds: 57.142857 unlinks/second - open/close 401 (time 1743498059.68 total 10.01 last 40.08) - open/close 807 (time 1743498069.70 total 20.02 last 40.53) - open/close 1236 (time 1743498079.71 total 30.03 last 42.87) - open/close 1678 (time 1743498089.71 total 40.04 last 44.18) - open/close 2110 (time 1743498099.72 total 50.05 last 43.15) - open/close 2546 (time 1743498109.74 total 60.06 last 43.55) - open/close 2970 (time 1743498119.74 total 70.06 last 42.39) - open/close 3388 (time 1743498129.77 total 80.09 last 41.69) - open/close 3838 (time 1743498139.78 total 90.10 last 44.94) - open/close 4288 (time 1743498149.78 total 100.10 last 45.00) - open/close 4682 (time 1743498159.80 total 110.12 last 39.33) - open/close 5119 (time 1743498169.89 total 120.21 last 43.30) - open/close 5541 (time 1743498179.90 total 130.22 last 42.17) - open/close 5953 (time 1743498189.93 total 140.25 last 41.06) - open/close 6374 (time 1743498199.93 total 150.25 last 42.10) - open/close 6784 (time 1743498209.93 total 160.26 last 40.99) - open/close 7244 (time 1743498219.96 total 170.28 last 45.87) - open/close 7659 (time 1743498229.97 total 180.30 last 41.45) - open/close 8114 (time 1743498239.97 total 190.30 last 45.50) - open/close 8543 (time 1743498249.99 total 200.31 last 42.85) - open/close 8922 (time 1743498259.99 total 210.31 last 37.89) - open/close 9319 (time 1743498270.02 total 220.34 last 39.57) - open/close 9717 (time 1743498280.02 total 230.35 last 39.79) total: 10000 open/close in 237.16 seconds: 42.17 ops/second - unlinked 0 (time 1743498294 ; total 0 ; last 0) total: 10000 unlinks in 158 seconds: 63.291138 unlinks/second - open/close 452 (time 1743498470.45 total 10.01 last 45.16) - open/close 909 (time 1743498480.46 total 20.02 last 45.65) - open/close 1378 (time 1743498490.48 total 30.03 last 46.83) - open/close 1826 (time 1743498500.49 total 40.05 last 44.74) - open/close 2265 (time 1743498510.50 total 50.06 last 43.85) - open/close 2724 (time 1743498520.52 total 60.08 last 45.79) - open/close 3152 (time 1743498530.53 total 70.09 last 42.77) - open/close 3599 (time 1743498540.54 total 80.10 last 44.66) - open/close 4043 (time 1743498550.55 total 90.11 last 44.34) - open/close 4461 (time 1743498560.57 total 100.12 last 41.74) - open/close 4869 (time 1743498570.58 total 110.13 last 40.76) - open/close 5296 (time 1743498580.60 total 120.15 last 42.61) - open/close 5707 (time 1743498590.61 total 130.17 last 41.04) - open/close 6130 (time 1743498600.64 total 140.20 last 42.17) - open/close 6552 (time 1743498610.65 total 150.20 last 42.19) - open/close 6984 (time 1743498620.68 total 160.24 last 43.06) - open/close 7427 (time 1743498630.68 total 170.24 last 44.28) - open/close 7885 (time 1743498640.70 total 180.26 last 45.72) - open/close 8338 (time 1743498650.70 total 190.26 last 45.30) - open/close 8743 (time 1743498660.71 total 200.26 last 40.48) - open/close 9175 (time 1743498670.72 total 210.28 last 43.13) - open/close 9603 (time 1743498680.74 total 220.29 last 42.74) total: 10000 open/close in 229.65 seconds: 43.54 ops/second - unlinked 0 (time 1743498697 ; total 0 ; last 0) total: 10000 unlinks in 146 seconds: 68.493149 unlinks/second - open/close 478 (time 1743498861.27 total 10.01 last 47.75) - open/close 941 (time 1743498871.29 total 20.04 last 46.18) - open/close 1393 (time 1743498881.30 total 30.04 last 45.18) - open/close 1853 (time 1743498891.30 total 40.05 last 45.98) - open/close 2347 (time 1743498901.31 total 50.06 last 49.36) - open/close 2852 (time 1743498911.32 total 60.06 last 50.47) - open/close 3367 (time 1743498921.32 total 70.07 last 51.47) - open/close 3871 (time 1743498931.32 total 80.07 last 50.40) - open/close 4359 (time 1743498941.34 total 90.08 last 48.72) - open/close 4836 (time 1743498951.34 total 100.09 last 47.67) - open/close 5347 (time 1743498961.35 total 110.09 last 51.08) - open/close 5824 (time 1743498971.36 total 120.10 last 47.65) - open/close 6250 (time 1743498981.38 total 130.13 last 42.51) - open/close 6698 (time 1743498991.39 total 140.14 last 44.75) - open/close 7140 (time 1743499001.41 total 150.16 last 44.11) - open/close 7622 (time 1743499011.42 total 160.17 last 48.13) - open/close 8080 (time 1743499021.44 total 170.18 last 45.75) - open/close 8504 (time 1743499031.44 total 180.19 last 42.36) - open/close 8922 (time 1743499041.46 total 190.21 last 41.72) - open/close 9355 (time 1743499051.47 total 200.21 last 43.29) - open/close 9776 (time 1743499061.48 total 210.23 last 42.03) total: 10000 open/close in 215.43 seconds: 46.42 ops/second - unlinked 0 (time 1743499073 ; total 0 ; last 0) total: 10000 unlinks in 149 seconds: 67.114098 unlinks/second - open/close 513 (time 1743499238.72 total 10.01 last 51.23) - open/close 1000 (time 1743499248.73 total 20.03 last 48.64) - open/close 1569 (time 1743499258.75 total 30.04 last 56.79) - open/close 2128 (time 1743499268.76 total 40.05 last 55.84) - open/close 2683 (time 1743499278.77 total 50.06 last 55.45) - open/close 3247 (time 1743499288.78 total 60.07 last 56.34) - open/close 3774 (time 1743499298.80 total 70.10 last 52.59) - open/close 4264 (time 1743499308.80 total 80.10 last 48.98) - open/close 4774 (time 1743499318.82 total 90.11 last 50.92) - open/close 5266 (time 1743499328.83 total 100.12 last 49.17) - open/close 5792 (time 1743499338.84 total 110.13 last 52.54) - open/close 6312 (time 1743499348.84 total 120.14 last 51.97) - open/close 6818 (time 1743499358.86 total 130.15 last 50.52) - open/close 7319 (time 1743499368.87 total 140.17 last 50.04) - open/close 7899 (time 1743499378.88 total 150.17 last 57.96) - open/close 8431 (time 1743499388.88 total 160.17 last 53.20) - open/close 8998 (time 1743499398.90 total 170.20 last 56.55) - open/close 9605 (time 1743499408.95 total 180.25 last 60.40) total: 10000 open/close in 187.58 seconds: 53.31 ops/second - unlinked 0 (time 1743499421 ; total 0 ; last 0) total: 10000 unlinks in 123 seconds: 81.300812 unlinks/second - open/close 634 (time 1743499560.55 total 10.00 last 63.37) - open/close 1227 (time 1743499570.57 total 20.02 last 59.21) - open/close 1832 (time 1743499580.58 total 30.04 last 60.41) - open/close 2467 (time 1743499590.60 total 40.05 last 63.41) - open/close 3089 (time 1743499600.62 total 50.07 last 62.09) - open/close 3665 (time 1743499610.62 total 60.07 last 57.58) - open/close 4258 (time 1743499620.62 total 70.07 last 59.29) - open/close 4772 (time 1743499630.63 total 80.08 last 51.35) - open/close 5284 (time 1743499640.63 total 90.08 last 51.19) - open/close 5812 (time 1743499650.65 total 100.11 last 52.69) - open/close 6349 (time 1743499660.67 total 110.12 last 53.62) - open/close 6884 (time 1743499670.70 total 120.15 last 53.33) - open/close 7472 (time 1743499680.71 total 130.16 last 58.73) - open/close 8057 (time 1743499690.73 total 140.18 last 58.42) - open/close 8667 (time 1743499700.75 total 150.20 last 60.83) - open/close 9252 (time 1743499710.77 total 160.22 last 58.39) - open/close 9804 (time 1743499720.78 total 170.23 last 55.19) total: 10000 open/close in 173.50 seconds: 57.64 ops/second - unlinked 0 (time 1743499729 ; total 0 ; last 0) total: 10000 unlinks in 117 seconds: 85.470085 unlinks/second - open/close 570 (time 1743499862.49 total 10.01 last 56.97) - open/close 1157 (time 1743499872.50 total 20.01 last 58.68) - open/close 1747 (time 1743499882.51 total 30.02 last 58.94) - open/close 2296 (time 1743499892.52 total 40.03 last 54.83) - open/close 2843 (time 1743499902.52 total 50.03 last 54.70) - open/close 3402 (time 1743499912.52 total 60.04 last 55.89) - open/close 3974 (time 1743499922.52 total 70.04 last 57.19) - open/close 4562 (time 1743499932.53 total 80.05 last 58.74) - open/close 5096 (time 1743499942.54 total 90.06 last 53.35) - open/close 5639 (time 1743499952.56 total 100.07 last 54.24) - open/close 6136 (time 1743499962.56 total 110.08 last 49.67) - open/close 6682 (time 1743499972.57 total 120.08 last 54.55) - open/close 7308 (time 1743499982.58 total 130.10 last 62.51) - open/close 7948 (time 1743499992.63 total 140.15 last 63.70) - open/close 8593 (time 1743500002.64 total 150.15 last 64.46) - open/close 9170 (time 1743500012.65 total 160.16 last 57.63) - open/close 9701 (time 1743500022.67 total 170.19 last 52.98) total: 10000 open/close in 176.46 seconds: 56.67 ops/second - unlinked 0 (time 1743500034 ; total 0 ; last 0) total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg660-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg660-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg660-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.206.160@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.160@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg660-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg660-server: oleg660-server.virtnet: executing set_default_debug -1 all pdsh@oleg660-client: oleg660-server: ssh exited with exit code 1 Started lustre-OST0000 oleg660-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg660-server: oleg660-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg660-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg660-client.virtnet: -o user_xattr,flock 192.168.206.160@tcp:/lustre /mnt/lustre On OST0, 16469 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg660-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg660-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg660-server stop mds service on oleg660-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg660-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg660-server' oleg660-server: oleg660-server.virtnet: executing unload_modules_local oleg660-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg660-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg660-client: oleg660-client: ssh exited with exit code 2 pdsh@oleg660-client: oleg660-server: ssh exited with exit code 2 pdsh@oleg660-client: oleg660-client: ssh exited with exit code 2 pdsh@oleg660-client: oleg660-server: ssh exited with exit code 2 PASS 69 (4352s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 05:38:03 (1743500283) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (5s) Stopping clients: oleg660-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg660-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg660-client: oleg660-server: ssh exited with exit code 2 oleg660-server: oleg660-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg660-server' oleg660-server: oleg660-server.virtnet: executing load_modules_local oleg660-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg660-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg660-server: Loading modules from /home/green/git/lustre-release/lustre oleg660-server: detected 4 online CPUs by sysfs oleg660-server: MODOPTS_LIBCFS= oleg660-server: Force libcfs to create 2 CPU partitions oleg660-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg660-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 5148 sec ========== 05:39:42 (1743500382) === conf-sanity: start cleanup 05:39:45 (1743500385) === === conf-sanity: finish cleanup 05:39:49 (1743500389) ===