== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:20:17 (1787836817) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg339-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 195 (time 1787836930.64 total 10.18 last 19.15) - open/close 423 (time 1787836942.01 total 21.56 last 20.05) - open/close 652 (time 1787836952.04 total 31.58 last 22.84) - open/close 836 (time 1787836963.02 total 42.57 last 16.75) - open/close 1059 (time 1787836973.51 total 53.05 last 21.26) - open/close 1203 (time 1787836983.55 total 63.10 last 14.34) - open/close 1429 (time 1787836994.42 total 73.96 last 20.80) - open/close 1626 (time 1787837004.45 total 84.00 last 19.64) - open/close 1750 (time 1787837014.45 total 94.00 last 12.39) - open/close 1958 (time 1787837025.06 total 104.60 last 19.62) - open/close 2133 (time 1787837035.46 total 115.01 last 16.82) - open/close 2321 (time 1787837045.47 total 125.01 last 18.79) - open/close 2434 (time 1787837055.93 total 135.48 last 10.80) - open/close 2602 (time 1787837066.50 total 146.04 last 15.90) - open/close 2783 (time 1787837076.51 total 156.05 last 18.08) - open/close 2977 (time 1787837086.66 total 166.21 last 19.10) - open/close 3200 (time 1787837096.68 total 176.22 last 22.26) - open/close 3283 (time 1787837106.68 total 186.23 last 8.30) - open/close 3491 (time 1787837116.70 total 196.24 last 20.77) - open/close 3709 (time 1787837126.70 total 206.25 last 21.78) - open/close 3971 (time 1787837136.72 total 216.27 last 26.16) - open/close 4139 (time 1787837146.88 total 226.43 last 16.53) - open/close 4333 (time 1787837156.91 total 236.45 last 19.36) - open/close 4453 (time 1787837167.63 total 247.18 last 11.19) - open/close 4722 (time 1787837178.49 total 258.04 last 24.77) - open/close 4855 (time 1787837189.31 total 268.86 last 12.30) - open/close 5062 (time 1787837199.71 total 279.26 last 19.90) - open/close 5251 (time 1787837209.75 total 289.30 last 18.83) - open/close 5441 (time 1787837222.15 total 301.69 last 15.33) - open/close 5649 (time 1787837232.15 total 311.70 last 20.79) - open/close 5892 (time 1787837242.16 total 321.71 last 24.28) - open/close 6004 (time 1787837252.42 total 331.96 last 10.92) - open/close 6232 (time 1787837262.42 total 341.97 last 22.80) - open/close 6463 (time 1787837273.35 total 352.90 last 21.14) - open/close 6671 (time 1787837283.35 total 362.90 last 20.80) - open/close 6792 (time 1787837293.65 total 373.20 last 11.75) - open/close 6956 (time 1787837304.34 total 383.89 last 15.34) - open/close 7107 (time 1787837314.84 total 394.39 last 14.38) - open/close 7356 (time 1787837325.77 total 405.32 last 22.78) - open/close 7515 (time 1787837335.94 total 415.49 last 15.64) - open/close 7664 (time 1787837345.96 total 425.50 last 14.88) - open/close 7851 (time 1787837355.96 total 435.51 last 18.70) - open/close 8042 (time 1787837366.92 total 446.47 last 17.43) - open/close 8167 (time 1787837378.20 total 457.75 last 11.08) - open/close 8391 (time 1787837388.22 total 467.77 last 22.36) - open/close 8581 (time 1787837398.22 total 477.77 last 18.99) - open/close 8821 (time 1787837408.86 total 488.40 last 22.57) - open/close 9046 (time 1787837419.44 total 498.98 last 21.27) - open/close 9359 (time 1787837429.44 total 508.99 last 31.27) - open/close 9597 (time 1787837439.85 total 519.39 last 22.88) - open/close 9833 (time 1787837450.63 total 530.18 last 21.88) - open/close 10000 (time 1787837458.97 total 538.51 last 20.04) - open/close 10202 (time 1787837468.98 total 548.53 last 20.17) - open/close 10437 (time 1787837479.51 total 559.05 last 22.32) - open/close 10706 (time 1787837490.37 total 569.92 last 24.77) - open/close 10981 (time 1787837501.19 total 580.74 last 25.40) - open/close 11179 (time 1787837512.84 total 592.39 last 16.99) - open/close 11340 (time 1787837522.86 total 602.40 last 16.08) - open/close 11460 (time 1787837532.87 total 612.42 last 11.98) - open/close 11643 (time 1787837543.19 total 622.74 last 17.73) - open/close 11775 (time 1787837553.61 total 633.16 last 12.67) - open/close 12015 (time 1787837564.24 total 643.79 last 22.58) - open/close 12261 (time 1787837574.27 total 653.82 last 24.52) - open/close 12430 (time 1787837584.64 total 664.19 last 16.29) - open/close 12665 (time 1787837594.67 total 674.21 last 23.44) - open/close 12873 (time 1787837606.72 total 686.27 last 17.25) - open/close 13015 (time 1787837617.53 total 697.08 last 13.14) - open/close 13159 (time 1787837627.77 total 707.32 last 14.06) - open/close 13319 (time 1787837638.72 total 718.27 last 14.61) - open/close 13464 (time 1787837648.74 total 728.29 last 14.47) - open/close 13552 (time 1787837658.75 total 738.30 last 8.79) - open/close 13727 (time 1787837668.75 total 748.30 last 17.49) - open/close 13882 (time 1787837679.78 total 759.33 last 14.05) - open/close 14080 (time 1787837691.90 total 771.45 last 16.34) - open/close 14333 (time 1787837702.53 total 782.07 last 23.82) - open/close 14518 (time 1787837713.41 total 792.96 last 16.99) - open/close 14764 (time 1787837725.13 total 804.68 last 20.99) - open/close 14989 (time 1787837735.44 total 814.99 last 21.83) - open/close 15145 (time 1787837745.56 total 825.10 last 15.43) - open/close 15408 (time 1787837756.36 total 835.91 last 24.33) - open/close 15639 (time 1787837766.66 total 846.21 last 22.43) - open/close 15912 (time 1787837776.67 total 856.22 last 27.28) - open/close 16079 (time 1787837790.47 total 870.02 last 12.10) - open/close 16290 (time 1787837800.58 total 880.13 last 20.87) - open/close 16562 (time 1787837810.59 total 890.13 last 27.17) - open/close 16804 (time 1787837821.49 total 901.04 last 22.19) - open/close 17054 (time 1787837831.55 total 911.10 last 24.86) - open/close 17320 (time 1787837842.27 total 921.82 last 24.82) - open/close 17576 (time 1787837852.27 total 931.82 last 25.59) - open/close 17776 (time 1787837862.42 total 941.97 last 19.70) - open/close 17982 (time 1787837874.82 total 954.37 last 16.62) - open/close 18154 (time 1787837884.84 total 964.39 last 17.16) - open/close 18333 (time 1787837895.19 total 974.74 last 17.29) - open/close 18410 (time 1787837905.28 total 984.82 last 7.64) - open/close 18561 (time 1787837915.91 total 995.46 last 14.20) - open/close 18698 (time 1787837925.92 total 1005.47 last 13.69) - open/close 19384 (time 1787837935.92 total 1015.47 last 68.59) - open/close 20000 (time 1787837944.17 total 1023.71 last 74.73) - open/close 20721 (time 1787837954.17 total 1033.72 last 72.09) - open/close 21474 (time 1787837964.17 total 1043.72 last 75.29) - open/close 22161 (time 1787837974.18 total 1053.73 last 68.63) - open/close 22878 (time 1787837984.19 total 1063.74 last 71.64) - open/close 23627 (time 1787837994.20 total 1073.75 last 74.77) - open/close 24321 (time 1787838004.22 total 1083.77 last 69.31) - open/close 25068 (time 1787838014.25 total 1093.79 last 74.48) - open/close 25769 (time 1787838024.27 total 1103.82 last 69.94) - open/close 26513 (time 1787838034.28 total 1113.83 last 74.32) - open/close 27237 (time 1787838044.28 total 1123.83 last 72.37) - open/close 27999 (time 1787838054.29 total 1133.84 last 76.17) - open/close 28668 (time 1787838064.29 total 1143.84 last 66.88) - open/close 29417 (time 1787838074.30 total 1153.85 last 74.84) - open/close 30000 (time 1787838082.60 total 1162.15 last 70.20) - open/close 30699 (time 1787838092.62 total 1172.17 last 69.80) - open/close 31378 (time 1787838102.62 total 1182.17 last 67.87) - open/close 32055 (time 1787838112.63 total 1192.17 last 67.68) - open/close 32767 (time 1787838122.63 total 1202.18 last 71.17) - open/close 33433 (time 1787838132.65 total 1212.20 last 66.47) - open/close 34193 (time 1787838142.66 total 1222.21 last 75.90) - open/close 34929 (time 1787838152.67 total 1232.22 last 73.54) - open/close 35632 (time 1787838162.70 total 1242.24 last 70.14) - open/close 36344 (time 1787838172.70 total 1252.25 last 71.18) - open/close 37123 (time 1787838182.71 total 1262.25 last 77.83) - open/close 37792 (time 1787838192.71 total 1272.26 last 66.88) - open/close 38478 (time 1787838202.71 total 1282.26 last 68.59) - open/close 39168 (time 1787838212.71 total 1292.26 last 68.99) - open/close 39891 (time 1787838222.72 total 1302.27 last 72.26) - open/close 40583 (time 1787838232.72 total 1312.27 last 69.18) - open/close 41347 (time 1787838242.72 total 1322.27 last 76.38) - open/close 42029 (time 1787838252.73 total 1332.28 last 68.13) - open/close 42644 (time 1787838262.75 total 1342.30 last 61.41) - open/close 43326 (time 1787838272.76 total 1352.30 last 68.14) - open/close 43942 (time 1787838282.76 total 1362.31 last 61.58) - open/close 44657 (time 1787838292.77 total 1372.32 last 71.44) - open/close 45313 (time 1787838302.78 total 1382.33 last 65.53) - open/close 45871 (time 1787838312.80 total 1392.34 last 55.70) - open/close 46452 (time 1787838322.81 total 1402.36 last 58.00) - open/close 47036 (time 1787838332.82 total 1412.37 last 58.36) - open/close 47490 (time 1787838342.82 total 1422.37 last 45.40) - open/close 48184 (time 1787838352.83 total 1432.37 last 69.37) - open/close 48815 (time 1787838362.84 total 1442.38 last 63.04) - open/close 49438 (time 1787838372.84 total 1452.39 last 62.26) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1455.20 seconds: 34.11 ops/second - unlinked 0 (time 1787838381 ; total 0 ; last 0) - unlinked 10000 (time 1787838531 ; total 150 ; last 150) - unlinked 20000 (time 1787838675 ; total 294 ; last 144) - unlinked 30000 (time 1787838826 ; total 445 ; last 151) - unlinked 40000 (time 1787838974 ; total 593 ; last 148) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 728 seconds: 68.174454 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2