== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:15:52 (1787861752) start mds service on oleg348-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing load_modules_local oleg348-server: Loading modules from /home/green/git/lustre-release/lustre oleg348-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg348-server: detected 4 online CPUs by sysfs oleg348-server: MODOPTS_LIBCFS= oleg348-server: Force libcfs to create 2 CPU partitions oleg348-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg348-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg348-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg348-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg348-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock 192.168.203.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 284 (time 1787861871.33 total 12.98 last 21.88) - open/close 525 (time 1787861881.85 total 23.51 last 22.90) - open/close 678 (time 1787861892.06 total 33.71 last 14.99) - open/close 924 (time 1787861902.07 total 43.72 last 24.58) - open/close 1065 (time 1787861912.07 total 53.72 last 14.10) - open/close 1285 (time 1787861922.59 total 64.25 last 20.90) - open/close 1509 (time 1787861932.66 total 74.32 last 22.24) - open/close 1807 (time 1787861942.90 total 84.56 last 29.10) - open/close 2028 (time 1787861953.00 total 94.65 last 21.89) - open/close 2179 (time 1787861963.02 total 104.67 last 15.07) - open/close 2284 (time 1787861973.04 total 114.70 last 10.48) - open/close 2544 (time 1787861983.04 total 124.70 last 25.99) - open/close 2718 (time 1787861994.02 total 135.68 last 15.85) - open/close 2932 (time 1787862004.64 total 146.30 last 20.15) - open/close 3139 (time 1787862014.95 total 156.61 last 20.08) - open/close 3321 (time 1787862024.97 total 166.63 last 18.16) - open/close 3517 (time 1787862035.96 total 177.61 last 17.85) - open/close 3780 (time 1787862046.71 total 188.36 last 24.47) - open/close 3998 (time 1787862056.71 total 198.37 last 21.79) - open/close 4218 (time 1787862066.71 total 208.37 last 21.99) - open/close 4473 (time 1787862077.19 total 218.84 last 24.35) - open/close 4641 (time 1787862088.63 total 230.28 last 14.69) - open/close 4901 (time 1787862101.23 total 242.89 last 20.62) - open/close 5140 (time 1787862111.64 total 253.30 last 22.96) - open/close 5336 (time 1787862121.67 total 263.33 last 19.55) - open/close 5549 (time 1787862131.67 total 273.33 last 21.29) - open/close 5771 (time 1787862142.07 total 283.73 last 21.34) - open/close 5887 (time 1787862152.74 total 294.40 last 10.88) - open/close 6113 (time 1787862162.75 total 304.41 last 22.57) - open/close 6359 (time 1787862173.17 total 314.82 last 23.62) - open/close 6610 (time 1787862183.52 total 325.17 last 24.24) - open/close 6880 (time 1787862194.22 total 335.88 last 25.23) - open/close 7143 (time 1787862205.10 total 346.76 last 24.17) - open/close 7405 (time 1787862215.41 total 357.06 last 25.43) - open/close 7661 (time 1787862226.09 total 367.75 last 23.96) - open/close 7924 (time 1787862236.58 total 378.23 last 25.08) - open/close 8207 (time 1787862246.60 total 388.26 last 28.22) - open/close 8302 (time 1787862257.14 total 398.80 last 9.02) - open/close 8465 (time 1787862267.95 total 409.61 last 15.07) - open/close 8667 (time 1787862278.17 total 419.83 last 19.77) - open/close 8927 (time 1787862288.18 total 429.83 last 25.99) - open/close 9056 (time 1787862298.19 total 439.84 last 12.89) - open/close 9190 (time 1787862308.73 total 450.39 last 12.70) - open/close 9407 (time 1787862319.48 total 461.13 last 20.20) - open/close 9601 (time 1787862329.48 total 471.14 last 19.39) - open/close 9810 (time 1787862339.49 total 481.15 last 20.88) - open/close 9928 (time 1787862350.43 total 492.09 last 10.79) - open/close 10000 (time 1787862355.18 total 496.84 last 15.17) - open/close 10175 (time 1787862365.19 total 506.85 last 17.48) - open/close 10294 (time 1787862375.33 total 516.99 last 11.73) - open/close 10423 (time 1787862386.35 total 528.01 last 11.71) - open/close 10645 (time 1787862397.22 total 538.87 last 20.43) - open/close 10902 (time 1787862407.22 total 548.88 last 25.69) - open/close 11164 (time 1787862417.22 total 558.88 last 26.20) - open/close 11443 (time 1787862427.74 total 569.40 last 26.52) - open/close 11720 (time 1787862437.76 total 579.42 last 27.64) - open/close 11964 (time 1787862448.21 total 589.87 last 23.35) - open/close 12201 (time 1787862459.83 total 601.48 last 20.41) - open/close 12318 (time 1787862470.84 total 612.49 last 10.63) - open/close 12555 (time 1787862480.84 total 622.49 last 23.69) - open/close 12780 (time 1787862490.84 total 632.50 last 22.49) - open/close 12981 (time 1787862501.87 total 643.53 last 18.23) - open/close 13215 (time 1787862511.88 total 653.53 last 23.39) - open/close 13364 (time 1787862521.89 total 663.55 last 14.87) - open/close 13599 (time 1787862531.91 total 673.57 last 23.46) - open/close 13740 (time 1787862544.18 total 685.84 last 11.49) - open/close 13944 (time 1787862554.19 total 695.85 last 20.37) - open/close 14163 (time 1787862564.52 total 706.17 last 21.21) - open/close 14401 (time 1787862574.53 total 716.18 last 23.77) - open/close 14618 (time 1787862584.53 total 726.18 last 21.70) - open/close 14857 (time 1787862594.75 total 736.40 last 23.39) - open/close 15057 (time 1787862607.66 total 749.31 last 15.50) - open/close 15324 (time 1787862617.66 total 759.31 last 26.69) - open/close 15569 (time 1787862630.24 total 771.90 last 19.47) - open/close 15833 (time 1787862640.26 total 781.91 last 26.37) - open/close 16050 (time 1787862650.26 total 791.91 last 21.70) - open/close 16276 (time 1787862660.57 total 802.22 last 21.92) - open/close 16562 (time 1787862670.63 total 812.28 last 28.43) - open/close 16768 (time 1787862680.69 total 822.34 last 20.48) - open/close 16957 (time 1787862691.42 total 833.08 last 17.61) - open/close 17212 (time 1787862701.85 total 843.51 last 24.45) - open/close 17481 (time 1787862711.86 total 853.52 last 26.88) - open/close 17722 (time 1787862721.88 total 863.54 last 24.05) - open/close 17969 (time 1787862732.60 total 874.26 last 23.04) - open/close 18246 (time 1787862743.65 total 885.31 last 25.07) - open/close 18498 (time 1787862754.34 total 896.00 last 23.58) - open/close 18713 (time 1787862764.34 total 906.00 last 21.50) - open/close 18972 (time 1787862774.36 total 916.01 last 25.86) - open/close 19212 (time 1787862784.80 total 926.45 last 22.99) - open/close 19461 (time 1787862794.81 total 936.46 last 24.87) - open/close 19616 (time 1787862805.72 total 947.38 last 14.20) - open/close 19894 (time 1787862815.73 total 957.38 last 27.79) - open/close 20673 (time 1787862825.75 total 967.40 last 77.75) - open/close 21450 (time 1787862835.75 total 977.40 last 77.68) - open/close 22193 (time 1787862845.76 total 987.41 last 74.24) - open/close 22974 (time 1787862855.76 total 997.41 last 78.09) - open/close 23692 (time 1787862865.77 total 1007.42 last 71.73) - open/close 24440 (time 1787862875.78 total 1017.44 last 74.71) - open/close 25151 (time 1787862885.79 total 1027.44 last 71.05) - open/close 25830 (time 1787862895.80 total 1037.45 last 67.84) - open/close 26601 (time 1787862905.81 total 1047.46 last 77.03) - open/close 27347 (time 1787862915.81 total 1057.46 last 74.58) - open/close 28144 (time 1787862925.82 total 1067.48 last 79.58) - open/close 28873 (time 1787862935.83 total 1077.48 last 72.87) - open/close 29658 (time 1787862945.84 total 1087.49 last 78.40) - open/close 30000 (time 1787862950.10 total 1091.76 last 80.17) - open/close 30740 (time 1787862960.12 total 1101.78 last 73.86) - open/close 31505 (time 1787862970.13 total 1111.79 last 76.43) - open/close 32245 (time 1787862980.15 total 1121.81 last 73.87) - open/close 33035 (time 1787862990.17 total 1131.82 last 78.87) - open/close 33780 (time 1787863000.18 total 1141.83 last 74.44) - open/close 34562 (time 1787863010.24 total 1151.89 last 77.72) - open/close 35335 (time 1787863020.24 total 1161.90 last 77.29) - open/close 36082 (time 1787863030.29 total 1171.95 last 74.33) - open/close 36848 (time 1787863040.31 total 1181.97 last 76.44) - open/close 37589 (time 1787863050.31 total 1191.97 last 74.08) - open/close 38369 (time 1787863060.33 total 1201.98 last 77.88) - open/close 39141 (time 1787863070.33 total 1211.99 last 77.18) - open/close 39907 (time 1787863080.33 total 1221.99 last 76.59) - open/close 40734 (time 1787863090.34 total 1232.00 last 82.65) - open/close 41384 (time 1787863100.36 total 1242.02 last 64.86) - open/close 42076 (time 1787863110.37 total 1252.02 last 69.15) - open/close 42796 (time 1787863120.38 total 1262.03 last 71.95) - open/close 43534 (time 1787863130.39 total 1272.04 last 73.73) - open/close 44235 (time 1787863140.40 total 1282.05 last 70.03) - open/close 44878 (time 1787863150.40 total 1292.06 last 64.25) - open/close 45563 (time 1787863160.41 total 1302.07 last 68.46) - open/close 46320 (time 1787863170.42 total 1312.07 last 75.63) - open/close 47061 (time 1787863180.43 total 1322.09 last 74.00) - open/close 47819 (time 1787863190.44 total 1332.10 last 75.72) - open/close 48512 (time 1787863200.45 total 1342.10 last 69.27) - open/close 49224 (time 1787863210.46 total 1352.11 last 71.14) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1356.99 seconds: 36.57 ops/second - unlinked 0 (time 1787863219 ; total 0 ; last 0) - unlinked 10000 (time 1787863362 ; total 143 ; last 143) - unlinked 20000 (time 1787863513 ; total 294 ; last 151) - unlinked 30000 (time 1787863669 ; total 450 ; last 156) - unlinked 40000 (time 1787863814 ; total 595 ; last 145) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 734 seconds: 67.617165 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg348-server unloading modules via unload_modules_local on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing unload_modules_local oleg348-server: modules unloaded. pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2