== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:05:32 (1781611532) start mds service on oleg410-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing load_modules_local oleg410-server: Loading modules from /home/green/git/lustre-release/lustre oleg410-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg410-server: detected 4 online CPUs by sysfs oleg410-server: MODOPTS_LIBCFS= oleg410-server: Force libcfs to create 2 CPU partitions oleg410-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg410-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg410-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 333 (time 1781611596.54 total 10.01 last 33.25) - open/close 658 (time 1781611606.54 total 20.01 last 32.50) - open/close 867 (time 1781611616.58 total 30.05 last 20.82) - open/close 1131 (time 1781611626.87 total 40.34 last 25.66) - open/close 1522 (time 1781611637.42 total 50.90 last 37.04) - open/close 1793 (time 1781611647.74 total 61.21 last 26.27) - open/close 2130 (time 1781611657.85 total 71.33 last 33.33) - open/close 2494 (time 1781611667.85 total 81.33 last 36.39) - open/close 2771 (time 1781611678.78 total 92.26 last 25.35) - open/close 2846 (time 1781611688.95 total 102.43 last 7.37) - open/close 3076 (time 1781611700.34 total 113.82 last 20.20) - open/close 3413 (time 1781611710.35 total 123.83 last 33.66) - open/close 3788 (time 1781611720.37 total 133.84 last 37.44) - open/close 4135 (time 1781611731.05 total 144.53 last 32.47) - open/close 4421 (time 1781611741.30 total 154.78 last 27.92) - open/close 4777 (time 1781611751.30 total 164.78 last 35.60) - open/close 4987 (time 1781611761.30 total 174.78 last 20.99) - open/close 5253 (time 1781611771.37 total 184.85 last 26.41) - open/close 5597 (time 1781611781.43 total 194.91 last 34.20) - open/close 5939 (time 1781611792.04 total 205.52 last 32.22) - open/close 6285 (time 1781611802.05 total 215.53 last 34.59) - open/close 6613 (time 1781611812.05 total 225.53 last 32.79) - open/close 6972 (time 1781611822.77 total 236.25 last 33.50) - open/close 7241 (time 1781611834.16 total 247.64 last 23.62) - open/close 7570 (time 1781611844.86 total 258.34 last 30.74) - open/close 7854 (time 1781611855.41 total 268.89 last 26.91) - open/close 8171 (time 1781611865.53 total 279.01 last 31.34) - open/close 8531 (time 1781611875.63 total 289.11 last 35.64) - open/close 8884 (time 1781611886.39 total 299.87 last 32.80) - open/close 9246 (time 1781611897.46 total 310.94 last 32.70) - open/close 9508 (time 1781611908.21 total 321.69 last 24.37) - open/close 9782 (time 1781611918.26 total 331.74 last 27.28) - open/close 10000 (time 1781611925.16 total 338.64 last 31.59) - open/close 10315 (time 1781611935.87 total 349.35 last 29.40) - open/close 10659 (time 1781611946.67 total 360.15 last 31.85) - open/close 10995 (time 1781611956.67 total 370.15 last 33.59) - open/close 11313 (time 1781611967.60 total 381.08 last 29.11) - open/close 11625 (time 1781611978.61 total 392.09 last 28.32) - open/close 11878 (time 1781611989.17 total 402.65 last 23.96) - open/close 12233 (time 1781611999.28 total 412.76 last 35.12) - open/close 12534 (time 1781612009.91 total 423.38 last 28.33) - open/close 12808 (time 1781612020.78 total 434.26 last 25.19) - open/close 13136 (time 1781612031.34 total 444.82 last 31.06) - open/close 13416 (time 1781612041.46 total 454.94 last 27.67) - open/close 13747 (time 1781612052.34 total 465.81 last 30.43) - open/close 14985 (time 1781612062.34 total 475.82 last 123.77) - open/close 16408 (time 1781612072.34 total 485.82 last 142.25) - open/close 17885 (time 1781612082.34 total 495.82 last 147.65) - open/close 19400 (time 1781612092.35 total 505.83 last 151.44) - open/close 20000 (time 1781612096.27 total 509.75 last 153.01) - open/close 21642 (time 1781612106.27 total 519.75 last 164.12) - open/close 23215 (time 1781612116.28 total 529.76 last 157.25) - open/close 24649 (time 1781612126.28 total 539.76 last 143.37) - open/close 25970 (time 1781612136.29 total 549.76 last 132.01) - open/close 27591 (time 1781612146.29 total 559.77 last 162.09) - open/close 29296 (time 1781612156.29 total 569.77 last 170.43) - open/close 30000 (time 1781612159.63 total 573.11 last 210.53) - open/close 31940 (time 1781612169.64 total 583.11 last 193.98) - open/close 33891 (time 1781612179.64 total 593.12 last 195.08) - open/close 35675 (time 1781612189.64 total 603.12 last 178.37) - open/close 37270 (time 1781612199.64 total 613.12 last 159.42) - open/close 38617 (time 1781612209.65 total 623.13 last 134.61) - open/close 40000 (time 1781612219.51 total 632.98 last 140.33) - open/close 41890 (time 1781612229.51 total 642.99 last 188.97) - open/close 43548 (time 1781612239.51 total 652.99 last 165.79) - open/close 45293 (time 1781612249.51 total 662.99 last 174.47) - open/close 46936 (time 1781612259.51 total 672.99 last 164.25) - open/close 48799 (time 1781612269.51 total 682.99 last 186.30) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49729) error: No space left on device total: 49729 open/close in 688.78 seconds: 72.20 ops/second - unlinked 0 (time 1781612277 ; total 0 ; last 0) - unlinked 10000 (time 1781612348 ; total 71 ; last 71) - unlinked 20000 (time 1781612393 ; total 116 ; last 45) - unlinked 30000 (time 1781612440 ; total 163 ; last 47) - unlinked 40000 (time 1781612483 ; total 206 ; last 43) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49729) error: No such file or directory total: 49729 unlinks in 247 seconds: 201.331985 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg410-server unloading modules via unload_modules_local on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing unload_modules_local oleg410-server: modules unloaded. pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2