== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:09:47 (1782482987) start mds service on oleg240-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing load_modules_local oleg240-server: Loading modules from /home/green/git/lustre-release/lustre oleg240-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg240-server: detected 4 online CPUs by sysfs oleg240-server: MODOPTS_LIBCFS= oleg240-server: Force libcfs to create 2 CPU partitions oleg240-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg240-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg240-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg240-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 255 (time 1782483038.98 total 10.80 last 23.61) - open/close 447 (time 1782483049.03 total 20.85 last 19.10) - open/close 722 (time 1782483059.96 total 31.79 last 25.15) - open/close 1005 (time 1782483070.91 total 42.73 last 25.85) - open/close 1314 (time 1782483081.35 total 53.18 last 29.59) - open/close 1538 (time 1782483091.39 total 63.21 last 22.32) - open/close 1792 (time 1782483101.95 total 73.77 last 24.05) - open/close 2046 (time 1782483112.51 total 84.33 last 24.05) - open/close 2383 (time 1782483122.51 total 94.34 last 33.68) - open/close 2627 (time 1782483133.25 total 105.07 last 22.73) - open/close 2894 (time 1782483143.93 total 115.76 last 24.99) - open/close 3050 (time 1782483154.94 total 126.76 last 14.17) - open/close 3249 (time 1782483164.94 total 136.76 last 19.90) - open/close 3547 (time 1782483175.62 total 147.44 last 27.92) - open/close 3882 (time 1782483185.62 total 157.44 last 33.48) - open/close 4153 (time 1782483196.47 total 168.29 last 24.98) - open/close 4443 (time 1782483207.03 total 178.86 last 27.46) - open/close 4729 (time 1782483217.79 total 189.61 last 26.60) - open/close 4963 (time 1782483228.41 total 200.24 last 22.02) - open/close 5230 (time 1782483238.42 total 210.25 last 26.67) - open/close 5474 (time 1782483249.15 total 220.98 last 22.74) - open/close 5665 (time 1782483259.57 total 231.40 last 18.33) - open/close 5804 (time 1782483270.21 total 242.03 last 13.07) - open/close 6039 (time 1782483281.85 total 253.67 last 20.19) - open/close 6288 (time 1782483292.29 total 264.12 last 23.84) - open/close 6577 (time 1782483303.37 total 275.20 last 26.09) - open/close 6856 (time 1782483314.30 total 286.13 last 25.52) - open/close 7093 (time 1782483326.66 total 298.49 last 19.17) - open/close 7339 (time 1782483337.40 total 309.23 last 22.91) - open/close 7525 (time 1782483347.53 total 319.35 last 18.37) - open/close 7854 (time 1782483357.63 total 329.46 last 32.56) - open/close 8151 (time 1782483370.24 total 342.07 last 23.55) - open/close 8488 (time 1782483380.25 total 352.07 last 33.67) - open/close 8730 (time 1782483392.64 total 364.46 last 19.54) - open/close 9038 (time 1782483402.94 total 374.77 last 29.88) - open/close 9226 (time 1782483413.88 total 385.71 last 17.19) - open/close 9540 (time 1782483424.70 total 396.52 last 29.02) - open/close 9725 (time 1782483434.71 total 406.54 last 18.48) - open/close 9950 (time 1782483444.73 total 416.55 last 22.47) - open/close 10000 (time 1782483447.23 total 419.06 last 19.93) - open/close 10346 (time 1782483457.99 total 429.81 last 32.18) - open/close 10658 (time 1782483468.86 total 440.68 last 28.69) - open/close 10906 (time 1782483479.30 total 451.12 last 23.76) - open/close 11158 (time 1782483489.65 total 461.48 last 24.34) - open/close 11398 (time 1782483499.66 total 471.48 last 23.98) - open/close 11768 (time 1782483509.66 total 481.49 last 36.99) - open/close 13220 (time 1782483519.67 total 491.49 last 145.10) - open/close 14822 (time 1782483529.67 total 501.49 last 160.18) - open/close 16578 (time 1782483539.67 total 511.50 last 175.57) - open/close 18183 (time 1782483549.68 total 521.50 last 160.46) - open/close 19874 (time 1782483559.68 total 531.50 last 169.09) - open/close 21725 (time 1782483569.68 total 541.50 last 185.06) - open/close 23291 (time 1782483579.68 total 551.50 last 156.59) - open/close 24947 (time 1782483589.68 total 561.50 last 165.58) - open/close 26297 (time 1782483599.68 total 571.51 last 134.96) - open/close 27796 (time 1782483609.69 total 581.52 last 149.74) - open/close 29282 (time 1782483619.70 total 591.52 last 148.52) - open/close 30000 (time 1782483625.00 total 596.83 last 135.39) - open/close 31255 (time 1782483635.00 total 606.83 last 125.49) - open/close 32776 (time 1782483645.01 total 616.83 last 152.04) - open/close 34476 (time 1782483655.01 total 626.83 last 169.95) - open/close 36359 (time 1782483665.01 total 636.83 last 188.26) - open/close 38367 (time 1782483675.02 total 646.84 last 200.69) - open/close 40000 (time 1782483682.89 total 654.71 last 207.53) - open/close 42191 (time 1782483692.89 total 664.71 last 219.08) - open/close 44353 (time 1782483702.89 total 674.71 last 216.17) - open/close 46841 (time 1782483712.89 total 684.71 last 248.80) - open/close 49266 (time 1782483722.89 total 694.71 last 242.45) total: 50000 open/close in 698.34 seconds: 71.60 ops/second - unlinked 0 (time 1782483728 ; total 0 ; last 0) - unlinked 10000 (time 1782483758 ; total 30 ; last 30) - unlinked 20000 (time 1782483783 ; total 55 ; last 25) - unlinked 30000 (time 1782483814 ; total 86 ; last 31) - unlinked 40000 (time 1782483843 ; total 115 ; last 29) total: 50000 unlinks in 141 seconds: 354.609924 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg240-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg240-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg240-server stop mds service on oleg240-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg240-server unloading modules via unload_modules_local on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing unload_modules_local oleg240-server: modules unloaded. pdsh@oleg240-client: oleg240-client: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-server: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-client: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-server: ssh exited with exit code 2