== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:12:20 (1782483140) start mds service on oleg448-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing load_modules_local oleg448-server: Loading modules from /home/green/git/lustre-release/lustre oleg448-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg448-server: detected 4 online CPUs by sysfs oleg448-server: MODOPTS_LIBCFS= oleg448-server: Force libcfs to create 2 CPU partitions oleg448-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg448-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg448-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg448-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 97 (time 1782483207.92 total 10.56 last 9.19) - open/close 197 (time 1782483218.54 total 21.17 last 9.42) - open/close 342 (time 1782483228.65 total 31.29 last 14.33) - open/close 487 (time 1782483239.53 total 42.16 last 13.33) - open/close 596 (time 1782483250.16 total 52.79 last 10.26) - open/close 778 (time 1782483260.58 total 63.22 last 17.45) - open/close 884 (time 1782483271.48 total 74.11 last 9.73) - open/close 991 (time 1782483282.22 total 84.86 last 9.96) - open/close 1051 (time 1782483292.22 total 94.86 last 6.00) - open/close 1112 (time 1782483302.37 total 105.01 last 6.01) - open/close 1273 (time 1782483312.38 total 115.02 last 16.09) - open/close 1377 (time 1782483322.99 total 125.62 last 9.81) - open/close 1488 (time 1782483333.75 total 136.38 last 10.31) - open/close 1588 (time 1782483344.17 total 146.81 last 9.59) - open/close 1765 (time 1782483354.35 total 156.99 last 17.39) - open/close 1965 (time 1782483364.46 total 167.09 last 19.79) - open/close 2186 (time 1782483375.02 total 177.66 last 20.91) - open/close 2473 (time 1782483385.03 total 187.67 last 28.68) - open/close 2697 (time 1782483395.70 total 198.34 last 20.99) - open/close 2854 (time 1782483405.70 total 208.34 last 15.70) - open/close 3151 (time 1782483415.70 total 218.34 last 29.69) - open/close 3440 (time 1782483426.35 total 228.98 last 27.15) - open/close 3757 (time 1782483436.96 total 239.60 last 29.86) - open/close 3952 (time 1782483447.40 total 250.03 last 18.69) - open/close 4275 (time 1782483458.09 total 260.73 last 30.20) - open/close 4601 (time 1782483469.10 total 271.73 last 29.62) - open/close 4912 (time 1782483479.11 total 281.75 last 31.05) - open/close 5189 (time 1782483490.02 total 292.66 last 25.39) - open/close 5490 (time 1782483500.02 total 302.66 last 30.09) - open/close 5766 (time 1782483510.95 total 313.59 last 25.25) - open/close 5947 (time 1782483521.90 total 324.54 last 16.53) - open/close 6267 (time 1782483532.20 total 334.84 last 31.06) - open/close 6598 (time 1782483543.14 total 345.78 last 30.26) - open/close 6813 (time 1782483553.51 total 356.15 last 20.73) - open/close 7089 (time 1782483564.27 total 366.90 last 25.66) - open/close 7233 (time 1782483574.89 total 377.52 last 13.56) - open/close 7471 (time 1782483585.89 total 388.53 last 21.63) - open/close 7816 (time 1782483596.13 total 398.77 last 33.69) - open/close 8142 (time 1782483606.13 total 408.77 last 32.60) - open/close 8458 (time 1782483616.13 total 418.77 last 31.60) - open/close 8737 (time 1782483626.14 total 428.77 last 27.88) - open/close 9036 (time 1782483636.14 total 438.78 last 29.89) - open/close 9313 (time 1782483646.76 total 449.39 last 26.09) - open/close 9545 (time 1782483657.77 total 460.40 last 21.07) - open/close 9845 (time 1782483668.40 total 471.03 last 28.23) - open/close 10000 (time 1782483671.55 total 474.19 last 49.10) - open/close 10223 (time 1782483682.60 total 485.24 last 20.18) - open/close 10584 (time 1782483692.61 total 495.25 last 36.06) - open/close 12081 (time 1782483702.61 total 505.25 last 149.67) - open/close 13546 (time 1782483712.61 total 515.25 last 146.49) - open/close 14987 (time 1782483722.62 total 525.26 last 144.03) - open/close 16520 (time 1782483732.62 total 535.26 last 153.23) - open/close 18061 (time 1782483742.63 total 545.26 last 154.08) - open/close 19700 (time 1782483752.63 total 555.27 last 163.85) - open/close 21476 (time 1782483762.63 total 565.27 last 177.52) - open/close 23144 (time 1782483772.64 total 575.27 last 166.74) - open/close 25065 (time 1782483782.65 total 585.28 last 191.93) - open/close 26800 (time 1782483792.65 total 595.29 last 173.42) - open/close 28304 (time 1782483802.65 total 605.29 last 150.34) - open/close 29710 (time 1782483812.67 total 615.30 last 140.42) - open/close 31130 (time 1782483822.67 total 625.31 last 141.95) - open/close 32458 (time 1782483832.68 total 635.31 last 132.73) - open/close 33803 (time 1782483842.68 total 645.31 last 134.46) - open/close 35256 (time 1782483852.68 total 655.32 last 145.28) - open/close 36730 (time 1782483862.68 total 665.32 last 147.37) - open/close 38229 (time 1782483872.68 total 675.32 last 149.86) - open/close 39824 (time 1782483882.69 total 685.33 last 159.40) - open/close 41307 (time 1782483892.69 total 695.33 last 148.27) - open/close 42541 (time 1782483902.70 total 705.34 last 123.30) - open/close 44097 (time 1782483912.70 total 715.34 last 155.58) - open/close 45541 (time 1782483922.71 total 725.35 last 144.29) - open/close 46919 (time 1782483932.72 total 735.35 last 137.70) - open/close 48286 (time 1782483942.72 total 745.36 last 136.66) - open/close 49660 (time 1782483952.72 total 755.36 last 137.37) total: 50000 open/close in 757.97 seconds: 65.97 ops/second - unlinked 0 (time 1782483957 ; total 0 ; last 0) - unlinked 10000 (time 1782484004 ; total 47 ; last 47) - unlinked 20000 (time 1782484040 ; total 83 ; last 36) - unlinked 30000 (time 1782484070 ; total 113 ; last 30) - unlinked 40000 (time 1782484092 ; total 135 ; last 22) total: 50000 unlinks in 152 seconds: 328.947357 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg448-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server unloading modules via unload_modules_local on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing unload_modules_local oleg448-server: modules unloaded. pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2