== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:46:01 (1783140361) start mds service on oleg320-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing load_modules_local oleg320-server: Loading modules from /home/green/git/lustre-release/lustre oleg320-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg320-server: detected 4 online CPUs by sysfs oleg320-server: MODOPTS_LIBCFS= oleg320-server: Force libcfs to create 2 CPU partitions oleg320-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg320-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg320-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg320-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0000 oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 188 (time 1783140442.69 total 10.00 last 18.79) - open/close 373 (time 1783140453.53 total 20.85 last 17.06) - open/close 570 (time 1783140463.95 total 31.26 last 18.92) - open/close 792 (time 1783140473.97 total 41.28 last 22.15) - open/close 1047 (time 1783140483.98 total 51.29 last 25.47) - open/close 1304 (time 1783140494.28 total 61.59 last 24.95) - open/close 1489 (time 1783140505.16 total 72.47 last 17.01) - open/close 1726 (time 1783140515.28 total 82.59 last 23.41) - open/close 1907 (time 1783140526.61 total 93.92 last 15.98) - open/close 2213 (time 1783140537.22 total 104.54 last 28.82) - open/close 2434 (time 1783140547.66 total 114.97 last 21.18) - open/close 2666 (time 1783140558.03 total 125.34 last 22.36) - open/close 2870 (time 1783140568.08 total 135.39 last 20.30) - open/close 3122 (time 1783140578.95 total 146.26 last 23.19) - open/close 3338 (time 1783140588.96 total 156.27 last 21.59) - open/close 3547 (time 1783140599.86 total 167.17 last 19.17) - open/close 3781 (time 1783140609.87 total 177.18 last 23.38) - open/close 4054 (time 1783140620.84 total 188.15 last 24.88) - open/close 4310 (time 1783140631.56 total 198.87 last 23.88) - open/close 4561 (time 1783140641.56 total 208.87 last 25.09) - open/close 4736 (time 1783140651.57 total 218.88 last 17.49) - open/close 4922 (time 1783140661.77 total 229.08 last 18.22) - open/close 5155 (time 1783140672.72 total 240.03 last 21.29) - open/close 5344 (time 1783140682.74 total 250.05 last 18.86) - open/close 5474 (time 1783140692.74 total 260.05 last 12.99) - open/close 5681 (time 1783140702.75 total 270.06 last 20.69) - open/close 5881 (time 1783140712.92 total 280.23 last 19.67) - open/close 6095 (time 1783140723.34 total 290.66 last 20.52) - open/close 6260 (time 1783140733.96 total 301.28 last 15.54) - open/close 6432 (time 1783140744.58 total 311.89 last 16.21) - open/close 6616 (time 1783140754.59 total 321.91 last 18.37) - open/close 6796 (time 1783140765.63 total 332.95 last 16.30) - open/close 6989 (time 1783140775.87 total 343.18 last 18.85) - open/close 7112 (time 1783140785.88 total 353.19 last 12.29) - open/close 7320 (time 1783140796.35 total 363.67 last 19.85) - open/close 7495 (time 1783140807.17 total 374.48 last 16.18) - open/close 7652 (time 1783140817.74 total 385.05 last 14.85) - open/close 7836 (time 1783140828.68 total 395.99 last 16.83) - open/close 8018 (time 1783140838.68 total 406.00 last 18.19) - open/close 8210 (time 1783140849.67 total 416.98 last 17.48) - open/close 8382 (time 1783140860.62 total 427.93 last 15.72) - open/close 8561 (time 1783140871.62 total 438.93 last 16.27) - open/close 8788 (time 1783140882.13 total 449.44 last 21.60) - open/close 8960 (time 1783140892.94 total 460.26 last 15.90) - open/close 9230 (time 1783140903.75 total 471.06 last 24.99) - open/close 9446 (time 1783140913.99 total 481.30 last 21.09) - open/close 9568 (time 1783140924.05 total 491.36 last 12.13) - open/close 9680 (time 1783140934.85 total 502.16 last 10.37) - open/close 9827 (time 1783140944.92 total 512.23 last 14.60) - open/close 10000 (time 1783140954.57 total 521.88 last 17.93) - open/close 10082 (time 1783140965.12 total 532.43 last 7.77) - open/close 10231 (time 1783140975.94 total 543.25 last 13.78) - open/close 10373 (time 1783140986.95 total 554.27 last 12.89) - open/close 10461 (time 1783140997.19 total 564.50 last 8.60) - open/close 10645 (time 1783141007.75 total 575.06 last 17.44) - open/close 10778 (time 1783141017.75 total 585.07 last 13.29) - open/close 10850 (time 1783141028.03 total 595.35 last 7.00) - open/close 10994 (time 1783141038.04 total 605.35 last 14.40) - open/close 11119 (time 1783141048.85 total 616.16 last 11.56) - open/close 11239 (time 1783141059.84 total 627.15 last 10.91) - open/close 11314 (time 1783141069.95 total 637.27 last 7.42) - open/close 11478 (time 1783141080.26 total 647.57 last 15.92) - open/close 11614 (time 1783141090.26 total 657.57 last 13.59) - open/close 11759 (time 1783141101.32 total 668.63 last 13.12) - open/close 11938 (time 1783141113.61 total 680.92 last 14.56) - open/close 12004 (time 1783141123.63 total 690.94 last 6.59) - open/close 12974 (time 1783141133.64 total 700.95 last 96.91) - open/close 14018 (time 1783141143.64 total 710.95 last 104.38) - open/close 15084 (time 1783141153.67 total 720.98 last 106.24) - open/close 16169 (time 1783141163.67 total 730.98 last 108.48) - open/close 17426 (time 1783141173.67 total 740.99 last 125.69) - open/close 18555 (time 1783141183.68 total 750.99 last 112.84) - open/close 19565 (time 1783141193.68 total 760.99 last 100.97) - open/close 20000 (time 1783141197.93 total 765.24 last 102.37) - open/close 21072 (time 1783141207.94 total 775.25 last 107.12) - open/close 22265 (time 1783141217.94 total 785.25 last 119.28) - open/close 23629 (time 1783141227.94 total 795.25 last 136.38) - open/close 24744 (time 1783141237.94 total 805.26 last 111.48) - open/close 25876 (time 1783141247.95 total 815.26 last 113.14) - open/close 27038 (time 1783141257.96 total 825.27 last 116.07) - open/close 28320 (time 1783141267.96 total 835.28 last 128.16) - open/close 29881 (time 1783141277.97 total 845.28 last 156.03) - open/close 31303 (time 1783141287.97 total 855.28 last 142.19) - open/close 32672 (time 1783141297.97 total 865.28 last 136.84) - open/close 33840 (time 1783141307.98 total 875.29 last 116.69) - open/close 35177 (time 1783141317.99 total 885.30 last 133.66) - open/close 36377 (time 1783141327.99 total 895.30 last 119.93) - open/close 37662 (time 1783141338.00 total 905.31 last 128.45) - open/close 38870 (time 1783141348.00 total 915.31 last 120.75) - open/close 40000 (time 1783141357.85 total 925.16 last 114.72) - open/close 41361 (time 1783141367.85 total 935.16 last 136.05) - open/close 42699 (time 1783141377.86 total 945.18 last 133.66) - open/close 43881 (time 1783141387.87 total 955.18 last 118.19) - open/close 45051 (time 1783141397.87 total 965.18 last 116.97) - open/close 46165 (time 1783141407.88 total 975.19 last 111.30) - open/close 47262 (time 1783141417.88 total 985.19 last 109.64) - open/close 48346 (time 1783141427.88 total 995.19 last 108.38) - open/close 49372 (time 1783141437.89 total 1005.20 last 102.58) total: 50000 open/close in 1011.58 seconds: 49.43 ops/second - unlinked 0 (time 1783141446 ; total 0 ; last 0) - unlinked 10000 (time 1783141619 ; total 173 ; last 173) - unlinked 20000 (time 1783141725 ; total 279 ; last 106) - unlinked 30000 (time 1783141830 ; total 384 ; last 105) - unlinked 40000 (time 1783141923 ; total 477 ; last 93) total: 50000 unlinks in 570 seconds: 87.719299 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg320-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg320-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server unloading modules via unload_modules_local on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing unload_modules_local oleg320-server: modules unloaded. pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2