== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:54:34 (1780329274) start mds service on oleg239-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing load_modules_local oleg239-server: Loading modules from /home/green/git/lustre-release/lustre oleg239-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg239-server: detected 4 online CPUs by sysfs oleg239-server: MODOPTS_LIBCFS= oleg239-server: Force libcfs to create 2 CPU partitions oleg239-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg239-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg239-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg239-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 156 (time 1780329323.39 total 10.00 last 15.60) - open/close 334 (time 1780329333.39 total 20.00 last 17.79) - open/close 476 (time 1780329344.06 total 30.68 last 13.30) - open/close 812 (time 1780329355.08 total 41.70 last 30.50) - open/close 1145 (time 1780329365.08 total 51.70 last 33.30) - open/close 1413 (time 1780329375.82 total 62.43 last 24.96) - open/close 1719 (time 1780329386.17 total 72.79 last 29.55) - open/close 2009 (time 1780329396.43 total 83.04 last 28.28) - open/close 2346 (time 1780329406.53 total 93.14 last 33.37) - open/close 2600 (time 1780329417.28 total 103.90 last 23.62) - open/close 2768 (time 1780329427.53 total 114.14 last 16.40) - open/close 2845 (time 1780329437.57 total 124.19 last 7.67) - open/close 2985 (time 1780329448.45 total 135.06 last 12.88) - open/close 3265 (time 1780329458.45 total 145.07 last 27.98) - open/close 3416 (time 1780329468.49 total 155.11 last 15.04) - open/close 3740 (time 1780329478.79 total 165.40 last 31.46) - open/close 4069 (time 1780329488.79 total 175.41 last 32.89) - open/close 4279 (time 1780329500.22 total 186.84 last 18.37) - open/close 4460 (time 1780329512.32 total 198.94 last 14.96) - open/close 4590 (time 1780329522.76 total 209.37 last 12.45) - open/close 4683 (time 1780329533.12 total 219.73 last 8.98) - open/close 4805 (time 1780329543.68 total 230.30 last 11.55) - open/close 5170 (time 1780329554.18 total 240.79 last 34.77) - open/close 5590 (time 1780329564.18 total 250.80 last 41.99) - open/close 5894 (time 1780329576.65 total 263.26 last 24.39) - open/close 6219 (time 1780329587.27 total 273.89 last 30.58) - open/close 6513 (time 1780329600.06 total 286.68 last 22.99) - open/close 6817 (time 1780329610.43 total 297.05 last 29.32) - open/close 7148 (time 1780329620.44 total 307.05 last 33.09) - open/close 7395 (time 1780329631.36 total 317.97 last 22.61) - open/close 7680 (time 1780329641.73 total 328.34 last 27.48) - open/close 8016 (time 1780329652.35 total 338.97 last 31.63) - open/close 8182 (time 1780329662.91 total 349.53 last 15.72) - open/close 8443 (time 1780329672.92 total 359.54 last 26.07) - open/close 8714 (time 1780329683.48 total 370.10 last 25.66) - open/close 8854 (time 1780329693.49 total 380.10 last 14.00) - open/close 8995 (time 1780329704.42 total 391.03 last 12.90) - open/close 9137 (time 1780329714.45 total 401.07 last 14.15) - open/close 9311 (time 1780329724.85 total 411.46 last 16.74) - open/close 9412 (time 1780329734.85 total 421.46 last 10.10) - open/close 9541 (time 1780329745.55 total 432.16 last 12.06) - open/close 9602 (time 1780329755.55 total 442.17 last 6.10) - open/close 9706 (time 1780329766.36 total 452.97 last 9.63) - open/close 9849 (time 1780329777.03 total 463.65 last 13.39) - open/close 9957 (time 1780329787.09 total 473.71 last 10.74) - open/close 10000 (time 1780329789.36 total 475.98 last 18.97) - open/close 10163 (time 1780329799.37 total 485.98 last 16.29) - open/close 10373 (time 1780329812.04 total 498.65 last 16.57) - open/close 10567 (time 1780329822.10 total 508.71 last 19.29) - open/close 10768 (time 1780329832.10 total 518.71 last 20.10) - open/close 10970 (time 1780329842.76 total 529.37 last 18.94) - open/close 11207 (time 1780329853.21 total 539.83 last 22.67) - open/close 11351 (time 1780329863.22 total 549.84 last 14.38) - open/close 11489 (time 1780329873.81 total 560.43 last 13.03) - open/close 11687 (time 1780329884.75 total 571.36 last 18.11) - open/close 11841 (time 1780329895.20 total 581.81 last 14.74) - open/close 12043 (time 1780329905.20 total 591.81 last 20.20) - open/close 12280 (time 1780329915.27 total 601.89 last 23.52) - open/close 12543 (time 1780329925.28 total 611.90 last 26.27) - open/close 12771 (time 1780329935.29 total 621.91 last 22.78) - open/close 13008 (time 1780329945.67 total 632.29 last 22.83) - open/close 13255 (time 1780329955.69 total 642.30 last 24.67) - open/close 14092 (time 1780329965.69 total 652.31 last 83.66) - open/close 14875 (time 1780329975.71 total 662.32 last 78.18) - open/close 15587 (time 1780329985.72 total 672.33 last 71.11) - open/close 16262 (time 1780329995.72 total 682.33 last 67.50) - open/close 16883 (time 1780330005.74 total 692.36 last 61.97) - open/close 17463 (time 1780330015.75 total 702.36 last 57.97) - open/close 18218 (time 1780330025.77 total 712.39 last 75.30) - open/close 18707 (time 1780330035.78 total 722.40 last 48.86) - open/close 19256 (time 1780330045.79 total 732.40 last 54.87) - open/close 20000 (time 1780330053.95 total 740.56 last 91.16) - open/close 20588 (time 1780330063.96 total 750.58 last 58.70) - open/close 21092 (time 1780330073.97 total 760.59 last 50.36) - open/close 21562 (time 1780330084.00 total 770.62 last 46.86) - open/close 21947 (time 1780330094.05 total 780.66 last 38.33) - open/close 22452 (time 1780330104.05 total 790.67 last 50.48) - open/close 23001 (time 1780330114.06 total 800.68 last 54.86) - open/close 23378 (time 1780330124.07 total 810.68 last 37.68) - open/close 23904 (time 1780330134.09 total 820.70 last 52.48) - open/close 24357 (time 1780330144.11 total 830.73 last 45.18) - open/close 24966 (time 1780330154.12 total 840.73 last 60.89) - open/close 25604 (time 1780330164.12 total 850.73 last 63.77) - open/close 26298 (time 1780330174.13 total 860.74 last 69.33) - open/close 27052 (time 1780330184.14 total 870.75 last 75.35) - open/close 27910 (time 1780330194.15 total 880.77 last 85.67) - open/close 28748 (time 1780330204.16 total 890.77 last 83.75) - open/close 29539 (time 1780330214.17 total 900.78 last 79.04) - open/close 30000 (time 1780330219.42 total 906.04 last 87.69) - open/close 30878 (time 1780330229.43 total 916.04 last 87.74) - open/close 31715 (time 1780330239.44 total 926.05 last 83.63) - open/close 32560 (time 1780330249.45 total 936.06 last 84.43) - open/close 33499 (time 1780330259.45 total 946.06 last 93.89) - open/close 34390 (time 1780330269.45 total 956.07 last 89.06) - open/close 35292 (time 1780330279.46 total 966.07 last 90.14) - open/close 36200 (time 1780330289.47 total 976.08 last 90.71) - open/close 37077 (time 1780330299.47 total 986.09 last 87.64) - open/close 37897 (time 1780330309.48 total 996.10 last 81.94) - open/close 38703 (time 1780330319.49 total 1006.11 last 80.52) - open/close 39514 (time 1780330329.49 total 1016.11 last 81.08) - open/close 40000 (time 1780330335.42 total 1022.03 last 82.06) - open/close 40852 (time 1780330345.42 total 1032.04 last 85.15) - open/close 41798 (time 1780330355.43 total 1042.05 last 94.50) - open/close 42611 (time 1780330365.43 total 1052.05 last 81.28) - open/close 43539 (time 1780330375.44 total 1062.06 last 92.74) - open/close 44371 (time 1780330385.45 total 1072.06 last 83.13) - open/close 45237 (time 1780330395.45 total 1082.07 last 86.59) - open/close 46175 (time 1780330405.45 total 1092.07 last 93.78) - open/close 47132 (time 1780330415.46 total 1102.07 last 95.64) - open/close 48059 (time 1780330425.47 total 1112.08 last 92.64) - open/close 49156 (time 1780330435.47 total 1122.09 last 109.64) total: 50000 open/close in 1129.92 seconds: 44.25 ops/second - unlinked 0 (time 1780330446 ; total 0 ; last 0) - unlinked 10000 (time 1780330531 ; total 85 ; last 85) - unlinked 20000 (time 1780330677 ; total 231 ; last 146) - unlinked 30000 (time 1780330792 ; total 346 ; last 115) - unlinked 40000 (time 1780330887 ; total 441 ; last 95) total: 50000 unlinks in 532 seconds: 93.984962 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg239-server unloading modules via unload_modules_local on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing unload_modules_local oleg239-server: modules unloaded. pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2