== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:59:59 (1783951199) start mds service on oleg129-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing load_modules_local oleg129-server: Loading modules from /home/green/git/lustre-release/lustre oleg129-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg129-server: detected 4 online CPUs by sysfs oleg129-server: MODOPTS_LIBCFS= oleg129-server: Force libcfs to create 2 CPU partitions oleg129-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg129-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg129-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg129-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 217 (time 1783951284.18 total 10.53 last 20.61) - open/close 441 (time 1783951296.33 total 22.68 last 18.44) - open/close 649 (time 1783951306.33 total 32.68 last 20.78) - open/close 832 (time 1783951316.96 total 43.31 last 17.22) - open/close 1015 (time 1783951327.42 total 53.77 last 17.50) - open/close 1271 (time 1783951337.44 total 63.79 last 25.54) - open/close 1442 (time 1783951348.13 total 74.48 last 15.99) - open/close 1658 (time 1783951358.15 total 84.49 last 21.58) - open/close 1893 (time 1783951368.15 total 94.50 last 23.49) - open/close 2158 (time 1783951378.24 total 104.58 last 26.28) - open/close 2290 (time 1783951388.25 total 114.60 last 13.18) - open/close 2483 (time 1783951398.26 total 124.60 last 19.29) - open/close 2702 (time 1783951408.27 total 134.62 last 21.86) - open/close 2935 (time 1783951418.99 total 145.34 last 21.75) - open/close 3102 (time 1783951429.73 total 156.07 last 15.55) - open/close 3264 (time 1783951440.81 total 167.16 last 14.61) - open/close 3423 (time 1783951450.81 total 177.16 last 15.90) - open/close 3460 (time 1783951460.82 total 187.17 last 3.70) - open/close 3697 (time 1783951470.89 total 197.24 last 23.54) - open/close 3915 (time 1783951480.90 total 207.25 last 21.78) - open/close 4176 (time 1783951491.80 total 218.15 last 23.93) - open/close 4399 (time 1783951502.57 total 228.92 last 20.72) - open/close 4644 (time 1783951512.59 total 238.94 last 24.44) - open/close 4912 (time 1783951522.60 total 248.95 last 26.77) - open/close 5165 (time 1783951533.10 total 259.45 last 24.11) - open/close 5338 (time 1783951543.11 total 269.46 last 17.28) - open/close 5531 (time 1783951553.11 total 279.46 last 19.29) - open/close 5728 (time 1783951563.13 total 289.47 last 19.68) - open/close 6015 (time 1783951574.05 total 300.40 last 26.27) - open/close 6237 (time 1783951584.08 total 310.43 last 22.14) - open/close 6458 (time 1783951594.08 total 320.43 last 22.09) - open/close 6685 (time 1783951604.10 total 330.45 last 22.65) - open/close 6899 (time 1783951614.34 total 340.69 last 20.91) - open/close 7142 (time 1783951624.98 total 351.33 last 22.83) - open/close 7354 (time 1783951635.15 total 361.50 last 20.85) - open/close 7564 (time 1783951645.33 total 371.68 last 20.63) - open/close 7726 (time 1783951655.35 total 381.70 last 16.18) - open/close 7849 (time 1783951665.35 total 391.70 last 12.30) - open/close 8048 (time 1783951675.56 total 401.91 last 19.49) - open/close 8224 (time 1783951686.36 total 412.71 last 16.29) - open/close 8379 (time 1783951697.15 total 423.50 last 14.38) - open/close 8468 (time 1783951707.54 total 433.89 last 8.57) - open/close 8662 (time 1783951718.48 total 444.83 last 17.73) - open/close 8889 (time 1783951728.52 total 454.87 last 22.62) - open/close 9120 (time 1783951738.53 total 464.88 last 23.07) - open/close 9371 (time 1783951748.55 total 474.90 last 25.05) - open/close 9517 (time 1783951759.52 total 485.87 last 13.31) - open/close 9745 (time 1783951770.14 total 496.49 last 21.47) - open/close 9970 (time 1783951780.15 total 506.50 last 22.48) - open/close 10231 (time 1783951790.16 total 516.51 last 26.08) - open/close 10450 (time 1783951800.69 total 527.03 last 20.81) - open/close 10650 (time 1783951810.71 total 537.06 last 19.95) - open/close 10876 (time 1783951820.73 total 547.07 last 22.57) - open/close 11048 (time 1783951831.33 total 557.68 last 16.22) - open/close 11329 (time 1783951841.74 total 568.09 last 26.98) - open/close 11486 (time 1783951851.89 total 578.24 last 15.47) - open/close 11729 (time 1783951862.62 total 588.97 last 22.65) - open/close 11873 (time 1783951872.75 total 599.10 last 14.22) - open/close 12066 (time 1783951883.73 total 610.08 last 17.58) - open/close 12239 (time 1783951893.75 total 620.10 last 17.28) - open/close 12453 (time 1783951904.50 total 630.85 last 19.90) - open/close 12682 (time 1783951915.36 total 641.71 last 21.08) - open/close 12862 (time 1783951925.39 total 651.74 last 17.95) - open/close 13036 (time 1783951936.02 total 662.37 last 16.37) - open/close 13163 (time 1783951946.03 total 672.37 last 12.69) - open/close 13388 (time 1783951956.44 total 682.78 last 21.61) - open/close 13589 (time 1783951966.45 total 692.80 last 20.08) - open/close 13739 (time 1783951977.36 total 703.70 last 13.75) - open/close 13948 (time 1783951987.36 total 713.71 last 20.89) - open/close 14149 (time 1783951997.48 total 723.83 last 19.87) - open/close 14405 (time 1783952008.35 total 734.70 last 23.54) - open/close 14640 (time 1783952018.35 total 744.70 last 23.50) - open/close 14836 (time 1783952029.02 total 755.37 last 18.37) - open/close 15074 (time 1783952039.60 total 765.95 last 22.49) - open/close 15305 (time 1783952049.62 total 775.97 last 23.06) - open/close 15555 (time 1783952060.69 total 787.04 last 22.59) - open/close 15729 (time 1783952071.39 total 797.74 last 16.27) - open/close 15967 (time 1783952082.34 total 808.69 last 21.73) - open/close 16206 (time 1783952092.66 total 819.01 last 23.15) - open/close 16439 (time 1783952102.88 total 829.23 last 22.80) - open/close 16660 (time 1783952113.44 total 839.79 last 20.92) - open/close 16874 (time 1783952123.45 total 849.80 last 21.38) - open/close 17068 (time 1783952133.46 total 859.81 last 19.39) - open/close 17281 (time 1783952143.82 total 870.17 last 20.56) - open/close 17463 (time 1783952153.82 total 880.17 last 18.20) - open/close 17588 (time 1783952163.82 total 890.17 last 12.49) - open/close 17819 (time 1783952173.84 total 900.19 last 23.06) - open/close 18044 (time 1783952184.59 total 910.94 last 20.94) - open/close 18172 (time 1783952194.60 total 920.95 last 12.78) - open/close 18417 (time 1783952204.63 total 930.98 last 24.43) - open/close 19091 (time 1783952214.64 total 940.99 last 67.34) - open/close 19585 (time 1783952224.65 total 951.00 last 49.37) - open/close 20000 (time 1783952230.15 total 956.50 last 75.47) - open/close 20684 (time 1783952240.16 total 966.50 last 68.34) - open/close 21363 (time 1783952250.16 total 976.51 last 67.86) - open/close 22149 (time 1783952260.17 total 986.52 last 78.53) - open/close 22878 (time 1783952270.17 total 996.52 last 72.87) - open/close 23595 (time 1783952280.19 total 1006.54 last 71.60) - open/close 24318 (time 1783952290.19 total 1016.54 last 72.25) - open/close 25094 (time 1783952300.20 total 1026.55 last 77.56) - open/close 25872 (time 1783952310.20 total 1036.55 last 77.78) - open/close 26494 (time 1783952320.21 total 1046.55 last 62.18) - open/close 27252 (time 1783952330.22 total 1056.57 last 75.69) - open/close 27954 (time 1783952340.22 total 1066.57 last 70.19) - open/close 28685 (time 1783952350.23 total 1076.58 last 73.04) - open/close 29449 (time 1783952360.24 total 1086.59 last 76.32) - open/close 30000 (time 1783952367.08 total 1093.43 last 80.52) - open/close 30687 (time 1783952377.09 total 1103.44 last 68.67) - open/close 31451 (time 1783952387.09 total 1113.44 last 76.38) - open/close 32183 (time 1783952397.09 total 1123.44 last 73.17) - open/close 32982 (time 1783952407.10 total 1133.45 last 79.86) - open/close 33691 (time 1783952417.11 total 1143.45 last 70.86) - open/close 34513 (time 1783952427.11 total 1153.46 last 82.18) - open/close 35196 (time 1783952437.11 total 1163.46 last 68.28) - open/close 36032 (time 1783952447.11 total 1173.46 last 83.59) - open/close 36802 (time 1783952457.12 total 1183.47 last 76.90) - open/close 37538 (time 1783952467.12 total 1193.47 last 73.60) - open/close 38136 (time 1783952477.13 total 1203.48 last 59.77) - open/close 38833 (time 1783952487.14 total 1213.49 last 69.65) - open/close 39504 (time 1783952497.16 total 1223.50 last 66.98) - open/close 40000 (time 1783952503.64 total 1229.99 last 76.48) - open/close 40809 (time 1783952513.65 total 1240.00 last 80.82) - open/close 41446 (time 1783952523.65 total 1250.00 last 63.70) - open/close 42011 (time 1783952533.67 total 1260.02 last 56.42) - open/close 42618 (time 1783952543.70 total 1270.05 last 60.50) - open/close 43259 (time 1783952553.70 total 1280.05 last 64.09) - open/close 43916 (time 1783952563.70 total 1290.05 last 65.68) - open/close 44677 (time 1783952573.70 total 1300.05 last 76.09) - open/close 45387 (time 1783952583.71 total 1310.06 last 70.97) - open/close 46126 (time 1783952593.71 total 1320.06 last 73.86) - open/close 46875 (time 1783952603.72 total 1330.07 last 74.88) - open/close 47553 (time 1783952613.73 total 1340.07 last 67.74) - open/close 48306 (time 1783952623.73 total 1350.08 last 75.24) - open/close 49007 (time 1783952633.74 total 1360.09 last 70.07) - open/close 49622 (time 1783952643.76 total 1370.11 last 61.39) total: 50000 open/close in 1374.92 seconds: 36.37 ops/second - unlinked 0 (time 1783952653 ; total 0 ; last 0) - unlinked 10000 (time 1783952803 ; total 150 ; last 150) - unlinked 20000 (time 1783952957 ; total 304 ; last 154) - unlinked 30000 (time 1783953104 ; total 451 ; last 147) - unlinked 40000 (time 1783953248 ; total 595 ; last 144) total: 50000 unlinks in 763 seconds: 65.530800 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server unloading modules via unload_modules_local on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing unload_modules_local oleg129-server: modules unloaded. pdsh@oleg129-client: oleg129-client: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-client: ssh exited with exit code 2