== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:12:03 (1787865123) start mds service on oleg654-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg654-server' oleg654-server: oleg654-server.virtnet: executing load_modules_local oleg654-server: Loading modules from /home/green/git/lustre-release/lustre oleg654-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg654-server: detected 4 online CPUs by sysfs oleg654-server: MODOPTS_LIBCFS= oleg654-server: Force libcfs to create 2 CPU partitions oleg654-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg654-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg654-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg654-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0000 oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 240 (time 1787865192.34 total 10.14 last 23.68) - open/close 541 (time 1787865203.08 total 20.87 last 28.04) - open/close 792 (time 1787865213.38 total 31.18 last 24.35) - open/close 1055 (time 1787865224.14 total 41.94 last 24.44) - open/close 1314 (time 1787865234.16 total 51.96 last 25.84) - open/close 1551 (time 1787865244.18 total 61.97 last 23.67) - open/close 1810 (time 1787865255.11 total 72.90 last 23.70) - open/close 2117 (time 1787865266.06 total 83.86 last 28.02) - open/close 2309 (time 1787865276.22 total 94.01 last 18.91) - open/close 2578 (time 1787865286.22 total 104.01 last 26.90) - open/close 2778 (time 1787865296.22 total 114.02 last 20.00) - open/close 3026 (time 1787865306.87 total 124.67 last 23.28) - open/close 3254 (time 1787865317.70 total 135.49 last 21.06) - open/close 3493 (time 1787865327.70 total 145.50 last 23.89) - open/close 3664 (time 1787865337.71 total 155.50 last 17.09) - open/close 3921 (time 1787865347.72 total 165.52 last 25.66) - open/close 4141 (time 1787865358.29 total 176.09 last 20.82) - open/close 4307 (time 1787865371.79 total 189.59 last 12.30) - open/close 4563 (time 1787865382.34 total 200.13 last 24.28) - open/close 4850 (time 1787865392.35 total 210.14 last 28.67) - open/close 5063 (time 1787865402.35 total 220.14 last 21.30) - open/close 5301 (time 1787865412.36 total 230.16 last 23.77) - open/close 5538 (time 1787865422.54 total 240.33 last 23.29) - open/close 5852 (time 1787865435.21 total 253.01 last 24.78) - open/close 6066 (time 1787865445.37 total 263.17 last 21.07) - open/close 6346 (time 1787865456.07 total 273.87 last 26.16) - open/close 6598 (time 1787865466.37 total 284.16 last 24.48) - open/close 6866 (time 1787865476.74 total 294.53 last 25.85) - open/close 7049 (time 1787865486.74 total 304.53 last 18.29) - open/close 7303 (time 1787865496.74 total 314.54 last 25.39) - open/close 7553 (time 1787865506.94 total 324.74 last 24.51) - open/close 7791 (time 1787865517.02 total 334.81 last 23.62) - open/close 7948 (time 1787865527.06 total 344.85 last 15.64) - open/close 8184 (time 1787865537.17 total 354.96 last 23.35) - open/close 8376 (time 1787865547.27 total 365.07 last 19.00) - open/close 8613 (time 1787865558.27 total 376.06 last 21.56) - open/close 8916 (time 1787865568.27 total 386.06 last 30.29) - open/close 9164 (time 1787865579.27 total 397.06 last 22.55) - open/close 9405 (time 1787865589.31 total 407.10 last 24.01) - open/close 9643 (time 1787865599.31 total 417.11 last 23.79) - open/close 9898 (time 1787865609.33 total 427.12 last 25.46) - open/close 10000 (time 1787865612.96 total 430.76 last 28.04) - open/close 10237 (time 1787865623.81 total 441.60 last 21.86) - open/close 10448 (time 1787865633.81 total 451.60 last 21.10) - open/close 10780 (time 1787865643.82 total 461.62 last 33.15) - open/close 10948 (time 1787865653.97 total 471.77 last 16.55) - open/close 11243 (time 1787865666.96 total 484.76 last 22.71) - open/close 11537 (time 1787865677.12 total 494.92 last 28.93) - open/close 11806 (time 1787865687.44 total 505.23 last 26.08) - open/close 12036 (time 1787865697.67 total 515.46 last 22.49) - open/close 12219 (time 1787865707.93 total 525.72 last 17.83) - open/close 12380 (time 1787865718.80 total 536.60 last 14.81) - open/close 12604 (time 1787865728.81 total 546.60 last 22.39) - open/close 12806 (time 1787865739.27 total 557.07 last 19.30) - open/close 12989 (time 1787865752.00 total 569.79 last 14.38) - open/close 13198 (time 1787865762.32 total 580.12 last 20.25) - open/close 13453 (time 1787865772.35 total 590.14 last 25.43) - open/close 13746 (time 1787865783.31 total 601.11 last 26.72) - open/close 13935 (time 1787865793.86 total 611.65 last 17.92) - open/close 14015 (time 1787865804.28 total 622.07 last 7.68) - open/close 14180 (time 1787865814.29 total 632.08 last 16.49) - open/close 14321 (time 1787865824.29 total 642.09 last 14.09) - open/close 14486 (time 1787865835.85 total 653.64 last 14.28) - open/close 14748 (time 1787865845.85 total 663.64 last 26.20) - open/close 14976 (time 1787865858.19 total 675.98 last 18.48) - open/close 15199 (time 1787865868.19 total 685.99 last 22.29) - open/close 15388 (time 1787865878.66 total 696.46 last 18.05) - open/close 15509 (time 1787865890.18 total 707.98 last 10.50) - open/close 15642 (time 1787865900.29 total 718.08 last 13.16) - open/close 15815 (time 1787865910.78 total 728.58 last 16.48) - open/close 15948 (time 1787865921.19 total 738.99 last 12.78) - open/close 16045 (time 1787865931.27 total 749.06 last 9.63) - open/close 16305 (time 1787865941.31 total 759.10 last 25.90) - open/close 16526 (time 1787865951.32 total 769.11 last 22.07) - open/close 16770 (time 1787865962.25 total 780.04 last 22.33) - open/close 16995 (time 1787865972.68 total 790.48 last 21.56) - open/close 17242 (time 1787865983.29 total 801.09 last 23.28) - open/close 17450 (time 1787865993.34 total 811.13 last 20.71) - open/close 17566 (time 1787866003.96 total 821.75 last 10.92) - open/close 17699 (time 1787866014.40 total 832.19 last 12.74) - open/close 17797 (time 1787866026.72 total 844.52 last 7.95) - open/close 17944 (time 1787866036.80 total 854.60 last 14.59) - open/close 18168 (time 1787866046.99 total 864.78 last 21.99) - open/close 18389 (time 1787866056.99 total 874.79 last 22.09) - open/close 18606 (time 1787866067.86 total 885.66 last 19.97) - open/close 18831 (time 1787866077.86 total 895.66 last 22.50) - open/close 19017 (time 1787866087.87 total 905.67 last 18.58) - open/close 19803 (time 1787866097.88 total 915.67 last 78.58) - open/close 20000 (time 1787866100.54 total 918.34 last 73.94) - open/close 20772 (time 1787866110.55 total 928.34 last 77.14) - open/close 21541 (time 1787866120.55 total 938.34 last 76.89) - open/close 22357 (time 1787866130.55 total 948.35 last 81.59) - open/close 23108 (time 1787866140.56 total 958.36 last 75.02) - open/close 23893 (time 1787866150.58 total 968.37 last 78.39) - open/close 24625 (time 1787866160.58 total 978.37 last 73.19) - open/close 25379 (time 1787866170.59 total 988.38 last 75.34) - open/close 26141 (time 1787866180.59 total 998.38 last 76.17) - open/close 26942 (time 1787866190.60 total 1008.39 last 80.03) - open/close 27719 (time 1787866200.60 total 1018.40 last 77.65) - open/close 28576 (time 1787866210.61 total 1028.40 last 85.69) - open/close 29384 (time 1787866220.61 total 1038.41 last 80.76) - open/close 30000 (time 1787866229.19 total 1046.98 last 71.81) - open/close 30780 (time 1787866239.19 total 1056.99 last 77.97) - open/close 31546 (time 1787866249.20 total 1067.00 last 76.51) - open/close 32302 (time 1787866259.21 total 1077.00 last 75.57) - open/close 32960 (time 1787866269.22 total 1087.02 last 65.72) - open/close 33604 (time 1787866279.23 total 1097.03 last 64.33) - open/close 34369 (time 1787866289.23 total 1107.03 last 76.50) - open/close 35175 (time 1787866299.24 total 1117.03 last 80.57) - open/close 35999 (time 1787866309.24 total 1127.03 last 82.38) - open/close 36829 (time 1787866319.25 total 1137.04 last 82.95) - open/close 37615 (time 1787866329.25 total 1147.04 last 78.58) - open/close 38552 (time 1787866339.25 total 1157.04 last 93.70) - open/close 39290 (time 1787866349.25 total 1167.05 last 73.77) - open/close 40000 (time 1787866358.59 total 1176.38 last 76.07) - open/close 40766 (time 1787866368.59 total 1186.39 last 76.54) - open/close 41620 (time 1787866378.61 total 1196.40 last 85.28) - open/close 42467 (time 1787866388.62 total 1206.41 last 84.65) - open/close 43307 (time 1787866398.62 total 1216.42 last 83.93) - open/close 44128 (time 1787866408.64 total 1226.43 last 81.98) - open/close 44921 (time 1787866418.64 total 1236.44 last 79.25) - open/close 45665 (time 1787866428.65 total 1246.45 last 74.34) - open/close 46425 (time 1787866438.66 total 1256.45 last 75.97) - open/close 47184 (time 1787866448.66 total 1266.45 last 75.88) - open/close 47968 (time 1787866458.67 total 1276.47 last 78.27) - open/close 48746 (time 1787866468.68 total 1286.48 last 77.73) - open/close 49459 (time 1787866478.69 total 1296.49 last 71.24) total: 50000 open/close in 1304.19 seconds: 38.34 ops/second - unlinked 0 (time 1787866491 ; total 0 ; last 0) - unlinked 10000 (time 1787866626 ; total 135 ; last 135) - unlinked 20000 (time 1787866759 ; total 268 ; last 133) - unlinked 30000 (time 1787866888 ; total 397 ; last 129) - unlinked 40000 (time 1787867023 ; total 532 ; last 135) total: 50000 unlinks in 661 seconds: 75.642967 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg654-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg654-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg654-server stop mds service on oleg654-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg654-server unloading modules via unload_modules_local on: 'oleg654-server' oleg654-server: oleg654-server.virtnet: executing unload_modules_local oleg654-server: modules unloaded. pdsh@oleg654-client: oleg654-server: ssh exited with exit code 2 pdsh@oleg654-client: oleg654-client: ssh exited with exit code 2 pdsh@oleg654-client: oleg654-client: ssh exited with exit code 2 pdsh@oleg654-client: oleg654-server: ssh exited with exit code 2