== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:22:59 (1786803779) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 229 (time 1786803861.38 total 10.00 last 22.89) - open/close 512 (time 1786803871.40 total 20.03 last 28.23) - open/close 732 (time 1786803884.04 total 32.66 last 17.41) - open/close 1004 (time 1786803894.06 total 42.68 last 27.14) - open/close 1255 (time 1786803904.39 total 53.01 last 24.31) - open/close 1463 (time 1786803915.41 total 64.03 last 18.87) - open/close 1714 (time 1786803925.99 total 74.61 last 23.73) - open/close 1947 (time 1786803936.52 total 85.14 last 22.13) - open/close 2167 (time 1786803947.14 total 95.76 last 20.71) - open/close 2348 (time 1786803958.86 total 107.48 last 15.45) - open/close 2570 (time 1786803969.18 total 117.80 last 21.51) - open/close 2780 (time 1786803979.18 total 127.81 last 20.99) - open/close 2950 (time 1786803991.89 total 140.52 last 13.38) - open/close 3203 (time 1786804002.94 total 151.56 last 22.91) - open/close 3472 (time 1786804013.06 total 161.68 last 26.58) - open/close 3680 (time 1786804023.69 total 172.31 last 19.56) - open/close 3915 (time 1786804033.71 total 182.33 last 23.45) - open/close 4121 (time 1786804043.73 total 192.35 last 20.57) - open/close 4387 (time 1786804053.73 total 202.35 last 26.60) - open/close 4641 (time 1786804064.01 total 212.63 last 24.71) - open/close 4875 (time 1786804074.84 total 223.46 last 21.60) - open/close 5104 (time 1786804084.87 total 233.49 last 22.82) - open/close 5227 (time 1786804094.87 total 243.50 last 12.30) - open/close 5377 (time 1786804105.61 total 254.23 last 13.97) - open/close 5482 (time 1786804115.94 total 264.56 last 10.17) - open/close 5747 (time 1786804126.65 total 275.28 last 24.73) - open/close 6019 (time 1786804136.67 total 285.29 last 27.16) - open/close 6225 (time 1786804147.40 total 296.03 last 19.19) - open/close 6437 (time 1786804157.91 total 306.53 last 20.19) - open/close 6520 (time 1786804167.94 total 316.56 last 8.27) - open/close 6665 (time 1786804178.57 total 327.19 last 13.64) - open/close 6817 (time 1786804188.58 total 337.21 last 15.18) - open/close 6999 (time 1786804198.80 total 347.42 last 17.82) - open/close 7217 (time 1786804208.80 total 357.43 last 21.79) - open/close 7308 (time 1786804218.81 total 367.44 last 9.09) - open/close 7512 (time 1786804229.11 total 377.73 last 19.81) - open/close 7758 (time 1786804239.11 total 387.74 last 24.59) - open/close 7973 (time 1786804249.13 total 397.75 last 21.47) - open/close 8191 (time 1786804259.13 total 407.76 last 21.79) - open/close 8393 (time 1786804269.14 total 417.76 last 20.20) - open/close 8622 (time 1786804279.31 total 427.94 last 22.50) - open/close 8865 (time 1786804289.49 total 438.11 last 23.88) - open/close 9088 (time 1786804300.23 total 448.85 last 20.76) - open/close 9341 (time 1786804311.00 total 459.62 last 23.49) - open/close 9489 (time 1786804321.02 total 469.64 last 14.77) - open/close 9675 (time 1786804331.02 total 479.64 last 18.59) - open/close 9887 (time 1786804341.02 total 489.65 last 21.19) - open/close 10000 (time 1786804346.37 total 495.00 last 21.12) - open/close 10207 (time 1786804356.38 total 505.01 last 20.68) - open/close 10437 (time 1786804366.40 total 515.03 last 22.95) - open/close 10592 (time 1786804376.59 total 525.21 last 15.22) - open/close 10781 (time 1786804387.56 total 536.18 last 17.23) - open/close 10992 (time 1786804397.57 total 546.19 last 21.08) - open/close 11180 (time 1786804408.39 total 557.01 last 17.38) - open/close 11316 (time 1786804418.39 total 567.01 last 13.59) - open/close 11581 (time 1786804429.04 total 577.67 last 24.87) - open/close 11815 (time 1786804439.22 total 587.85 last 22.99) - open/close 12077 (time 1786804450.24 total 598.86 last 23.79) - open/close 12323 (time 1786804460.84 total 609.46 last 23.21) - open/close 12556 (time 1786804470.85 total 619.47 last 23.28) - open/close 12774 (time 1786804481.10 total 629.72 last 21.27) - open/close 13031 (time 1786804491.97 total 640.60 last 23.63) - open/close 13297 (time 1786804503.06 total 651.68 last 24.00) - open/close 13524 (time 1786804513.34 total 661.97 last 22.07) - open/close 13746 (time 1786804523.35 total 671.97 last 22.18) - open/close 14031 (time 1786804534.21 total 682.83 last 26.25) - open/close 14266 (time 1786804544.86 total 693.48 last 22.06) - open/close 14516 (time 1786804555.40 total 704.02 last 23.73) - open/close 14762 (time 1786804566.27 total 714.90 last 22.62) - open/close 14988 (time 1786804577.22 total 725.85 last 20.64) - open/close 15159 (time 1786804587.23 total 735.86 last 17.08) - open/close 15250 (time 1786804597.55 total 746.17 last 8.82) - open/close 15394 (time 1786804607.63 total 756.25 last 14.29) - open/close 15674 (time 1786804618.71 total 767.33 last 25.27) - open/close 15975 (time 1786804628.88 total 777.50 last 29.61) - open/close 16258 (time 1786804639.24 total 787.87 last 27.30) - open/close 16518 (time 1786804649.54 total 798.16 last 25.25) - open/close 16811 (time 1786804659.55 total 808.17 last 29.28) - open/close 17061 (time 1786804669.57 total 818.19 last 24.95) - open/close 17332 (time 1786804679.57 total 828.19 last 27.09) - open/close 17550 (time 1786804690.30 total 838.92 last 20.32) - open/close 17812 (time 1786804700.45 total 849.07 last 25.82) - open/close 18006 (time 1786804710.78 total 859.41 last 18.78) - open/close 18152 (time 1786804720.78 total 869.41 last 14.60) - open/close 18366 (time 1786804730.80 total 879.42 last 21.37) - open/close 18571 (time 1786804740.82 total 889.45 last 20.46) - open/close 18782 (time 1786804751.80 total 900.43 last 19.21) - open/close 19019 (time 1786804762.37 total 911.00 last 22.42) - open/close 19571 (time 1786804772.38 total 921.01 last 55.14) - open/close 20000 (time 1786804780.63 total 929.25 last 52.05) - open/close 20694 (time 1786804790.63 total 939.26 last 69.35) - open/close 21239 (time 1786804800.64 total 949.26 last 54.48) - open/close 22003 (time 1786804810.64 total 959.26 last 76.38) - open/close 22526 (time 1786804820.65 total 969.27 last 52.25) - open/close 23039 (time 1786804830.66 total 979.28 last 51.27) - open/close 23774 (time 1786804840.66 total 989.28 last 73.46) - open/close 24585 (time 1786804850.67 total 999.29 last 81.02) - open/close 25206 (time 1786804860.68 total 1009.30 last 62.05) - open/close 25809 (time 1786804870.69 total 1019.31 last 60.24) - open/close 26415 (time 1786804880.70 total 1029.32 last 60.53) - open/close 26936 (time 1786804890.70 total 1039.33 last 52.08) - open/close 27616 (time 1786804900.71 total 1049.33 last 67.96) - open/close 28348 (time 1786804910.72 total 1059.34 last 73.12) - open/close 29097 (time 1786804920.74 total 1069.36 last 74.77) - open/close 29789 (time 1786804930.75 total 1079.37 last 69.13) - open/close 30000 (time 1786804933.67 total 1082.29 last 72.18) - open/close 30749 (time 1786804943.68 total 1092.30 last 74.83) - open/close 31514 (time 1786804953.69 total 1102.32 last 76.40) - open/close 32290 (time 1786804963.70 total 1112.33 last 77.52) - open/close 33025 (time 1786804973.71 total 1122.33 last 73.47) - open/close 33726 (time 1786804983.71 total 1132.34 last 70.05) - open/close 34446 (time 1786804993.72 total 1142.34 last 71.95) - open/close 35094 (time 1786805003.72 total 1152.35 last 64.78) - open/close 35790 (time 1786805013.72 total 1162.35 last 69.59) - open/close 36477 (time 1786805023.74 total 1172.36 last 68.63) - open/close 37167 (time 1786805033.74 total 1182.36 last 68.98) - open/close 37871 (time 1786805043.75 total 1192.37 last 70.35) - open/close 38536 (time 1786805053.75 total 1202.37 last 66.49) - open/close 39309 (time 1786805063.75 total 1212.37 last 77.28) - open/close 39960 (time 1786805073.76 total 1222.39 last 65.00) - open/close 40671 (time 1786805083.77 total 1232.39 last 71.09) - open/close 41420 (time 1786805093.77 total 1242.39 last 74.89) - open/close 42152 (time 1786805103.78 total 1252.40 last 73.13) - open/close 42881 (time 1786805113.79 total 1262.42 last 72.79) - open/close 43626 (time 1786805123.80 total 1272.42 last 74.47) - open/close 44387 (time 1786805133.81 total 1282.43 last 76.01) - open/close 45114 (time 1786805143.81 total 1292.43 last 72.68) - open/close 45864 (time 1786805153.82 total 1302.44 last 74.95) - open/close 46545 (time 1786805163.83 total 1312.45 last 68.03) - open/close 47245 (time 1786805173.83 total 1322.45 last 69.97) - open/close 47944 (time 1786805183.84 total 1332.46 last 69.85) - open/close 48690 (time 1786805193.85 total 1342.47 last 74.54) - open/close 49401 (time 1786805203.85 total 1352.47 last 71.07) total: 50000 open/close in 1360.77 seconds: 36.74 ops/second - unlinked 0 (time 1786805216 ; total 0 ; last 0) - unlinked 10000 (time 1786805364 ; total 148 ; last 148) - unlinked 20000 (time 1786805510 ; total 294 ; last 146) - unlinked 30000 (time 1786805650 ; total 434 ; last 140) - unlinked 40000 (time 1786805798 ; total 582 ; last 148) total: 50000 unlinks in 723 seconds: 69.156296 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2