== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:53:20 (1786434800) start mds service on oleg222-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 46 (time 1786434889.80 total 10.50 last 4.38) - open/close 265 (time 1786434900.56 total 21.25 last 20.37) - open/close 437 (time 1786434910.56 total 31.25 last 17.20) - open/close 655 (time 1786434920.56 total 41.26 last 21.78) - open/close 900 (time 1786434930.58 total 51.28 last 24.45) - open/close 1016 (time 1786434940.59 total 61.28 last 11.60) - open/close 1226 (time 1786434951.18 total 71.87 last 19.83) - open/close 1522 (time 1786434961.98 total 82.68 last 27.39) - open/close 1671 (time 1786434972.30 total 92.99 last 14.45) - open/close 1805 (time 1786434983.02 total 103.72 last 12.50) - open/close 1932 (time 1786434993.49 total 114.19 last 12.12) - open/close 2045 (time 1786435003.64 total 124.33 last 11.14) - open/close 2273 (time 1786435015.94 total 136.63 last 18.54) - open/close 2498 (time 1786435026.44 total 147.13 last 21.43) - open/close 2758 (time 1786435036.93 total 157.63 last 24.76) - open/close 3006 (time 1786435046.94 total 167.64 last 24.79) - open/close 3093 (time 1786435058.37 total 179.06 last 7.61) - open/close 3284 (time 1786435068.51 total 189.21 last 18.83) - open/close 3396 (time 1786435079.49 total 200.19 last 10.20) - open/close 3523 (time 1786435089.86 total 210.55 last 12.25) - open/close 3639 (time 1786435100.17 total 220.86 last 11.25) - open/close 3803 (time 1786435110.34 total 231.04 last 16.12) - open/close 3871 (time 1786435122.19 total 242.89 last 5.74) - open/close 4059 (time 1786435132.20 total 252.90 last 18.79) - open/close 4265 (time 1786435144.13 total 264.83 last 17.27) - open/close 4468 (time 1786435154.88 total 275.58 last 18.88) - open/close 4672 (time 1786435165.32 total 286.01 last 19.55) - open/close 4942 (time 1786435175.32 total 296.02 last 26.99) - open/close 5104 (time 1786435186.91 total 307.61 last 13.97) - open/close 5326 (time 1786435197.83 total 318.52 last 20.34) - open/close 5508 (time 1786435207.86 total 328.56 last 18.13) - open/close 5628 (time 1786435217.93 total 338.62 last 11.93) - open/close 5717 (time 1786435228.05 total 348.75 last 8.79) - open/close 5782 (time 1786435238.26 total 358.96 last 6.37) - open/close 5917 (time 1786435248.52 total 369.22 last 13.16) - open/close 6096 (time 1786435258.55 total 379.25 last 17.86) - open/close 6308 (time 1786435268.55 total 389.25 last 21.19) - open/close 6431 (time 1786435278.58 total 399.27 last 12.27) - open/close 6611 (time 1786435288.59 total 409.29 last 17.97) - open/close 6796 (time 1786435301.71 total 422.41 last 14.11) - open/close 6970 (time 1786435311.71 total 432.41 last 17.39) - open/close 7229 (time 1786435322.75 total 443.44 last 23.47) - open/close 7398 (time 1786435332.78 total 453.48 last 16.84) - open/close 7602 (time 1786435343.63 total 464.33 last 18.81) - open/close 7851 (time 1786435353.88 total 474.58 last 24.29) - open/close 8090 (time 1786435366.99 total 487.69 last 18.22) - open/close 8201 (time 1786435377.00 total 497.69 last 11.10) - open/close 8386 (time 1786435387.16 total 507.85 last 18.21) - open/close 8539 (time 1786435397.44 total 518.14 last 14.88) - open/close 8673 (time 1786435408.83 total 529.53 last 11.76) - open/close 8793 (time 1786435419.79 total 540.49 last 10.95) - open/close 9019 (time 1786435429.80 total 550.49 last 22.59) - open/close 9197 (time 1786435440.13 total 560.82 last 17.23) - open/close 9444 (time 1786435451.07 total 571.76 last 22.58) - open/close 9655 (time 1786435461.70 total 582.40 last 19.85) - open/close 9906 (time 1786435471.74 total 592.44 last 25.00) - open/close 10000 (time 1786435476.54 total 597.24 last 19.57) - open/close 10207 (time 1786435486.58 total 607.28 last 20.61) - open/close 10363 (time 1786435497.22 total 617.91 last 14.67) - open/close 10595 (time 1786435507.24 total 627.93 last 23.16) - open/close 10843 (time 1786435517.25 total 637.95 last 24.76) - open/close 10963 (time 1786435527.26 total 647.96 last 11.98) - open/close 11047 (time 1786435537.73 total 658.43 last 8.03) - open/close 11233 (time 1786435547.77 total 668.47 last 18.52) - open/close 11489 (time 1786435557.77 total 678.47 last 25.59) - open/close 11677 (time 1786435567.78 total 688.48 last 18.78) - open/close 11818 (time 1786435577.79 total 698.49 last 14.08) - open/close 12042 (time 1786435588.24 total 708.94 last 21.44) - open/close 12233 (time 1786435598.53 total 719.23 last 18.57) - open/close 12414 (time 1786435608.56 total 729.26 last 18.05) - open/close 12598 (time 1786435619.33 total 740.03 last 17.08) - open/close 12754 (time 1786435630.20 total 750.90 last 14.35) - open/close 12999 (time 1786435640.20 total 760.90 last 24.49) - open/close 13146 (time 1786435650.24 total 770.94 last 14.64) - open/close 13341 (time 1786435660.26 total 780.95 last 19.47) - open/close 13573 (time 1786435670.61 total 791.31 last 22.40) - open/close 13730 (time 1786435680.77 total 801.47 last 15.46) - open/close 13796 (time 1786435691.01 total 811.71 last 6.44) - open/close 13919 (time 1786435701.25 total 821.95 last 12.01) - open/close 14152 (time 1786435711.49 total 832.18 last 22.77) - open/close 14397 (time 1786435721.94 total 842.64 last 23.44) - open/close 14628 (time 1786435731.95 total 852.64 last 23.09) - open/close 14731 (time 1786435742.35 total 863.05 last 9.90) - open/close 14896 (time 1786435752.61 total 873.31 last 16.08) - open/close 15157 (time 1786435764.88 total 885.58 last 21.27) - open/close 15396 (time 1786435774.89 total 895.59 last 23.89) - open/close 15549 (time 1786435785.59 total 906.29 last 14.29) - open/close 15672 (time 1786435796.03 total 916.72 last 11.79) - open/close 15772 (time 1786435806.86 total 927.55 last 9.23) - open/close 16024 (time 1786435816.86 total 937.56 last 25.19) - open/close 16214 (time 1786435826.88 total 947.58 last 18.96) - open/close 16403 (time 1786435837.96 total 958.66 last 17.06) - open/close 16523 (time 1786435848.12 total 968.81 last 11.81) - open/close 16714 (time 1786435858.12 total 978.82 last 19.10) - open/close 16925 (time 1786435868.30 total 988.99 last 20.73) - open/close 17197 (time 1786435878.34 total 999.03 last 27.09) - open/close 17403 (time 1786435888.64 total 1009.34 last 20.00) - open/close 18082 (time 1786435898.67 total 1019.37 last 67.70) - open/close 18821 (time 1786435908.68 total 1029.37 last 73.83) - open/close 19477 (time 1786435918.68 total 1039.38 last 65.56) - open/close 20000 (time 1786435926.21 total 1046.91 last 69.46) - open/close 20799 (time 1786435936.21 total 1056.91 last 79.88) - open/close 21486 (time 1786435946.22 total 1066.92 last 68.64) - open/close 22131 (time 1786435956.24 total 1076.94 last 64.40) - open/close 22873 (time 1786435966.24 total 1086.94 last 74.20) - open/close 23621 (time 1786435976.25 total 1096.95 last 74.70) - open/close 24292 (time 1786435986.29 total 1106.99 last 66.84) - open/close 24903 (time 1786435996.30 total 1117.00 last 61.03) - open/close 25504 (time 1786436006.32 total 1127.01 last 60.02) - open/close 26192 (time 1786436016.32 total 1137.02 last 68.74) - open/close 26818 (time 1786436026.33 total 1147.03 last 62.57) - open/close 27404 (time 1786436036.34 total 1157.03 last 58.57) - open/close 28148 (time 1786436046.34 total 1167.04 last 74.39) - open/close 28798 (time 1786436056.34 total 1177.04 last 64.97) - open/close 29374 (time 1786436066.35 total 1187.05 last 57.52) - open/close 30000 (time 1786436076.06 total 1196.76 last 64.47) - open/close 30736 (time 1786436086.07 total 1206.77 last 73.58) - open/close 31556 (time 1786436096.11 total 1216.80 last 81.68) - open/close 32122 (time 1786436106.11 total 1226.81 last 56.58) - open/close 32796 (time 1786436116.11 total 1236.81 last 67.37) - open/close 33613 (time 1786436126.11 total 1246.81 last 81.70) - open/close 34400 (time 1786436136.11 total 1256.81 last 78.70) - open/close 35193 (time 1786436146.12 total 1266.81 last 79.28) - open/close 35965 (time 1786436156.12 total 1276.82 last 77.15) - open/close 36729 (time 1786436166.15 total 1286.85 last 76.20) - open/close 37505 (time 1786436176.15 total 1296.85 last 77.59) - open/close 38224 (time 1786436186.17 total 1306.87 last 71.77) - open/close 38949 (time 1786436196.17 total 1316.87 last 72.46) - open/close 39719 (time 1786436206.18 total 1326.88 last 76.96) - open/close 40000 (time 1786436210.34 total 1331.04 last 67.55) - open/close 40719 (time 1786436220.34 total 1341.04 last 71.89) - open/close 41280 (time 1786436230.35 total 1351.05 last 56.05) - open/close 41834 (time 1786436240.36 total 1361.06 last 55.34) - open/close 42359 (time 1786436250.37 total 1371.07 last 52.45) - open/close 42987 (time 1786436260.37 total 1381.07 last 62.79) - open/close 43647 (time 1786436270.38 total 1391.08 last 65.96) - open/close 44329 (time 1786436280.39 total 1401.09 last 68.09) - open/close 45037 (time 1786436290.45 total 1411.15 last 70.41) - open/close 45651 (time 1786436300.45 total 1421.15 last 61.38) - open/close 46289 (time 1786436310.47 total 1431.16 last 63.72) - open/close 46908 (time 1786436320.48 total 1441.18 last 61.82) - open/close 47555 (time 1786436330.48 total 1451.18 last 64.66) - open/close 48145 (time 1786436340.49 total 1461.18 last 58.99) - open/close 48704 (time 1786436350.50 total 1471.20 last 55.84) - open/close 49380 (time 1786436360.51 total 1481.21 last 67.51) total: 50000 open/close in 1490.79 seconds: 33.54 ops/second - unlinked 0 (time 1786436376 ; total 0 ; last 0) - unlinked 10000 (time 1786436545 ; total 169 ; last 169) - unlinked 20000 (time 1786436716 ; total 340 ; last 171) - unlinked 30000 (time 1786436884 ; total 508 ; last 168) - unlinked 40000 (time 1786437053 ; total 677 ; last 169) total: 50000 unlinks in 836 seconds: 59.808613 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded. pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2