== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:13:14 (1787789594) start mds service on oleg122-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: detected 4 online CPUs by sysfs oleg122-server: MODOPTS_LIBCFS= oleg122-server: Force libcfs to create 2 CPU partitions oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg122-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg122-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 120 (time 1787789684.18 total 10.01 last 11.99) - open/close 343 (time 1787789694.20 total 20.03 last 22.26) - open/close 560 (time 1787789704.34 total 30.17 last 21.40) - open/close 786 (time 1787789716.44 total 42.26 last 18.69) - open/close 1038 (time 1787789727.27 total 53.09 last 23.27) - open/close 1316 (time 1787789737.28 total 63.11 last 27.75) - open/close 1537 (time 1787789747.77 total 73.60 last 21.07) - open/close 1781 (time 1787789757.80 total 83.63 last 24.32) - open/close 2042 (time 1787789768.86 total 94.69 last 23.60) - open/close 2271 (time 1787789779.16 total 104.99 last 22.23) - open/close 2530 (time 1787789789.17 total 115.00 last 25.88) - open/close 2789 (time 1787789799.18 total 125.01 last 25.88) - open/close 3013 (time 1787789810.19 total 136.02 last 20.34) - open/close 3255 (time 1787789820.20 total 146.02 last 24.18) - open/close 3512 (time 1787789830.20 total 156.03 last 25.69) - open/close 3737 (time 1787789840.21 total 166.04 last 22.48) - open/close 3945 (time 1787789850.22 total 176.05 last 20.77) - open/close 4207 (time 1787789860.23 total 186.06 last 26.17) - open/close 4399 (time 1787789870.26 total 196.08 last 19.16) - open/close 4687 (time 1787789880.26 total 206.09 last 28.79) - open/close 4888 (time 1787789890.29 total 216.12 last 20.04) - open/close 5089 (time 1787789900.30 total 226.13 last 20.08) - open/close 5189 (time 1787789910.31 total 236.13 last 9.99) - open/close 5441 (time 1787789921.11 total 246.94 last 23.31) - open/close 5700 (time 1787789931.14 total 256.96 last 25.84) - open/close 5930 (time 1787789941.65 total 267.48 last 21.88) - open/close 6133 (time 1787789951.65 total 277.48 last 20.29) - open/close 6358 (time 1787789961.68 total 287.51 last 22.44) - open/close 6592 (time 1787789971.69 total 297.52 last 23.38) - open/close 6769 (time 1787789981.73 total 307.55 last 17.64) - open/close 7011 (time 1787789992.08 total 317.90 last 23.38) - open/close 7219 (time 1787790002.58 total 328.40 last 19.81) - open/close 7399 (time 1787790012.60 total 338.43 last 17.95) - open/close 7698 (time 1787790022.61 total 348.44 last 29.87) - open/close 7855 (time 1787790032.86 total 358.68 last 15.33) - open/close 7980 (time 1787790043.04 total 368.87 last 12.27) - open/close 8141 (time 1787790053.14 total 378.96 last 15.95) - open/close 8320 (time 1787790063.14 total 388.97 last 17.89) - open/close 8481 (time 1787790073.15 total 398.98 last 16.08) - open/close 8710 (time 1787790084.12 total 409.95 last 20.87) - open/close 8979 (time 1787790094.93 total 420.76 last 24.89) - open/close 9232 (time 1787790105.58 total 431.41 last 23.76) - open/close 9472 (time 1787790115.75 total 441.57 last 23.61) - open/close 9723 (time 1787790125.75 total 451.58 last 25.09) - open/close 9987 (time 1787790135.76 total 461.59 last 26.38) - open/close 10186 (time 1787790145.79 total 471.62 last 19.84) - open/close 10456 (time 1787790156.19 total 482.02 last 25.96) - open/close 10702 (time 1787790166.93 total 492.76 last 22.90) - open/close 10948 (time 1787790176.93 total 502.76 last 24.60) - open/close 11158 (time 1787790187.74 total 513.56 last 19.44) - open/close 11350 (time 1787790198.64 total 524.47 last 17.61) - open/close 11530 (time 1787790209.32 total 535.15 last 16.85) - open/close 11745 (time 1787790219.49 total 545.32 last 21.14) - open/close 12026 (time 1787790229.50 total 555.33 last 28.07) - open/close 12243 (time 1787790240.03 total 565.85 last 20.62) - open/close 12412 (time 1787790250.03 total 575.86 last 16.89) - open/close 12644 (time 1787790260.12 total 585.95 last 22.99) - open/close 12916 (time 1787790271.06 total 596.89 last 24.87) - open/close 13188 (time 1787790281.37 total 607.19 last 26.40) - open/close 13491 (time 1787790291.41 total 617.23 last 30.18) - open/close 13721 (time 1787790301.60 total 627.42 last 22.57) - open/close 13940 (time 1787790312.29 total 638.12 last 20.47) - open/close 14172 (time 1787790323.11 total 648.94 last 21.45) - open/close 14336 (time 1787790333.60 total 659.43 last 15.63) - open/close 14586 (time 1787790344.45 total 670.27 last 23.05) - open/close 14796 (time 1787790354.46 total 680.29 last 20.97) - open/close 15042 (time 1787790364.47 total 690.30 last 24.57) - open/close 15296 (time 1787790374.87 total 700.70 last 24.44) - open/close 15505 (time 1787790384.87 total 710.70 last 20.89) - open/close 15723 (time 1787790394.99 total 720.81 last 21.55) - open/close 15943 (time 1787790406.04 total 731.87 last 19.90) - open/close 16185 (time 1787790416.35 total 742.18 last 23.48) - open/close 16436 (time 1787790426.86 total 752.68 last 23.89) - open/close 16687 (time 1787790436.86 total 762.69 last 25.09) - open/close 16943 (time 1787790446.88 total 772.71 last 25.56) - open/close 17134 (time 1787790457.22 total 783.05 last 18.47) - open/close 17378 (time 1787790468.26 total 794.09 last 22.10) - open/close 17566 (time 1787790478.71 total 804.53 last 18.00) - open/close 17814 (time 1787790488.71 total 814.54 last 24.80) - open/close 18070 (time 1787790498.72 total 824.55 last 25.56) - open/close 18280 (time 1787790509.68 total 835.51 last 19.17) - open/close 18503 (time 1787790519.70 total 845.52 last 22.26) - open/close 18714 (time 1787790529.78 total 855.61 last 20.93) - open/close 18935 (time 1787790540.06 total 865.89 last 21.49) - open/close 19173 (time 1787790550.57 total 876.39 last 22.66) - open/close 19408 (time 1787790560.89 total 886.71 last 22.77) - open/close 20000 (time 1787790570.31 total 896.14 last 62.78) - open/close 20724 (time 1787790580.33 total 906.16 last 72.28) - open/close 21409 (time 1787790590.34 total 916.16 last 68.46) - open/close 22071 (time 1787790600.35 total 926.17 last 66.14) - open/close 22796 (time 1787790610.35 total 936.18 last 72.44) - open/close 23450 (time 1787790620.36 total 946.19 last 65.37) - open/close 24314 (time 1787790630.36 total 956.19 last 86.38) - open/close 24983 (time 1787790640.37 total 966.19 last 66.87) - open/close 25632 (time 1787790650.37 total 976.19 last 64.89) - open/close 26233 (time 1787790660.38 total 986.20 last 60.05) - open/close 26994 (time 1787790670.39 total 996.21 last 76.01) - open/close 27713 (time 1787790680.39 total 1006.21 last 71.89) - open/close 28487 (time 1787790690.39 total 1016.22 last 77.37) - open/close 29241 (time 1787790700.40 total 1026.23 last 75.33) - open/close 29961 (time 1787790710.42 total 1036.25 last 71.84) - open/close 30684 (time 1787790720.43 total 1046.26 last 72.26) - open/close 31398 (time 1787790730.44 total 1056.27 last 71.32) - open/close 32103 (time 1787790740.44 total 1066.27 last 70.50) - open/close 32801 (time 1787790750.45 total 1076.28 last 69.75) - open/close 33578 (time 1787790760.46 total 1086.29 last 77.60) - open/close 34371 (time 1787790770.46 total 1096.29 last 79.29) - open/close 35119 (time 1787790780.47 total 1106.29 last 74.77) - open/close 35892 (time 1787790790.48 total 1116.30 last 77.23) - open/close 36643 (time 1787790800.57 total 1126.40 last 74.40) - open/close 37343 (time 1787790810.58 total 1136.41 last 69.93) - open/close 38118 (time 1787790820.58 total 1146.41 last 77.48) - open/close 38950 (time 1787790830.59 total 1156.42 last 83.14) - open/close 39816 (time 1787790840.60 total 1166.42 last 86.55) - open/close 40000 (time 1787790843.09 total 1168.92 last 73.83) - open/close 40755 (time 1787790853.10 total 1178.92 last 75.44) - open/close 41571 (time 1787790863.10 total 1188.92 last 81.60) - open/close 42290 (time 1787790873.11 total 1198.93 last 71.82) - open/close 43023 (time 1787790883.11 total 1208.94 last 73.27) - open/close 43669 (time 1787790893.12 total 1218.94 last 64.58) - open/close 44371 (time 1787790903.12 total 1228.95 last 70.14) - open/close 45152 (time 1787790913.13 total 1238.96 last 78.05) - open/close 45982 (time 1787790923.14 total 1248.96 last 82.96) - open/close 46771 (time 1787790933.14 total 1258.97 last 78.86) - open/close 47572 (time 1787790943.14 total 1268.97 last 80.09) - open/close 48269 (time 1787790953.18 total 1279.01 last 69.44) - open/close 48933 (time 1787790963.20 total 1289.03 last 66.26) - open/close 49552 (time 1787790973.20 total 1299.03 last 61.89) total: 50000 open/close in 1304.28 seconds: 38.34 ops/second - unlinked 0 (time 1787790982 ; total 0 ; last 0) - unlinked 10000 (time 1787791123 ; total 141 ; last 141) - unlinked 20000 (time 1787791265 ; total 283 ; last 142) - unlinked 30000 (time 1787791397 ; total 415 ; last 132) - unlinked 40000 (time 1787791539 ; total 557 ; last 142) total: 50000 unlinks in 694 seconds: 72.046112 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server unloading modules via unload_modules_local on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing unload_modules_local oleg122-server: modules unloaded. pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2