== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:29:59 (1786814999) start mds service on oleg128-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions oleg128-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg128-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg128-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg128-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-OST0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 186 (time 1786815087.04 total 10.09 last 18.44) - open/close 391 (time 1786815099.27 total 22.32 last 16.76) - open/close 530 (time 1786815109.34 total 32.39 last 13.80) - open/close 614 (time 1786815120.13 total 43.18 last 7.79) - open/close 766 (time 1786815130.15 total 53.20 last 15.17) - open/close 970 (time 1786815140.15 total 63.20 last 20.40) - open/close 1203 (time 1786815150.72 total 73.77 last 22.04) - open/close 1272 (time 1786815161.67 total 84.72 last 6.30) - open/close 1385 (time 1786815171.69 total 94.74 last 11.28) - open/close 1601 (time 1786815181.72 total 104.77 last 21.53) - open/close 1902 (time 1786815192.44 total 115.49 last 28.06) - open/close 2125 (time 1786815202.45 total 125.50 last 22.29) - open/close 2271 (time 1786815212.68 total 135.73 last 14.27) - open/close 2386 (time 1786815223.77 total 146.82 last 10.37) - open/close 2611 (time 1786815233.77 total 156.82 last 22.50) - open/close 2878 (time 1786815243.77 total 166.82 last 26.69) - open/close 3051 (time 1786815253.79 total 176.84 last 17.27) - open/close 3141 (time 1786815263.80 total 186.85 last 8.99) - open/close 3391 (time 1786815273.92 total 196.97 last 24.70) - open/close 3644 (time 1786815284.36 total 207.41 last 24.24) - open/close 3870 (time 1786815294.38 total 217.43 last 22.57) - open/close 4010 (time 1786815304.38 total 227.43 last 13.99) - open/close 4116 (time 1786815314.51 total 237.56 last 10.47) - open/close 4281 (time 1786815327.55 total 250.60 last 12.65) - open/close 4479 (time 1786815337.57 total 260.62 last 19.76) - open/close 4771 (time 1786815347.59 total 270.64 last 29.15) - open/close 4977 (time 1786815357.62 total 280.67 last 20.53) - open/close 5208 (time 1786815368.42 total 291.47 last 21.40) - open/close 5359 (time 1786815378.42 total 301.47 last 15.09) - open/close 5564 (time 1786815388.43 total 311.48 last 20.48) - open/close 5746 (time 1786815398.95 total 322.00 last 17.31) - open/close 5965 (time 1786815408.95 total 332.00 last 21.88) - open/close 6107 (time 1786815418.96 total 342.01 last 14.20) - open/close 6181 (time 1786815429.90 total 352.95 last 6.76) - open/close 6319 (time 1786815439.92 total 362.97 last 13.77) - open/close 6475 (time 1786815450.38 total 373.43 last 14.92) - open/close 6698 (time 1786815460.38 total 383.43 last 22.29) - open/close 6965 (time 1786815470.39 total 393.44 last 26.68) - open/close 7175 (time 1786815480.91 total 403.96 last 19.96) - open/close 7364 (time 1786815494.02 total 417.07 last 14.42) - open/close 7483 (time 1786815504.03 total 427.08 last 11.89) - open/close 7635 (time 1786815514.06 total 437.11 last 15.16) - open/close 7782 (time 1786815524.88 total 447.93 last 13.58) - open/close 7876 (time 1786815535.78 total 458.83 last 8.63) - open/close 8037 (time 1786815545.78 total 468.83 last 16.10) - open/close 8133 (time 1786815556.11 total 479.16 last 9.29) - open/close 8389 (time 1786815567.11 total 490.16 last 23.26) - open/close 8587 (time 1786815577.12 total 500.17 last 19.78) - open/close 8790 (time 1786815587.16 total 510.21 last 20.22) - open/close 9027 (time 1786815597.20 total 520.25 last 23.61) - open/close 9241 (time 1786815608.19 total 531.24 last 19.47) - open/close 9489 (time 1786815620.37 total 543.42 last 20.36) - open/close 9569 (time 1786815630.37 total 553.42 last 8.00) - open/close 9820 (time 1786815640.80 total 563.85 last 24.08) - open/close 10000 (time 1786815650.68 total 573.73 last 18.22) - open/close 10228 (time 1786815660.69 total 583.74 last 22.77) - open/close 10456 (time 1786815671.51 total 594.56 last 21.08) - open/close 10689 (time 1786815681.52 total 604.57 last 23.27) - open/close 10875 (time 1786815691.73 total 614.78 last 18.22) - open/close 11112 (time 1786815701.76 total 624.81 last 23.64) - open/close 11324 (time 1786815711.80 total 634.85 last 21.11) - open/close 11490 (time 1786815722.59 total 645.64 last 15.38) - open/close 11594 (time 1786815735.68 total 658.73 last 7.95) - open/close 11887 (time 1786815745.69 total 668.74 last 29.27) - open/close 12036 (time 1786815756.00 total 679.05 last 14.46) - open/close 12286 (time 1786815766.00 total 689.06 last 24.98) - open/close 12436 (time 1786815776.99 total 700.05 last 13.65) - open/close 12625 (time 1786815787.13 total 710.18 last 18.64) - open/close 12823 (time 1786815797.15 total 720.20 last 19.77) - open/close 13066 (time 1786815807.54 total 730.59 last 23.37) - open/close 13279 (time 1786815817.63 total 740.68 last 21.12) - open/close 13492 (time 1786815828.15 total 751.20 last 20.24) - open/close 13736 (time 1786815838.33 total 761.38 last 23.97) - open/close 13957 (time 1786815849.22 total 772.27 last 20.30) - open/close 14131 (time 1786815859.23 total 782.28 last 17.39) - open/close 14387 (time 1786815869.23 total 792.28 last 25.59) - open/close 14651 (time 1786815879.24 total 802.29 last 26.38) - open/close 14920 (time 1786815889.25 total 812.30 last 26.87) - open/close 15189 (time 1786815899.26 total 822.31 last 26.87) - open/close 15438 (time 1786815910.05 total 833.10 last 23.09) - open/close 15698 (time 1786815920.05 total 843.10 last 25.98) - open/close 15949 (time 1786815930.76 total 853.81 last 23.44) - open/close 16220 (time 1786815941.09 total 864.14 last 26.23) - open/close 16496 (time 1786815951.22 total 874.27 last 27.24) - open/close 16758 (time 1786815961.70 total 884.75 last 25.01) - open/close 17007 (time 1786815972.32 total 895.37 last 23.43) - open/close 17289 (time 1786815983.06 total 906.11 last 26.27) - open/close 17448 (time 1786815993.98 total 917.03 last 14.56) - open/close 17592 (time 1786816004.99 total 928.04 last 13.09) - open/close 17733 (time 1786816015.57 total 938.62 last 13.32) - open/close 17832 (time 1786816025.87 total 948.92 last 9.62) - open/close 18065 (time 1786816036.75 total 959.80 last 21.40) - open/close 18292 (time 1786816047.44 total 970.49 last 21.25) - open/close 18459 (time 1786816058.51 total 981.56 last 15.08) - open/close 18675 (time 1786816068.52 total 991.57 last 21.59) - open/close 19500 (time 1786816078.52 total 1001.57 last 82.47) - open/close 20000 (time 1786816083.82 total 1006.87 last 94.34) - open/close 20823 (time 1786816093.84 total 1016.89 last 82.15) - open/close 21611 (time 1786816103.86 total 1026.91 last 78.69) - open/close 22437 (time 1786816113.86 total 1036.91 last 82.54) - open/close 23251 (time 1786816123.87 total 1046.92 last 81.32) - open/close 24048 (time 1786816133.89 total 1056.94 last 79.58) - open/close 24863 (time 1786816143.89 total 1066.94 last 81.45) - open/close 25682 (time 1786816153.90 total 1076.95 last 81.85) - open/close 26509 (time 1786816163.91 total 1086.96 last 82.65) - open/close 27365 (time 1786816173.92 total 1096.97 last 85.52) - open/close 28164 (time 1786816183.92 total 1106.97 last 79.87) - open/close 28911 (time 1786816193.92 total 1116.97 last 74.69) - open/close 29836 (time 1786816203.93 total 1126.98 last 92.41) - open/close 30619 (time 1786816213.93 total 1136.98 last 78.29) - open/close 31396 (time 1786816223.93 total 1146.98 last 77.69) - open/close 32247 (time 1786816233.95 total 1157.00 last 84.95) - open/close 33087 (time 1786816243.96 total 1167.01 last 83.92) - open/close 33975 (time 1786816253.97 total 1177.02 last 88.72) - open/close 34890 (time 1786816263.99 total 1187.04 last 91.31) - open/close 35688 (time 1786816273.99 total 1197.04 last 79.77) - open/close 36557 (time 1786816284.02 total 1207.07 last 86.69) - open/close 37226 (time 1786816294.02 total 1217.07 last 66.87) - open/close 37963 (time 1786816304.02 total 1227.07 last 73.70) - open/close 38829 (time 1786816314.02 total 1237.07 last 86.59) - open/close 39767 (time 1786816324.03 total 1247.08 last 93.74) - open/close 40000 (time 1786816326.69 total 1249.74 last 87.65) - open/close 40830 (time 1786816336.69 total 1259.74 last 82.99) - open/close 41734 (time 1786816346.70 total 1269.75 last 90.33) - open/close 42625 (time 1786816356.70 total 1279.75 last 89.07) - open/close 43508 (time 1786816366.72 total 1289.77 last 88.14) - open/close 44361 (time 1786816376.73 total 1299.78 last 85.26) - open/close 45248 (time 1786816386.73 total 1309.78 last 88.66) - open/close 46128 (time 1786816396.76 total 1319.81 last 87.69) - open/close 46985 (time 1786816406.84 total 1329.89 last 85.05) - open/close 47770 (time 1786816416.85 total 1339.90 last 78.46) - open/close 48630 (time 1786816426.85 total 1349.90 last 85.95) - open/close 49501 (time 1786816436.85 total 1359.90 last 87.09) total: 50000 open/close in 1366.30 seconds: 36.60 ops/second - unlinked 0 (time 1786816448 ; total 0 ; last 0) - unlinked 10000 (time 1786816572 ; total 124 ; last 124) - unlinked 20000 (time 1786816706 ; total 258 ; last 134) - unlinked 30000 (time 1786816831 ; total 383 ; last 125) - unlinked 40000 (time 1786816962 ; total 514 ; last 131) total: 50000 unlinks in 647 seconds: 77.279755 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg128-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg128-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg128-server stop mds service on oleg128-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg128-server unloading modules via unload_modules_local on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing unload_modules_local oleg128-server: modules unloaded. pdsh@oleg128-client: oleg128-client: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-client: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2