== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:53:30 (1786830810) start mds service on oleg418-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing load_modules_local oleg418-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg418-server: Loading modules from /home/green/git/lustre-release/lustre oleg418-server: detected 4 online CPUs by sysfs oleg418-server: MODOPTS_LIBCFS= oleg418-server: Force libcfs to create 2 CPU partitions oleg418-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg418-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg418-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg418-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock 192.168.204.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 126 (time 1786830900.24 total 10.01 last 12.59) - open/close 383 (time 1786830910.26 total 20.03 last 25.64) - open/close 617 (time 1786830920.26 total 30.03 last 23.40) - open/close 627 (time 1786830930.68 total 40.44 last 0.96) - open/close 759 (time 1786830940.90 total 50.67 last 12.91) - open/close 976 (time 1786830951.74 total 61.50 last 20.03) - open/close 1206 (time 1786830962.34 total 72.11 last 21.69) - open/close 1408 (time 1786830973.03 total 82.79 last 18.90) - open/close 1620 (time 1786830983.78 total 93.55 last 19.72) - open/close 1807 (time 1786830993.79 total 103.55 last 18.69) - open/close 1987 (time 1786831004.72 total 114.49 last 16.46) - open/close 2259 (time 1786831015.28 total 125.05 last 25.76) - open/close 2463 (time 1786831027.83 total 137.59 last 16.26) - open/close 2640 (time 1786831038.71 total 148.47 last 16.27) - open/close 2828 (time 1786831048.71 total 158.47 last 18.80) - open/close 2964 (time 1786831059.17 total 168.93 last 13.00) - open/close 3161 (time 1786831069.42 total 179.18 last 19.22) - open/close 3283 (time 1786831080.31 total 190.08 last 11.20) - open/close 3523 (time 1786831091.31 total 201.07 last 21.83) - open/close 3751 (time 1786831101.32 total 211.08 last 22.78) - open/close 3959 (time 1786831112.26 total 222.02 last 19.01) - open/close 4117 (time 1786831122.52 total 232.29 last 15.39) - open/close 4238 (time 1786831133.29 total 243.05 last 11.24) - open/close 4442 (time 1786831143.81 total 253.58 last 19.38) - open/close 4668 (time 1786831153.82 total 263.59 last 22.58) - open/close 4909 (time 1786831163.83 total 273.59 last 24.10) - open/close 5068 (time 1786831174.26 total 284.03 last 15.23) - open/close 5312 (time 1786831184.49 total 294.26 last 23.85) - open/close 5495 (time 1786831194.50 total 304.27 last 18.28) - open/close 5768 (time 1786831204.52 total 314.29 last 27.25) - open/close 5965 (time 1786831214.55 total 324.31 last 19.65) - open/close 6109 (time 1786831224.87 total 334.63 last 13.95) - open/close 6331 (time 1786831235.14 total 344.90 last 21.62) - open/close 6639 (time 1786831245.16 total 354.93 last 30.74) - open/close 6894 (time 1786831255.92 total 365.68 last 23.70) - open/close 7059 (time 1786831266.15 total 375.92 last 16.12) - open/close 7314 (time 1786831277.00 total 386.76 last 23.51) - open/close 7543 (time 1786831287.41 total 397.18 last 21.99) - open/close 7680 (time 1786831297.92 total 407.69 last 13.03) - open/close 7833 (time 1786831307.94 total 417.71 last 15.27) - open/close 7962 (time 1786831318.06 total 427.83 last 12.75) - open/close 8096 (time 1786831328.75 total 438.51 last 12.54) - open/close 8214 (time 1786831338.77 total 448.53 last 11.78) - open/close 8422 (time 1786831349.35 total 459.12 last 19.65) - open/close 8672 (time 1786831359.36 total 469.12 last 24.99) - open/close 8808 (time 1786831369.92 total 479.69 last 12.87) - open/close 8941 (time 1786831380.59 total 490.35 last 12.47) - open/close 9067 (time 1786831390.95 total 500.72 last 12.16) - open/close 9191 (time 1786831401.08 total 510.84 last 12.25) - open/close 9385 (time 1786831411.09 total 520.85 last 19.38) - open/close 9621 (time 1786831421.74 total 531.51 last 22.15) - open/close 9860 (time 1786831431.76 total 541.53 last 23.85) - open/close 10000 (time 1786831439.66 total 549.42 last 17.72) - open/close 10219 (time 1786831450.21 total 559.97 last 20.76) - open/close 10459 (time 1786831461.11 total 570.88 last 22.01) - open/close 10715 (time 1786831472.17 total 581.93 last 23.16) - open/close 10981 (time 1786831482.17 total 591.94 last 26.58) - open/close 11150 (time 1786831493.99 total 603.76 last 14.30) - open/close 11302 (time 1786831504.55 total 614.31 last 14.40) - open/close 11524 (time 1786831514.59 total 624.36 last 22.10) - open/close 11793 (time 1786831525.56 total 635.32 last 24.53) - open/close 12040 (time 1786831535.72 total 645.48 last 24.32) - open/close 12159 (time 1786831546.03 total 655.79 last 11.54) - open/close 12399 (time 1786831556.21 total 665.97 last 23.58) - open/close 12603 (time 1786831566.58 total 676.34 last 19.67) - open/close 12827 (time 1786831577.33 total 687.09 last 20.84) - open/close 13052 (time 1786831588.43 total 698.19 last 20.27) - open/close 13182 (time 1786831598.75 total 708.52 last 12.59) - open/close 13305 (time 1786831609.09 total 718.86 last 11.90) - open/close 13385 (time 1786831619.16 total 728.92 last 7.95) - open/close 13517 (time 1786831629.70 total 739.46 last 12.52) - open/close 13704 (time 1786831639.74 total 749.51 last 18.61) - open/close 13900 (time 1786831649.75 total 759.52 last 19.59) - open/close 14094 (time 1786831659.77 total 769.54 last 19.35) - open/close 14233 (time 1786831669.80 total 779.56 last 13.87) - open/close 14499 (time 1786831680.05 total 789.82 last 25.94) - open/close 14689 (time 1786831690.06 total 799.82 last 18.99) - open/close 14801 (time 1786831700.52 total 810.29 last 10.70) - open/close 15104 (time 1786831711.08 total 820.84 last 28.71) - open/close 15350 (time 1786831721.72 total 831.48 last 23.12) - open/close 15645 (time 1786831732.04 total 841.80 last 28.58) - open/close 15903 (time 1786831742.07 total 851.83 last 25.73) - open/close 16147 (time 1786831752.87 total 862.63 last 22.59) - open/close 16307 (time 1786831763.75 total 873.52 last 14.70) - open/close 16447 (time 1786831774.12 total 883.88 last 13.50) - open/close 16770 (time 1786831784.13 total 893.89 last 32.27) - open/close 16981 (time 1786831794.41 total 904.17 last 20.52) - open/close 17212 (time 1786831805.43 total 915.19 last 20.96) - open/close 17328 (time 1786831815.53 total 925.29 last 11.49) - open/close 17485 (time 1786831825.55 total 935.31 last 15.67) - open/close 17735 (time 1786831836.28 total 946.04 last 23.30) - open/close 17958 (time 1786831847.34 total 957.10 last 20.16) - open/close 18149 (time 1786831857.38 total 967.14 last 19.02) - open/close 18339 (time 1786831868.09 total 977.85 last 17.74) - open/close 18407 (time 1786831878.09 total 987.85 last 6.80) - open/close 18580 (time 1786831890.60 total 1000.36 last 13.83) - open/close 19153 (time 1786831900.61 total 1010.37 last 57.26) - open/close 19781 (time 1786831910.62 total 1020.38 last 62.74) - open/close 20000 (time 1786831913.84 total 1023.60 last 67.95) - open/close 20599 (time 1786831923.84 total 1033.61 last 59.88) - open/close 21209 (time 1786831933.85 total 1043.61 last 60.98) - open/close 21764 (time 1786831943.85 total 1053.61 last 55.47) - open/close 22390 (time 1786831953.86 total 1063.63 last 62.51) - open/close 23045 (time 1786831963.88 total 1073.65 last 65.40) - open/close 23690 (time 1786831973.90 total 1083.66 last 64.41) - open/close 24153 (time 1786831983.91 total 1093.67 last 46.25) - open/close 24814 (time 1786831993.93 total 1103.70 last 65.92) - open/close 25469 (time 1786832003.94 total 1113.71 last 65.44) - open/close 26182 (time 1786832013.95 total 1123.71 last 71.27) - open/close 26882 (time 1786832023.95 total 1133.72 last 69.95) - open/close 27486 (time 1786832033.97 total 1143.73 last 60.31) - open/close 28153 (time 1786832043.97 total 1153.74 last 66.67) - open/close 28844 (time 1786832053.98 total 1163.74 last 69.05) - open/close 29571 (time 1786832063.99 total 1173.76 last 72.61) - open/close 30000 (time 1786832070.82 total 1180.58 last 62.83) - open/close 30654 (time 1786832080.82 total 1190.59 last 65.36) - open/close 31286 (time 1786832090.83 total 1200.60 last 63.16) - open/close 32023 (time 1786832100.84 total 1210.60 last 73.63) - open/close 32700 (time 1786832110.84 total 1220.61 last 67.67) - open/close 33437 (time 1786832120.87 total 1230.63 last 73.53) - open/close 34094 (time 1786832130.87 total 1240.64 last 65.65) - open/close 34784 (time 1786832140.89 total 1250.65 last 68.91) - open/close 35336 (time 1786832150.90 total 1260.67 last 55.10) - open/close 35834 (time 1786832160.91 total 1270.68 last 49.76) - open/close 36313 (time 1786832170.93 total 1280.69 last 47.83) - open/close 36894 (time 1786832180.95 total 1290.71 last 57.98) - open/close 37441 (time 1786832190.97 total 1300.73 last 54.61) - open/close 38058 (time 1786832200.98 total 1310.74 last 61.63) - open/close 38748 (time 1786832211.00 total 1320.77 last 68.82) - open/close 39372 (time 1786832221.02 total 1330.78 last 62.33) - open/close 39990 (time 1786832231.03 total 1340.80 last 61.69) - open/close 40402 (time 1786832241.06 total 1350.82 last 41.10) - open/close 40981 (time 1786832251.07 total 1360.84 last 57.81) - open/close 41573 (time 1786832261.07 total 1370.84 last 59.19) - open/close 42270 (time 1786832271.08 total 1380.84 last 69.69) - open/close 43002 (time 1786832281.08 total 1390.84 last 73.18) - open/close 43677 (time 1786832291.08 total 1400.84 last 67.50) - open/close 44368 (time 1786832301.10 total 1410.86 last 68.98) - open/close 45031 (time 1786832311.11 total 1420.87 last 66.21) - open/close 45722 (time 1786832321.12 total 1430.88 last 69.04) - open/close 46442 (time 1786832331.12 total 1440.89 last 71.97) - open/close 47122 (time 1786832341.13 total 1450.90 last 67.92) - open/close 47781 (time 1786832351.14 total 1460.90 last 65.87) - open/close 48355 (time 1786832361.15 total 1470.91 last 57.35) - open/close 48942 (time 1786832371.15 total 1480.91 last 58.69) - open/close 49504 (time 1786832381.15 total 1490.92 last 56.17) total: 50000 open/close in 1498.01 seconds: 33.38 ops/second - unlinked 0 (time 1786832393 ; total 0 ; last 0) - unlinked 10000 (time 1786832547 ; total 154 ; last 154) - unlinked 20000 (time 1786832720 ; total 327 ; last 173) - unlinked 30000 (time 1786832895 ; total 502 ; last 175) - unlinked 40000 (time 1786833068 ; total 675 ; last 173) total: 50000 unlinks in 841 seconds: 59.453033 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server unloading modules via unload_modules_local on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing unload_modules_local oleg418-server: modules unloaded. pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2