== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:55:54 (1779695754) start mds service on oleg355-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing load_modules_local oleg355-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg355-server: Loading modules from /home/green/git/lustre-release/lustre oleg355-server: detected 4 online CPUs by sysfs oleg355-server: MODOPTS_LIBCFS= oleg355-server: Force libcfs to create 2 CPU partitions oleg355-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg355-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg355-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg355-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 221 (time 1779695825.16 total 10.54 last 20.98) - open/close 492 (time 1779695836.05 total 21.43 last 24.87) - open/close 738 (time 1779695846.06 total 31.44 last 24.58) - open/close 939 (time 1779695856.08 total 41.45 last 20.07) - open/close 1185 (time 1779695866.10 total 51.47 last 24.55) - open/close 1383 (time 1779695876.10 total 61.48 last 19.79) - open/close 1632 (time 1779695886.11 total 71.48 last 24.89) - open/close 1875 (time 1779695896.11 total 81.49 last 24.29) - open/close 2115 (time 1779695906.12 total 91.50 last 23.98) - open/close 2356 (time 1779695916.13 total 101.50 last 24.08) - open/close 2586 (time 1779695926.66 total 112.04 last 21.84) - open/close 2788 (time 1779695936.66 total 122.04 last 20.19) - open/close 3026 (time 1779695946.67 total 132.04 last 23.80) - open/close 3244 (time 1779695957.33 total 142.71 last 20.44) - open/close 3509 (time 1779695967.34 total 152.71 last 26.49) - open/close 3721 (time 1779695977.34 total 162.72 last 21.19) - open/close 3997 (time 1779695987.59 total 172.96 last 26.94) - open/close 4207 (time 1779695997.90 total 183.28 last 20.36) - open/close 4452 (time 1779696008.26 total 193.64 last 23.64) - open/close 4753 (time 1779696019.01 total 204.39 last 28.01) - open/close 4971 (time 1779696029.08 total 214.45 last 21.65) - open/close 5284 (time 1779696039.95 total 225.33 last 28.77) - open/close 5554 (time 1779696050.43 total 235.81 last 25.77) - open/close 5737 (time 1779696060.44 total 245.82 last 18.28) - open/close 5938 (time 1779696071.01 total 256.39 last 19.02) - open/close 6094 (time 1779696081.01 total 266.39 last 15.59) - open/close 6350 (time 1779696091.40 total 276.78 last 24.65) - open/close 6618 (time 1779696102.06 total 287.43 last 25.15) - open/close 6871 (time 1779696112.06 total 297.44 last 25.29) - open/close 7128 (time 1779696122.88 total 308.26 last 23.75) - open/close 7335 (time 1779696133.07 total 318.45 last 20.31) - open/close 7557 (time 1779696143.08 total 328.46 last 22.19) - open/close 7814 (time 1779696153.36 total 338.73 last 25.00) - open/close 7986 (time 1779696163.37 total 348.75 last 17.18) - open/close 8242 (time 1779696173.38 total 358.76 last 25.56) - open/close 8485 (time 1779696184.14 total 369.51 last 22.60) - open/close 8759 (time 1779696194.69 total 380.07 last 25.96) - open/close 8991 (time 1779696205.71 total 391.09 last 21.05) - open/close 9219 (time 1779696215.71 total 401.09 last 22.80) - open/close 9480 (time 1779696226.38 total 411.76 last 24.47) - open/close 9732 (time 1779696237.07 total 422.45 last 23.57) - open/close 9991 (time 1779696247.30 total 432.67 last 25.32) - open/close 10233 (time 1779696257.31 total 442.68 last 24.18) - open/close 10484 (time 1779696268.36 total 453.74 last 22.70) - open/close 10735 (time 1779696278.55 total 463.92 last 24.65) - open/close 10927 (time 1779696289.36 total 474.74 last 17.75) - open/close 11168 (time 1779696299.36 total 484.74 last 24.10) - open/close 11378 (time 1779696309.37 total 494.74 last 20.99) - open/close 11627 (time 1779696319.37 total 504.75 last 24.90) - open/close 11806 (time 1779696329.48 total 514.86 last 17.70) - open/close 12043 (time 1779696339.49 total 524.87 last 23.68) - open/close 12281 (time 1779696349.65 total 535.03 last 23.42) - open/close 12516 (time 1779696359.81 total 545.19 last 23.13) - open/close 12801 (time 1779696370.85 total 556.22 last 25.83) - open/close 13071 (time 1779696381.26 total 566.64 last 25.93) - open/close 13331 (time 1779696391.64 total 577.02 last 25.04) - open/close 13635 (time 1779696401.67 total 587.05 last 30.31) - open/close 13882 (time 1779696412.13 total 597.51 last 23.62) - open/close 14141 (time 1779696422.54 total 607.91 last 24.89) - open/close 14387 (time 1779696432.83 total 618.21 last 23.89) - open/close 14628 (time 1779696442.84 total 628.22 last 24.08) - open/close 14875 (time 1779696453.39 total 638.76 last 23.42) - open/close 15124 (time 1779696463.63 total 649.00 last 24.31) - open/close 15346 (time 1779696473.65 total 659.03 last 22.14) - open/close 15556 (time 1779696483.87 total 669.25 last 20.55) - open/close 15774 (time 1779696494.00 total 679.38 last 21.51) - open/close 16002 (time 1779696504.01 total 689.39 last 22.79) - open/close 16248 (time 1779696514.38 total 699.76 last 23.71) - open/close 16516 (time 1779696524.39 total 709.76 last 26.79) - open/close 16782 (time 1779696534.40 total 719.78 last 26.57) - open/close 17032 (time 1779696544.41 total 729.78 last 24.98) - open/close 17261 (time 1779696554.51 total 739.89 last 22.66) - open/close 17539 (time 1779696564.51 total 749.89 last 27.79) - open/close 17755 (time 1779696574.51 total 759.89 last 21.60) - open/close 17915 (time 1779696584.86 total 770.23 last 15.47) - open/close 18143 (time 1779696594.86 total 780.24 last 22.79) - open/close 18314 (time 1779696605.07 total 790.45 last 16.75) - open/close 18545 (time 1779696615.19 total 800.56 last 22.83) - open/close 18754 (time 1779696625.35 total 810.73 last 20.56) - open/close 18839 (time 1779696635.98 total 821.36 last 8.00) - open/close 19098 (time 1779696645.99 total 831.37 last 25.87) - open/close 19323 (time 1779696656.00 total 841.38 last 22.48) - open/close 20000 (time 1779696665.03 total 850.41 last 74.97) - open/close 20750 (time 1779696675.04 total 860.42 last 74.91) - open/close 21561 (time 1779696685.05 total 870.43 last 81.04) - open/close 22318 (time 1779696695.06 total 880.44 last 75.60) - open/close 23004 (time 1779696705.08 total 890.46 last 68.49) - open/close 23812 (time 1779696715.09 total 900.46 last 80.74) - open/close 24697 (time 1779696725.09 total 910.47 last 88.45) - open/close 25567 (time 1779696735.10 total 920.48 last 86.92) - open/close 26444 (time 1779696745.11 total 930.48 last 87.66) - open/close 27302 (time 1779696755.12 total 940.50 last 85.70) - open/close 28159 (time 1779696765.13 total 950.50 last 85.65) - open/close 28987 (time 1779696775.14 total 960.52 last 82.69) - open/close 29862 (time 1779696785.14 total 970.52 last 87.50) - open/close 30590 (time 1779696795.15 total 980.53 last 72.70) - open/close 31383 (time 1779696805.16 total 990.54 last 79.25) - open/close 32207 (time 1779696815.16 total 1000.54 last 82.39) - open/close 33014 (time 1779696825.17 total 1010.55 last 80.62) - open/close 33850 (time 1779696835.18 total 1020.56 last 83.51) - open/close 34709 (time 1779696845.19 total 1030.56 last 85.86) - open/close 35553 (time 1779696855.20 total 1040.57 last 84.31) - open/close 36351 (time 1779696865.21 total 1050.59 last 79.68) - open/close 37169 (time 1779696875.21 total 1060.59 last 81.79) - open/close 38012 (time 1779696885.22 total 1070.60 last 84.25) - open/close 38692 (time 1779696895.23 total 1080.61 last 67.91) - open/close 39280 (time 1779696905.24 total 1090.62 last 58.75) - open/close 40000 (time 1779696914.81 total 1100.19 last 75.25) - open/close 40750 (time 1779696924.81 total 1110.19 last 74.99) - open/close 41433 (time 1779696934.82 total 1120.20 last 68.22) - open/close 42046 (time 1779696944.83 total 1130.21 last 61.24) - open/close 42838 (time 1779696954.84 total 1140.21 last 79.17) - open/close 43543 (time 1779696964.85 total 1150.23 last 70.41) - open/close 44291 (time 1779696974.86 total 1160.23 last 74.73) - open/close 45040 (time 1779696984.86 total 1170.24 last 74.85) - open/close 45849 (time 1779696994.87 total 1180.25 last 80.84) - open/close 46598 (time 1779697004.89 total 1190.27 last 74.74) - open/close 47396 (time 1779697014.90 total 1200.27 last 79.77) - open/close 48215 (time 1779697024.90 total 1210.27 last 81.90) - open/close 48995 (time 1779697034.90 total 1220.28 last 77.99) - open/close 49769 (time 1779697044.91 total 1230.29 last 77.31) total: 50000 open/close in 1233.08 seconds: 40.55 ops/second - unlinked 0 (time 1779697052 ; total 0 ; last 0) - unlinked 10000 (time 1779697185 ; total 133 ; last 133) - unlinked 20000 (time 1779697321 ; total 269 ; last 136) - unlinked 30000 (time 1779697462 ; total 410 ; last 141) - unlinked 40000 (time 1779697610 ; total 558 ; last 148) total: 50000 unlinks in 707 seconds: 70.721359 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg355-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg355-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg355-server unloading modules via unload_modules_local on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing unload_modules_local oleg355-server: modules unloaded. pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2