== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:15:31 (1786436131) start mds service on oleg309-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg309-server' oleg309-server: oleg309-server.virtnet: executing load_modules_local oleg309-server: Loading modules from /home/green/git/lustre-release/lustre oleg309-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg309-server: detected 4 online CPUs by sysfs oleg309-server: MODOPTS_LIBCFS= oleg309-server: Force libcfs to create 2 CPU partitions oleg309-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg309-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg309-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg309-server: oleg309-server.virtnet: executing set_default_debug -1 all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg309-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg309-server: oleg309-server.virtnet: executing set_default_debug -1 all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg309-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug -1 all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 216 (time 1786436236.19 total 10.79 last 20.02) - open/close 332 (time 1786436246.20 total 20.80 last 11.59) - open/close 557 (time 1786436256.21 total 30.81 last 22.49) - open/close 798 (time 1786436266.31 total 40.91 last 23.86) - open/close 1087 (time 1786436276.32 total 50.92 last 28.87) - open/close 1277 (time 1786436286.78 total 61.38 last 18.16) - open/close 1478 (time 1786436297.19 total 71.79 last 19.30) - open/close 1728 (time 1786436307.20 total 81.80 last 24.98) - open/close 1896 (time 1786436317.30 total 91.90 last 16.63) - open/close 2077 (time 1786436327.31 total 101.91 last 18.07) - open/close 2324 (time 1786436337.43 total 112.03 last 24.40) - open/close 2572 (time 1786436347.44 total 122.04 last 24.78) - open/close 2642 (time 1786436358.03 total 132.63 last 6.61) - open/close 2879 (time 1786436368.05 total 142.65 last 23.66) - open/close 3105 (time 1786436378.55 total 153.15 last 21.53) - open/close 3350 (time 1786436389.53 total 164.13 last 22.31) - open/close 3628 (time 1786436400.60 total 175.20 last 25.11) - open/close 3887 (time 1786436410.60 total 185.20 last 25.89) - open/close 4123 (time 1786436420.71 total 195.31 last 23.34) - open/close 4284 (time 1786436430.73 total 205.33 last 16.07) - open/close 4560 (time 1786436440.74 total 215.34 last 27.59) - open/close 4817 (time 1786436450.76 total 225.36 last 25.65) - open/close 5050 (time 1786436460.76 total 235.36 last 23.29) - open/close 5307 (time 1786436471.10 total 245.71 last 24.85) - open/close 5538 (time 1786436481.46 total 256.06 last 22.31) - open/close 5742 (time 1786436491.47 total 266.07 last 20.37) - open/close 5964 (time 1786436501.49 total 276.09 last 22.17) - open/close 6158 (time 1786436511.63 total 286.24 last 19.12) - open/close 6361 (time 1786436521.94 total 296.54 last 19.69) - open/close 6569 (time 1786436532.84 total 307.45 last 19.08) - open/close 6804 (time 1786436542.85 total 317.45 last 23.49) - open/close 6973 (time 1786436552.93 total 327.53 last 16.77) - open/close 7194 (time 1786436562.96 total 337.56 last 22.03) - open/close 7428 (time 1786436573.13 total 347.73 last 23.01) - open/close 7689 (time 1786436583.13 total 357.73 last 26.09) - open/close 7883 (time 1786436593.25 total 367.85 last 19.17) - open/close 8133 (time 1786436603.51 total 378.11 last 24.37) - open/close 8398 (time 1786436613.51 total 388.11 last 26.50) - open/close 8665 (time 1786436623.59 total 398.19 last 26.50) - open/close 8903 (time 1786436634.36 total 408.96 last 22.10) - open/close 9093 (time 1786436644.36 total 418.96 last 18.99) - open/close 9333 (time 1786436654.40 total 429.00 last 23.92) - open/close 9495 (time 1786436664.40 total 439.00 last 16.19) - open/close 9740 (time 1786436674.41 total 449.01 last 24.47) - open/close 9993 (time 1786436684.44 total 459.04 last 25.24) - open/close 10212 (time 1786436694.44 total 469.05 last 21.89) - open/close 10396 (time 1786436704.98 total 479.58 last 17.46) - open/close 10691 (time 1786436714.99 total 489.59 last 29.48) - open/close 10890 (time 1786436725.05 total 499.65 last 19.78) - open/close 11085 (time 1786436735.64 total 510.24 last 18.41) - open/close 11321 (time 1786436745.87 total 520.47 last 23.07) - open/close 11575 (time 1786436756.18 total 530.78 last 24.65) - open/close 11761 (time 1786436766.36 total 540.96 last 18.27) - open/close 11973 (time 1786436776.68 total 551.28 last 20.54) - open/close 12137 (time 1786436786.69 total 561.29 last 16.38) - open/close 12379 (time 1786436796.70 total 571.30 last 24.18) - open/close 12625 (time 1786436806.82 total 581.42 last 24.31) - open/close 12865 (time 1786436816.82 total 591.42 last 24.00) - open/close 13090 (time 1786436826.83 total 601.43 last 22.47) - open/close 13317 (time 1786436836.84 total 611.44 last 22.69) - open/close 13549 (time 1786436846.84 total 621.44 last 23.19) - open/close 13738 (time 1786436857.00 total 631.60 last 18.62) - open/close 13968 (time 1786436867.80 total 642.40 last 21.29) - open/close 14228 (time 1786436878.62 total 653.22 last 24.03) - open/close 14428 (time 1786436888.62 total 663.22 last 19.99) - open/close 14654 (time 1786436898.89 total 673.49 last 22.01) - open/close 14865 (time 1786436908.91 total 683.52 last 21.05) - open/close 15122 (time 1786436919.46 total 694.06 last 24.37) - open/close 15362 (time 1786436929.69 total 704.29 last 23.47) - open/close 15568 (time 1786436940.08 total 714.68 last 19.82) - open/close 15841 (time 1786436950.37 total 724.97 last 26.52) - open/close 16067 (time 1786436960.42 total 735.02 last 22.50) - open/close 16292 (time 1786436970.45 total 745.05 last 22.44) - open/close 16604 (time 1786436980.46 total 755.06 last 31.15) - open/close 16806 (time 1786436991.09 total 765.69 last 19.01) - open/close 17070 (time 1786437001.95 total 776.55 last 24.31) - open/close 17343 (time 1786437012.84 total 787.44 last 25.07) - open/close 17619 (time 1786437022.85 total 797.45 last 27.58) - open/close 17825 (time 1786437032.85 total 807.45 last 20.59) - open/close 18042 (time 1786437042.98 total 817.58 last 21.44) - open/close 18273 (time 1786437054.76 total 829.36 last 19.60) - open/close 18477 (time 1786437065.05 total 839.65 last 19.82) - open/close 18676 (time 1786437075.06 total 849.66 last 19.89) - open/close 18908 (time 1786437086.42 total 861.03 last 20.41) - open/close 19137 (time 1786437096.79 total 871.39 last 22.09) - open/close 19444 (time 1786437107.49 total 882.09 last 28.71) - open/close 19646 (time 1786437118.02 total 892.62 last 19.19) - open/close 19875 (time 1786437128.03 total 902.63 last 22.87) - open/close 20000 (time 1786437135.28 total 909.88 last 17.25) - open/close 20243 (time 1786437145.29 total 919.89 last 24.27) - open/close 20761 (time 1786437155.29 total 929.89 last 51.78) - open/close 21659 (time 1786437165.30 total 939.90 last 89.77) - open/close 22691 (time 1786437175.31 total 949.91 last 103.10) - open/close 23703 (time 1786437185.31 total 959.91 last 101.15) - open/close 24577 (time 1786437195.33 total 969.93 last 87.21) - open/close 25589 (time 1786437205.34 total 979.94 last 101.14) - open/close 26563 (time 1786437215.35 total 989.95 last 97.32) - open/close 27423 (time 1786437225.52 total 1000.13 last 84.49) - open/close 28395 (time 1786437235.53 total 1010.13 last 97.19) - open/close 29382 (time 1786437245.53 total 1020.13 last 98.68) - open/close 30000 (time 1786437251.08 total 1025.68 last 111.37) - open/close 31023 (time 1786437261.08 total 1035.68 last 102.22) - open/close 31930 (time 1786437271.09 total 1045.69 last 90.69) - open/close 32801 (time 1786437281.11 total 1055.71 last 86.92) - open/close 33773 (time 1786437291.12 total 1065.72 last 97.05) - open/close 34724 (time 1786437301.13 total 1075.73 last 95.06) - open/close 35595 (time 1786437311.13 total 1085.73 last 87.04) - open/close 36686 (time 1786437321.14 total 1095.74 last 108.97) - open/close 37699 (time 1786437331.15 total 1105.75 last 101.26) - open/close 38695 (time 1786437341.16 total 1115.76 last 99.46) - open/close 39692 (time 1786437351.17 total 1125.77 last 99.63) - open/close 40000 (time 1786437354.04 total 1128.64 last 107.18) - open/close 40978 (time 1786437364.05 total 1138.66 last 97.69) - open/close 41953 (time 1786437374.12 total 1148.72 last 96.85) - open/close 42986 (time 1786437384.14 total 1158.74 last 103.15) - open/close 43955 (time 1786437394.14 total 1168.74 last 96.88) - open/close 44938 (time 1786437404.15 total 1178.75 last 98.21) - open/close 45946 (time 1786437414.15 total 1188.75 last 100.79) - open/close 46920 (time 1786437424.16 total 1198.76 last 97.25) - open/close 47909 (time 1786437434.31 total 1208.91 last 97.50) - open/close 48797 (time 1786437444.32 total 1218.92 last 88.73) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1227.81 seconds: 40.42 ops/second - unlinked 0 (time 1786437457 ; total 0 ; last 0) - unlinked 10000 (time 1786437574 ; total 117 ; last 117) - unlinked 20000 (time 1786437676 ; total 219 ; last 102) - unlinked 30000 (time 1786437736 ; total 279 ; last 60) - unlinked 40000 (time 1786437801 ; total 344 ; last 65) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 404 seconds: 122.849007 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg309-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg309-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg309-server stop mds service on oleg309-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg309-server stop mds service on oleg309-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg309-server unloading modules via unload_modules_local on: 'oleg309-server' oleg309-server: oleg309-server.virtnet: executing unload_modules_local oleg309-server: modules unloaded. pdsh@oleg309-client: oleg309-client: ssh exited with exit code 2 pdsh@oleg309-client: oleg309-server: ssh exited with exit code 2 pdsh@oleg309-client: oleg309-server: ssh exited with exit code 2 pdsh@oleg309-client: oleg309-client: ssh exited with exit code 2