== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:33:42 (1753443222) start mds service on oleg143-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg143-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg143-server: detected 4 online CPUs by sysfs oleg143-server: MODOPTS_LIBCFS= oleg143-server: Force libcfs to create 2 CPU partitions oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg143-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg143-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 163 (time 1753443293.99 total 10.66 last 15.30) - open/close 335 (time 1753443304.61 total 21.28 last 16.20) - open/close 523 (time 1753443315.63 total 32.30 last 17.06) - open/close 703 (time 1753443325.64 total 42.31 last 17.98) - open/close 909 (time 1753443336.04 total 52.70 last 19.82) - open/close 1068 (time 1753443346.40 total 63.07 last 15.34) - open/close 1241 (time 1753443357.17 total 73.84 last 16.06) - open/close 1436 (time 1753443367.26 total 83.93 last 19.32) - open/close 1565 (time 1753443377.50 total 94.17 last 12.60) - open/close 1709 (time 1753443388.01 total 104.68 last 13.71) - open/close 1923 (time 1753443398.01 total 114.68 last 21.40) - open/close 2010 (time 1753443408.04 total 124.71 last 8.67) - open/close 2110 (time 1753443418.05 total 134.72 last 9.99) - open/close 2206 (time 1753443428.05 total 144.72 last 9.60) - open/close 2339 (time 1753443438.37 total 155.04 last 12.89) - open/close 2445 (time 1753443448.51 total 165.18 last 10.45) - open/close 2567 (time 1753443458.55 total 175.22 last 12.16) - open/close 2808 (time 1753443469.04 total 185.71 last 22.98) - open/close 3039 (time 1753443479.47 total 196.14 last 22.15) - open/close 3241 (time 1753443490.10 total 206.77 last 19.00) - open/close 3430 (time 1753443500.33 total 217.00 last 18.47) - open/close 3646 (time 1753443510.34 total 227.01 last 21.59) - open/close 3816 (time 1753443520.56 total 237.23 last 16.63) - open/close 4043 (time 1753443530.68 total 247.35 last 22.42) - open/close 4306 (time 1753443540.69 total 257.35 last 26.30) - open/close 4525 (time 1753443550.90 total 267.56 last 21.45) - open/close 4789 (time 1753443561.13 total 277.79 last 25.80) - open/close 5032 (time 1753443572.01 total 288.67 last 22.33) - open/close 5244 (time 1753443582.01 total 298.68 last 21.19) - open/close 5452 (time 1753443592.12 total 308.78 last 20.58) - open/close 5668 (time 1753443602.12 total 318.79 last 21.58) - open/close 5886 (time 1753443612.46 total 329.13 last 21.10) - open/close 6100 (time 1753443623.46 total 340.13 last 19.45) - open/close 6375 (time 1753443633.47 total 350.14 last 27.48) - open/close 6526 (time 1753443643.60 total 360.27 last 14.91) - open/close 6716 (time 1753443654.60 total 371.26 last 17.28) - open/close 6892 (time 1753443664.60 total 381.27 last 17.59) - open/close 7069 (time 1753443674.61 total 391.28 last 17.69) - open/close 7289 (time 1753443685.69 total 402.36 last 19.86) - open/close 7495 (time 1753443695.85 total 412.52 last 20.27) - open/close 7613 (time 1753443706.21 total 422.87 last 11.40) - open/close 7737 (time 1753443716.21 total 432.88 last 12.40) - open/close 7946 (time 1753443726.26 total 442.93 last 20.79) - open/close 8150 (time 1753443736.44 total 453.10 last 20.05) - open/close 8411 (time 1753443746.44 total 463.11 last 26.09) - open/close 8579 (time 1753443756.45 total 473.12 last 16.78) - open/close 8774 (time 1753443766.89 total 483.56 last 18.68) - open/close 8984 (time 1753443777.14 total 493.80 last 20.49) - open/close 9203 (time 1753443788.13 total 504.80 last 19.91) - open/close 9405 (time 1753443798.14 total 514.81 last 20.19) - open/close 9591 (time 1753443808.68 total 525.34 last 17.65) - open/close 9823 (time 1753443819.75 total 536.42 last 20.95) - open/close 9986 (time 1753443829.85 total 546.52 last 16.14) - open/close 10179 (time 1753443840.09 total 556.76 last 18.84) - open/close 10393 (time 1753443850.78 total 567.45 last 20.02) - open/close 10632 (time 1753443860.79 total 577.46 last 23.88) - open/close 10840 (time 1753443871.65 total 588.32 last 19.15) - open/close 11016 (time 1753443881.71 total 598.37 last 17.50) - open/close 11260 (time 1753443891.71 total 608.38 last 24.39) - open/close 11485 (time 1753443902.75 total 619.42 last 20.38) - open/close 11706 (time 1753443913.79 total 630.46 last 20.02) - open/close 11919 (time 1753443923.88 total 640.55 last 21.11) - open/close 12130 (time 1753443933.88 total 650.55 last 21.10) - open/close 12335 (time 1753443944.23 total 660.90 last 19.81) - open/close 12564 (time 1753443954.25 total 670.91 last 22.87) - open/close 12780 (time 1753443964.26 total 680.93 last 21.56) - open/close 13005 (time 1753443975.22 total 691.88 last 20.54) - open/close 13235 (time 1753443985.57 total 702.24 last 22.21) - open/close 13498 (time 1753443995.58 total 712.24 last 26.29) - open/close 13713 (time 1753444005.60 total 722.26 last 21.45) - open/close 13935 (time 1753444015.61 total 732.28 last 22.17) - open/close 14165 (time 1753444025.62 total 742.29 last 22.97) - open/close 14389 (time 1753444035.63 total 752.30 last 22.38) - open/close 14622 (time 1753444045.64 total 762.31 last 23.27) - open/close 14842 (time 1753444055.65 total 772.31 last 21.99) - open/close 15064 (time 1753444065.65 total 782.32 last 22.18) - open/close 15295 (time 1753444076.14 total 792.81 last 22.03) - open/close 15520 (time 1753444086.31 total 802.97 last 22.13) - open/close 15747 (time 1753444096.34 total 813.01 last 22.62) - open/close 15970 (time 1753444106.40 total 823.07 last 22.17) - open/close 16779 (time 1753444116.40 total 833.07 last 80.88) - open/close 17525 (time 1753444126.42 total 843.09 last 74.43) - open/close 18271 (time 1753444136.43 total 853.10 last 74.54) - open/close 19013 (time 1753444146.45 total 863.11 last 74.09) - open/close 19740 (time 1753444156.46 total 873.12 last 72.64) - open/close 20000 (time 1753444160.02 total 876.69 last 72.98) - open/close 20732 (time 1753444170.04 total 886.70 last 73.07) - open/close 21448 (time 1753444180.04 total 896.70 last 71.59) - open/close 22148 (time 1753444190.05 total 906.72 last 69.92) - open/close 22869 (time 1753444200.06 total 916.73 last 72.02) - open/close 23624 (time 1753444210.07 total 926.74 last 75.41) - open/close 24369 (time 1753444220.08 total 936.74 last 74.46) - open/close 25085 (time 1753444230.08 total 946.75 last 71.57) - open/close 25823 (time 1753444240.10 total 956.76 last 73.69) - open/close 26569 (time 1753444250.10 total 966.77 last 74.58) - open/close 27313 (time 1753444260.10 total 976.77 last 74.38) - open/close 28056 (time 1753444270.12 total 986.78 last 74.20) - open/close 28817 (time 1753444280.12 total 996.79 last 76.09) - open/close 29549 (time 1753444290.13 total 1006.80 last 73.11) - open/close 30000 (time 1753444296.41 total 1013.08 last 71.80) - open/close 30758 (time 1753444306.43 total 1023.09 last 75.70) - open/close 31520 (time 1753444316.43 total 1033.10 last 76.18) - open/close 32304 (time 1753444326.44 total 1043.11 last 78.30) - open/close 33040 (time 1753444336.46 total 1053.13 last 73.45) - open/close 33787 (time 1753444346.47 total 1063.13 last 74.66) - open/close 34506 (time 1753444356.49 total 1073.15 last 71.77) - open/close 35213 (time 1753444366.49 total 1083.16 last 70.64) - open/close 35954 (time 1753444376.50 total 1093.16 last 74.09) - open/close 36652 (time 1753444386.52 total 1103.18 last 69.65) - open/close 37357 (time 1753444396.52 total 1113.18 last 70.49) - open/close 38116 (time 1753444406.52 total 1123.19 last 75.89) - open/close 38848 (time 1753444416.53 total 1133.19 last 73.15) - open/close 39586 (time 1753444426.53 total 1143.20 last 73.75) - open/close 40000 (time 1753444431.84 total 1148.51 last 78.00) - open/close 40745 (time 1753444441.85 total 1158.52 last 74.42) - open/close 41490 (time 1753444451.86 total 1168.53 last 74.42) - open/close 42205 (time 1753444461.87 total 1178.53 last 71.47) - open/close 42979 (time 1753444471.87 total 1188.54 last 77.38) - open/close 43732 (time 1753444481.87 total 1198.54 last 75.27) - open/close 44446 (time 1753444491.88 total 1208.55 last 71.36) - open/close 45167 (time 1753444501.89 total 1218.56 last 72.01) - open/close 45994 (time 1753444511.90 total 1228.56 last 82.65) - open/close 46925 (time 1753444521.91 total 1238.57 last 93.03) - open/close 47842 (time 1753444531.91 total 1248.58 last 91.64) - open/close 48703 (time 1753444541.93 total 1258.59 last 85.98) - open/close 49502 (time 1753444551.93 total 1268.60 last 79.88) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1270.22 seconds: 39.07 ops/second - unlinked 0 (time 1753444556 ; total 0 ; last 0) - unlinked 10000 (time 1753444668 ; total 112 ; last 112) - unlinked 20000 (time 1753444768 ; total 212 ; last 100) - unlinked 30000 (time 1753444816 ; total 260 ; last 48) - unlinked 40000 (time 1753444862 ; total 306 ; last 46) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 347 seconds: 143.031708 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local oleg143-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg143-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2