== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:43:22 (1761241402) start mds service on oleg627-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg627-server' oleg627-server: oleg627-server.virtnet: executing load_modules_local oleg627-server: Loading modules from /home/green/git/lustre-release/lustre oleg627-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg627-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg627-server: detected 4 online CPUs by sysfs oleg627-server: MODOPTS_LIBCFS= oleg627-server: Force libcfs to create 2 CPU partitions oleg627-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg627-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg627-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg627-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 134 (time 1761241475.76 total 10.34 last 12.96) - open/close 296 (time 1761241485.82 total 20.40 last 16.10) - open/close 508 (time 1761241496.30 total 30.88 last 20.23) - open/close 746 (time 1761241506.31 total 40.89 last 23.78) - open/close 1007 (time 1761241516.32 total 50.90 last 26.08) - open/close 1134 (time 1761241527.15 total 61.73 last 11.72) - open/close 1195 (time 1761241537.18 total 71.77 last 6.08) - open/close 1420 (time 1761241548.15 total 82.73 last 20.53) - open/close 1624 (time 1761241559.14 total 93.72 last 18.56) - open/close 1874 (time 1761241569.15 total 103.73 last 24.98) - open/close 2137 (time 1761241579.91 total 114.49 last 24.43) - open/close 2386 (time 1761241589.93 total 124.51 last 24.85) - open/close 2545 (time 1761241599.95 total 134.53 last 15.87) - open/close 2723 (time 1761241610.72 total 145.31 last 16.52) - open/close 2919 (time 1761241620.97 total 155.55 last 19.13) - open/close 3100 (time 1761241631.02 total 165.60 last 18.01) - open/close 3290 (time 1761241641.03 total 175.61 last 18.99) - open/close 3505 (time 1761241651.04 total 185.62 last 21.47) - open/close 3698 (time 1761241662.05 total 196.63 last 17.53) - open/close 3871 (time 1761241672.75 total 207.34 last 16.16) - open/close 4068 (time 1761241683.76 total 218.34 last 17.90) - open/close 4254 (time 1761241693.77 total 228.35 last 18.59) - open/close 4451 (time 1761241703.78 total 238.36 last 19.69) - open/close 4693 (time 1761241713.92 total 248.50 last 23.86) - open/close 4968 (time 1761241724.27 total 258.86 last 26.55) - open/close 5153 (time 1761241735.28 total 269.86 last 16.82) - open/close 5382 (time 1761241745.97 total 280.55 last 21.41) - open/close 5633 (time 1761241756.20 total 290.78 last 24.54) - open/close 5866 (time 1761241766.45 total 301.03 last 22.73) - open/close 6129 (time 1761241777.13 total 311.72 last 24.62) - open/close 6372 (time 1761241787.14 total 321.72 last 24.30) - open/close 6584 (time 1761241797.80 total 332.38 last 19.87) - open/close 6716 (time 1761241807.97 total 342.56 last 12.98) - open/close 6880 (time 1761241818.35 total 352.93 last 15.81) - open/close 7054 (time 1761241828.45 total 363.03 last 17.22) - open/close 7232 (time 1761241838.47 total 373.05 last 17.77) - open/close 7466 (time 1761241848.47 total 383.05 last 23.40) - open/close 7679 (time 1761241858.48 total 393.06 last 21.29) - open/close 7942 (time 1761241868.48 total 403.06 last 26.29) - open/close 8175 (time 1761241878.49 total 413.07 last 23.27) - open/close 8429 (time 1761241888.51 total 423.09 last 25.36) - open/close 8675 (time 1761241898.55 total 433.13 last 24.50) - open/close 8943 (time 1761241908.84 total 443.43 last 26.03) - open/close 9182 (time 1761241918.85 total 453.43 last 23.88) - open/close 9422 (time 1761241929.02 total 463.60 last 23.61) - open/close 9699 (time 1761241939.02 total 473.61 last 27.68) - open/close 9793 (time 1761241949.09 total 483.68 last 9.33) - open/close 10000 (time 1761241956.93 total 491.51 last 26.42) - open/close 10279 (time 1761241967.54 total 502.12 last 26.30) - open/close 10531 (time 1761241978.23 total 512.81 last 23.57) - open/close 10790 (time 1761241988.24 total 522.82 last 25.88) - open/close 11035 (time 1761241998.60 total 533.18 last 23.64) - open/close 11258 (time 1761242008.76 total 543.34 last 21.96) - open/close 11448 (time 1761242018.92 total 553.50 last 18.70) - open/close 11735 (time 1761242029.56 total 564.14 last 26.98) - open/close 12017 (time 1761242040.50 total 575.09 last 25.76) - open/close 12289 (time 1761242051.30 total 585.88 last 25.19) - open/close 12590 (time 1761242061.98 total 596.57 last 28.18) - open/close 12839 (time 1761242072.89 total 607.47 last 22.83) - open/close 13091 (time 1761242083.13 total 617.71 last 24.62) - open/close 13280 (time 1761242093.75 total 628.33 last 17.79) - open/close 13562 (time 1761242104.64 total 639.22 last 25.90) - open/close 13830 (time 1761242115.69 total 650.28 last 24.24) - open/close 14074 (time 1761242125.70 total 660.28 last 24.40) - open/close 14300 (time 1761242135.79 total 670.37 last 22.40) - open/close 14530 (time 1761242145.80 total 680.38 last 22.97) - open/close 14754 (time 1761242155.88 total 690.46 last 22.23) - open/close 14963 (time 1761242166.18 total 700.77 last 20.28) - open/close 15217 (time 1761242177.07 total 711.65 last 23.34) - open/close 15439 (time 1761242187.63 total 722.21 last 21.02) - open/close 15711 (time 1761242198.57 total 733.15 last 24.86) - open/close 15948 (time 1761242208.82 total 743.40 last 23.12) - open/close 16174 (time 1761242218.84 total 753.43 last 22.54) - open/close 16446 (time 1761242229.54 total 764.13 last 25.42) - open/close 16652 (time 1761242239.79 total 774.37 last 20.11) - open/close 17382 (time 1761242249.79 total 784.37 last 72.98) - open/close 18363 (time 1761242259.79 total 794.37 last 98.09) - open/close 19295 (time 1761242269.79 total 804.37 last 93.20) - open/close 20000 (time 1761242277.13 total 811.72 last 95.99) - open/close 20858 (time 1761242287.14 total 821.72 last 85.77) - open/close 21636 (time 1761242297.14 total 831.72 last 77.78) - open/close 22324 (time 1761242307.15 total 841.73 last 68.72) - open/close 23162 (time 1761242317.16 total 851.74 last 83.74) - open/close 23949 (time 1761242327.17 total 861.75 last 78.65) - open/close 24783 (time 1761242337.18 total 871.76 last 83.33) - open/close 25653 (time 1761242347.18 total 881.77 last 86.93) - open/close 26408 (time 1761242357.20 total 891.78 last 75.40) - open/close 27326 (time 1761242367.20 total 901.78 last 91.75) - open/close 28086 (time 1761242377.21 total 911.79 last 75.98) - open/close 28909 (time 1761242387.21 total 921.80 last 82.23) - open/close 29759 (time 1761242397.22 total 931.80 last 84.94) - open/close 30000 (time 1761242400.08 total 934.66 last 84.30) - open/close 30919 (time 1761242410.08 total 944.67 last 91.87) - open/close 31770 (time 1761242420.08 total 954.67 last 85.09) - open/close 32614 (time 1761242430.09 total 964.67 last 84.34) - open/close 33445 (time 1761242440.10 total 974.68 last 83.02) - open/close 34140 (time 1761242450.10 total 984.68 last 69.49) - open/close 35026 (time 1761242460.11 total 994.69 last 88.53) - open/close 35948 (time 1761242470.11 total 1004.69 last 92.20) - open/close 36751 (time 1761242480.11 total 1014.69 last 80.28) - open/close 37638 (time 1761242490.13 total 1024.71 last 88.59) - open/close 38436 (time 1761242500.14 total 1034.72 last 79.70) - open/close 39251 (time 1761242510.14 total 1044.72 last 81.48) - open/close 40000 (time 1761242518.35 total 1052.93 last 91.22) - open/close 40957 (time 1761242528.36 total 1062.94 last 95.62) - open/close 41922 (time 1761242538.36 total 1072.94 last 96.48) - open/close 42754 (time 1761242548.37 total 1082.95 last 83.13) - open/close 43719 (time 1761242558.38 total 1092.96 last 96.41) - open/close 44594 (time 1761242568.39 total 1102.97 last 87.43) - open/close 45524 (time 1761242578.40 total 1112.98 last 92.92) - open/close 46274 (time 1761242588.42 total 1123.00 last 74.82) - open/close 47045 (time 1761242598.44 total 1133.02 last 76.95) - open/close 47700 (time 1761242608.50 total 1143.08 last 65.09) - open/close 48190 (time 1761242618.51 total 1153.09 last 48.95) - open/close 49009 (time 1761242628.52 total 1163.10 last 81.81) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1169.87 seconds: 42.43 ops/second - unlinked 0 (time 1761242638 ; total 0 ; last 0) - unlinked 10000 (time 1761242746 ; total 108 ; last 108) - unlinked 20000 (time 1761242838 ; total 200 ; last 92) - unlinked 30000 (time 1761242898 ; total 260 ; last 60) - unlinked 40000 (time 1761242961 ; total 323 ; last 63) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 382 seconds: 129.926697 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg627-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg627-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg627-server stop mds service on oleg627-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg627-server stop mds service on oleg627-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg627-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg627-server' oleg627-server: oleg627-server.virtnet: executing unload_modules_local oleg627-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg627-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg627-client: oleg627-client: ssh exited with exit code 2 pdsh@oleg627-client: oleg627-server: ssh exited with exit code 2 pdsh@oleg627-client: oleg627-client: ssh exited with exit code 2 pdsh@oleg627-client: oleg627-server: ssh exited with exit code 2