== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:05:06 (1763406306) start mds service on oleg105-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing load_modules_local oleg105-server: Loading modules from /home/green/git/lustre-release/lustre oleg105-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg105-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg105-server: detected 4 online CPUs by sysfs oleg105-server: MODOPTS_LIBCFS= oleg105-server: Force libcfs to create 2 CPU partitions oleg105-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg105-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg105-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg105-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0000 oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 204 (time 1763406402.33 total 10.94 last 18.65) - open/close 386 (time 1763406415.11 total 23.72 last 14.24) - open/close 555 (time 1763406425.36 total 33.97 last 16.49) - open/close 732 (time 1763406435.38 total 44.00 last 17.66) - open/close 994 (time 1763406446.33 total 54.95 last 23.93) - open/close 1245 (time 1763406456.34 total 64.96 last 25.08) - open/close 1433 (time 1763406467.33 total 75.94 last 17.11) - open/close 1535 (time 1763406477.52 total 86.14 last 10.01) - open/close 1795 (time 1763406487.85 total 96.47 last 25.17) - open/close 1989 (time 1763406497.89 total 106.50 last 19.33) - open/close 2139 (time 1763406507.89 total 116.50 last 15.00) - open/close 2332 (time 1763406517.91 total 126.52 last 19.26) - open/close 2557 (time 1763406528.27 total 136.89 last 21.70) - open/close 2712 (time 1763406538.69 total 147.31 last 14.88) - open/close 2944 (time 1763406549.08 total 157.69 last 22.34) - open/close 3194 (time 1763406559.94 total 168.56 last 23.01) - open/close 3414 (time 1763406569.94 total 178.56 last 22.00) - open/close 3662 (time 1763406579.95 total 188.56 last 24.79) - open/close 3875 (time 1763406590.15 total 198.77 last 20.87) - open/close 4053 (time 1763406600.47 total 209.09 last 17.25) - open/close 4174 (time 1763406610.95 total 219.56 last 11.55) - open/close 4269 (time 1763406621.62 total 230.24 last 8.90) - open/close 4492 (time 1763406632.35 total 240.97 last 20.78) - open/close 4719 (time 1763406642.35 total 250.97 last 22.70) - open/close 4983 (time 1763406652.37 total 260.99 last 26.36) - open/close 5212 (time 1763406662.92 total 271.54 last 21.70) - open/close 5370 (time 1763406672.94 total 281.55 last 15.78) - open/close 5440 (time 1763406683.77 total 292.39 last 6.46) - open/close 5627 (time 1763406693.77 total 302.39 last 18.70) - open/close 5759 (time 1763406703.79 total 312.41 last 13.18) - open/close 5942 (time 1763406714.22 total 322.83 last 17.55) - open/close 6054 (time 1763406724.43 total 333.05 last 10.96) - open/close 6221 (time 1763406735.19 total 343.80 last 15.53) - open/close 6432 (time 1763406746.20 total 354.81 last 19.16) - open/close 6625 (time 1763406756.20 total 364.82 last 19.30) - open/close 6736 (time 1763406766.81 total 375.43 last 10.46) - open/close 6893 (time 1763406777.25 total 385.86 last 15.05) - open/close 7056 (time 1763406787.26 total 395.87 last 16.28) - open/close 7132 (time 1763406797.28 total 405.89 last 7.59) - open/close 7253 (time 1763406807.30 total 415.92 last 12.07) - open/close 7489 (time 1763406817.32 total 425.93 last 23.57) - open/close 7740 (time 1763406828.17 total 436.79 last 23.12) - open/close 7960 (time 1763406839.18 total 447.80 last 19.98) - open/close 8139 (time 1763406849.24 total 457.85 last 17.80) - open/close 8358 (time 1763406859.25 total 467.87 last 21.86) - open/close 8506 (time 1763406870.32 total 478.93 last 13.38) - open/close 8758 (time 1763406880.32 total 488.94 last 25.19) - open/close 8947 (time 1763406890.32 total 498.94 last 18.90) - open/close 9140 (time 1763406902.06 total 510.68 last 16.44) - open/close 9396 (time 1763406913.17 total 521.78 last 23.04) - open/close 9618 (time 1763406923.18 total 531.79 last 22.18) - open/close 9696 (time 1763406935.36 total 543.98 last 6.40) - open/close 9911 (time 1763406945.62 total 554.24 last 20.96) - open/close 10000 (time 1763406949.29 total 557.90 last 24.27) - open/close 10137 (time 1763406960.09 total 568.70 last 12.68) - open/close 10234 (time 1763406970.95 total 579.56 last 8.93) - open/close 10419 (time 1763406981.19 total 589.80 last 18.07) - open/close 10647 (time 1763406991.21 total 599.82 last 22.75) - open/close 10866 (time 1763407001.81 total 610.43 last 20.65) - open/close 11054 (time 1763407012.10 total 620.71 last 18.27) - open/close 11291 (time 1763407022.12 total 630.73 last 23.66) - open/close 11471 (time 1763407032.99 total 641.60 last 16.56) - open/close 11603 (time 1763407043.99 total 652.60 last 12.00) - open/close 11790 (time 1763407054.00 total 662.62 last 18.68) - open/close 12027 (time 1763407064.67 total 673.28 last 22.22) - open/close 12132 (time 1763407075.21 total 683.82 last 9.97) - open/close 12248 (time 1763407085.47 total 694.09 last 11.30) - open/close 12376 (time 1763407096.22 total 704.84 last 11.91) - open/close 12590 (time 1763407106.23 total 714.85 last 21.38) - open/close 12710 (time 1763407116.24 total 724.85 last 12.00) - open/close 12986 (time 1763407126.99 total 735.61 last 25.66) - open/close 13226 (time 1763407137.86 total 746.48 last 22.08) - open/close 13344 (time 1763407147.92 total 756.53 last 11.73) - open/close 13440 (time 1763407157.94 total 766.56 last 9.57) - open/close 13536 (time 1763407168.01 total 776.63 last 9.53) - open/close 13752 (time 1763407178.05 total 786.66 last 21.53) - open/close 13878 (time 1763407188.08 total 796.70 last 12.55) - open/close 14100 (time 1763407198.09 total 806.70 last 22.19) - open/close 14161 (time 1763407209.15 total 817.76 last 5.52) - open/close 14305 (time 1763407219.16 total 827.78 last 14.38) - open/close 14477 (time 1763407229.26 total 837.88 last 17.03) - open/close 14697 (time 1763407239.28 total 847.90 last 21.95) - open/close 14939 (time 1763407249.28 total 857.90 last 24.20) - open/close 15148 (time 1763407259.81 total 868.42 last 19.86) - open/close 15336 (time 1763407269.84 total 878.46 last 18.73) - open/close 15389 (time 1763407280.78 total 889.40 last 4.84) - open/close 15490 (time 1763407290.80 total 899.41 last 10.09) - open/close 15725 (time 1763407301.77 total 910.39 last 21.41) - open/close 15902 (time 1763407313.61 total 922.23 last 14.95) - open/close 16125 (time 1763407323.62 total 932.24 last 22.28) - open/close 16362 (time 1763407334.13 total 942.74 last 22.56) - open/close 16567 (time 1763407344.14 total 952.75 last 20.48) - open/close 16731 (time 1763407354.31 total 962.93 last 16.12) - open/close 16955 (time 1763407364.87 total 973.49 last 21.21) - open/close 17174 (time 1763407375.83 total 984.45 last 19.98) - open/close 17254 (time 1763407386.18 total 994.80 last 7.73) - open/close 17365 (time 1763407397.15 total 1005.76 last 10.13) - open/close 17690 (time 1763407407.18 total 1015.80 last 32.38) - open/close 18334 (time 1763407417.19 total 1025.80 last 64.37) - open/close 18960 (time 1763407427.20 total 1035.82 last 62.50) - open/close 19542 (time 1763407437.22 total 1045.83 last 58.12) - open/close 20000 (time 1763407444.93 total 1053.55 last 59.39) - open/close 20656 (time 1763407454.94 total 1063.56 last 65.53) - open/close 21262 (time 1763407464.95 total 1073.56 last 60.55) - open/close 21846 (time 1763407474.95 total 1083.57 last 58.39) - open/close 22557 (time 1763407484.95 total 1093.57 last 71.07) - open/close 23345 (time 1763407494.96 total 1103.57 last 78.79) - open/close 24072 (time 1763407504.96 total 1113.57 last 72.69) - open/close 24837 (time 1763407514.96 total 1123.57 last 76.49) - open/close 25608 (time 1763407524.96 total 1133.58 last 77.06) - open/close 26351 (time 1763407534.97 total 1143.58 last 74.29) - open/close 27037 (time 1763407544.98 total 1153.60 last 68.50) - open/close 27773 (time 1763407555.00 total 1163.61 last 73.49) - open/close 28500 (time 1763407565.01 total 1173.62 last 72.63) - open/close 29278 (time 1763407575.02 total 1183.63 last 77.70) - open/close 29937 (time 1763407585.02 total 1193.64 last 65.85) - open/close 30681 (time 1763407595.03 total 1203.65 last 74.34) - open/close 31372 (time 1763407605.04 total 1213.65 last 69.07) - open/close 32092 (time 1763407615.04 total 1223.65 last 71.98) - open/close 32742 (time 1763407625.05 total 1233.67 last 64.92) - open/close 33353 (time 1763407635.05 total 1243.67 last 61.09) - open/close 34033 (time 1763407645.06 total 1253.67 last 67.97) - open/close 34723 (time 1763407655.06 total 1263.67 last 69.00) - open/close 35432 (time 1763407665.07 total 1273.68 last 70.82) - open/close 36157 (time 1763407675.07 total 1283.69 last 72.45) - open/close 36927 (time 1763407685.08 total 1293.69 last 77.00) - open/close 37485 (time 1763407695.08 total 1303.69 last 55.78) - open/close 38096 (time 1763407705.09 total 1313.70 last 61.06) - open/close 38664 (time 1763407715.10 total 1323.71 last 56.73) - open/close 39397 (time 1763407725.10 total 1333.72 last 73.26) - open/close 40000 (time 1763407733.05 total 1341.67 last 75.85) - open/close 40764 (time 1763407743.07 total 1351.69 last 76.26) - open/close 41534 (time 1763407753.07 total 1361.69 last 76.97) - open/close 42238 (time 1763407763.08 total 1371.69 last 70.38) - open/close 42819 (time 1763407773.08 total 1381.70 last 58.08) - open/close 43396 (time 1763407783.08 total 1391.70 last 57.69) - open/close 44109 (time 1763407793.09 total 1401.71 last 71.24) - open/close 44733 (time 1763407803.10 total 1411.72 last 62.32) - open/close 45496 (time 1763407813.10 total 1421.72 last 76.29) - open/close 46214 (time 1763407823.12 total 1431.73 last 71.71) - open/close 46902 (time 1763407833.13 total 1441.74 last 68.72) - open/close 47562 (time 1763407843.13 total 1451.75 last 65.99) - open/close 48257 (time 1763407853.13 total 1461.75 last 69.48) - open/close 48950 (time 1763407863.14 total 1471.76 last 69.23) - open/close 49453 (time 1763407873.18 total 1481.79 last 50.13) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1485.35 seconds: 33.41 ops/second - unlinked 0 (time 1763407884 ; total 0 ; last 0) - unlinked 10000 (time 1763408030 ; total 146 ; last 146) - unlinked 20000 (time 1763408182 ; total 298 ; last 152) - unlinked 30000 (time 1763408338 ; total 454 ; last 156) - unlinked 40000 (time 1763408494 ; total 610 ; last 156) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 758 seconds: 65.477570 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg105-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg105-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing unload_modules_local oleg105-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg105-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2