== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:48:45 (1761248925) start mds service on oleg419-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg419-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg419-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg419-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg419-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 95 (time 1761249003.88 total 10.10 last 9.41) - open/close 318 (time 1761249014.20 total 20.42 last 21.59) - open/close 551 (time 1761249024.98 total 31.20 last 21.62) - open/close 663 (time 1761249036.04 total 42.26 last 10.13) - open/close 806 (time 1761249046.25 total 52.47 last 14.00) - open/close 1040 (time 1761249056.91 total 63.13 last 21.96) - open/close 1188 (time 1761249067.21 total 73.43 last 14.37) - open/close 1312 (time 1761249077.97 total 84.19 last 11.53) - open/close 1442 (time 1761249088.99 total 95.21 last 11.79) - open/close 1627 (time 1761249099.01 total 105.23 last 18.47) - open/close 1835 (time 1761249109.39 total 115.61 last 20.04) - open/close 2095 (time 1761249119.72 total 125.94 last 25.17) - open/close 2275 (time 1761249130.44 total 136.67 last 16.78) - open/close 2408 (time 1761249140.88 total 147.11 last 12.74) - open/close 2631 (time 1761249151.31 total 157.53 last 21.39) - open/close 2886 (time 1761249162.32 total 168.54 last 23.17) - open/close 3090 (time 1761249172.69 total 178.91 last 19.67) - open/close 3333 (time 1761249182.69 total 188.91 last 24.29) - open/close 3524 (time 1761249192.78 total 199.00 last 18.93) - open/close 3656 (time 1761249202.79 total 209.01 last 13.19) - open/close 3784 (time 1761249212.82 total 219.04 last 12.77) - open/close 3958 (time 1761249223.58 total 229.80 last 16.17) - open/close 4078 (time 1761249234.33 total 240.55 last 11.16) - open/close 4214 (time 1761249244.33 total 250.55 last 13.60) - open/close 4394 (time 1761249254.38 total 260.61 last 17.91) - open/close 4595 (time 1761249264.65 total 270.88 last 19.57) - open/close 4759 (time 1761249274.96 total 281.19 last 15.90) - open/close 4893 (time 1761249286.04 total 292.27 last 12.10) - open/close 5099 (time 1761249296.27 total 302.49 last 20.15) - open/close 5286 (time 1761249306.52 total 312.74 last 18.25) - open/close 5480 (time 1761249316.69 total 322.91 last 19.07) - open/close 5678 (time 1761249326.86 total 333.09 last 19.46) - open/close 5863 (time 1761249336.91 total 343.13 last 18.42) - open/close 6067 (time 1761249346.91 total 353.13 last 20.39) - open/close 6205 (time 1761249357.15 total 363.37 last 13.48) - open/close 6346 (time 1761249367.20 total 373.42 last 14.03) - open/close 6515 (time 1761249377.81 total 384.03 last 15.93) - open/close 6743 (time 1761249387.81 total 394.03 last 22.79) - open/close 6895 (time 1761249398.29 total 404.52 last 14.50) - open/close 7116 (time 1761249408.92 total 415.15 last 20.79) - open/close 7345 (time 1761249419.92 total 426.14 last 20.83) - open/close 7587 (time 1761249430.17 total 436.40 last 23.60) - open/close 7829 (time 1761249440.19 total 446.41 last 24.16) - open/close 8009 (time 1761249451.16 total 457.39 last 16.40) - open/close 8241 (time 1761249461.17 total 467.40 last 23.17) - open/close 8488 (time 1761249471.20 total 477.42 last 24.63) - open/close 8720 (time 1761249481.21 total 487.43 last 23.18) - open/close 8924 (time 1761249491.73 total 497.96 last 19.38) - open/close 9115 (time 1761249502.74 total 508.96 last 17.35) - open/close 9334 (time 1761249513.17 total 519.39 last 21.00) - open/close 9557 (time 1761249523.18 total 529.40 last 22.28) - open/close 9802 (time 1761249533.20 total 539.42 last 24.45) - open/close 10000 (time 1761249542.48 total 548.71 last 21.32) - open/close 10148 (time 1761249552.73 total 558.96 last 14.44) - open/close 10383 (time 1761249563.54 total 569.76 last 21.75) - open/close 10548 (time 1761249574.12 total 580.34 last 15.60) - open/close 10777 (time 1761249584.91 total 591.14 last 21.21) - open/close 10928 (time 1761249595.22 total 601.45 last 14.65) - open/close 11152 (time 1761249605.64 total 611.86 last 21.51) - open/close 11369 (time 1761249615.65 total 621.87 last 21.67) - open/close 11597 (time 1761249625.67 total 631.89 last 22.76) - open/close 11856 (time 1761249635.67 total 641.89 last 25.90) - open/close 12082 (time 1761249645.81 total 652.03 last 22.30) - open/close 12374 (time 1761249656.87 total 663.09 last 26.40) - open/close 12599 (time 1761249666.87 total 673.09 last 22.49) - open/close 12831 (time 1761249676.88 total 683.10 last 23.18) - open/close 13064 (time 1761249686.95 total 693.17 last 23.14) - open/close 13328 (time 1761249696.98 total 703.20 last 26.32) - open/close 13567 (time 1761249707.58 total 713.80 last 22.55) - open/close 13806 (time 1761249717.73 total 723.95 last 23.55) - open/close 14496 (time 1761249727.74 total 733.96 last 68.92) - open/close 15282 (time 1761249737.74 total 743.97 last 78.59) - open/close 16185 (time 1761249747.75 total 753.97 last 90.27) - open/close 17131 (time 1761249757.75 total 763.97 last 94.58) - open/close 18023 (time 1761249767.75 total 773.97 last 89.17) - open/close 18832 (time 1761249777.76 total 783.98 last 80.85) - open/close 19721 (time 1761249787.76 total 793.98 last 88.87) - open/close 20000 (time 1761249790.92 total 797.14 last 88.28) - open/close 20848 (time 1761249800.94 total 807.17 last 84.61) - open/close 21740 (time 1761249810.97 total 817.19 last 89.00) - open/close 22613 (time 1761249820.98 total 827.20 last 87.17) - open/close 23499 (time 1761249830.99 total 837.21 last 88.53) - open/close 24364 (time 1761249840.99 total 847.21 last 86.49) - open/close 25192 (time 1761249850.99 total 857.21 last 82.78) - open/close 26062 (time 1761249861.00 total 867.22 last 86.96) - open/close 26961 (time 1761249871.02 total 877.24 last 89.73) - open/close 27748 (time 1761249881.02 total 887.25 last 78.64) - open/close 28664 (time 1761249891.02 total 897.25 last 91.60) - open/close 29591 (time 1761249901.03 total 907.25 last 92.68) - open/close 30000 (time 1761249905.66 total 911.89 last 88.18) - open/close 30831 (time 1761249915.68 total 921.90 last 82.98) - open/close 31758 (time 1761249925.69 total 931.91 last 92.63) - open/close 32718 (time 1761249935.69 total 941.91 last 95.97) - open/close 33608 (time 1761249945.69 total 951.92 last 88.94) - open/close 34584 (time 1761249955.70 total 961.92 last 97.59) - open/close 35608 (time 1761249965.70 total 971.92 last 102.33) - open/close 36681 (time 1761249975.70 total 981.93 last 107.29) - open/close 37636 (time 1761249985.71 total 991.94 last 95.40) - open/close 38544 (time 1761249995.73 total 1001.95 last 90.65) - open/close 39465 (time 1761250005.73 total 1011.96 last 92.08) - open/close 40000 (time 1761250011.32 total 1017.55 last 95.70) - open/close 40961 (time 1761250021.33 total 1027.56 last 95.99) - open/close 41976 (time 1761250031.34 total 1037.56 last 101.49) - open/close 42916 (time 1761250041.34 total 1047.56 last 93.99) - open/close 43870 (time 1761250051.34 total 1057.56 last 95.36) - open/close 44803 (time 1761250061.34 total 1067.56 last 93.29) - open/close 45735 (time 1761250071.35 total 1077.57 last 93.15) - open/close 46621 (time 1761250081.35 total 1087.57 last 88.54) - open/close 47527 (time 1761250091.36 total 1097.58 last 90.55) - open/close 48315 (time 1761250101.36 total 1107.58 last 78.77) - open/close 49215 (time 1761250111.36 total 1117.59 last 89.97) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1122.47 seconds: 44.22 ops/second - unlinked 0 (time 1761250120 ; total 0 ; last 0) - unlinked 10000 (time 1761250232 ; total 112 ; last 112) - unlinked 20000 (time 1761250307 ; total 187 ; last 75) - unlinked 30000 (time 1761250369 ; total 249 ; last 62) - unlinked 40000 (time 1761250434 ; total 314 ; last 65) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 377 seconds: 131.649872 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg419-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local oleg419-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg419-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2