== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:02:37 (1743501757) start mds service on oleg607-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg607-server' oleg607-server: oleg607-server.virtnet: executing load_modules_local oleg607-server: Loading modules from /home/green/git/lustre-release/lustre oleg607-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg607-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg607-server: detected 4 online CPUs by sysfs oleg607-server: MODOPTS_LIBCFS= oleg607-server: Force libcfs to create 2 CPU partitions oleg607-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg607-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg607-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg607-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Started lustre-OST0000 oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg607-client.virtnet: -o user_xattr,flock 192.168.206.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 249 (time 1743501841.06 total 10.09 last 24.67) - open/close 452 (time 1743501851.88 total 20.91 last 18.77) - open/close 660 (time 1743501862.76 total 31.79 last 19.11) - open/close 835 (time 1743501873.32 total 42.35 last 16.57) - open/close 1053 (time 1743501884.20 total 53.23 last 20.04) - open/close 1224 (time 1743501894.70 total 63.74 last 16.28) - open/close 1323 (time 1743501906.73 total 75.76 last 8.23) - open/close 1512 (time 1743501917.29 total 86.32 last 17.90) - open/close 1632 (time 1743501927.40 total 96.43 last 11.87) - open/close 1755 (time 1743501937.51 total 106.54 last 12.17) - open/close 1915 (time 1743501949.99 total 119.02 last 12.82) - open/close 2136 (time 1743501962.54 total 131.57 last 17.62) - open/close 2336 (time 1743501973.29 total 142.32 last 18.60) - open/close 2534 (time 1743501984.17 total 153.20 last 18.20) - open/close 2776 (time 1743501994.21 total 163.25 last 24.09) - open/close 2920 (time 1743502004.64 total 173.68 last 13.81) - open/close 3070 (time 1743502015.08 total 184.11 last 14.37) - open/close 3277 (time 1743502025.08 total 194.12 last 20.69) - open/close 3478 (time 1743502036.14 total 205.17 last 18.18) - open/close 3629 (time 1743502046.51 total 215.54 last 14.56) - open/close 3804 (time 1743502057.33 total 226.36 last 16.17) - open/close 4022 (time 1743502068.20 total 237.23 last 20.05) - open/close 4196 (time 1743502078.89 total 247.92 last 16.28) - open/close 4416 (time 1743502089.89 total 258.93 last 19.99) - open/close 4523 (time 1743502099.94 total 268.97 last 10.65) - open/close 4678 (time 1743502110.19 total 279.22 last 15.12) - open/close 4834 (time 1743502120.62 total 289.65 last 14.95) - open/close 4975 (time 1743502131.11 total 300.14 last 13.44) - open/close 5136 (time 1743502141.67 total 310.70 last 15.24) - open/close 5319 (time 1743502152.49 total 321.52 last 16.92) - open/close 5544 (time 1743502163.50 total 332.53 last 20.44) - open/close 5744 (time 1743502174.38 total 343.41 last 18.38) - open/close 5986 (time 1743502184.38 total 353.41 last 24.19) - open/close 6140 (time 1743502194.80 total 363.83 last 14.78) - open/close 6272 (time 1743502205.16 total 374.19 last 12.74) - open/close 6423 (time 1743502215.59 total 384.63 last 14.47) - open/close 6570 (time 1743502225.96 total 394.99 last 14.18) - open/close 6721 (time 1743502236.45 total 405.49 last 14.39) - open/close 6962 (time 1743502246.76 total 415.79 last 23.39) - open/close 7125 (time 1743502257.34 total 426.37 last 15.41) - open/close 7310 (time 1743502268.07 total 437.10 last 17.23) - open/close 7550 (time 1743502278.18 total 447.21 last 23.74) - open/close 7774 (time 1743502288.19 total 457.22 last 22.38) - open/close 7999 (time 1743502298.34 total 467.37 last 22.17) - open/close 8238 (time 1743502308.58 total 477.61 last 23.34) - open/close 8443 (time 1743502318.58 total 487.62 last 20.49) - open/close 8667 (time 1743502329.57 total 498.61 last 20.38) - open/close 8842 (time 1743502340.07 total 509.10 last 16.67) - open/close 8975 (time 1743502350.38 total 519.41 last 12.91) - open/close 9189 (time 1743502360.42 total 529.46 last 21.30) - open/close 9443 (time 1743502370.73 total 539.76 last 24.65) - open/close 9656 (time 1743502380.84 total 549.88 last 21.06) - open/close 9883 (time 1743502391.85 total 560.88 last 20.63) - open/close 10000 (time 1743502395.89 total 564.92 last 28.92) - open/close 10204 (time 1743502406.76 total 575.79 last 18.77) - open/close 10372 (time 1743502417.26 total 586.29 last 16.00) - open/close 11031 (time 1743502427.27 total 596.30 last 65.84) - open/close 12400 (time 1743502437.27 total 606.31 last 136.86) - open/close 14013 (time 1743502447.28 total 616.31 last 161.24) - open/close 15531 (time 1743502457.28 total 626.31 last 151.75) - open/close 17197 (time 1743502467.28 total 636.32 last 166.53) - open/close 18743 (time 1743502477.29 total 646.32 last 154.56) - open/close 20000 (time 1743502484.63 total 653.67 last 171.08) - open/close 21401 (time 1743502494.64 total 663.67 last 140.03) - open/close 23080 (time 1743502504.65 total 673.68 last 167.81) - open/close 24718 (time 1743502514.65 total 683.68 last 163.76) - open/close 26454 (time 1743502524.65 total 693.68 last 173.52) - open/close 28369 (time 1743502534.65 total 703.69 last 191.49) - open/close 29873 (time 1743502544.66 total 713.69 last 150.31) - open/close 31433 (time 1743502554.66 total 723.70 last 155.90) - open/close 32868 (time 1743502564.67 total 733.70 last 143.48) - open/close 34533 (time 1743502574.67 total 743.70 last 166.43) - open/close 36462 (time 1743502584.67 total 753.71 last 192.85) - open/close 38372 (time 1743502594.67 total 763.71 last 190.97) - open/close 40000 (time 1743502604.31 total 773.35 last 168.90) - open/close 41511 (time 1743502614.31 total 783.35 last 151.09) - open/close 43151 (time 1743502624.32 total 793.35 last 163.92) - open/close 45057 (time 1743502634.32 total 803.35 last 190.55) - open/close 46967 (time 1743502644.32 total 813.36 last 190.99) - open/close 48717 (time 1743502654.33 total 823.36 last 174.93) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 828.60 seconds: 59.90 ops/second - unlinked 0 (time 1743502663 ; total 0 ; last 0) - unlinked 10000 (time 1743502705 ; total 42 ; last 42) - unlinked 20000 (time 1743502735 ; total 72 ; last 30) - unlinked 30000 (time 1743502767 ; total 104 ; last 32) - unlinked 40000 (time 1743502797 ; total 134 ; last 30) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 162 seconds: 306.370361 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg607-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg607-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg607-server stop mds service on oleg607-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg607-server stop mds service on oleg607-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg607-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg607-server' oleg607-server: oleg607-server.virtnet: executing unload_modules_local oleg607-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg607-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg607-client: oleg607-client: ssh exited with exit code 2 pdsh@oleg607-client: oleg607-server: ssh exited with exit code 2 pdsh@oleg607-client: oleg607-server: ssh exited with exit code 2 pdsh@oleg607-client: oleg607-client: ssh exited with exit code 2