== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:24:32 (1752690272) start mds service on oleg424-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions oleg424-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg424-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg424-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg424-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 310 (time 1752690331.78 total 10.91 last 28.42) - open/close 838 (time 1752690341.78 total 20.91 last 52.79) - open/close 1254 (time 1752690352.52 total 31.65 last 38.73) - open/close 1313 (time 1752690363.07 total 42.20 last 5.59) - open/close 1707 (time 1752690373.83 total 52.96 last 36.62) - open/close 2210 (time 1752690385.79 total 64.92 last 42.05) - open/close 2880 (time 1752690396.09 total 75.23 last 65.03) - open/close 3679 (time 1752690406.33 total 85.47 last 78.03) - open/close 4402 (time 1752690416.77 total 95.90 last 69.29) - open/close 4871 (time 1752690427.71 total 106.84 last 42.85) - open/close 5097 (time 1752690437.76 total 116.89 last 22.49) - open/close 5579 (time 1752690449.73 total 128.86 last 40.28) - open/close 5730 (time 1752690460.67 total 139.80 last 13.80) - open/close 6178 (time 1752690471.42 total 150.56 last 41.66) - open/close 6601 (time 1752690483.90 total 163.03 last 33.90) - open/close 7422 (time 1752690494.27 total 173.40 last 79.18) - open/close 8117 (time 1752690504.27 total 183.41 last 69.49) - open/close 8769 (time 1752690515.14 total 194.27 last 60.03) - open/close 9501 (time 1752690526.01 total 205.15 last 67.29) - open/close 10000 (time 1752690532.50 total 211.63 last 76.98) - open/close 10688 (time 1752690543.30 total 222.43 last 63.70) - open/close 11409 (time 1752690553.30 total 232.43 last 72.10) - open/close 12041 (time 1752690563.33 total 242.46 last 62.99) - open/close 12805 (time 1752690573.44 total 252.57 last 75.57) - open/close 13573 (time 1752690583.44 total 262.57 last 76.79) - open/close 14137 (time 1752690593.92 total 273.05 last 53.84) - open/close 14762 (time 1752690604.61 total 283.74 last 58.46) - open/close 15474 (time 1752690614.61 total 293.74 last 71.19) - open/close 16112 (time 1752690625.28 total 304.41 last 59.77) - open/close 16820 (time 1752690636.22 total 315.35 last 64.72) - open/close 18540 (time 1752690646.22 total 325.36 last 171.98) - open/close 20000 (time 1752690649.20 total 328.34 last 489.88) - open/close 24274 (time 1752690659.21 total 338.34 last 427.33) - open/close 29252 (time 1752690669.21 total 348.34 last 497.79) - open/close 33635 (time 1752690679.21 total 358.34 last 438.24) - open/close 37834 (time 1752690689.21 total 368.34 last 419.79) - open/close 40000 (time 1752690694.09 total 373.22 last 444.25) - open/close 44176 (time 1752690704.09 total 383.22 last 417.59) - open/close 47439 (time 1752690714.09 total 393.22 last 326.12) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 401.22 seconds: 123.70 ops/second - unlinked 0 (time 1752690723 ; total 0 ; last 0) - unlinked 10000 (time 1752690746 ; total 23 ; last 23) - unlinked 20000 (time 1752690772 ; total 49 ; last 26) - unlinked 30000 (time 1752690795 ; total 72 ; last 23) - unlinked 40000 (time 1752690819 ; total 96 ; last 24) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 117 seconds: 424.205139 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg424-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local oleg424-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg424-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2