== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:57:28 (1752512248) start mds service on oleg334-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg334-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions oleg334-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg334-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg334-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg334-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 309 (time 1752512307.69 total 10.78 last 28.65) - open/close 660 (time 1752512318.31 total 21.41 last 33.04) - open/close 1285 (time 1752512328.42 total 31.52 last 61.81) - open/close 1670 (time 1752512339.12 total 42.22 last 35.99) - open/close 1884 (time 1752512349.75 total 52.84 last 20.14) - open/close 2092 (time 1752512361.27 total 64.36 last 18.06) - open/close 2398 (time 1752512371.95 total 75.04 last 28.65) - open/close 2613 (time 1752512382.19 total 85.28 last 20.99) - open/close 2715 (time 1752512394.09 total 97.19 last 8.57) - open/close 3041 (time 1752512404.52 total 107.62 last 31.26) - open/close 3453 (time 1752512416.49 total 119.59 last 34.42) - open/close 3833 (time 1752512427.31 total 130.40 last 35.13) - open/close 4399 (time 1752512437.31 total 140.41 last 56.58) - open/close 4571 (time 1752512448.75 total 151.84 last 15.04) - open/close 4746 (time 1752512458.81 total 161.91 last 17.39) - open/close 4979 (time 1752512468.82 total 171.92 last 23.28) - open/close 5183 (time 1752512478.99 total 182.08 last 20.06) - open/close 5453 (time 1752512489.97 total 193.06 last 24.60) - open/close 5651 (time 1752512500.39 total 203.49 last 18.99) - open/close 5817 (time 1752512510.83 total 213.93 last 15.90) - open/close 6011 (time 1752512521.26 total 224.36 last 18.61) - open/close 6273 (time 1752512531.26 total 234.36 last 26.20) - open/close 6518 (time 1752512542.19 total 245.28 last 22.42) - open/close 6761 (time 1752512552.88 total 255.97 last 22.73) - open/close 6916 (time 1752512563.18 total 266.27 last 15.05) - open/close 7237 (time 1752512573.48 total 276.58 last 31.15) - open/close 7548 (time 1752512586.42 total 289.51 last 24.05) - open/close 7864 (time 1752512596.78 total 299.88 last 30.48) - open/close 8091 (time 1752512607.53 total 310.63 last 21.12) - open/close 8255 (time 1752512617.64 total 320.74 last 16.22) - open/close 8849 (time 1752512627.82 total 330.91 last 58.39) - open/close 9386 (time 1752512637.86 total 340.96 last 53.44) - open/close 9742 (time 1752512648.49 total 351.58 last 33.50) - open/close 10000 (time 1752512653.58 total 356.68 last 50.65) - open/close 10136 (time 1752512664.55 total 367.65 last 12.40) - open/close 10374 (time 1752512674.99 total 378.08 last 22.81) - open/close 10773 (time 1752512684.99 total 388.09 last 39.88) - open/close 13001 (time 1752512695.00 total 398.09 last 222.76) - open/close 15042 (time 1752512705.00 total 408.09 last 204.06) - open/close 17145 (time 1752512715.00 total 418.09 last 210.25) - open/close 19693 (time 1752512725.01 total 428.10 last 254.53) - open/close 21817 (time 1752512735.01 total 438.11 last 212.35) - open/close 23677 (time 1752512745.01 total 448.11 last 185.99) - open/close 25817 (time 1752512755.02 total 458.11 last 213.94) - open/close 29088 (time 1752512765.02 total 468.11 last 327.05) - open/close 30000 (time 1752512767.76 total 470.86 last 332.04) - open/close 32595 (time 1752512777.77 total 480.86 last 259.41) - open/close 35730 (time 1752512787.77 total 490.86 last 313.49) - open/close 38936 (time 1752512797.77 total 500.86 last 320.56) - open/close 40000 (time 1752512800.45 total 503.55 last 396.18) - open/close 43756 (time 1752512810.46 total 513.55 last 375.56) - open/close 47742 (time 1752512820.46 total 523.55 last 398.59) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 527.90 seconds: 94.02 ops/second - unlinked 0 (time 1752512826 ; total 0 ; last 0) - unlinked 10000 (time 1752512839 ; total 13 ; last 13) - unlinked 20000 (time 1752512851 ; total 25 ; last 12) - unlinked 30000 (time 1752512864 ; total 38 ; last 13) - unlinked 40000 (time 1752512876 ; total 50 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 61 seconds: 813.639343 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg334-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing unload_modules_local oleg334-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg334-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2