== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:53:08 (1756677188) start mds service on oleg230-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions oleg230-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg230-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg230-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 872 (time 1756677230.72 total 10.35 last 84.22) - open/close 1616 (time 1756677240.72 total 20.36 last 74.38) - open/close 2433 (time 1756677250.95 total 30.58 last 79.91) - open/close 3222 (time 1756677261.06 total 40.69 last 78.04) - open/close 3964 (time 1756677271.06 total 50.69 last 74.19) - open/close 4693 (time 1756677281.16 total 60.79 last 72.20) - open/close 5422 (time 1756677291.20 total 70.84 last 72.57) - open/close 6037 (time 1756677301.96 total 81.59 last 57.19) - open/close 6733 (time 1756677312.83 total 92.47 last 63.98) - open/close 7465 (time 1756677323.85 total 103.48 last 66.48) - open/close 8198 (time 1756677334.85 total 114.48 last 66.59) - open/close 9031 (time 1756677345.29 total 124.92 last 79.80) - open/close 9817 (time 1756677355.53 total 135.16 last 76.77) - open/close 10000 (time 1756677358.03 total 137.67 last 73.08) - open/close 10807 (time 1756677368.52 total 148.15 last 76.96) - open/close 11575 (time 1756677378.75 total 158.39 last 75.03) - open/close 12321 (time 1756677388.76 total 168.39 last 74.58) - open/close 12982 (time 1756677398.76 total 178.39 last 66.10) - open/close 13656 (time 1756677408.76 total 188.39 last 67.39) - open/close 14413 (time 1756677418.82 total 198.46 last 75.21) - open/close 15226 (time 1756677429.31 total 208.95 last 77.50) - open/close 15939 (time 1756677439.32 total 218.95 last 71.29) - open/close 16654 (time 1756677450.31 total 229.94 last 65.03) - open/close 17437 (time 1756677460.55 total 240.18 last 76.49) - open/close 18168 (time 1756677470.55 total 250.18 last 73.08) - open/close 18840 (time 1756677480.55 total 260.18 last 67.20) - open/close 19526 (time 1756677490.57 total 270.20 last 68.49) - open/close 20000 (time 1756677496.94 total 276.58 last 74.32) - open/close 20738 (time 1756677506.95 total 286.58 last 73.74) - open/close 21470 (time 1756677517.06 total 296.70 last 72.39) - open/close 22180 (time 1756677527.17 total 306.80 last 70.24) - open/close 23344 (time 1756677537.17 total 316.80 last 116.40) - open/close 26901 (time 1756677547.17 total 326.81 last 355.68) - open/close 30000 (time 1756677554.86 total 334.49 last 403.01) - open/close 34363 (time 1756677564.86 total 344.50 last 436.27) - open/close 37757 (time 1756677574.86 total 354.50 last 339.37) - open/close 40000 (time 1756677580.32 total 359.95 last 411.33) - open/close 44060 (time 1756677590.32 total 369.95 last 405.95) - open/close 47848 (time 1756677600.32 total 379.95 last 378.80) total: 50000 open/close in 385.69 seconds: 129.64 ops/second - unlinked 0 (time 1756677607 ; total 0 ; last 0) - unlinked 10000 (time 1756677622 ; total 15 ; last 15) - unlinked 20000 (time 1756677637 ; total 30 ; last 15) - unlinked 30000 (time 1756677653 ; total 46 ; last 16) - unlinked 40000 (time 1756677665 ; total 58 ; last 12) total: 50000 unlinks in 72 seconds: 694.444458 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg230-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg230-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg230-server stop mds service on oleg230-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg230-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing unload_modules_local oleg230-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg230-client: oleg230-client: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-client: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2