== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:13:15 (1786482795) start mds service on oleg213-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg213-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg213-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg213-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 100 (time 1786482858.05 total 10.62 last 9.42) - open/close 231 (time 1786482868.68 total 21.25 last 12.33) - open/close 407 (time 1786482878.91 total 31.48 last 17.20) - open/close 593 (time 1786482888.91 total 41.48 last 18.59) - open/close 728 (time 1786482898.91 total 51.49 last 13.49) - open/close 918 (time 1786482909.37 total 61.94 last 18.18) - open/close 1045 (time 1786482919.38 total 71.96 last 12.68) - open/close 1180 (time 1786482929.39 total 81.96 last 13.50) - open/close 1346 (time 1786482939.39 total 91.97 last 16.59) - open/close 1477 (time 1786482949.39 total 101.97 last 13.10) - open/close 1668 (time 1786482959.80 total 112.37 last 18.36) - open/close 1830 (time 1786482969.97 total 122.54 last 15.92) - open/close 1997 (time 1786482980.02 total 132.59 last 16.62) - open/close 2066 (time 1786482990.26 total 142.83 last 6.74) - open/close 2211 (time 1786483000.27 total 152.84 last 14.49) - open/close 2438 (time 1786483010.36 total 162.93 last 22.48) - open/close 2642 (time 1786483021.11 total 173.68 last 18.98) - open/close 2774 (time 1786483032.06 total 184.63 last 12.06) - open/close 2878 (time 1786483042.62 total 195.19 last 9.85) - open/close 3013 (time 1786483052.62 total 205.20 last 13.49) - open/close 3114 (time 1786483062.98 total 215.55 last 9.75) - open/close 3193 (time 1786483075.33 total 227.91 last 6.39) - open/close 3368 (time 1786483085.44 total 238.01 last 17.31) - open/close 3492 (time 1786483095.93 total 248.50 last 11.83) - open/close 3830 (time 1786483106.55 total 259.12 last 31.82) - open/close 4123 (time 1786483116.67 total 269.24 last 28.96) - open/close 4257 (time 1786483127.09 total 279.67 last 12.85) - open/close 4445 (time 1786483137.10 total 289.67 last 18.79) - open/close 4718 (time 1786483147.63 total 300.20 last 25.93) - open/close 5013 (time 1786483158.51 total 311.09 last 27.11) - open/close 5304 (time 1786483168.52 total 321.09 last 29.08) - open/close 5574 (time 1786483178.86 total 331.44 last 26.10) - open/close 5841 (time 1786483188.86 total 341.44 last 26.70) - open/close 6083 (time 1786483199.09 total 351.66 last 23.67) - open/close 6294 (time 1786483209.78 total 362.36 last 19.73) - open/close 6493 (time 1786483220.73 total 373.30 last 18.18) - open/close 6740 (time 1786483231.30 total 383.87 last 23.37) - open/close 7118 (time 1786483241.98 total 394.55 last 35.38) - open/close 7288 (time 1786483252.03 total 404.61 last 16.91) - open/close 7513 (time 1786483262.33 total 414.90 last 21.85) - open/close 7679 (time 1786483272.33 total 424.91 last 16.59) - open/close 7970 (time 1786483282.42 total 434.99 last 28.85) - open/close 8261 (time 1786483293.44 total 446.01 last 26.41) - open/close 8505 (time 1786483303.74 total 456.31 last 23.69) - open/close 8845 (time 1786483313.74 total 466.32 last 33.99) - open/close 9077 (time 1786483323.89 total 476.46 last 22.87) - open/close 9343 (time 1786483334.34 total 486.91 last 25.46) - open/close 9572 (time 1786483344.34 total 496.91 last 22.90) - open/close 9772 (time 1786483354.87 total 507.44 last 18.99) - open/close 10000 (time 1786483357.28 total 509.85 last 94.70) - open/close 11554 (time 1786483367.28 total 519.86 last 155.31) - open/close 12838 (time 1786483377.29 total 529.87 last 128.29) - open/close 14373 (time 1786483387.29 total 539.87 last 153.47) - open/close 15917 (time 1786483397.30 total 549.87 last 154.31) - open/close 17323 (time 1786483407.30 total 559.87 last 140.58) - open/close 18820 (time 1786483417.30 total 569.88 last 149.67) - open/close 20000 (time 1786483423.25 total 575.82 last 198.57) - open/close 21671 (time 1786483433.25 total 585.82 last 167.01) - open/close 23595 (time 1786483443.25 total 595.83 last 192.39) - open/close 25553 (time 1786483453.25 total 605.83 last 195.78) - open/close 27389 (time 1786483463.25 total 615.83 last 183.59) - open/close 28843 (time 1786483473.26 total 625.83 last 145.31) - open/close 30000 (time 1786483481.49 total 634.07 last 140.50) - open/close 31435 (time 1786483491.50 total 644.07 last 143.48) - open/close 33609 (time 1786483501.50 total 654.07 last 217.39) - open/close 35787 (time 1786483511.50 total 664.07 last 217.73) - open/close 37834 (time 1786483521.50 total 674.07 last 204.68) - open/close 39749 (time 1786483531.51 total 684.08 last 191.29) - open/close 41623 (time 1786483541.51 total 694.09 last 187.37) - open/close 44067 (time 1786483551.51 total 704.09 last 244.37) - open/close 45945 (time 1786483561.52 total 714.09 last 187.79) - open/close 47840 (time 1786483571.52 total 724.09 last 189.38) - open/close 49478 (time 1786483581.53 total 734.10 last 163.67) total: 50000 open/close in 736.76 seconds: 67.87 ops/second - unlinked 0 (time 1786483585 ; total 0 ; last 0) - unlinked 10000 (time 1786483624 ; total 39 ; last 39) - unlinked 20000 (time 1786483652 ; total 67 ; last 28) - unlinked 30000 (time 1786483677 ; total 92 ; last 25) - unlinked 40000 (time 1786483701 ; total 116 ; last 24) total: 50000 unlinks in 138 seconds: 362.318848 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg213-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg213-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg213-server unloading modules via unload_modules_local on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing unload_modules_local oleg213-server: modules unloaded. pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2