== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:16:39 (1753442199) start mds service on oleg341-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg341-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg341-server: detected 4 online CPUs by sysfs oleg341-server: MODOPTS_LIBCFS= oleg341-server: Force libcfs to create 2 CPU partitions oleg341-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg341-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg341-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 254 (time 1753442245.47 total 10.54 last 24.10) - open/close 449 (time 1753442256.03 total 21.10 last 18.46) - open/close 612 (time 1753442266.27 total 31.35 last 15.91) - open/close 932 (time 1753442276.32 total 41.39 last 31.86) - open/close 1233 (time 1753442287.27 total 52.34 last 27.49) - open/close 1450 (time 1753442297.77 total 62.84 last 20.67) - open/close 1623 (time 1753442308.76 total 73.84 last 15.73) - open/close 1934 (time 1753442318.77 total 83.84 last 31.09) - open/close 2231 (time 1753442328.77 total 93.84 last 29.69) - open/close 2381 (time 1753442338.77 total 103.85 last 14.99) - open/close 2667 (time 1753442349.66 total 114.73 last 26.27) - open/close 2999 (time 1753442359.77 total 124.85 last 32.82) - open/close 3302 (time 1753442369.79 total 134.86 last 30.26) - open/close 3590 (time 1753442380.77 total 145.84 last 26.22) - open/close 3648 (time 1753442390.89 total 155.97 last 5.73) - open/close 3751 (time 1753442401.64 total 166.71 last 9.58) - open/close 3853 (time 1753442414.06 total 179.13 last 8.22) - open/close 4114 (time 1753442425.24 total 190.32 last 23.33) - open/close 4397 (time 1753442436.19 total 201.26 last 25.86) - open/close 4683 (time 1753442446.81 total 211.89 last 26.92) - open/close 4994 (time 1753442456.82 total 221.89 last 31.08) - open/close 5263 (time 1753442466.82 total 231.89 last 26.89) - open/close 5477 (time 1753442477.16 total 242.23 last 20.71) - open/close 5765 (time 1753442487.90 total 252.97 last 26.81) - open/close 6009 (time 1753442497.90 total 262.97 last 24.40) - open/close 6217 (time 1753442507.93 total 273.00 last 20.73) - open/close 6465 (time 1753442518.10 total 283.18 last 24.37) - open/close 6750 (time 1753442528.73 total 293.81 last 26.81) - open/close 7050 (time 1753442539.42 total 304.49 last 28.07) - open/close 7332 (time 1753442549.78 total 314.86 last 27.21) - open/close 7593 (time 1753442560.02 total 325.10 last 25.49) - open/close 7879 (time 1753442570.79 total 335.86 last 26.58) - open/close 8128 (time 1753442580.79 total 345.86 last 24.89) - open/close 8406 (time 1753442590.87 total 355.95 last 27.57) - open/close 8602 (time 1753442601.50 total 366.58 last 18.44) - open/close 8837 (time 1753442611.80 total 376.87 last 22.83) - open/close 9146 (time 1753442622.83 total 387.90 last 28.02) - open/close 9397 (time 1753442633.24 total 398.32 last 24.10) - open/close 9685 (time 1753442643.61 total 408.68 last 27.78) - open/close 9953 (time 1753442653.62 total 418.69 last 26.79) - open/close 10000 (time 1753442655.92 total 420.99 last 20.40) - open/close 10188 (time 1753442666.33 total 431.41 last 18.05) - open/close 10462 (time 1753442676.95 total 442.03 last 25.80) - open/close 10680 (time 1753442687.32 total 452.39 last 21.03) - open/close 11950 (time 1753442697.32 total 462.40 last 126.95) - open/close 14029 (time 1753442707.33 total 472.40 last 207.79) - open/close 16423 (time 1753442717.33 total 482.40 last 239.37) - open/close 19116 (time 1753442727.33 total 492.40 last 269.25) - open/close 20000 (time 1753442731.23 total 496.30 last 226.60) - open/close 22407 (time 1753442741.23 total 506.31 last 240.64) - open/close 24286 (time 1753442751.24 total 516.31 last 187.85) - open/close 26100 (time 1753442761.24 total 526.31 last 181.35) - open/close 28237 (time 1753442771.25 total 536.32 last 213.56) - open/close 30000 (time 1753442779.19 total 544.26 last 222.04) - open/close 32160 (time 1753442789.19 total 554.26 last 215.96) - open/close 34241 (time 1753442799.19 total 564.26 last 208.03) - open/close 36105 (time 1753442809.19 total 574.27 last 186.39) - open/close 38288 (time 1753442819.21 total 584.28 last 217.97) - open/close 40000 (time 1753442826.12 total 591.20 last 247.54) - open/close 42343 (time 1753442836.12 total 601.20 last 234.29) - open/close 45112 (time 1753442846.12 total 611.20 last 276.89) - open/close 47716 (time 1753442856.13 total 621.20 last 260.35) total: 50000 open/close in 629.23 seconds: 79.46 ops/second - unlinked 0 (time 1753442865 ; total 0 ; last 0) - unlinked 10000 (time 1753442890 ; total 25 ; last 25) - unlinked 20000 (time 1753442914 ; total 49 ; last 24) - unlinked 30000 (time 1753442937 ; total 72 ; last 23) - unlinked 40000 (time 1753442960 ; total 95 ; last 23) total: 50000 unlinks in 121 seconds: 413.223145 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing unload_modules_local oleg341-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg341-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2