== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:03:58 (1766415838) start mds service on oleg241-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg241-server: detected 4 online CPUs by sysfs oleg241-server: MODOPTS_LIBCFS= oleg241-server: Force libcfs to create 2 CPU partitions oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg241-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 281 (time 1766415890.61 total 10.47 last 26.85) - open/close 558 (time 1766415901.17 total 21.03 last 26.21) - open/close 874 (time 1766415911.18 total 31.04 last 31.58) - open/close 1104 (time 1766415922.41 total 42.27 last 20.48) - open/close 1407 (time 1766415933.11 total 52.97 last 28.32) - open/close 1671 (time 1766415944.11 total 63.96 last 24.00) - open/close 2000 (time 1766415955.00 total 74.86 last 30.19) - open/close 2323 (time 1766415965.54 total 85.40 last 30.64) - open/close 2609 (time 1766415976.23 total 96.09 last 26.76) - open/close 2799 (time 1766415986.73 total 106.59 last 18.10) - open/close 3066 (time 1766415996.74 total 116.59 last 26.69) - open/close 3328 (time 1766416007.35 total 127.21 last 24.69) - open/close 3638 (time 1766416018.35 total 138.21 last 28.18) - open/close 3848 (time 1766416028.85 total 148.71 last 20.00) - open/close 4180 (time 1766416039.34 total 159.20 last 31.64) - open/close 4479 (time 1766416049.96 total 169.82 last 28.14) - open/close 4691 (time 1766416060.52 total 180.38 last 20.08) - open/close 4933 (time 1766416070.53 total 190.39 last 24.19) - open/close 5188 (time 1766416081.39 total 201.25 last 23.48) - open/close 5382 (time 1766416091.63 total 211.49 last 18.95) - open/close 5688 (time 1766416103.47 total 223.33 last 25.85) - open/close 5876 (time 1766416113.64 total 233.50 last 18.48) - open/close 6287 (time 1766416123.69 total 243.55 last 40.90) - open/close 6574 (time 1766416134.70 total 254.56 last 26.07) - open/close 6766 (time 1766416145.20 total 265.06 last 18.29) - open/close 6956 (time 1766416155.63 total 275.49 last 18.22) - open/close 7212 (time 1766416165.99 total 285.85 last 24.69) - open/close 7411 (time 1766416178.35 total 298.21 last 16.10) - open/close 7626 (time 1766416188.35 total 308.21 last 21.50) - open/close 7881 (time 1766416198.51 total 318.37 last 25.11) - open/close 8155 (time 1766416208.69 total 328.55 last 26.91) - open/close 8409 (time 1766416218.93 total 338.78 last 24.81) - open/close 8955 (time 1766416229.87 total 349.73 last 49.89) - open/close 9175 (time 1766416240.56 total 360.42 last 20.58) - open/close 9398 (time 1766416251.37 total 371.23 last 20.63) - open/close 9613 (time 1766416261.87 total 381.73 last 20.47) - open/close 9786 (time 1766416272.04 total 391.90 last 17.02) - open/close 9992 (time 1766416282.41 total 402.27 last 19.87) - open/close 11903 (time 1766416292.41 total 412.27 last 191.05) - open/close 13554 (time 1766416302.41 total 422.27 last 165.09) - open/close 15502 (time 1766416312.41 total 432.27 last 194.78) - open/close 17324 (time 1766416322.41 total 442.27 last 182.19) - open/close 18951 (time 1766416332.42 total 452.28 last 162.64) - open/close 20000 (time 1766416337.23 total 457.09 last 218.05) - open/close 22470 (time 1766416347.23 total 467.09 last 246.98) - open/close 24585 (time 1766416357.23 total 477.09 last 211.48) - open/close 26904 (time 1766416367.23 total 487.09 last 231.83) - open/close 29259 (time 1766416377.23 total 497.09 last 235.50) - open/close 30000 (time 1766416380.90 total 500.76 last 201.83) - open/close 32001 (time 1766416390.91 total 510.77 last 200.05) - open/close 34235 (time 1766416400.91 total 520.77 last 223.34) - open/close 36339 (time 1766416410.91 total 530.77 last 210.35) - open/close 38659 (time 1766416420.91 total 540.77 last 231.97) - open/close 40000 (time 1766416426.95 total 546.81 last 222.04) - open/close 41844 (time 1766416436.96 total 556.82 last 184.33) - open/close 44044 (time 1766416446.96 total 566.82 last 219.96) - open/close 47139 (time 1766416456.96 total 576.82 last 309.46) - open/close 49838 (time 1766416466.96 total 586.82 last 269.83) total: 50000 open/close in 587.37 seconds: 85.12 ops/second - unlinked 0 (time 1766416469 ; total 0 ; last 0) - unlinked 10000 (time 1766416486 ; total 17 ; last 17) - unlinked 20000 (time 1766416505 ; total 36 ; last 19) - unlinked 30000 (time 1766416529 ; total 60 ; last 24) - unlinked 40000 (time 1766416550 ; total 81 ; last 21) total: 50000 unlinks in 99 seconds: 505.050507 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing unload_modules_local oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2