== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:21:13 (1767324073) start mds service on oleg335-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions oleg335-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg335-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 233 (time 1767324140.30 total 10.00 last 23.30) - open/close 461 (time 1767324151.03 total 20.73 last 21.25) - open/close 645 (time 1767324161.40 total 31.10 last 17.74) - open/close 974 (time 1767324172.09 total 41.79 last 30.78) - open/close 1189 (time 1767324182.27 total 51.97 last 21.11) - open/close 1482 (time 1767324192.82 total 62.52 last 27.78) - open/close 1776 (time 1767324203.25 total 72.95 last 28.18) - open/close 2096 (time 1767324214.26 total 83.96 last 29.06) - open/close 2413 (time 1767324224.82 total 94.52 last 30.03) - open/close 2682 (time 1767324235.70 total 105.40 last 24.71) - open/close 2876 (time 1767324246.07 total 115.77 last 18.72) - open/close 3104 (time 1767324256.07 total 125.77 last 22.79) - open/close 3430 (time 1767324266.49 total 136.19 last 31.30) - open/close 3730 (time 1767324277.18 total 146.88 last 28.06) - open/close 3959 (time 1767324287.92 total 157.62 last 21.31) - open/close 4178 (time 1767324297.93 total 167.63 last 21.89) - open/close 4355 (time 1767324308.28 total 177.98 last 17.10) - open/close 4573 (time 1767324318.39 total 188.09 last 21.58) - open/close 4849 (time 1767324329.27 total 198.97 last 25.36) - open/close 5166 (time 1767324339.28 total 208.98 last 31.67) - open/close 5446 (time 1767324349.56 total 219.26 last 27.23) - open/close 5727 (time 1767324360.19 total 229.89 last 26.44) - open/close 6037 (time 1767324370.55 total 240.25 last 29.92) - open/close 6343 (time 1767324381.18 total 250.88 last 28.81) - open/close 6650 (time 1767324391.68 total 261.38 last 29.23) - open/close 6855 (time 1767324401.79 total 271.48 last 20.28) - open/close 7187 (time 1767324412.66 total 282.36 last 30.52) - open/close 7462 (time 1767324423.22 total 292.92 last 26.05) - open/close 7710 (time 1767324433.34 total 303.04 last 24.51) - open/close 8006 (time 1767324444.34 total 314.04 last 26.91) - open/close 8215 (time 1767324454.84 total 324.54 last 19.91) - open/close 8371 (time 1767324465.21 total 334.91 last 15.05) - open/close 8600 (time 1767324476.02 total 345.72 last 21.17) - open/close 8736 (time 1767324486.07 total 355.77 last 13.54) - open/close 9013 (time 1767324497.08 total 366.78 last 25.14) - open/close 9266 (time 1767324507.96 total 377.66 last 23.27) - open/close 9352 (time 1767324518.97 total 388.67 last 7.81) - open/close 9649 (time 1767324529.14 total 398.84 last 29.19) - open/close 9858 (time 1767324539.15 total 408.85 last 20.88) - open/close 10000 (time 1767324543.47 total 413.17 last 32.92) - open/close 11607 (time 1767324553.47 total 423.17 last 160.61) - open/close 13457 (time 1767324563.47 total 433.17 last 185.00) - open/close 15620 (time 1767324573.48 total 443.18 last 216.28) - open/close 17153 (time 1767324583.48 total 453.18 last 153.25) - open/close 19500 (time 1767324593.48 total 463.18 last 234.63) - open/close 20000 (time 1767324596.26 total 465.96 last 180.27) - open/close 21731 (time 1767324606.26 total 475.96 last 173.07) - open/close 23725 (time 1767324616.26 total 485.96 last 199.35) - open/close 25533 (time 1767324626.26 total 495.96 last 180.76) - open/close 27792 (time 1767324636.26 total 505.96 last 225.83) - open/close 30000 (time 1767324645.79 total 515.49 last 231.71) - open/close 32381 (time 1767324655.80 total 525.50 last 237.87) - open/close 34387 (time 1767324665.81 total 535.51 last 200.51) - open/close 36424 (time 1767324675.81 total 545.51 last 203.68) - open/close 38556 (time 1767324685.82 total 555.52 last 213.01) - open/close 40000 (time 1767324694.32 total 564.02 last 169.87) - open/close 41635 (time 1767324704.33 total 574.03 last 163.29) - open/close 43910 (time 1767324714.33 total 584.03 last 227.44) - open/close 46245 (time 1767324724.33 total 594.03 last 233.49) - open/close 48538 (time 1767324734.34 total 604.04 last 229.29) total: 50000 open/close in 611.30 seconds: 81.79 ops/second - unlinked 0 (time 1767324743 ; total 0 ; last 0) - unlinked 10000 (time 1767324764 ; total 21 ; last 21) - unlinked 20000 (time 1767324785 ; total 42 ; last 21) - unlinked 30000 (time 1767324807 ; total 64 ; last 22) - unlinked 40000 (time 1767324829 ; total 86 ; last 22) total: 50000 unlinks in 106 seconds: 471.698120 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing unload_modules_local oleg335-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg335-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2