== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 05:54:53 (1774518893) Stopping clients: oleg141-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg141-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 oleg141-server: oleg141-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg141-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: detected 4 online CPUs by sysfs oleg141-server: Force libcfs to create 2 CPU partitions oleg141-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg141-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg141-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg141-server: detected 4 online CPUs by sysfs oleg141-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg141-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg141-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg141-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock oleg141-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 49667 inodes available. Want 100500. rc=0 - open/close 1199 (time 1774519000.10 total 10.01 last 119.82) - open/close 2410 (time 1774519010.10 total 20.01 last 121.03) - open/close 3673 (time 1774519020.11 total 30.02 last 126.20) - open/close 4859 (time 1774519030.11 total 40.02 last 118.58) - open/close 6144 (time 1774519040.11 total 50.02 last 128.49) - open/close 7383 (time 1774519050.12 total 60.03 last 123.83) - open/close 8619 (time 1774519060.12 total 70.03 last 123.54) - open/close 9785 (time 1774519070.12 total 80.03 last 116.57) total: 10000 open/close in 81.79 seconds: 122.27 ops/second - unlinked 0 (time 1774519073 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1384 (time 1774519121.71 total 10.00 last 138.35) - open/close 2735 (time 1774519131.72 total 20.01 last 135.01) - open/close 4287 (time 1774519141.72 total 30.02 last 155.11) - open/close 5740 (time 1774519151.73 total 40.02 last 145.27) - open/close 6937 (time 1774519161.74 total 50.03 last 119.57) - open/close 8122 (time 1774519171.74 total 60.03 last 118.49) - open/close 9385 (time 1774519181.74 total 70.04 last 126.22) total: 10000 open/close in 75.01 seconds: 133.31 ops/second - unlinked 0 (time 1774519189 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 1682 (time 1774519239.15 total 10.01 last 168.05) - open/close 2965 (time 1774519249.15 total 20.01 last 128.30) - open/close 4346 (time 1774519259.15 total 30.01 last 138.09) - open/close 5676 (time 1774519269.16 total 40.02 last 132.89) - open/close 6830 (time 1774519279.16 total 50.02 last 115.34) - open/close 8110 (time 1774519289.16 total 60.02 last 128.00) - open/close 9467 (time 1774519299.17 total 70.03 last 135.61) total: 10000 open/close in 73.44 seconds: 136.16 ops/second - unlinked 0 (time 1774519304 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1374 (time 1774519358.53 total 10.00 last 137.35) - open/close 2763 (time 1774519368.53 total 20.00 last 138.90) - open/close 4140 (time 1774519378.53 total 30.01 last 137.66) - open/close 5518 (time 1774519388.54 total 40.01 last 137.72) - open/close 6959 (time 1774519398.54 total 50.02 last 144.04) - open/close 8345 (time 1774519408.55 total 60.03 last 138.49) - open/close 9746 (time 1774519418.55 total 70.03 last 140.06) total: 10000 open/close in 71.54 seconds: 139.77 ops/second - unlinked 0 (time 1774519421 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1715 (time 1774519467.77 total 10.00 last 171.46) - open/close 3206 (time 1774519477.77 total 20.00 last 149.07) - open/close 4624 (time 1774519487.77 total 30.01 last 141.75) - open/close 6287 (time 1774519497.78 total 40.01 last 166.22) - open/close 7937 (time 1774519507.78 total 50.01 last 164.99) - open/close 9622 (time 1774519517.78 total 60.02 last 168.46) total: 10000 open/close in 62.57 seconds: 159.82 ops/second - unlinked 0 (time 1774519522 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1302 (time 1774519569.10 total 10.01 last 130.05) - open/close 2674 (time 1774519579.10 total 20.02 last 137.15) - open/close 3991 (time 1774519589.11 total 30.02 last 131.64) - open/close 5506 (time 1774519599.11 total 40.02 last 151.43) - open/close 6874 (time 1774519609.12 total 50.03 last 136.77) - open/close 8418 (time 1774519619.12 total 60.03 last 154.27) total: 10000 open/close in 68.18 seconds: 146.68 ops/second - unlinked 0 (time 1774519629 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1866 (time 1774519670.73 total 10.00 last 186.59) - open/close 3777 (time 1774519680.73 total 20.01 last 190.97) - open/close 5902 (time 1774519690.73 total 30.01 last 212.44) - open/close 7497 (time 1774519700.74 total 40.01 last 159.49) - open/close 9333 (time 1774519710.74 total 50.01 last 183.56) total: 10000 open/close in 53.12 seconds: 188.25 ops/second - unlinked 0 (time 1774519715 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2017 (time 1774519746.86 total 10.00 last 201.62) - open/close 3813 (time 1774519756.86 total 20.01 last 179.57) - open/close 5539 (time 1774519766.86 total 30.01 last 172.56) - open/close 7644 (time 1774519776.86 total 40.01 last 210.45) - open/close 9732 (time 1774519786.87 total 50.01 last 208.77) total: 10000 open/close in 51.34 seconds: 194.76 ops/second - unlinked 0 (time 1774519789 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 1798 (time 1774519820.84 total 10.01 last 179.70) - open/close 3642 (time 1774519830.84 total 20.01 last 184.33) - open/close 5358 (time 1774519840.84 total 30.01 last 171.57) - open/close 7004 (time 1774519850.84 total 40.01 last 164.59) - open/close 8875 (time 1774519860.85 total 50.01 last 187.05) total: 10000 open/close in 55.37 seconds: 180.61 ops/second - unlinked 0 (time 1774519867 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1689 (time 1774519908.62 total 10.00 last 168.87) - open/close 3469 (time 1774519918.63 total 20.01 last 177.91) - open/close 5220 (time 1774519928.64 total 30.01 last 174.96) - open/close 7600 (time 1774519938.64 total 40.02 last 237.93) total: 10000 open/close in 49.95 seconds: 200.20 ops/second - unlinked 0 (time 1774519950 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2090 (time 1774519987.83 total 10.00 last 208.98) - open/close 4139 (time 1774519997.83 total 20.00 last 204.89) - open/close 5788 (time 1774520007.83 total 30.00 last 164.86) - open/close 7489 (time 1774520017.84 total 40.01 last 169.96) - open/close 9407 (time 1774520027.84 total 50.01 last 191.80) total: 10000 open/close in 52.48 seconds: 190.57 ops/second - unlinked 0 (time 1774520031 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 4176 82416 5% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1504 85088 2% /mnt/lustre[MDT:1] lustre-OST0000_UUID 142216 1644 126572 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1644 126572 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109907 s, 9.5 MB/s fail_loc=0x80000204 Failing ost1 on oleg141-server Stopping /mnt/lustre-ost1 (opts:) on oleg141-server reboot facets: ost1 Failover ost1 to oleg141-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 100 open/close in 0.85 seconds: 117.51 ops/second umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg141-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local oleg141-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg141-server: LNET ready to unload oleg141-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.