== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:21:34 (1767320494)
Stopping clients: oleg227-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg227-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg227-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server
pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1
pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1
Stopping /mnt/lustre-ost2 (opts:-f) on oleg227-server
oleg227-server: oleg227-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg227-server'
oleg227-server: oleg227-server.virtnet: executing load_modules_local
oleg227-server: Loading modules from /home/green/git/lustre-release/lustre
oleg227-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg227-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg227-server: detected 4 online CPUs by sysfs
oleg227-server: MODOPTS_LIBCFS=
oleg227-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/vdc
Format mds2: /dev/vdd
Format ost1: /dev/vde
Format ost2: /dev/vdf
start mds service on oleg227-server
pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all
pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1
Commit the device label on /dev/vdc
Started lustre-MDT0000
start mds service on oleg227-server
pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all
pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1
Commit the device label on /dev/vdd
Started lustre-MDT0001
oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg227-server
pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all
pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1
Commit the device label on /dev/vde
Started lustre-OST0000
oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg227-client.virtnet:  -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 771 (time 1767320773.93 total 10.00 last 77.09)
 - open/close 1454 (time 1767320783.93 total 20.01 last 68.26)
 - open/close 2195 (time 1767320793.93 total 30.01 last 74.08)
 - open/close 2966 (time 1767320803.96 total 40.03 last 76.93)
 - open/close 3727 (time 1767320813.98 total 50.06 last 75.91)
 - open/close 4478 (time 1767320823.99 total 60.06 last 75.05)
total: 4500 open/close in 60.36 seconds: 74.55 ops/second
 - /unlink 1510 (time 1767320838.77 total 10.00 last 150.98)
 - /unlink 2919 (time 1767320848.78 total 20.01 last 140.79)
 - /unlink 4288 (time 1767320858.79 total 30.02 last 136.74)
total: 4500 /unlink in 31.81 seconds: 141.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 757 (time 1767320874.92 total 10.02 last 75.54)
 - open/close 1495 (time 1767320884.93 total 20.03 last 73.75)
 - open/close 2212 (time 1767320894.93 total 30.04 last 71.65)
 - open/close 3008 (time 1767320904.94 total 40.05 last 79.52)
 - open/close 3782 (time 1767320914.96 total 50.06 last 77.31)
total: 4500 open/close in 59.72 seconds: 75.35 ops/second
 - /unlink 1429 (time 1767320939.14 total 10.00 last 142.84)
 - /unlink 2839 (time 1767320949.14 total 20.01 last 140.95)
 - /unlink 4327 (time 1767320959.14 total 30.01 last 148.75)
total: 4500 /unlink in 31.31 seconds: 143.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 699 (time 1767320975.08 total 10.00 last 69.89)
 - open/close 1483 (time 1767320985.08 total 20.00 last 78.40)
 - open/close 2199 (time 1767320995.08 total 30.01 last 71.57)
 - open/close 2928 (time 1767321005.09 total 40.01 last 72.86)
 - open/close 3673 (time 1767321015.09 total 50.01 last 74.48)
total: 4500 open/close in 59.87 seconds: 75.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1386 (time 1767321039.30 total 10.03 last 138.24)
 - /unlink 2672 (time 1767321049.30 total 20.03 last 128.54)
 - /unlink 4216 (time 1767321059.31 total 30.04 last 154.27)
total: 4500 /unlink in 31.87 seconds: 141.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 778 (time 1767321075.19 total 10.01 last 77.70)
 - open/close 1568 (time 1767321085.19 total 20.02 last 78.97)
 - open/close 2333 (time 1767321095.20 total 30.02 last 76.47)
 - open/close 3013 (time 1767321105.21 total 40.03 last 67.93)
 - open/close 3638 (time 1767321115.21 total 50.03 last 62.49)
 - open/close 4295 (time 1767321125.21 total 60.04 last 65.67)
total: 4500 open/close in 63.05 seconds: 71.37 ops/second
 - /unlink 1345 (time 1767321142.96 total 10.01 last 134.38)
 - /unlink 2752 (time 1767321152.97 total 20.01 last 140.68)
 - /unlink 4083 (time 1767321162.97 total 30.01 last 133.09)
total: 4500 /unlink in 32.86 seconds: 136.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 741 (time 1767321181.20 total 10.02 last 73.96)
 - open/close 1523 (time 1767321191.21 total 20.02 last 78.15)
 - open/close 2317 (time 1767321201.22 total 30.03 last 79.36)
 - open/close 3132 (time 1767321211.22 total 40.03 last 81.45)
 - open/close 3995 (time 1767321221.23 total 50.05 last 86.21)
total: 4500 open/close in 57.27 seconds: 78.58 ops/second
 - /unlink 1232 (time 1767321242.88 total 10.00 last 123.17)
 - /unlink 2660 (time 1767321252.89 total 20.01 last 142.64)
 - /unlink 4163 (time 1767321262.90 total 30.02 last 150.25)
total: 4500 /unlink in 32.51 seconds: 138.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 742 (time 1767321280.22 total 10.01 last 74.12)
 - open/close 1441 (time 1767321290.23 total 20.02 last 69.84)
 - open/close 2158 (time 1767321300.23 total 30.02 last 71.66)
 - open/close 2855 (time 1767321310.24 total 40.03 last 69.64)
 - open/close 3561 (time 1767321320.25 total 50.04 last 70.55)
 - open/close 4163 (time 1767321330.26 total 60.05 last 60.12)
total: 4500 open/close in 65.08 seconds: 69.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1002 (time 1767321349.91 total 10.00 last 100.17)
 - /unlink 2224 (time 1767321359.91 total 20.00 last 122.19)
 - /unlink 3423 (time 1767321369.91 total 30.00 last 119.89)
total: 4500 /unlink in 39.53 seconds: 113.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 758 (time 1767321393.76 total 10.01 last 75.75)
 - open/close 1324 (time 1767321403.78 total 20.03 last 56.49)
 - open/close 1916 (time 1767321413.79 total 30.04 last 59.13)
 - open/close 2701 (time 1767321423.79 total 40.04 last 78.47)
 - open/close 3491 (time 1767321433.80 total 50.04 last 78.98)
 - open/close 4252 (time 1767321443.80 total 60.05 last 76.04)
total: 4500 open/close in 63.21 seconds: 71.19 ops/second
 - /unlink 1460 (time 1767321461.10 total 10.00 last 145.95)
 - /unlink 2806 (time 1767321471.10 total 20.00 last 134.60)
 - /unlink 4251 (time 1767321481.10 total 30.00 last 144.49)
total: 4500 /unlink in 31.76 seconds: 141.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 827 (time 1767321496.44 total 10.00 last 82.67)
 - open/close 1708 (time 1767321506.45 total 20.01 last 88.08)
 - open/close 2527 (time 1767321516.45 total 30.01 last 81.84)
 - open/close 3316 (time 1767321526.46 total 40.01 last 78.89)
 - open/close 4142 (time 1767321536.46 total 50.02 last 82.53)
total: 4500 open/close in 54.46 seconds: 82.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1425 (time 1767321555.00 total 10.00 last 142.44)
 - /unlink 2765 (time 1767321565.01 total 20.01 last 133.91)
 - /unlink 4195 (time 1767321575.03 total 30.03 last 142.70)
total: 4500 /unlink in 32.33 seconds: 139.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 819 (time 1767321591.95 total 10.00 last 81.88)
 - open/close 1608 (time 1767321601.95 total 20.01 last 78.86)
 - open/close 2449 (time 1767321611.96 total 30.02 last 84.03)
 - open/close 3329 (time 1767321621.97 total 40.02 last 87.95)
 - open/close 4015 (time 1767321631.97 total 50.03 last 68.55)
total: 4500 open/close in 56.47 seconds: 79.68 ops/second
 - /unlink 1552 (time 1767321651.91 total 10.00 last 155.15)
 - /unlink 2966 (time 1767321661.92 total 20.01 last 141.29)
 - /unlink 4259 (time 1767321671.92 total 30.01 last 129.26)
total: 4500 /unlink in 32.33 seconds: 139.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 775 (time 1767321689.72 total 10.00 last 77.48)
 - open/close 1523 (time 1767321699.73 total 20.01 last 74.72)
 - open/close 2292 (time 1767321709.73 total 30.02 last 76.87)
 - open/close 3199 (time 1767321719.74 total 40.03 last 90.62)
 - open/close 3991 (time 1767321729.75 total 50.04 last 79.12)
total: 4500 open/close in 55.88 seconds: 80.53 ops/second
 - /unlink 1446 (time 1767321750.00 total 10.00 last 144.59)
 - /unlink 2980 (time 1767321760.00 total 20.00 last 153.34)
 - /unlink 4458 (time 1767321770.01 total 30.01 last 147.73)
total: 4500 /unlink in 30.49 seconds: 147.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 873 (time 1767321784.69 total 10.01 last 87.21)
 - open/close 1727 (time 1767321794.70 total 20.02 last 85.34)
 - open/close 2575 (time 1767321804.70 total 30.03 last 84.73)
 - open/close 3284 (time 1767321814.72 total 40.04 last 70.81)
 - open/close 4044 (time 1767321824.73 total 50.05 last 75.90)
total: 4500 open/close in 56.05 seconds: 80.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1331 (time 1767321844.79 total 10.01 last 133.01)
 - /unlink 2598 (time 1767321854.79 total 20.01 last 126.62)
 - /unlink 4033 (time 1767321864.79 total 30.01 last 143.49)
total: 4500 /unlink in 33.51 seconds: 134.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 818 (time 1767321882.29 total 10.01 last 81.71)
 - open/close 1581 (time 1767321892.30 total 20.02 last 76.21)
 - open/close 2334 (time 1767321902.31 total 30.03 last 75.21)
 - open/close 3097 (time 1767321912.32 total 40.04 last 76.24)
 - open/close 3782 (time 1767321922.32 total 50.05 last 68.47)
 - open/close 4361 (time 1767321932.35 total 60.07 last 57.74)
total: 4500 open/close in 62.63 seconds: 71.85 ops/second
 - /unlink 1155 (time 1767321951.06 total 10.00 last 115.49)
 - /unlink 2555 (time 1767321961.07 total 20.01 last 139.94)
 - /unlink 3784 (time 1767321971.07 total 30.01 last 122.85)
total: 4500 /unlink in 35.69 seconds: 126.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 748 (time 1767321990.80 total 10.01 last 74.76)
 - open/close 1401 (time 1767322000.81 total 20.02 last 65.18)
 - open/close 2196 (time 1767322010.82 total 30.03 last 79.48)
 - open/close 2936 (time 1767322020.83 total 40.04 last 73.88)
 - open/close 3714 (time 1767322030.84 total 50.05 last 77.74)
 - open/close 4460 (time 1767322040.84 total 60.05 last 74.59)
total: 4500 open/close in 60.58 seconds: 74.29 ops/second
 - /unlink 1417 (time 1767322055.94 total 10.00 last 141.66)
 - /unlink 2880 (time 1767322065.94 total 20.00 last 146.29)
 - /unlink 4133 (time 1767322075.94 total 30.00 last 125.28)
total: 4500 /unlink in 32.52 seconds: 138.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 676 (time 1767322093.44 total 10.00 last 67.57)
 - open/close 1455 (time 1767322103.47 total 20.03 last 77.67)
 - open/close 2271 (time 1767322113.48 total 30.04 last 81.52)
 - open/close 3110 (time 1767322123.49 total 40.05 last 83.85)
 - open/close 3977 (time 1767322133.50 total 50.06 last 86.63)
total: 4500 open/close in 56.94 seconds: 79.03 ops/second
 - /unlink 1398 (time 1767322155.42 total 10.00 last 139.77)
 - /unlink 2742 (time 1767322165.42 total 20.01 last 134.31)
 - /unlink 4038 (time 1767322175.43 total 30.01 last 129.54)
total: 4500 /unlink in 33.48 seconds: 134.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 815 (time 1767322193.46 total 10.00 last 81.46)
 - open/close 1536 (time 1767322203.47 total 20.02 last 72.02)
 - open/close 2200 (time 1767322213.47 total 30.02 last 66.37)
 - open/close 2906 (time 1767322223.49 total 40.03 last 70.51)
 - open/close 3576 (time 1767322233.51 total 50.06 last 66.85)
 - open/close 4159 (time 1767322243.52 total 60.07 last 58.24)
total: 4500 open/close in 65.43 seconds: 68.78 ops/second
 - /unlink 1383 (time 1767322263.56 total 10.01 last 138.17)
 - /unlink 2834 (time 1767322273.57 total 20.01 last 145.07)
 - /unlink 4366 (time 1767322283.57 total 30.01 last 153.17)
total: 4500 /unlink in 31.04 seconds: 144.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 830 (time 1767322299.16 total 10.01 last 82.93)
 - open/close 1596 (time 1767322309.17 total 20.02 last 76.50)
 - open/close 2393 (time 1767322319.18 total 30.03 last 79.64)
 - open/close 3257 (time 1767322329.19 total 40.03 last 86.36)
 - open/close 4083 (time 1767322339.19 total 50.04 last 82.55)
total: 4500 open/close in 54.97 seconds: 81.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1494 (time 1767322358.14 total 10.00 last 149.37)
 - /unlink 2982 (time 1767322368.14 total 20.01 last 148.72)
total: 4500 /unlink in 29.02 seconds: 155.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 815 (time 1767322391.98 total 10.01 last 81.42)
 - open/close 1713 (time 1767322401.99 total 20.01 last 89.76)
 - open/close 2480 (time 1767322412.00 total 30.03 last 76.58)
 - open/close 3305 (time 1767322422.01 total 40.03 last 82.46)
 - open/close 4089 (time 1767322432.02 total 50.04 last 78.33)
total: 4500 open/close in 55.38 seconds: 81.26 ops/second
 - /unlink 1225 (time 1767322451.60 total 10.00 last 122.47)
 - /unlink 2770 (time 1767322461.61 total 20.01 last 154.35)
 - /unlink 4275 (time 1767322471.61 total 30.01 last 150.48)
total: 4500 /unlink in 31.44 seconds: 143.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 877 (time 1767322486.41 total 10.01 last 87.65)
 - open/close 1703 (time 1767322496.42 total 20.02 last 82.49)
 - open/close 2530 (time 1767322506.44 total 30.03 last 82.60)
 - open/close 3398 (time 1767322516.45 total 40.04 last 86.69)
 - open/close 4275 (time 1767322526.45 total 50.05 last 87.69)
total: 4500 open/close in 52.71 seconds: 85.37 ops/second
 - /unlink 1538 (time 1767322542.82 total 10.00 last 153.79)
 - /unlink 3080 (time 1767322552.82 total 20.01 last 154.12)
total: 4500 /unlink in 29.30 seconds: 153.60 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11271: 111697 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg227-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg227-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server
stop mds service on oleg227-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server
stop mds service on oleg227-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg227-server'
oleg227-server: oleg227-server.virtnet: executing unload_modules_local
oleg227-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg227-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.