== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:23:34 (1781238214)
Stopping clients: oleg240-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg240-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg240-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg240-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg240-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg240-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg240-server
oleg240-server: oleg240-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg240-server'
oleg240-server: oleg240-server.virtnet: executing load_modules_local
oleg240-server: Loading modules from /home/green/git/lustre-release/lustre
oleg240-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg240-server: detected 4 online CPUs by sysfs
oleg240-server: MODOPTS_LIBCFS=
oleg240-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg240-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all
pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg240-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all
pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg240-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all
pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg240-client.virtnet:  -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 725 (time 1781238425.60 total 10.00 last 72.50)
 - open/close 1497 (time 1781238435.62 total 20.02 last 77.03)
 - open/close 2326 (time 1781238445.64 total 30.04 last 82.75)
 - open/close 3072 (time 1781238455.64 total 40.05 last 74.56)
 - open/close 3765 (time 1781238465.65 total 50.05 last 69.25)
 - open/close 4450 (time 1781238475.66 total 60.06 last 68.44)
total: 4500 open/close in 60.82 seconds: 73.99 ops/second
 - /unlink 1228 (time 1781238492.04 total 10.01 last 122.73)
 - /unlink 2282 (time 1781238502.05 total 20.02 last 105.22)
 - /unlink 3385 (time 1781238512.06 total 30.02 last 110.28)
total: 4500 /unlink in 37.47 seconds: 120.10 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 704 (time 1781238534.85 total 10.01 last 70.31)
 - open/close 1563 (time 1781238544.85 total 20.02 last 85.87)
 - open/close 2372 (time 1781238554.85 total 30.02 last 80.86)
 - open/close 3229 (time 1781238564.86 total 40.03 last 85.65)
 - open/close 4059 (time 1781238574.87 total 50.04 last 82.92)
total: 4500 open/close in 55.17 seconds: 81.56 ops/second
 - /unlink 1370 (time 1781238594.36 total 10.00 last 136.95)
 - /unlink 2805 (time 1781238604.36 total 20.01 last 143.47)
 - /unlink 4302 (time 1781238614.37 total 30.01 last 149.62)
total: 4500 /unlink in 31.55 seconds: 142.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 810 (time 1781238630.44 total 10.01 last 80.93)
 - open/close 1532 (time 1781238640.45 total 20.01 last 72.19)
 - open/close 2390 (time 1781238650.45 total 30.01 last 85.79)
 - open/close 3174 (time 1781238660.45 total 40.02 last 78.35)
 - open/close 3927 (time 1781238670.46 total 50.03 last 75.24)
total: 4500 open/close in 58.67 seconds: 76.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1231 (time 1781238693.22 total 10.01 last 123.04)
 - /unlink 2463 (time 1781238703.22 total 20.01 last 123.10)
 - /unlink 3780 (time 1781238713.22 total 30.01 last 131.70)
total: 4500 /unlink in 35.99 seconds: 125.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 710 (time 1781238734.85 total 10.01 last 70.92)
 - open/close 1419 (time 1781238744.86 total 20.01 last 70.87)
 - open/close 2027 (time 1781238754.87 total 30.02 last 60.75)
 - open/close 2676 (time 1781238764.87 total 40.03 last 64.88)
 - open/close 3413 (time 1781238774.90 total 50.05 last 73.50)
 - open/close 4143 (time 1781238784.90 total 60.06 last 72.96)
total: 4500 open/close in 65.88 seconds: 68.31 ops/second
 - /unlink 1224 (time 1781238806.41 total 10.01 last 122.33)
 - /unlink 2492 (time 1781238816.41 total 20.01 last 126.75)
 - /unlink 3789 (time 1781238826.41 total 30.01 last 129.69)
total: 4500 /unlink in 35.91 seconds: 125.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 657 (time 1781238847.98 total 10.00 last 65.69)
 - open/close 1409 (time 1781238857.99 total 20.01 last 75.14)
 - open/close 2092 (time 1781238868.06 total 30.07 last 67.87)
 - open/close 2713 (time 1781238878.06 total 40.07 last 62.09)
 - open/close 3415 (time 1781238888.08 total 50.09 last 70.06)
 - open/close 4047 (time 1781238898.08 total 60.10 last 63.16)
total: 4500 open/close in 66.43 seconds: 67.74 ops/second
 - /unlink 1277 (time 1781238919.10 total 10.00 last 127.69)
 - /unlink 2629 (time 1781238929.12 total 20.01 last 135.02)
 - /unlink 3928 (time 1781238939.12 total 30.02 last 129.89)
total: 4500 /unlink in 34.61 seconds: 130.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 738 (time 1781238958.98 total 10.01 last 73.70)
 - open/close 1512 (time 1781238968.98 total 20.02 last 77.39)
 - open/close 2167 (time 1781238978.99 total 30.03 last 65.42)
 - open/close 2904 (time 1781238988.99 total 40.03 last 73.68)
 - open/close 3543 (time 1781238999.03 total 50.06 last 63.67)
 - open/close 4087 (time 1781239009.06 total 60.10 last 54.21)
total: 4500 open/close in 67.62 seconds: 66.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1103 (time 1781239032.22 total 10.00 last 110.29)
 - /unlink 2200 (time 1781239042.23 total 20.01 last 109.64)
 - /unlink 3374 (time 1781239052.24 total 30.01 last 117.34)
total: 4500 /unlink in 39.32 seconds: 114.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 612 (time 1781239076.57 total 10.00 last 61.17)
 - open/close 1174 (time 1781239086.57 total 20.01 last 56.19)
 - open/close 1695 (time 1781239096.58 total 30.01 last 52.06)
 - open/close 2272 (time 1781239106.58 total 40.01 last 57.70)
 - open/close 2773 (time 1781239116.61 total 50.05 last 49.94)
 - open/close 3195 (time 1781239126.62 total 60.06 last 42.16)
 - open/close 3762 (time 1781239136.63 total 70.07 last 56.63)
 - open/close 4448 (time 1781239146.64 total 80.07 last 68.58)
total: 4500 open/close in 80.63 seconds: 55.81 ops/second
 - /unlink 1180 (time 1781239163.16 total 10.00 last 117.96)
 - /unlink 2519 (time 1781239173.17 total 20.01 last 133.85)
 - /unlink 3996 (time 1781239183.17 total 30.01 last 147.65)
total: 4500 /unlink in 33.83 seconds: 133.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 612 (time 1781239202.71 total 10.00 last 61.20)
 - open/close 1133 (time 1781239212.73 total 20.02 last 51.98)
 - open/close 1683 (time 1781239222.75 total 30.04 last 54.93)
 - open/close 2162 (time 1781239232.76 total 40.05 last 47.84)
 - open/close 2686 (time 1781239242.77 total 50.06 last 52.32)
 - open/close 3323 (time 1781239252.79 total 60.08 last 63.61)
 - open/close 3929 (time 1781239262.80 total 70.09 last 60.54)
total: 4500 open/close in 78.26 seconds: 57.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1293 (time 1781239286.41 total 10.01 last 129.17)
 - /unlink 2370 (time 1781239296.41 total 20.01 last 107.65)
 - /unlink 3517 (time 1781239306.42 total 30.02 last 114.66)
total: 4500 /unlink in 39.19 seconds: 114.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 733 (time 1781239330.84 total 10.02 last 73.13)
 - open/close 1431 (time 1781239340.86 total 20.05 last 69.62)
 - open/close 2074 (time 1781239350.86 total 30.05 last 64.28)
 - open/close 2783 (time 1781239360.87 total 40.06 last 70.87)
 - open/close 3519 (time 1781239370.87 total 50.06 last 73.55)
 - open/close 4210 (time 1781239380.90 total 60.08 last 68.96)
total: 4500 open/close in 64.49 seconds: 69.78 ops/second
 - /unlink 1420 (time 1781239399.69 total 10.01 last 141.87)
 - /unlink 2627 (time 1781239409.70 total 20.03 last 120.51)
 - /unlink 3948 (time 1781239419.70 total 30.03 last 132.09)
total: 4500 /unlink in 33.78 seconds: 133.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 532 (time 1781239439.49 total 10.00 last 53.19)
 - open/close 1189 (time 1781239449.50 total 20.01 last 65.65)
 - open/close 1808 (time 1781239459.50 total 30.02 last 61.86)
 - open/close 2577 (time 1781239469.51 total 40.02 last 76.85)
 - open/close 3299 (time 1781239479.52 total 50.03 last 72.13)
 - open/close 3975 (time 1781239489.53 total 60.04 last 67.56)
total: 4500 open/close in 68.14 seconds: 66.04 ops/second
 - /unlink 1165 (time 1781239512.27 total 10.01 last 116.43)
 - /unlink 2421 (time 1781239522.27 total 20.01 last 125.58)
 - /unlink 3681 (time 1781239532.28 total 30.01 last 125.96)
total: 4500 /unlink in 36.77 seconds: 122.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 744 (time 1781239554.25 total 10.01 last 74.32)
 - open/close 1396 (time 1781239564.28 total 20.04 last 65.04)
 - open/close 2098 (time 1781239574.31 total 30.07 last 69.95)
 - open/close 2783 (time 1781239584.32 total 40.08 last 68.44)
 - open/close 3440 (time 1781239594.33 total 50.09 last 65.66)
 - open/close 4086 (time 1781239604.34 total 60.10 last 64.54)
total: 4500 open/close in 68.77 seconds: 65.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1345 (time 1781239629.40 total 10.00 last 134.44)
 - /unlink 2402 (time 1781239639.40 total 20.01 last 105.66)
 - /unlink 3662 (time 1781239649.41 total 30.01 last 125.95)
total: 4500 /unlink in 37.24 seconds: 120.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 661 (time 1781239672.21 total 10.00 last 66.08)
 - open/close 1353 (time 1781239682.22 total 20.01 last 69.19)
 - open/close 1958 (time 1781239692.22 total 30.01 last 60.48)
 - open/close 2687 (time 1781239702.23 total 40.02 last 72.85)
 - open/close 3496 (time 1781239712.24 total 50.03 last 80.81)
 - open/close 4085 (time 1781239722.24 total 60.03 last 58.88)
total: 4500 open/close in 66.74 seconds: 67.42 ops/second
 - /unlink 1398 (time 1781239744.65 total 10.00 last 139.80)
 - /unlink 2751 (time 1781239754.66 total 20.01 last 135.16)
 - /unlink 4040 (time 1781239764.66 total 30.01 last 128.88)
total: 4500 /unlink in 33.62 seconds: 133.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 687 (time 1781239783.38 total 10.05 last 68.39)
 - open/close 1377 (time 1781239793.38 total 20.05 last 68.97)
 - open/close 2126 (time 1781239803.38 total 30.05 last 74.87)
 - open/close 2818 (time 1781239813.39 total 40.05 last 69.19)
 - open/close 3477 (time 1781239823.39 total 50.06 last 65.86)
 - open/close 3983 (time 1781239833.40 total 60.07 last 50.55)
total: 4500 open/close in 69.95 seconds: 64.33 ops/second
 - /unlink 1181 (time 1781239858.35 total 10.00 last 118.10)
 - /unlink 2461 (time 1781239868.35 total 20.00 last 127.94)
 - /unlink 3375 (time 1781239878.36 total 30.01 last 91.34)
 - /unlink 4164 (time 1781239888.36 total 40.01 last 78.89)
total: 4500 /unlink in 42.95 seconds: 104.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 586 (time 1781239907.44 total 10.02 last 58.46)
 - open/close 1148 (time 1781239917.44 total 20.03 last 56.15)
 - open/close 1863 (time 1781239927.45 total 30.04 last 71.47)
 - open/close 2494 (time 1781239937.46 total 40.05 last 63.04)
 - open/close 3145 (time 1781239947.51 total 50.09 last 64.79)
 - open/close 3784 (time 1781239957.51 total 60.10 last 63.87)
 - open/close 4444 (time 1781239967.51 total 70.10 last 65.98)
total: 4500 open/close in 70.85 seconds: 63.52 ops/second
 - /unlink 1378 (time 1781239983.52 total 10.00 last 137.74)
 - /unlink 2664 (time 1781239993.52 total 20.01 last 128.59)
 - /unlink 3797 (time 1781240003.52 total 30.01 last 113.26)
total: 4500 /unlink in 36.16 seconds: 124.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 768 (time 1781240025.05 total 10.00 last 76.77)
 - open/close 1315 (time 1781240035.06 total 20.01 last 54.64)
 - open/close 2032 (time 1781240045.07 total 30.02 last 71.68)
 - open/close 2590 (time 1781240055.08 total 40.03 last 55.73)
 - open/close 3203 (time 1781240065.08 total 50.03 last 61.30)
 - open/close 3830 (time 1781240075.08 total 60.03 last 62.70)
total: 4500 open/close in 69.73 seconds: 64.53 ops/second
 - /unlink 1302 (time 1781240100.02 total 10.00 last 130.19)
 - /unlink 2561 (time 1781240110.02 total 20.00 last 125.86)
 - /unlink 3930 (time 1781240120.02 total 30.01 last 136.88)
total: 4500 /unlink in 34.13 seconds: 131.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 776 (time 1781240138.70 total 10.00 last 77.56)
 - open/close 1496 (time 1781240148.70 total 20.01 last 71.99)
 - open/close 2323 (time 1781240158.70 total 30.01 last 82.65)
 - open/close 3207 (time 1781240168.71 total 40.02 last 88.36)
 - open/close 3908 (time 1781240178.71 total 50.02 last 70.08)
total: 4500 open/close in 59.59 seconds: 75.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1251 (time 1781240203.51 total 10.02 last 124.82)
 - /unlink 2353 (time 1781240213.52 total 20.03 last 110.14)
 - /unlink 3584 (time 1781240223.52 total 30.03 last 123.10)
total: 4500 /unlink in 37.23 seconds: 120.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 806 (time 1781240245.38 total 10.00 last 80.56)
 - open/close 1431 (time 1781240255.38 total 20.01 last 62.48)
 - open/close 2072 (time 1781240265.41 total 30.04 last 63.91)
 - open/close 2678 (time 1781240275.41 total 40.04 last 60.60)
 - open/close 3355 (time 1781240285.43 total 50.05 last 67.60)
 - open/close 4077 (time 1781240295.43 total 60.06 last 72.17)
total: 4500 open/close in 65.03 seconds: 69.20 ops/second
 - /unlink 1414 (time 1781240314.93 total 10.00 last 141.37)
 - /unlink 2655 (time 1781240324.93 total 20.01 last 124.06)
 - /unlink 3994 (time 1781240334.94 total 30.01 last 133.78)
total: 4500 /unlink in 33.36 seconds: 134.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 844 (time 1781240353.30 total 10.02 last 84.19)
 - open/close 1720 (time 1781240363.30 total 20.03 last 87.58)
 - open/close 2465 (time 1781240373.31 total 30.03 last 74.48)
 - open/close 3211 (time 1781240383.31 total 40.03 last 74.58)
 - open/close 3920 (time 1781240393.31 total 50.04 last 70.86)
total: 4500 open/close in 57.23 seconds: 78.63 ops/second
 - /unlink 1447 (time 1781240415.14 total 10.01 last 144.55)
 - /unlink 2885 (time 1781240425.14 total 20.01 last 143.78)
 - /unlink 4368 (time 1781240435.14 total 30.01 last 148.29)
total: 4500 /unlink in 31.19 seconds: 144.28 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11209: 94156 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg240-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg240-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg240-server
stop mds service on oleg240-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg240-server
stop mds service on oleg240-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg240-server

unloading modules via unload_modules_local on: 'oleg240-server'
oleg240-server: oleg240-server.virtnet: executing unload_modules_local
oleg240-server: 
modules unloaded.