== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:11:50 (1788441110)
Stopping clients: oleg637-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg637-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg637-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg637-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg637-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg637-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg637-server
oleg637-server: oleg637-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg637-server'
oleg637-server: oleg637-server.virtnet: executing load_modules_local
oleg637-server: kmemleak disabled
oleg637-server: Loading modules from /home/green/git/lustre-release/lustre
oleg637-server: detected 4 online CPUs by sysfs
oleg637-server: MODOPTS_LIBCFS=
oleg637-server: Force libcfs to create 2 CPU partitions
oleg637-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg637-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg637-server: oleg637-server.virtnet: executing set_default_debug -1 all
pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg637-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg637-server: oleg637-server.virtnet: executing set_default_debug -1 all
pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg637-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg637-server: oleg637-server.virtnet: executing set_default_debug -1 all
pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg637-client.virtnet:  -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 755 (time 1788441312.29 total 10.01 last 75.43)
 - open/close 1483 (time 1788441322.30 total 20.02 last 72.76)
 - open/close 2221 (time 1788441332.31 total 30.02 last 73.76)
 - open/close 2998 (time 1788441342.31 total 40.03 last 77.63)
 - open/close 3768 (time 1788441352.32 total 50.03 last 76.98)
total: 4500 open/close in 59.82 seconds: 75.22 ops/second
 - /unlink 1363 (time 1788441376.33 total 10.01 last 136.22)
 - /unlink 2633 (time 1788441386.34 total 20.02 last 126.86)
 - /unlink 3879 (time 1788441396.34 total 30.02 last 124.57)
total: 4500 /unlink in 35.72 seconds: 125.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 824 (time 1788441417.51 total 10.00 last 82.39)
 - open/close 1554 (time 1788441427.53 total 20.02 last 72.87)
 - open/close 2308 (time 1788441437.53 total 30.02 last 75.35)
 - open/close 3130 (time 1788441447.55 total 40.04 last 82.05)
 - open/close 3919 (time 1788441457.55 total 50.05 last 78.88)
total: 4500 open/close in 57.24 seconds: 78.61 ops/second
 - /unlink 1438 (time 1788441479.11 total 10.00 last 143.73)
 - /unlink 2908 (time 1788441489.11 total 20.01 last 146.92)
 - /unlink 4348 (time 1788441499.12 total 30.01 last 143.93)
total: 4500 /unlink in 31.14 seconds: 144.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 714 (time 1788441516.15 total 10.01 last 71.34)
 - open/close 1410 (time 1788441526.15 total 20.01 last 69.59)
 - open/close 2109 (time 1788441536.17 total 30.03 last 69.76)
 - open/close 2800 (time 1788441546.18 total 40.04 last 69.05)
 - open/close 3522 (time 1788441556.19 total 50.04 last 72.17)
 - open/close 4227 (time 1788441566.19 total 60.05 last 70.44)
total: 4500 open/close in 63.93 seconds: 70.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1332 (time 1788441584.71 total 10.01 last 133.01)
 - /unlink 2680 (time 1788441594.72 total 20.03 last 134.64)
 - /unlink 4118 (time 1788441604.73 total 30.04 last 143.57)
total: 4500 /unlink in 32.81 seconds: 137.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 775 (time 1788441622.20 total 10.00 last 77.48)
 - open/close 1535 (time 1788441632.21 total 20.01 last 75.92)
 - open/close 2267 (time 1788441642.22 total 30.03 last 73.10)
 - open/close 2966 (time 1788441652.24 total 40.04 last 69.78)
 - open/close 3710 (time 1788441662.24 total 50.04 last 74.40)
 - open/close 4471 (time 1788441672.25 total 60.05 last 76.04)
total: 4500 open/close in 60.47 seconds: 74.41 ops/second
 - /unlink 1116 (time 1788441687.22 total 10.00 last 111.59)
 - /unlink 1995 (time 1788441697.22 total 20.01 last 87.85)
 - /unlink 2962 (time 1788441707.24 total 30.02 last 96.56)
 - /unlink 4293 (time 1788441717.24 total 40.02 last 133.05)
total: 4500 /unlink in 41.56 seconds: 108.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 733 (time 1788441733.13 total 10.00 last 73.28)
 - open/close 1495 (time 1788441743.13 total 20.00 last 76.19)
 - open/close 2190 (time 1788441753.13 total 30.00 last 69.50)
 - open/close 2864 (time 1788441763.13 total 40.01 last 67.39)
 - open/close 3587 (time 1788441773.14 total 50.02 last 72.22)
 - open/close 4356 (time 1788441783.15 total 60.03 last 76.81)
total: 4500 open/close in 61.82 seconds: 72.79 ops/second
 - /unlink 1524 (time 1788441799.87 total 10.00 last 152.34)
 - /unlink 2963 (time 1788441809.87 total 20.01 last 143.85)
 - /unlink 4387 (time 1788441819.87 total 30.01 last 142.37)
total: 4500 /unlink in 30.97 seconds: 145.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 753 (time 1788441834.61 total 10.01 last 75.25)
 - open/close 1465 (time 1788441844.62 total 20.02 last 71.10)
 - open/close 2186 (time 1788441854.62 total 30.02 last 72.09)
 - open/close 2909 (time 1788441864.63 total 40.03 last 72.26)
 - open/close 3648 (time 1788441874.63 total 50.03 last 73.86)
 - open/close 4357 (time 1788441884.64 total 60.04 last 70.84)
total: 4500 open/close in 61.94 seconds: 72.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1249 (time 1788441901.25 total 10.01 last 124.83)
 - /unlink 2570 (time 1788441911.26 total 20.01 last 132.06)
 - /unlink 3967 (time 1788441921.26 total 30.01 last 139.68)
total: 4500 /unlink in 34.28 seconds: 131.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 752 (time 1788441940.30 total 10.00 last 75.19)
 - open/close 1388 (time 1788441950.30 total 20.00 last 63.59)
 - open/close 2112 (time 1788441960.30 total 30.01 last 72.38)
 - open/close 2885 (time 1788441970.31 total 40.01 last 77.27)
 - open/close 3625 (time 1788441980.31 total 50.02 last 73.95)
 - open/close 4366 (time 1788441990.32 total 60.03 last 74.03)
total: 4500 open/close in 61.75 seconds: 72.87 ops/second
 - /unlink 1475 (time 1788442006.40 total 10.00 last 147.44)
 - /unlink 2901 (time 1788442016.40 total 20.01 last 142.51)
 - /unlink 4322 (time 1788442026.40 total 30.01 last 142.09)
total: 4500 /unlink in 31.49 seconds: 142.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 778 (time 1788442042.79 total 10.00 last 77.78)
 - open/close 1494 (time 1788442052.79 total 20.00 last 71.59)
 - open/close 2236 (time 1788442062.80 total 30.02 last 74.11)
 - open/close 2980 (time 1788442072.82 total 40.04 last 74.24)
 - open/close 3700 (time 1788442082.84 total 50.06 last 71.86)
 - open/close 4463 (time 1788442092.85 total 60.07 last 76.24)
total: 4500 open/close in 60.73 seconds: 74.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1474 (time 1788442108.33 total 10.00 last 147.37)
 - /unlink 2867 (time 1788442118.33 total 20.00 last 139.27)
 - /unlink 4175 (time 1788442128.33 total 30.01 last 130.78)
total: 4500 /unlink in 32.74 seconds: 137.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 724 (time 1788442146.23 total 10.02 last 72.29)
 - open/close 1387 (time 1788442156.23 total 20.02 last 66.29)
 - open/close 2049 (time 1788442166.25 total 30.04 last 66.05)
 - open/close 2723 (time 1788442176.27 total 40.06 last 67.29)
 - open/close 3470 (time 1788442186.29 total 50.07 last 74.57)
 - open/close 4187 (time 1788442196.29 total 60.08 last 71.66)
total: 4500 open/close in 64.44 seconds: 69.83 ops/second
 - /unlink 1460 (time 1788442214.74 total 10.00 last 145.97)
 - /unlink 2962 (time 1788442224.75 total 20.00 last 150.17)
 - /unlink 4366 (time 1788442234.75 total 30.01 last 140.34)
total: 4500 /unlink in 31.03 seconds: 145.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 746 (time 1788442250.00 total 10.01 last 74.55)
 - open/close 1493 (time 1788442260.02 total 20.03 last 74.56)
 - open/close 2252 (time 1788442270.03 total 30.03 last 75.86)
 - open/close 3050 (time 1788442280.04 total 40.04 last 79.73)
 - open/close 3773 (time 1788442290.04 total 50.05 last 72.26)
total: 4500 open/close in 59.27 seconds: 75.93 ops/second
 - /unlink 1417 (time 1788442313.14 total 10.01 last 141.61)
 - /unlink 2771 (time 1788442323.15 total 20.01 last 135.36)
 - /unlink 4148 (time 1788442333.15 total 30.01 last 137.69)
total: 4500 /unlink in 32.66 seconds: 137.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 679 (time 1788442350.52 total 10.02 last 67.77)
 - open/close 1362 (time 1788442360.54 total 20.04 last 68.17)
 - open/close 2131 (time 1788442370.55 total 30.05 last 76.85)
 - open/close 2808 (time 1788442380.56 total 40.06 last 67.63)
 - open/close 3550 (time 1788442390.57 total 50.06 last 74.13)
 - open/close 4274 (time 1788442400.57 total 60.07 last 72.40)
total: 4500 open/close in 63.24 seconds: 71.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1422 (time 1788442417.81 total 10.00 last 142.17)
 - /unlink 2846 (time 1788442427.82 total 20.00 last 142.36)
 - /unlink 4218 (time 1788442437.82 total 30.01 last 137.19)
total: 4500 /unlink in 32.24 seconds: 139.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 702 (time 1788442455.11 total 10.00 last 70.17)
 - open/close 1449 (time 1788442465.12 total 20.01 last 74.67)
 - open/close 2196 (time 1788442475.12 total 30.01 last 74.68)
 - open/close 2939 (time 1788442485.13 total 40.03 last 74.20)
 - open/close 3624 (time 1788442495.15 total 50.04 last 68.40)
 - open/close 4334 (time 1788442505.15 total 60.05 last 70.95)
total: 4500 open/close in 62.23 seconds: 72.31 ops/second
 - /unlink 1354 (time 1788442522.20 total 10.01 last 135.27)
 - /unlink 2729 (time 1788442532.20 total 20.01 last 137.49)
 - /unlink 4204 (time 1788442542.20 total 30.01 last 147.48)
total: 4500 /unlink in 32.19 seconds: 139.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 768 (time 1788442559.54 total 10.01 last 76.70)
 - open/close 1581 (time 1788442569.55 total 20.03 last 81.21)
 - open/close 2389 (time 1788442579.55 total 30.03 last 80.76)
 - open/close 3119 (time 1788442589.57 total 40.05 last 72.87)
 - open/close 3884 (time 1788442599.59 total 50.07 last 76.36)
total: 4500 open/close in 57.87 seconds: 77.76 ops/second
 - /unlink 1422 (time 1788442621.48 total 10.00 last 142.15)
 - /unlink 2823 (time 1788442631.48 total 20.01 last 140.00)
 - /unlink 4214 (time 1788442641.49 total 30.01 last 139.05)
total: 4500 /unlink in 32.43 seconds: 138.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 637 (time 1788442659.63 total 10.00 last 63.69)
 - open/close 1344 (time 1788442669.64 total 20.01 last 70.61)
 - open/close 2118 (time 1788442679.65 total 30.02 last 77.35)
 - open/close 2808 (time 1788442689.66 total 40.03 last 68.93)
 - open/close 3484 (time 1788442699.66 total 50.04 last 67.54)
 - open/close 4182 (time 1788442709.67 total 60.05 last 69.73)
total: 4500 open/close in 64.33 seconds: 69.95 ops/second
 - /unlink 1201 (time 1788442728.59 total 10.01 last 120.03)
 - /unlink 2528 (time 1788442738.61 total 20.03 last 132.42)
 - /unlink 3998 (time 1788442748.62 total 30.03 last 146.98)
total: 4500 /unlink in 34.11 seconds: 131.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 708 (time 1788442768.02 total 10.01 last 70.74)
 - open/close 1482 (time 1788442778.03 total 20.01 last 77.37)
 - open/close 2234 (time 1788442788.03 total 30.01 last 75.20)
 - open/close 2928 (time 1788442798.03 total 40.01 last 69.39)
 - open/close 3640 (time 1788442808.04 total 50.03 last 71.10)
 - open/close 4337 (time 1788442818.05 total 60.04 last 69.63)
total: 4500 open/close in 62.09 seconds: 72.48 ops/second
 - /unlink 1545 (time 1788442834.87 total 10.00 last 154.44)
 - /unlink 2836 (time 1788442844.87 total 20.00 last 129.09)
 - /unlink 4245 (time 1788442854.87 total 30.01 last 140.85)
total: 4500 /unlink in 31.90 seconds: 141.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 758 (time 1788442871.85 total 10.01 last 75.74)
 - open/close 1410 (time 1788442881.85 total 20.01 last 65.19)
 - open/close 2056 (time 1788442891.86 total 30.02 last 64.56)
 - open/close 2829 (time 1788442901.86 total 40.02 last 77.28)
 - open/close 3583 (time 1788442911.86 total 50.02 last 75.39)
 - open/close 4330 (time 1788442921.87 total 60.03 last 74.64)
total: 4500 open/close in 62.44 seconds: 72.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1286 (time 1788442938.58 total 10.01 last 128.46)
 - /unlink 2591 (time 1788442948.58 total 20.01 last 130.48)
 - /unlink 3974 (time 1788442958.59 total 30.02 last 138.24)
total: 4500 /unlink in 33.99 seconds: 132.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 653 (time 1788442978.07 total 10.01 last 65.21)
 - open/close 1404 (time 1788442988.07 total 20.02 last 75.09)
 - open/close 2127 (time 1788442998.08 total 30.02 last 72.28)
 - open/close 2882 (time 1788443008.10 total 40.04 last 75.34)
 - open/close 3627 (time 1788443018.10 total 50.05 last 74.43)
 - open/close 4344 (time 1788443028.11 total 60.05 last 71.67)
total: 4500 open/close in 62.59 seconds: 71.89 ops/second
 - /unlink 1296 (time 1788443045.59 total 10.02 last 129.37)
 - /unlink 2652 (time 1788443055.59 total 20.02 last 135.55)
 - /unlink 4114 (time 1788443065.62 total 30.05 last 145.85)
total: 4500 /unlink in 32.86 seconds: 136.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 734 (time 1788443083.36 total 10.01 last 73.36)
 - open/close 1366 (time 1788443093.37 total 20.02 last 63.14)
 - open/close 2104 (time 1788443103.37 total 30.02 last 73.76)
 - open/close 2843 (time 1788443113.38 total 40.03 last 73.84)
 - open/close 3433 (time 1788443123.39 total 50.04 last 58.94)
 - open/close 4042 (time 1788443133.39 total 60.04 last 60.90)
total: 4500 open/close in 65.78 seconds: 68.41 ops/second
 - /unlink 1371 (time 1788443153.79 total 10.00 last 137.10)
 - /unlink 2671 (time 1788443163.79 total 20.00 last 129.95)
 - /unlink 4002 (time 1788443173.80 total 30.01 last 133.07)
total: 4500 /unlink in 33.86 seconds: 132.92 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95229 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg637-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg637-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg637-server
stop mds service on oleg637-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg637-server
stop mds service on oleg637-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg637-server
unloading modules via unload_modules_local on: 'oleg637-server'
oleg637-server: oleg637-server.virtnet: executing unload_modules_local
oleg637-server: kmemleak disabled
modules unloaded.