== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:24:46 (1790976286)
Stopping clients: oleg128-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg128-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg128-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg128-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg128-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg128-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg128-server
oleg128-server: oleg128-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg128-server'
oleg128-server: oleg128-server.virtnet: executing load_modules_local
oleg128-server: kmemleak disabled
oleg128-server: Loading modules from /home/green/git/lustre-release/lustre
oleg128-server: detected 4 online CPUs by sysfs
oleg128-server: MODOPTS_LIBCFS=
oleg128-server: Force libcfs to create 2 CPU partitions
oleg128-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg128-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all
pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg128-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all
pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg128-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all
pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg128-client.virtnet:  -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 660 (time 1790976518.31 total 10.00 last 65.98)
 - open/close 1347 (time 1790976528.31 total 20.01 last 68.66)
 - open/close 2131 (time 1790976538.31 total 30.01 last 78.40)
 - open/close 2870 (time 1790976548.31 total 40.01 last 73.87)
 - open/close 3423 (time 1790976558.32 total 50.01 last 55.30)
 - open/close 3984 (time 1790976568.32 total 60.02 last 56.09)
total: 4500 open/close in 67.39 seconds: 66.78 ops/second
 - /unlink 1313 (time 1790976589.46 total 10.00 last 131.28)
 - /unlink 2493 (time 1790976599.46 total 20.00 last 118.00)
 - /unlink 3793 (time 1790976609.47 total 30.01 last 129.93)
total: 4500 /unlink in 36.76 seconds: 122.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 547 (time 1790976632.95 total 10.03 last 54.51)
 - open/close 1245 (time 1790976642.96 total 20.04 last 69.74)
 - open/close 1873 (time 1790976652.97 total 30.05 last 62.75)
 - open/close 2598 (time 1790976662.98 total 40.06 last 72.44)
 - open/close 3266 (time 1790976672.98 total 50.06 last 66.77)
 - open/close 3758 (time 1790976683.02 total 60.10 last 49.02)
 - open/close 4493 (time 1790976693.02 total 70.10 last 73.46)
total: 4500 open/close in 70.16 seconds: 64.14 ops/second
 - /unlink 1261 (time 1790976708.22 total 10.00 last 126.09)
 - /unlink 2660 (time 1790976718.22 total 20.00 last 139.89)
 - /unlink 3921 (time 1790976728.23 total 30.00 last 126.08)
total: 4500 /unlink in 34.41 seconds: 130.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 703 (time 1790976748.01 total 10.02 last 70.15)
 - open/close 1371 (time 1790976758.02 total 20.02 last 66.78)
 - open/close 2053 (time 1790976768.03 total 30.04 last 68.11)
 - open/close 2654 (time 1790976778.04 total 40.05 last 60.05)
 - open/close 3329 (time 1790976788.06 total 50.07 last 67.34)
 - open/close 4041 (time 1790976798.06 total 60.07 last 71.19)
total: 4500 open/close in 67.67 seconds: 66.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1129 (time 1790976821.66 total 10.00 last 112.90)
 - /unlink 2145 (time 1790976831.66 total 20.00 last 101.58)
 - /unlink 3287 (time 1790976841.67 total 30.01 last 114.15)
total: 4500 /unlink in 39.21 seconds: 114.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 655 (time 1790976866.24 total 10.01 last 65.46)
 - open/close 1313 (time 1790976876.25 total 20.02 last 65.72)
 - open/close 1980 (time 1790976886.25 total 30.02 last 66.66)
 - open/close 2582 (time 1790976896.25 total 40.02 last 60.20)
 - open/close 3213 (time 1790976906.26 total 50.03 last 63.07)
 - open/close 3878 (time 1790976916.27 total 60.04 last 66.43)
 - open/close 4446 (time 1790976926.28 total 70.05 last 56.75)
total: 4500 open/close in 70.73 seconds: 63.63 ops/second
 - /unlink 1380 (time 1790976940.42 total 10.00 last 138.00)
 - /unlink 2459 (time 1790976950.42 total 20.00 last 107.88)
 - /unlink 3726 (time 1790976960.43 total 30.01 last 126.54)
total: 4500 /unlink in 36.77 seconds: 122.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 570 (time 1790976983.08 total 10.01 last 56.95)
 - open/close 1322 (time 1790976993.09 total 20.02 last 75.14)
 - open/close 1968 (time 1790977003.10 total 30.03 last 64.52)
 - open/close 2682 (time 1790977013.10 total 40.03 last 71.38)
 - open/close 3426 (time 1790977023.11 total 50.04 last 74.38)
 - open/close 4069 (time 1790977033.11 total 60.04 last 64.25)
total: 4500 open/close in 66.92 seconds: 67.25 ops/second
 - /unlink 1312 (time 1790977055.28 total 10.00 last 131.18)
 - /unlink 2606 (time 1790977065.29 total 20.01 last 129.35)
 - /unlink 3925 (time 1790977075.29 total 30.01 last 131.85)
total: 4500 /unlink in 34.11 seconds: 131.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 651 (time 1790977095.56 total 10.03 last 64.94)
 - open/close 1339 (time 1790977105.58 total 20.04 last 68.70)
 - open/close 1890 (time 1790977115.58 total 30.04 last 55.10)
 - open/close 2468 (time 1790977125.58 total 40.04 last 57.80)
 - open/close 3201 (time 1790977135.59 total 50.05 last 73.22)
 - open/close 3875 (time 1790977145.60 total 60.06 last 67.32)
total: 4500 open/close in 69.43 seconds: 64.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1222 (time 1790977170.04 total 10.01 last 122.13)
 - /unlink 2438 (time 1790977180.05 total 20.01 last 121.53)
 - /unlink 3572 (time 1790977190.06 total 30.03 last 113.21)
total: 4500 /unlink in 36.93 seconds: 121.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 662 (time 1790977212.55 total 10.01 last 66.15)
 - open/close 1366 (time 1790977222.56 total 20.01 last 70.36)
 - open/close 2069 (time 1790977232.57 total 30.02 last 70.23)
 - open/close 2764 (time 1790977242.57 total 40.03 last 69.46)
 - open/close 3430 (time 1790977252.57 total 50.03 last 66.59)
 - open/close 4121 (time 1790977262.58 total 60.03 last 69.08)
total: 4500 open/close in 65.65 seconds: 68.55 ops/second
 - /unlink 1274 (time 1790977282.66 total 10.00 last 127.36)
 - /unlink 2354 (time 1790977292.67 total 20.01 last 107.92)
 - /unlink 3411 (time 1790977302.68 total 30.02 last 105.64)
total: 4500 /unlink in 38.78 seconds: 116.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 691 (time 1790977327.34 total 10.02 last 68.96)
 - open/close 1439 (time 1790977337.35 total 20.03 last 74.74)
 - open/close 2098 (time 1790977347.36 total 30.03 last 65.87)
 - open/close 2773 (time 1790977357.36 total 40.04 last 67.44)
 - open/close 3451 (time 1790977367.38 total 50.05 last 67.73)
 - open/close 4144 (time 1790977377.39 total 60.07 last 69.18)
total: 4500 open/close in 65.14 seconds: 69.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1199 (time 1790977397.96 total 10.01 last 119.74)
 - /unlink 2463 (time 1790977407.96 total 20.02 last 126.31)
 - /unlink 3781 (time 1790977417.97 total 30.02 last 131.77)
total: 4500 /unlink in 36.12 seconds: 124.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 638 (time 1790977439.79 total 10.01 last 63.75)
 - open/close 1259 (time 1790977449.79 total 20.01 last 62.10)
 - open/close 1891 (time 1790977459.79 total 30.01 last 63.17)
 - open/close 2545 (time 1790977469.80 total 40.02 last 65.35)
 - open/close 3166 (time 1790977479.80 total 50.02 last 62.09)
 - open/close 3785 (time 1790977489.81 total 60.02 last 61.89)
 - open/close 4489 (time 1790977499.81 total 70.03 last 70.39)
total: 4500 open/close in 70.13 seconds: 64.17 ops/second
 - /unlink 1304 (time 1790977514.77 total 10.01 last 130.28)
 - /unlink 2479 (time 1790977524.78 total 20.01 last 117.45)
 - /unlink 3631 (time 1790977534.78 total 30.02 last 115.17)
total: 4500 /unlink in 36.36 seconds: 123.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 669 (time 1790977556.54 total 10.01 last 66.87)
 - open/close 1375 (time 1790977566.55 total 20.01 last 70.57)
 - open/close 2098 (time 1790977576.56 total 30.02 last 72.20)
 - open/close 2771 (time 1790977586.57 total 40.03 last 67.27)
 - open/close 3476 (time 1790977596.57 total 50.03 last 70.48)
 - open/close 4203 (time 1790977606.58 total 60.04 last 72.63)
total: 4500 open/close in 65.48 seconds: 68.72 ops/second
 - /unlink 1234 (time 1790977628.83 total 10.00 last 123.35)
 - /unlink 2347 (time 1790977638.83 total 20.01 last 111.29)
 - /unlink 3576 (time 1790977648.86 total 30.04 last 122.52)
total: 4500 /unlink in 37.29 seconds: 120.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 530 (time 1790977672.93 total 10.01 last 52.93)
 - open/close 1112 (time 1790977682.94 total 20.03 last 58.12)
 - open/close 1724 (time 1790977692.94 total 30.03 last 61.20)
 - open/close 2287 (time 1790977702.97 total 40.05 last 56.16)
 - open/close 3070 (time 1790977712.97 total 50.05 last 78.29)
 - open/close 3834 (time 1790977722.97 total 60.06 last 76.38)
total: 4500 open/close in 69.00 seconds: 65.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1159 (time 1790977746.42 total 10.01 last 115.80)
 - /unlink 2097 (time 1790977756.42 total 20.01 last 93.75)
 - /unlink 3001 (time 1790977766.43 total 30.02 last 90.32)
 - /unlink 3997 (time 1790977776.44 total 40.02 last 99.58)
total: 4500 /unlink in 44.58 seconds: 100.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 659 (time 1790977796.37 total 10.01 last 65.85)
 - open/close 1325 (time 1790977806.39 total 20.02 last 66.52)
 - open/close 2042 (time 1790977816.39 total 30.03 last 71.65)
 - open/close 2762 (time 1790977826.39 total 40.03 last 71.99)
 - open/close 3520 (time 1790977836.40 total 50.03 last 75.77)
 - open/close 4253 (time 1790977846.42 total 60.05 last 73.16)
total: 4500 open/close in 63.65 seconds: 70.70 ops/second
 - /unlink 1260 (time 1790977864.26 total 10.00 last 125.98)
 - /unlink 2386 (time 1790977874.26 total 20.00 last 112.56)
 - /unlink 3532 (time 1790977884.28 total 30.02 last 114.42)
total: 4500 /unlink in 37.69 seconds: 119.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 534 (time 1790977910.12 total 10.00 last 53.38)
 - open/close 1155 (time 1790977920.14 total 20.02 last 62.00)
 - open/close 1854 (time 1790977930.14 total 30.02 last 69.88)
 - open/close 2568 (time 1790977940.15 total 40.03 last 71.34)
 - open/close 3310 (time 1790977950.17 total 50.05 last 74.06)
 - open/close 3958 (time 1790977960.17 total 60.05 last 64.78)
total: 4500 open/close in 67.71 seconds: 66.46 ops/second
 - /unlink 1442 (time 1790977982.90 total 10.01 last 144.12)
 - /unlink 2861 (time 1790977992.90 total 20.01 last 141.89)
 - /unlink 4338 (time 1790978002.91 total 30.01 last 147.61)
total: 4500 /unlink in 31.71 seconds: 141.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 479 (time 1790978021.27 total 10.03 last 47.74)
 - open/close 965 (time 1790978031.28 total 20.04 last 48.58)
 - open/close 1591 (time 1790978041.28 total 30.04 last 62.59)
 - open/close 2334 (time 1790978051.31 total 40.07 last 74.09)
 - open/close 3063 (time 1790978061.32 total 50.08 last 72.83)
 - open/close 3726 (time 1790978071.33 total 60.09 last 66.22)
 - open/close 4285 (time 1790978081.34 total 70.10 last 55.87)
total: 4500 open/close in 74.40 seconds: 60.48 ops/second
 - /unlink 1281 (time 1790978100.24 total 10.00 last 128.05)
 - /unlink 2480 (time 1790978110.25 total 20.01 last 119.79)
 - /unlink 3824 (time 1790978120.25 total 30.02 last 134.35)
total: 4500 /unlink in 34.84 seconds: 129.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 473 (time 1790978141.43 total 10.01 last 47.25)
 - open/close 1034 (time 1790978151.44 total 20.02 last 56.06)
 - open/close 1785 (time 1790978161.45 total 30.03 last 75.01)
 - open/close 2478 (time 1790978171.47 total 40.05 last 69.16)
 - open/close 3043 (time 1790978181.47 total 50.05 last 56.50)
 - open/close 3573 (time 1790978191.50 total 60.08 last 52.85)
 - open/close 4183 (time 1790978201.51 total 70.09 last 60.95)
total: 4500 open/close in 75.38 seconds: 59.69 ops/second
 - /unlink 1089 (time 1790978222.48 total 10.01 last 108.84)
 - /unlink 2141 (time 1790978232.49 total 20.01 last 105.16)
 - /unlink 3285 (time 1790978242.49 total 30.01 last 114.37)
 - /unlink 4429 (time 1790978252.51 total 40.03 last 114.20)
total: 4500 /unlink in 40.88 seconds: 110.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 657 (time 1790978269.57 total 10.02 last 65.56)
 - open/close 1280 (time 1790978279.58 total 20.04 last 62.20)
 - open/close 1904 (time 1790978289.59 total 30.04 last 62.35)
 - open/close 2552 (time 1790978299.61 total 40.06 last 64.71)
 - open/close 3123 (time 1790978309.62 total 50.07 last 57.03)
 - open/close 3721 (time 1790978319.63 total 60.08 last 59.75)
 - open/close 4334 (time 1790978329.63 total 70.09 last 61.24)
total: 4500 open/close in 72.64 seconds: 61.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1055 (time 1790978347.58 total 10.00 last 105.50)
 - /unlink 2206 (time 1790978357.58 total 20.00 last 115.09)
 - /unlink 3373 (time 1790978367.58 total 30.00 last 116.66)
 - /unlink 4378 (time 1790978377.59 total 40.02 last 100.38)
total: 4500 /unlink in 41.47 seconds: 108.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 379 (time 1790978395.85 total 10.01 last 37.88)
 - open/close 945 (time 1790978405.86 total 20.02 last 56.53)
 - open/close 1543 (time 1790978415.88 total 30.04 last 59.69)
 - open/close 2010 (time 1790978425.89 total 40.04 last 46.67)
 - open/close 2747 (time 1790978435.89 total 50.04 last 73.69)
 - open/close 3428 (time 1790978445.90 total 60.05 last 68.03)
 - open/close 4106 (time 1790978455.91 total 70.07 last 67.73)
total: 4500 open/close in 77.34 seconds: 58.19 ops/second
 - /unlink 1290 (time 1790978478.95 total 10.00 last 128.95)
 - /unlink 2410 (time 1790978488.96 total 20.01 last 111.95)
 - /unlink 3125 (time 1790978498.97 total 30.02 last 71.44)
 - /unlink 3949 (time 1790978508.97 total 40.02 last 82.37)
total: 4500 /unlink in 44.11 seconds: 102.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 751 (time 1790978529.50 total 10.02 last 74.97)
 - open/close 1352 (time 1790978539.51 total 20.02 last 60.07)
 - open/close 1933 (time 1790978549.53 total 30.04 last 57.99)
 - open/close 2581 (time 1790978559.54 total 40.05 last 64.71)
 - open/close 3133 (time 1790978569.56 total 50.07 last 55.11)
 - open/close 3713 (time 1790978579.57 total 60.09 last 57.90)
 - open/close 4234 (time 1790978589.61 total 70.12 last 51.93)
total: 4500 open/close in 74.47 seconds: 60.43 ops/second
 - /unlink 1169 (time 1790978609.39 total 10.01 last 116.82)
 - /unlink 2012 (time 1790978619.39 total 20.01 last 84.28)
 - /unlink 2966 (time 1790978629.39 total 30.01 last 95.37)
 - /unlink 4114 (time 1790978639.39 total 40.01 last 114.80)
total: 4500 /unlink in 43.27 seconds: 104.00 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11358: 95581 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg128-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg128-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg128-server
stop mds service on oleg128-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg128-server
stop mds service on oleg128-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg128-server
unloading modules via unload_modules_local on: 'oleg128-server'
oleg128-server: oleg128-server.virtnet: executing unload_modules_local
oleg128-server: kmemleak disabled
modules unloaded.