== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:59:03 (1790744343)
Stopping clients: oleg412-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg412-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg412-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg412-server
oleg412-server: oleg412-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing load_modules_local
oleg412-server: kmemleak disabled
oleg412-server: Loading modules from /home/green/git/lustre-release/lustre
oleg412-server: detected 4 online CPUs by sysfs
oleg412-server: MODOPTS_LIBCFS=
oleg412-server: Force libcfs to create 2 CPU partitions
oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg412-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg412-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg412-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all
pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg412-client.virtnet:  -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 725 (time 1790744554.78 total 10.01 last 72.46)
 - open/close 1358 (time 1790744564.79 total 20.01 last 63.25)
 - open/close 1948 (time 1790744574.80 total 30.03 last 58.93)
 - open/close 2615 (time 1790744584.80 total 40.03 last 66.67)
 - open/close 3280 (time 1790744594.80 total 50.03 last 66.49)
 - open/close 3983 (time 1790744604.81 total 60.03 last 70.29)
total: 4500 open/close in 68.63 seconds: 65.57 ops/second
 - /unlink 1249 (time 1790744628.11 total 10.00 last 124.87)
 - /unlink 2436 (time 1790744638.12 total 20.01 last 118.66)
 - /unlink 3660 (time 1790744648.12 total 30.01 last 122.35)
total: 4500 /unlink in 38.34 seconds: 117.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 646 (time 1790744673.52 total 10.00 last 64.59)
 - open/close 1191 (time 1790744683.53 total 20.01 last 54.46)
 - open/close 1798 (time 1790744693.54 total 30.02 last 60.63)
 - open/close 2444 (time 1790744703.55 total 40.03 last 64.57)
 - open/close 3111 (time 1790744713.56 total 50.04 last 66.60)
 - open/close 3662 (time 1790744723.59 total 60.07 last 54.94)
 - open/close 4364 (time 1790744733.62 total 70.09 last 70.01)
total: 4500 open/close in 72.19 seconds: 62.34 ops/second
 - /unlink 1007 (time 1790744751.75 total 10.01 last 100.62)
 - /unlink 2162 (time 1790744761.76 total 20.01 last 115.46)
 - /unlink 3396 (time 1790744771.77 total 30.02 last 123.27)
total: 4500 /unlink in 39.46 seconds: 114.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 644 (time 1790744797.45 total 10.00 last 64.40)
 - open/close 1312 (time 1790744807.48 total 20.03 last 66.62)
 - open/close 1943 (time 1790744817.49 total 30.04 last 63.02)
 - open/close 2456 (time 1790744827.50 total 40.05 last 51.27)
 - open/close 2986 (time 1790744837.51 total 50.06 last 52.94)
 - open/close 3543 (time 1790744847.53 total 60.08 last 55.58)
 - open/close 4126 (time 1790744857.55 total 70.10 last 58.20)
total: 4500 open/close in 76.35 seconds: 58.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1024 (time 1790744879.38 total 10.00 last 102.38)
 - /unlink 1945 (time 1790744889.39 total 20.01 last 92.02)
 - /unlink 2806 (time 1790744899.40 total 30.02 last 86.03)
 - /unlink 3786 (time 1790744909.40 total 40.02 last 97.96)
total: 4500 /unlink in 45.80 seconds: 98.26 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 705 (time 1790744930.13 total 10.00 last 70.48)
 - open/close 1333 (time 1790744940.14 total 20.01 last 62.76)
 - open/close 1962 (time 1790744950.14 total 30.01 last 62.89)
 - open/close 2651 (time 1790744960.15 total 40.02 last 68.82)
 - open/close 3280 (time 1790744970.16 total 50.03 last 62.82)
 - open/close 3871 (time 1790744980.17 total 60.04 last 59.06)
total: 4500 open/close in 68.87 seconds: 65.34 ops/second
 - /unlink 1238 (time 1790745003.57 total 10.01 last 123.70)
 - /unlink 2435 (time 1790745013.57 total 20.01 last 119.65)
 - /unlink 3422 (time 1790745023.57 total 30.01 last 98.67)
total: 4500 /unlink in 39.79 seconds: 113.10 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 619 (time 1790745048.24 total 10.00 last 61.89)
 - open/close 1241 (time 1790745058.25 total 20.01 last 62.13)
 - open/close 1862 (time 1790745068.26 total 30.02 last 62.05)
 - open/close 2446 (time 1790745078.27 total 40.02 last 58.39)
 - open/close 3062 (time 1790745088.27 total 50.03 last 61.58)
 - open/close 3699 (time 1790745098.28 total 60.04 last 63.62)
 - open/close 4145 (time 1790745108.29 total 70.05 last 44.56)
total: 4500 open/close in 76.51 seconds: 58.82 ops/second
 - /unlink 1289 (time 1790745130.56 total 10.00 last 128.88)
 - /unlink 2516 (time 1790745140.56 total 20.00 last 122.70)
 - /unlink 3709 (time 1790745150.56 total 30.01 last 119.24)
total: 4500 /unlink in 36.89 seconds: 121.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 530 (time 1790745173.78 total 10.00 last 52.98)
 - open/close 947 (time 1790745183.78 total 20.00 last 41.70)
 - open/close 1437 (time 1790745193.79 total 30.01 last 48.97)
 - open/close 1930 (time 1790745203.81 total 40.03 last 49.19)
 - open/close 2528 (time 1790745213.81 total 50.03 last 59.79)
 - open/close 3094 (time 1790745223.82 total 60.04 last 56.55)
 - open/close 3741 (time 1790745233.82 total 70.04 last 64.70)
 - open/close 4353 (time 1790745243.83 total 80.05 last 61.13)
total: 4500 open/close in 82.03 seconds: 54.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1283 (time 1790745260.84 total 10.01 last 128.14)
 - /unlink 2468 (time 1790745270.84 total 20.01 last 118.49)
 - /unlink 3834 (time 1790745280.84 total 30.02 last 136.56)
total: 4500 /unlink in 35.62 seconds: 126.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 678 (time 1790745300.89 total 10.01 last 67.77)
 - open/close 1366 (time 1790745310.90 total 20.02 last 68.72)
 - open/close 2062 (time 1790745320.91 total 30.02 last 69.56)
 - open/close 2708 (time 1790745330.91 total 40.03 last 64.56)
 - open/close 3401 (time 1790745340.92 total 50.03 last 69.26)
 - open/close 4133 (time 1790745350.92 total 60.04 last 73.18)
total: 4500 open/close in 65.42 seconds: 68.79 ops/second
 - /unlink 1232 (time 1790745371.00 total 10.00 last 123.18)
 - /unlink 2408 (time 1790745381.00 total 20.01 last 117.55)
 - /unlink 3632 (time 1790745391.01 total 30.01 last 122.32)
total: 4500 /unlink in 37.96 seconds: 118.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 588 (time 1790745413.86 total 10.01 last 58.77)
 - open/close 1124 (time 1790745423.87 total 20.01 last 53.56)
 - open/close 1700 (time 1790745433.89 total 30.03 last 57.48)
 - open/close 2272 (time 1790745443.90 total 40.05 last 57.13)
 - open/close 2866 (time 1790745453.92 total 50.06 last 59.31)
 - open/close 3465 (time 1790745463.92 total 60.06 last 59.88)
 - open/close 4139 (time 1790745473.92 total 70.07 last 67.39)
total: 4500 open/close in 75.41 seconds: 59.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1229 (time 1790745494.04 total 10.00 last 122.86)
 - /unlink 2426 (time 1790745504.05 total 20.01 last 119.57)
 - /unlink 3701 (time 1790745514.05 total 30.02 last 127.46)
total: 4500 /unlink in 37.35 seconds: 120.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 640 (time 1790745537.74 total 10.01 last 63.93)
 - open/close 1208 (time 1790745547.74 total 20.01 last 56.78)
 - open/close 1851 (time 1790745557.75 total 30.03 last 64.23)
 - open/close 2481 (time 1790745567.75 total 40.03 last 62.99)
 - open/close 3110 (time 1790745577.77 total 50.04 last 62.83)
 - open/close 3772 (time 1790745587.79 total 60.06 last 66.04)
 - open/close 4382 (time 1790745597.83 total 70.10 last 60.78)
total: 4500 open/close in 71.88 seconds: 62.60 ops/second
 - /unlink 1161 (time 1790745614.48 total 10.00 last 116.08)
 - /unlink 2372 (time 1790745624.48 total 20.00 last 121.06)
 - /unlink 3476 (time 1790745634.48 total 30.00 last 110.40)
total: 4500 /unlink in 38.26 seconds: 117.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 683 (time 1790745657.06 total 10.01 last 68.26)
 - open/close 1208 (time 1790745667.07 total 20.01 last 52.46)
 - open/close 1792 (time 1790745677.08 total 30.02 last 58.34)
 - open/close 2399 (time 1790745687.09 total 40.04 last 60.61)
 - open/close 3016 (time 1790745697.10 total 50.04 last 61.68)
 - open/close 3641 (time 1790745707.10 total 60.05 last 62.46)
 - open/close 4306 (time 1790745717.11 total 70.05 last 66.45)
total: 4500 open/close in 72.86 seconds: 61.76 ops/second
 - /unlink 1224 (time 1790745734.34 total 10.00 last 122.37)
 - /unlink 2523 (time 1790745744.34 total 20.01 last 129.85)
 - /unlink 3764 (time 1790745754.35 total 30.01 last 124.01)
total: 4500 /unlink in 35.95 seconds: 125.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 599 (time 1790745774.64 total 10.01 last 59.82)
 - open/close 1194 (time 1790745784.66 total 20.03 last 59.37)
 - open/close 1759 (time 1790745794.69 total 30.06 last 56.34)
 - open/close 2362 (time 1790745804.70 total 40.07 last 60.26)
 - open/close 3021 (time 1790745814.70 total 50.07 last 65.89)
 - open/close 3715 (time 1790745824.70 total 60.07 last 69.40)
 - open/close 4422 (time 1790745834.70 total 70.07 last 70.70)
total: 4500 open/close in 71.27 seconds: 63.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1201 (time 1790745849.70 total 10.01 last 120.02)
 - /unlink 2289 (time 1790745859.70 total 20.01 last 108.78)
 - /unlink 3472 (time 1790745869.70 total 30.01 last 118.26)
total: 4500 /unlink in 39.16 seconds: 114.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 612 (time 1790745893.37 total 10.01 last 61.17)
 - open/close 1208 (time 1790745903.39 total 20.02 last 59.52)
 - open/close 1793 (time 1790745913.39 total 30.02 last 58.47)
 - open/close 2420 (time 1790745923.40 total 40.03 last 62.66)
 - open/close 3058 (time 1790745933.40 total 50.04 last 63.76)
 - open/close 3649 (time 1790745943.40 total 60.04 last 59.10)
 - open/close 4327 (time 1790745953.41 total 70.04 last 67.78)
total: 4500 open/close in 72.43 seconds: 62.12 ops/second
 - /unlink 1186 (time 1790745970.68 total 10.00 last 118.59)
 - /unlink 2377 (time 1790745980.68 total 20.00 last 119.09)
 - /unlink 3637 (time 1790745990.68 total 30.00 last 125.97)
total: 4500 /unlink in 37.43 seconds: 120.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 646 (time 1790746012.45 total 10.00 last 64.58)
 - open/close 1298 (time 1790746022.46 total 20.02 last 65.09)
 - open/close 1935 (time 1790746032.49 total 30.05 last 63.50)
 - open/close 2597 (time 1790746042.50 total 40.06 last 66.15)
 - open/close 3288 (time 1790746052.52 total 50.08 last 68.94)
 - open/close 3977 (time 1790746062.54 total 60.09 last 68.82)
total: 4500 open/close in 68.13 seconds: 66.05 ops/second
 - /unlink 1239 (time 1790746085.58 total 10.01 last 123.84)
 - /unlink 2414 (time 1790746095.58 total 20.01 last 117.45)
 - /unlink 3385 (time 1790746105.59 total 30.02 last 97.05)
total: 4500 /unlink in 40.00 seconds: 112.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 665 (time 1790746130.51 total 10.00 last 66.47)
 - open/close 1294 (time 1790746140.52 total 20.02 last 62.83)
 - open/close 1948 (time 1790746150.52 total 30.02 last 65.38)
 - open/close 2617 (time 1790746160.53 total 40.03 last 66.83)
 - open/close 3327 (time 1790746170.55 total 50.04 last 70.91)
 - open/close 3938 (time 1790746180.55 total 60.04 last 61.09)
total: 4500 open/close in 68.86 seconds: 65.35 ops/second
 - /unlink 1154 (time 1790746203.90 total 10.00 last 115.40)
 - /unlink 2466 (time 1790746213.90 total 20.00 last 131.19)
 - /unlink 3614 (time 1790746223.90 total 30.00 last 114.77)
total: 4500 /unlink in 37.16 seconds: 121.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 657 (time 1790746246.31 total 10.01 last 65.64)
 - open/close 1260 (time 1790746256.32 total 20.02 last 60.24)
 - open/close 1893 (time 1790746266.34 total 30.03 last 63.20)
 - open/close 2614 (time 1790746276.34 total 40.04 last 72.09)
 - open/close 3261 (time 1790746286.35 total 50.05 last 64.63)
 - open/close 3913 (time 1790746296.35 total 60.05 last 65.17)
total: 4500 open/close in 69.08 seconds: 65.14 ops/second
 - /unlink 1310 (time 1790746320.53 total 10.02 last 130.78)
 - /unlink 2551 (time 1790746330.53 total 20.02 last 124.07)
 - /unlink 3824 (time 1790746340.54 total 30.02 last 127.25)
total: 4500 /unlink in 35.32 seconds: 127.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 682 (time 1790746360.22 total 10.02 last 68.09)
 - open/close 1367 (time 1790746370.23 total 20.02 last 68.46)
 - open/close 2023 (time 1790746380.24 total 30.03 last 65.54)
 - open/close 2687 (time 1790746390.24 total 40.03 last 66.37)
 - open/close 3370 (time 1790746400.24 total 50.04 last 68.28)
 - open/close 4056 (time 1790746410.25 total 60.04 last 68.56)
total: 4500 open/close in 66.52 seconds: 67.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1318 (time 1790746431.01 total 10.00 last 131.80)
 - /unlink 2567 (time 1790746441.01 total 20.00 last 124.87)
 - /unlink 3781 (time 1790746451.02 total 30.01 last 121.35)
total: 4500 /unlink in 35.69 seconds: 126.10 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 615 (time 1790746471.35 total 10.01 last 61.42)
 - open/close 1262 (time 1790746481.35 total 20.02 last 64.68)
 - open/close 1930 (time 1790746491.35 total 30.02 last 66.78)
 - open/close 2588 (time 1790746501.36 total 40.02 last 65.78)
 - open/close 3248 (time 1790746511.37 total 50.03 last 65.93)
 - open/close 3899 (time 1790746521.37 total 60.03 last 65.08)
total: 4500 open/close in 68.50 seconds: 65.69 ops/second
 - /unlink 1304 (time 1790746543.65 total 10.00 last 130.38)
 - /unlink 2602 (time 1790746553.66 total 20.01 last 129.73)
 - /unlink 3893 (time 1790746563.66 total 30.01 last 129.09)
total: 4500 /unlink in 34.96 seconds: 128.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 650 (time 1790746582.76 total 10.01 last 64.91)
 - open/close 1293 (time 1790746592.76 total 20.01 last 64.29)
 - open/close 1933 (time 1790746602.77 total 30.02 last 63.95)
 - open/close 2585 (time 1790746612.77 total 40.03 last 65.17)
 - open/close 3234 (time 1790746622.78 total 50.03 last 64.89)
 - open/close 3916 (time 1790746632.78 total 60.03 last 68.18)
total: 4500 open/close in 68.48 seconds: 65.71 ops/second
 - /unlink 1223 (time 1790746656.09 total 10.00 last 122.30)
 - /unlink 2435 (time 1790746666.09 total 20.00 last 121.18)
 - /unlink 3774 (time 1790746676.09 total 30.00 last 133.89)
total: 4500 /unlink in 35.74 seconds: 125.89 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11358: 95630 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg412-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg412-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server
stop mds service on oleg412-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg412-server
unloading modules via unload_modules_local on: 'oleg412-server'
oleg412-server: oleg412-server.virtnet: executing unload_modules_local
oleg412-server: kmemleak disabled
modules unloaded.