== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:45:54 (1789127154) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg257-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg257-server oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg257-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 803 (time 1789127364.84 total 10.00 last 80.29) - open/close 1676 (time 1789127374.85 total 20.01 last 87.24) - open/close 2552 (time 1789127384.85 total 30.01 last 87.58) - open/close 3550 (time 1789127394.86 total 40.02 last 99.72) - open/close 4333 (time 1789127404.87 total 50.02 last 78.25) total: 4500 open/close in 52.04 seconds: 86.47 ops/second - /unlink 1631 (time 1789127421.18 total 10.00 last 163.06) - /unlink 3451 (time 1789127431.18 total 20.00 last 181.97) total: 4500 /unlink in 27.64 seconds: 162.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 683 (time 1789127453.24 total 10.01 last 68.26) - open/close 1538 (time 1789127463.25 total 20.01 last 85.45) - open/close 2426 (time 1789127473.27 total 30.03 last 88.63) - open/close 3234 (time 1789127483.31 total 40.07 last 80.51) - open/close 4056 (time 1789127493.31 total 50.07 last 82.14) total: 4500 open/close in 55.57 seconds: 80.98 ops/second - /unlink 1554 (time 1789127513.19 total 10.00 last 155.39) - /unlink 3062 (time 1789127523.20 total 20.00 last 150.74) total: 4500 /unlink in 29.25 seconds: 153.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 788 (time 1789127547.10 total 10.01 last 78.74) - open/close 1614 (time 1789127557.10 total 20.01 last 82.57) - open/close 2469 (time 1789127567.11 total 30.02 last 85.39) - open/close 3321 (time 1789127577.12 total 40.03 last 85.13) - open/close 4237 (time 1789127587.12 total 50.03 last 91.58) total: 4500 open/close in 53.28 seconds: 84.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1727 (time 1789127605.07 total 10.00 last 172.69) - /unlink 2990 (time 1789127615.08 total 20.01 last 126.22) - /unlink 4472 (time 1789127625.08 total 30.01 last 148.10) total: 4500 /unlink in 30.42 seconds: 147.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 950 (time 1789127639.95 total 10.00 last 94.98) - open/close 1800 (time 1789127649.96 total 20.01 last 84.92) - open/close 2614 (time 1789127659.96 total 30.01 last 81.38) - open/close 3346 (time 1789127669.97 total 40.02 last 73.15) - open/close 4041 (time 1789127679.97 total 50.02 last 69.49) total: 4500 open/close in 57.60 seconds: 78.12 ops/second - /unlink 1720 (time 1789127702.82 total 10.02 last 171.70) - /unlink 3120 (time 1789127712.82 total 20.02 last 139.99) total: 4500 /unlink in 29.82 seconds: 150.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 762 (time 1789127737.03 total 10.02 last 76.07) - open/close 1621 (time 1789127747.06 total 20.05 last 85.61) - open/close 2324 (time 1789127757.08 total 30.06 last 70.20) - open/close 3069 (time 1789127767.08 total 40.07 last 74.47) - open/close 3754 (time 1789127777.09 total 50.07 last 68.47) total: 4500 open/close in 57.92 seconds: 77.69 ops/second - /unlink 1531 (time 1789127799.00 total 10.00 last 153.03) - /unlink 2861 (time 1789127809.01 total 20.01 last 132.91) - /unlink 4214 (time 1789127819.02 total 30.02 last 135.22) total: 4500 /unlink in 32.17 seconds: 139.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 781 (time 1789127835.73 total 10.01 last 78.03) - open/close 1767 (time 1789127845.73 total 20.02 last 98.54) - open/close 2714 (time 1789127855.75 total 30.03 last 94.58) - open/close 3448 (time 1789127865.75 total 40.03 last 73.37) - open/close 4339 (time 1789127875.76 total 50.04 last 89.02) total: 4500 open/close in 52.19 seconds: 86.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1466 (time 1789127892.51 total 10.01 last 146.47) - /unlink 2997 (time 1789127902.51 total 20.01 last 153.04) - /unlink 4433 (time 1789127912.51 total 30.01 last 143.58) total: 4500 /unlink in 30.58 seconds: 147.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 913 (time 1789127926.90 total 10.00 last 91.28) - open/close 1792 (time 1789127936.91 total 20.00 last 87.87) - open/close 2647 (time 1789127946.91 total 30.00 last 85.50) - open/close 3609 (time 1789127956.91 total 40.01 last 96.14) - open/close 4448 (time 1789127966.92 total 50.02 last 83.81) total: 4500 open/close in 50.97 seconds: 88.28 ops/second - /unlink 1436 (time 1789127983.98 total 10.00 last 143.57) - /unlink 3092 (time 1789127993.98 total 20.00 last 165.55) total: 4500 /unlink in 28.56 seconds: 157.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 743 (time 1789128018.08 total 10.00 last 74.27) - open/close 1656 (time 1789128028.09 total 20.02 last 91.15) - open/close 2490 (time 1789128038.10 total 30.02 last 83.37) - open/close 3256 (time 1789128048.11 total 40.04 last 76.49) - open/close 3898 (time 1789128058.12 total 50.05 last 64.13) total: 4500 open/close in 58.61 seconds: 76.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1488 (time 1789128081.45 total 10.00 last 148.79) - /unlink 2977 (time 1789128091.45 total 20.00 last 148.84) total: 4500 /unlink in 29.70 seconds: 151.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 745 (time 1789128115.72 total 10.00 last 74.47) - open/close 1643 (time 1789128125.73 total 20.01 last 89.74) - open/close 2496 (time 1789128135.73 total 30.01 last 85.28) - open/close 3318 (time 1789128145.73 total 40.01 last 82.19) - open/close 4219 (time 1789128155.74 total 50.02 last 90.04) total: 4500 open/close in 53.26 seconds: 84.49 ops/second - /unlink 1812 (time 1789128172.85 total 10.00 last 181.18) - /unlink 3443 (time 1789128182.85 total 20.00 last 163.10) total: 4500 /unlink in 26.60 seconds: 169.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 871 (time 1789128204.54 total 10.01 last 86.99) - open/close 1711 (time 1789128214.55 total 20.02 last 83.91) - open/close 2456 (time 1789128224.57 total 30.04 last 74.39) - open/close 3301 (time 1789128234.57 total 40.04 last 84.47) - open/close 4281 (time 1789128244.57 total 50.04 last 97.99) total: 4500 open/close in 52.92 seconds: 85.03 ops/second - /unlink 1819 (time 1789128261.90 total 10.01 last 181.78) - /unlink 3486 (time 1789128271.90 total 20.01 last 166.64) total: 4500 /unlink in 25.16 seconds: 178.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 760 (time 1789128290.83 total 10.01 last 75.94) - open/close 1582 (time 1789128300.83 total 20.01 last 82.19) - open/close 2549 (time 1789128310.83 total 30.01 last 96.67) - open/close 3439 (time 1789128320.84 total 40.01 last 88.99) - open/close 4363 (time 1789128330.84 total 50.02 last 92.31) total: 4500 open/close in 51.66 seconds: 87.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1805 (time 1789128346.93 total 10.00 last 180.48) - /unlink 3250 (time 1789128356.94 total 20.00 last 144.47) total: 4500 /unlink in 27.82 seconds: 161.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 951 (time 1789128378.64 total 10.01 last 95.00) - open/close 1801 (time 1789128388.64 total 20.02 last 84.92) - open/close 2721 (time 1789128398.65 total 30.03 last 91.92) - open/close 3484 (time 1789128408.66 total 40.03 last 76.25) - open/close 4305 (time 1789128418.66 total 50.04 last 82.08) total: 4500 open/close in 52.21 seconds: 86.19 ops/second - /unlink 1480 (time 1789128435.05 total 10.00 last 147.95) - /unlink 3012 (time 1789128445.06 total 20.01 last 153.06) total: 4500 /unlink in 28.46 seconds: 158.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 697 (time 1789128469.38 total 10.00 last 69.70) - open/close 1568 (time 1789128479.38 total 20.00 last 87.07) - open/close 2354 (time 1789128489.38 total 30.01 last 78.58) - open/close 3282 (time 1789128499.41 total 40.03 last 92.55) - open/close 4143 (time 1789128509.43 total 50.05 last 85.98) total: 4500 open/close in 54.27 seconds: 82.92 ops/second - /unlink 1486 (time 1789128528.58 total 10.02 last 148.35) - /unlink 2800 (time 1789128538.58 total 20.02 last 131.36) - /unlink 4392 (time 1789128548.58 total 30.02 last 159.19) total: 4500 /unlink in 30.83 seconds: 145.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 825 (time 1789128564.02 total 10.01 last 82.42) - open/close 1644 (time 1789128574.02 total 20.01 last 81.86) - open/close 2466 (time 1789128584.03 total 30.02 last 82.16) - open/close 3354 (time 1789128594.03 total 40.03 last 88.75) - open/close 4322 (time 1789128604.04 total 50.03 last 96.75) total: 4500 open/close in 52.15 seconds: 86.28 ops/second - /unlink 1810 (time 1789128620.20 total 10.00 last 180.96) - /unlink 3356 (time 1789128630.20 total 20.00 last 154.57) total: 4500 /unlink in 28.42 seconds: 158.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 721 (time 1789128653.58 total 10.01 last 72.03) - open/close 1617 (time 1789128663.58 total 20.01 last 89.60) - open/close 2580 (time 1789128673.58 total 30.02 last 96.25) - open/close 3454 (time 1789128683.59 total 40.02 last 87.37) - open/close 4490 (time 1789128693.59 total 50.03 last 103.54) total: 4500 open/close in 50.11 seconds: 89.80 ops/second - /unlink 1835 (time 1789128707.27 total 10.00 last 183.45) - /unlink 3408 (time 1789128717.27 total 20.00 last 157.27) total: 4500 /unlink in 27.55 seconds: 163.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 822 (time 1789128739.94 total 10.01 last 82.13) - open/close 1636 (time 1789128749.95 total 20.01 last 81.35) - open/close 2551 (time 1789128759.95 total 30.02 last 91.45) - open/close 3300 (time 1789128769.96 total 40.03 last 74.86) - open/close 4083 (time 1789128779.96 total 50.03 last 78.25) total: 4500 open/close in 55.62 seconds: 80.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1470 (time 1789128801.18 total 10.01 last 146.89) - /unlink 3041 (time 1789128811.18 total 20.01 last 157.04) - /unlink 4446 (time 1789128821.19 total 30.02 last 140.42) total: 4500 /unlink in 30.88 seconds: 145.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 707 (time 1789128837.53 total 10.00 last 70.68) - open/close 1377 (time 1789128847.54 total 20.01 last 66.92) - open/close 2084 (time 1789128857.55 total 30.02 last 70.66) - open/close 2958 (time 1789128867.55 total 40.02 last 87.36) - open/close 3775 (time 1789128877.56 total 50.04 last 81.61) total: 4500 open/close in 58.88 seconds: 76.43 ops/second - /unlink 1305 (time 1789128901.35 total 10.00 last 130.46) - /unlink 2785 (time 1789128911.35 total 20.01 last 147.92) - /unlink 4399 (time 1789128921.36 total 30.01 last 161.35) total: 4500 /unlink in 31.06 seconds: 144.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 856 (time 1789128936.96 total 10.00 last 85.58) - open/close 1687 (time 1789128946.97 total 20.01 last 83.04) - open/close 2509 (time 1789128956.97 total 30.01 last 82.18) - open/close 3289 (time 1789128966.97 total 40.01 last 77.98) - open/close 4151 (time 1789128976.98 total 50.03 last 86.08) total: 4500 open/close in 54.53 seconds: 82.53 ops/second - /unlink 1264 (time 1789128996.61 total 10.01 last 126.30) - /unlink 2797 (time 1789129006.62 total 20.01 last 153.22) - /unlink 4352 (time 1789129016.62 total 30.02 last 155.47) total: 4500 /unlink in 31.05 seconds: 144.92 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11449: 95244 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: modules unloaded.