151-200 of 10000 results (25ms)
2021-05-12 §
10:01 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host poolcounter2003.codfw.wmnet [production]
10:01 <effie> reboot poolcounter2003 and poolcounter2004 [production]
09:55 <volans@cumin2002> START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: Initial deploy to cumin2002 - volans@cumin2002 [production]
09:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1101:3317', diff saved to https://phabricator.wikimedia.org/P15940 and previous config saved to /var/cache/conftool/dbconfig/20210512-093333-marostegui.json [production]
09:33 <marostegui@cumin1001> dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 100%: Repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15939 and previous config saved to /var/cache/conftool/dbconfig/20210512-093308-root.json [production]
09:20 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1074.eqiad.wmnet [production]
09:18 <marostegui@cumin1001> dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 75%: Repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15938 and previous config saved to /var/cache/conftool/dbconfig/20210512-091804-root.json [production]
09:10 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission for hosts db1074.eqiad.wmnet [production]
09:03 <marostegui@cumin1001> dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 50%: Repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15937 and previous config saved to /var/cache/conftool/dbconfig/20210512-090301-root.json [production]
08:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 25%: Repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15936 and previous config saved to /var/cache/conftool/dbconfig/20210512-084757-root.json [production]
08:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1074 from dbctl T281959', diff saved to https://phabricator.wikimedia.org/P15935 and previous config saved to /var/cache/conftool/dbconfig/20210512-084755-marostegui.json [production]
08:23 <jbond42> rolling restart of ats [production]
07:10 <marostegui@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 100%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15934 and previous config saved to /var/cache/conftool/dbconfig/20210512-071017-root.json [production]
07:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15933 and previous config saved to /var/cache/conftool/dbconfig/20210512-070202-marostegui.json [production]
06:55 <marostegui@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 75%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15932 and previous config saved to /var/cache/conftool/dbconfig/20210512-065513-root.json [production]
06:40 <marostegui@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 50%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15931 and previous config saved to /var/cache/conftool/dbconfig/20210512-064009-root.json [production]
06:25 <marostegui@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 25%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15930 and previous config saved to /var/cache/conftool/dbconfig/20210512-062506-root.json [production]
06:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P15929 and previous config saved to /var/cache/conftool/dbconfig/20210512-062118-marostegui.json [production]
06:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2121 and db2108 in s7 T282535', diff saved to https://phabricator.wikimedia.org/P15928 and previous config saved to /var/cache/conftool/dbconfig/20210512-062046-marostegui.json [production]
06:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1181 (re)pooling @ 100%: Repool db1181', diff saved to https://phabricator.wikimedia.org/P15927 and previous config saved to /var/cache/conftool/dbconfig/20210512-061702-root.json [production]
06:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Move db2148 to also serve vslow in s2 T282535', diff saved to https://phabricator.wikimedia.org/P15926 and previous config saved to /var/cache/conftool/dbconfig/20210512-060817-marostegui.json [production]
06:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db1181 (re)pooling @ 75%: Repool db1181', diff saved to https://phabricator.wikimedia.org/P15925 and previous config saved to /var/cache/conftool/dbconfig/20210512-060158-root.json [production]
05:46 <marostegui@cumin1001> dbctl commit (dc=all): 'db1181 (re)pooling @ 50%: Repool db1181', diff saved to https://phabricator.wikimedia.org/P15924 and previous config saved to /var/cache/conftool/dbconfig/20210512-054655-root.json [production]
05:31 <marostegui@cumin1001> dbctl commit (dc=all): 'db1181 (re)pooling @ 25%: Repool db1181', diff saved to https://phabricator.wikimedia.org/P15923 and previous config saved to /var/cache/conftool/dbconfig/20210512-053151-root.json [production]
05:00 <marostegui> Stop MySQL on labsdb1009 labsdb1010 labsdb1011 T282524 T282523 T282522 [production]
04:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1181', diff saved to https://phabricator.wikimedia.org/P15922 and previous config saved to /var/cache/conftool/dbconfig/20210512-044728-marostegui.json [production]
04:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2121 T282535', diff saved to https://phabricator.wikimedia.org/P15920 and previous config saved to /var/cache/conftool/dbconfig/20210512-044222-marostegui.json [production]
04:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2108 T282535', diff saved to https://phabricator.wikimedia.org/P15919 and previous config saved to /var/cache/conftool/dbconfig/20210512-044109-marostegui.json [production]
04:38 <marostegui> Drop testing mailman3 databases T281548 [production]
04:36 <Amir1> importing archives of wikitech-l (T280322) [production]
01:42 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1 day, 0:00:00 on people2002.codfw.wmnet with reason: new host [production]
01:42 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on people2002.codfw.wmnet with reason: new host [production]
01:35 <mutante> people2002 - created new VM resembling people2001, signed puppet cert request, initial puppet run T280989 [production]
01:19 <tstarling@deploy1002> Synchronized php-1.37.0-wmf.5/includes/specialpage/ChangesListSpecialPage.php: T282183 fix hidemyself in RC and watchlist (duration: 01m 08s) [production]
01:17 <tstarling@deploy1002> Synchronized php-1.37.0-wmf.4/includes/specialpage/ChangesListSpecialPage.php: T282183 fix hidemyself in RC and watchlist (duration: 01m 16s) [production]
00:54 <mutante> made public_html dirs on people1002 readonly to make it obvious it is not the active backend anymore [production]
00:51 <mutante> [people1002:/home] $ sudo find . -type d -name public_html -exec chmod 555 {} \; [production]
2021-05-11 §
23:19 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: ec37795eba5faa0c1a1dddb29504941205e155b4: Change namespace names and aliases on tiwiki and tiwiktionary (T263840) (duration: 01m 07s) [production]
23:16 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 5bc40acfa6514f7940af0a6cef9974140680f4b9: ptwiki: Use celebration logos in new vector (T281925) (duration: 01m 06s) [production]
23:08 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: eac843a69574feebbf962959e5eb9811a2a83bc4: Make DT source mode toolbar available as beta on all wikis (T279124) (duration: 01m 12s) [production]
23:06 <urbanecm@deploy1002> Synchronized static/images/mobile/copyright/wikipedia-pt-20.png: 60e6e4e960ee6cb31df9ce08fdeaedb647ce3afb: ptwiki: Add wikipedia-pt-20.png (T281925) (duration: 01m 08s) [production]
23:02 <urbanecm@deploy1002> Synchronized static/images/mobile/copyright/: e35199baf01f423015905b8fec9e419ed3529787: Adding square logo and wordmark for ptwiki 20 years celebration (T281925) (duration: 01m 50s) [production]
22:14 <legoktm@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts lists1002.wikimedia.org [production]
22:05 <legoktm@cumin1001> START - Cookbook sre.hosts.decommission for hosts lists1002.wikimedia.org [production]
21:37 <Urbanecm> Start server-side upload for 3 video files (T282566, T282565, T282559) [production]
21:37 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1012.eqiad.wmnet with reason: REIMAGE [production]
21:34 <herron@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1012.eqiad.wmnet with reason: REIMAGE [production]
20:52 <legoktm> upgraded mailman3 on lists1001 [production]
20:37 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host people2002.codfw.wmnet [production]
20:24 <mforns@deploy1002> Finished deploy [analytics/refinery@270c753] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@270c753fc746b979cf90e1537f9a67ede6372795] (duration: 06m 57s) [production]