2201-2250 of 10000 results (41ms)
2017-06-05 §
06:13 <marostegui> Deploy alter table on s4 - db1053 - T166206 [production]
06:12 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1053 - T166206 (duration: 00m 39s) [production]
05:58 <marostegui> Stop MySQL on db1095 to take a backup - T153743 [production]
05:56 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add coments to db1089's current status (duration: 00m 39s) [production]
02:27 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Jun 5 02:27:53 UTC 2017 (duration 6m 2s) [production]
02:21 <l10nupdate@tin> scap sync-l10n completed (1.30.0-wmf.2) (duration: 08m 14s) [production]
2017-06-04 §
10:31 <ema> mw2256 down, console stuck on 'Starti'. power cycled. [production]
02:23 <l10nupdate@tin> scap sync-l10n completed (1.30.0-wmf.2) (duration: 09m 12s) [production]
2017-06-03 §
05:20 <marostegui> Reboot db1089 - T166933 [production]
05:08 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1059 - it is broken (duration: 00m 41s) [production]
02:30 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Jun 3 02:30:27 UTC 2017 (duration 6m 24s) [production]
02:24 <l10nupdate@tin> scap sync-l10n completed (1.30.0-wmf.2) (duration: 07m 47s) [production]
00:04 <mutante> wikitech-static-iad: mv /etc/acme/cert/wikitech-static-iad-signed.csr /etc/acme/cert/wikitech-static-iad.chained.crt ; wikitech-static-ord: copy wiki logo: /srv/mediawiki/images# wget https://wikitech-static-iad.wikimedia.org/w/images/labswiki.png [production]
2017-06-02 §
23:53 <demon@tin> Synchronized wmf-config/throttle.php: pruning some old throttle exceptions (duration: 00m 40s) [production]
23:46 <mutante> wikitech-static-iad: edited acme_tiny.py to adjust URL to agreement PDF, to fix ""Provided agreement URL [https://letsencrypt.org/documents/LE-SA-v1.0.1-July-27-2015.pdf] does not match current agreement URL[https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf]" [production]
23:45 <mutante> wikitech-static-iad: create new cert for "iad" hostname, using acme-setup/acme-tiny: /usr/local/sbin# acme-setup -i "wikitech-static-iad" -s "wikitech-static-iad.wikimedia.org" ; python acme_tiny.py --account-key /etc/acme/acct/acct.key --csr /etc/acme/csr/wikitech-static-iad.pem --acme-dir /var/acme/challenge/ > /etc/acme/cert/wikitech-static-iad-signed.csr ; had to hack acme_tiny.py [production]
23:22 <mutante> wikitech-static-ord copied Lets-Encrypt intermediate certs from /usr/local/share/ca-certificates on old server [production]
23:19 <mutante> wikitech-static (iad): adjust Apache config to use wikitech-static-iad [production]
23:18 <mutante> wikitech-static-ord: installed package upgrades, installed vim, removing "ord" from Apache config after DNS change .. [production]
23:14 <mutante> maintenance on status.wikimedia.org and wikitech-static.wikimedia.org [production]
20:08 <ejegg> re-enabled AstroPay/dLocal payment methods [production]
19:36 <ejegg> updated payments-wiki from 5edd788728e422fcbeba77cf1effb92c454f80df to 7a5054238169834da0336e805946c7ed73f4795b [production]
19:23 <ejegg> updated CiviCRM from 9c06bd2f4c2298607de995efc114baae79d6476a to c70ae650bdf2e670d1dcb77f471b3ac9c5fb05f9 [production]
18:29 <mobrovac@tin> Finished deploy [restbase/deploy@4b14527]: (no justification provided) (duration: 00m 41s) [production]
18:29 <mobrovac@tin> Started deploy [restbase/deploy@4b14527]: (no justification provided) [production]
18:28 <mobrovac@tin> Started deploy [restbase/deploy@4b14527]: h [production]
17:01 <bblack> starting wmf-auto-reimage on lvs1007-10 [production]
16:16 <RainbowSprinkles> gerrit2001: gerrit updated to 2.13.8+git1-wmf.4 [production]
16:03 <bblack> start wmf-auto-reimage of lvs1011, lvs1012 [production]
15:01 <jynus> restarting ircecho on tegment [production]
14:32 <mobrovac@tin> Finished deploy [restbase/deploy@4b14527]: Add the extract_html property to the summary end point for T165017 (duration: 06m 43s) [production]
14:25 <mobrovac@tin> Started deploy [restbase/deploy@4b14527]: Add the extract_html property to the summary end point for T165017 [production]
13:28 <gehel> restart elastic2003 to reload logging configuration [production]
12:11 <hashar> restarting Jenkins to upgrade the logstash plugin [production]
09:49 <jynus> stopping db2041 to prepare it for reimage [production]
09:18 <marostegui> Deploy alter table s3 - db1015 - T166278 [production]
09:12 <marostegui> Deploy alter table s3 - labsdb1003 - T166278 [production]
07:47 <marostegui> Resume alter table on db1047 enwiki.revision - T166452 [production]
07:45 <moritzm> uploaded gerrit 2.13.8+git1-wmf4 to apt.wikimedia.org [production]
07:43 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1059 - T166206 (duration: 00m 39s) [production]
07:36 <marostegui> Deploy alter table on s4 - labsdb1009 - T166206 [production]
07:02 <akosiaris> starting fleet wide PCC for gerrit change 356030. Should take a while to complete [production]
05:25 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Emergency pool of db1049 (duration: 00m 48s) [production]
04:42 <elukey> removed some old scap revs for the Analytics refinery on stat1002 to free space (git fat jars replicating after each deployment, known issue) [production]
02:46 <bd808> Loadavg on mw1198 very high (44+) and nginx/hhvm checks flapping [production]
2017-06-01 §
23:33 <twentyafterfour> phabricator upgrade complete. [production]
23:29 <twentyafterfour> Performing phabricator update, expect momentary downtime. [production]
23:25 <twentyafterfour> Preparing phabricator update to tag release/2017-06-01/1 [ https://phabricator.wikimedia.org/project/view/2802/ ] [production]
23:20 <ebernhardson@tin> Synchronized wmf-config/CirrusSearch-common.php: T163463: apply sister search restrictions requested by enwiki (duration: 00m 39s) [production]
23:18 <ebernhardson@tin> Synchronized wmf-config/InitialiseSettings.php: T163463: apply sister search restrictions requested by enwiki (duration: 00m 40s) [production]