4001-4050 of 10000 results (49ms)
2017-03-24 §
02:24 <MaxSem> Killed l10nupdate on tin, was blocking emergency pushes [production]
02:22 <Krinkle> Hard-killed all l10nupdate processes and rm'ed scap lock [production]
02:11 <Krinkle> Removing xhgui.results entries from before 1 December 2016 in MongoDB on tungsten (T161196) [production]
01:45 <mutante> bacula - on helium, attempt to start bacula-director process, attempt to fix permissions on key files as codified in director.pp [production]
01:40 <catrope@tin> Finished scap: Wikidata cherry-picks (with i18n) (duration: 25m 03s) [production]
01:15 <catrope@tin> Started scap: Wikidata cherry-picks (with i18n) [production]
2017-03-23 §
23:26 <Krinkle> Removing xhgui.results entries from before 1 June 2016 (T161196) [production]
23:12 <thcipriani@tin> Synchronized php-1.29.0-wmf.17/extensions/ORES: SWAT: [[gerrit:344555|Stats: Invert "false" thresholds so they are correct]] T161250 (duration: 00m 52s) [production]
23:05 <Pchelolo> update RESTBase to 2536b25c7 - eqiad [production]
22:56 <Pchelolo> update RESTBase to 2536b25c7 - staging [production]
22:39 <Pchelolo> update RESTBase to 2536b25c7 - codfw [production]
21:36 <krinkle@tin> Synchronized wmf-config/StartProfiler.php: (no justification provided) (duration: 00m 53s) [production]
21:05 <ejegg> rolled back payments-wiki to 9622a4bf331426ce85362e2fdb442f226068e971 [production]
21:00 <ejegg> updated payments from 9622a4bf331426ce85362e2fdb442f226068e971 to bb956bf1ea7f0c7a9b08877cce7f25b201bd2bbf [production]
20:16 <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.17 [production]
19:34 <thcipriani@tin> Synchronized php: Swap symlink for 1.29.0-wmf.17 (duration: 00m 43s) [production]
19:11 <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.17 [production]
18:51 <bblack> systemctl enable+start of lldpd on cp2009, cp1051, cp1061 (mysteriously dead and disabled) [production]
18:16 <dereckson@tin> Synchronized wmf-config/InitialiseSettings.php: Remove exception on Other Projects sidebar for Dutch Wikipedia (T159634) (duration: 00m 47s) [production]
18:04 <Pchelolo> update RESTBase to 9d2b393fb - production [production]
17:52 <Pchelolo> update RESTBase to 9d2b393fb - staging [production]
16:46 <mobrovac@tin> Started restart [parsoid/deploy@0c22f72]: (no justification provided) [production]
16:45 <_joe_> reenabling puppet on all jobqueue redises [production]
16:24 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase db1082 weight - T137191 (duration: 00m 43s) [production]
16:22 <hashar> Merged operations/puppet.git Jenkins job in a single one that runs tox then rake - T160923 [production]
16:10 <urandom> T111113: Live-hacking client encryption to be non-optional, to verify cqlsh encryption, restbase1007-a.eqiad.wmnet [production]
16:07 <mobrovac> restbase deploy 752ca4b7 [production]
15:36 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase db1082 weight - T137191 (duration: 00m 43s) [production]
15:32 <moritzm> upgrading restbase-test* to Linux 4.9 [production]
14:59 <akosiaris> enabling and running puppet on rdb200X fleet in a rolling restart scheme [production]
14:59 <akosiaris> disabled puppet on rdb* fleet [production]
14:56 <andrewbogott> dist-upgrading labvirt1001 and rebooting it a few times [production]
14:22 <moritzm> installing exim4 updates from jessie point release [production]
14:03 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1082 with low weight - T137191 (duration: 00m 48s) [production]
13:53 <dcausse> cirrus: refreshing comp suggest indices in elastic@eqiad to measure times [production]
12:59 <marostegui> Deploy schema change s4 on db1064 https://phabricator.wikimedia.org/T160415 - https://phabricator.wikimedia.org/T73563 [production]
12:58 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1059, depool db1064 T160415 - T73563 (duration: 00m 43s) [production]
12:27 <moritzm> installing libxml2 security updates [production]
12:21 <marostegui> Deploy schema change s4 on labsdb1003 https://phabricator.wikimedia.org/T160415 - https://phabricator.wikimedia.org/T73563 [production]
12:04 <jynus> converting es2014 tables back to uncompressed InnoDB T129350 [production]
11:08 <godog> codfw-prod: bump ms-be2028 ms-be2039 object weight to 2000 T158337 [production]
11:01 <godog> pool prometheus200[34] / depool prometheus200[12] - T148408 [production]
11:01 <jynus@tin> Synchronized wmf-config/db-codfw.php: Pool all es2XXX servers, depool es2014 for maintenance (duration: 00m 43s) [production]
10:59 <hashar> Actually restarting Jenkins for email plugins upgrades [production]
10:20 <hashar> Jenkins jobs got slightly blocked because I forgot to cancel the shutdown when jobs had to run. [production]
09:58 <hashar> Jenkins: upgrading plugins email-ext and mailer [production]
09:14 <hashar> Jenkins upgrading SSH Slaves plugin. Might cause disruption in CI [production]
08:47 <moritzm> repooled mw1261 now that T161095 is deployed [production]
08:29 <marostegui> Stop db1070 MySQL db1070 for maintenance - T137191 [production]
08:06 <moritzm> installing audiofile security updates [production]