1551-1600 of 10000 results (27ms)
2014-11-10 §
08:52 <_joe_> dist-upgrading mw1189 to use the latest kernel available, then rebooting [production]
08:32 <YuviPanda> ran mklost+found on /srv/postgres for reducing cronspam [production]
08:21 <paravoid> force-rebooting ms-be2011, kernel "xfs stuck" [production]
01:55 <ori> depooled mw1114 after it became unresponsive, likely <https://phabricator.wikimedia.org/T1195> [production]
2014-11-09 §
23:44 <hoo> Changed the email for a global account. Bug 73014. [production]
21:56 <_joe_> depooling mw1189 from the api pool, see https://phabricator.wikimedia.org/T1194 [production]
19:12 <_joe_> restarted apache on mw1192, this time an hard restart [production]
17:11 <hoo> mw1192 stuck with almost no idle workers as most workers are in the "Gracefully finishing" state. Attempted to gracefully restart it, but that (to no surprise) didn't help. [production]
2014-11-08 §
20:17 <Krinkle> Jenkins/Zuul was still stuck. Disconnected and relaunched slave agents on lanthanum and gallium. This fixed it (slaves in labs were fine). [production]
20:01 <Krinkle> Jenkins/Zuul appear stuck. Disconnect/Re-enable Gearman from Jenkins. [production]
15:30 <demon> Synchronized wmf-config/InitialiseSettings.php: Gerrit I46b151ff: Reverting addition of Draft namespace to enwiki (duration: 00m 04s) [production]
11:09 <YuviPanda> ran makelost+found on /srv/postgres on labsdb1004 to kill cronspam [production]
11:07 <YuviPanda> ran makelost+found on /srv/postgres on labsdb1007 to kill cronspam [production]
2014-11-07 §
20:53 <ebernhardson> Synchronized php-1.25wmf7/extensions/Flow: Bump flow submodule for bug 71858 (duration: 00m 08s) [production]
20:33 <ori> Synchronized php-1.25wmf6/includes/WebResponse.php: I569b2ebbc: Add WebResponse::getHeader() (duration: 00m 09s) [production]
20:13 <ori> Synchronized php-1.25wmf7/includes/WebResponse.php: I569b2ebbc: Add WebResponse::getHeader() (duration: 00m 07s) [production]
20:03 <YuviPanda> restarted gitblit on antimony [production]
19:36 <YuviPanda> upgraded php5-fss to 1.0-2 on virt1000 to prevent cronspam [production]
19:36 <demon> Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 04s) [production]
16:27 <godog> shut db1017 briefly for cmjohnson to look [production]
14:46 <_joe_> installing hhvm package built with full debug symbols on mw1114 [production]
09:00 <oblivian> Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 07s) [production]
08:00 <_joe_> powercycled mw1169, console unresponsive, not responding to pings [production]
07:52 <_joe_> killed the master apache process on mw1191, stuck in a futex wait, restarted apache [production]
07:44 <_joe_> upgrading the hhvm appservers to the new package version, it seems stable enough [production]
01:01 <reedy> Synchronized php-1.25wmf7/extensions/Flow/: (no message) (duration: 00m 16s) [production]
00:49 <reedy> Synchronized php-1.25wmf6/includes/api/: (no message) (duration: 00m 14s) [production]
00:36 <reedy> Synchronized php-1.25wmf7/extensions/MobileFrontend/: (no message) (duration: 00m 16s) [production]
00:27 <reedy> Synchronized php-1.25wmf7/extensions/VisualEditor/: (no message) (duration: 00m 15s) [production]
00:26 <reedy> Synchronized php-1.25wmf6/extensions/GeoData: (no message) (duration: 00m 14s) [production]
00:03 <Reedy> running foreachwikiindblist wikidataclient.dblist extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --strip-protocols [production]
00:00 <reedy> Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 14s) [production]
00:00 <reedy> Synchronized langlist: mai (duration: 00m 14s) [production]
2014-11-06 §
23:26 <bd808> deleted corrupt mediawki/core clone in workspace/mwext-MobileFrontend-qunit-mobile on gallium [production]
23:24 <bd808> Killed 3 hung /usr/local/bin/logstash_optimize_index.sh processes on logstash1002 [production]
23:22 <bd808> restarted logstash on logstash1002 [production]
23:21 <bd808> restarted logstash on logstash1003 [production]
23:02 <bd808> restarted logstash on logstash1001 for the usual reason (no events making it to elasticsearch) [production]
22:19 <subbu> updated parsoid to d23d2be6 (+ a hotfix to the production localsettings config file) [production]
21:26 <ottomata> added Range header field to varnishkafka webrequest logs [production]
19:45 <andrewbogott> restarted ntp on labstore1001 [production]
19:33 <manybubbles> manybubbles is done with SWAT [production]
16:11 <manybubbles> Synchronized php-1.25wmf7/extensions/MultimediaViewer/: SWAT revert layout changes (duration: 00m 06s) [production]
16:02 <manybubbles> Synchronized wmf-config/: SWAT deploy some beta configs. Should be noop. (duration: 00m 04s) [production]
15:49 <_joe_> load-testing hhvm, in particular the servers with the new package [production]
15:43 <_joe_> upgrading mw1031,mw1032 to the new package, no crashes seeen since reinstall [production]
15:24 <manybubbles> finished with performance testing for cirrus - new servers look like way way more than enough power [production]
15:01 <manybubbles> dewiki is fine. trying enwiki. [production]
14:57 <manybubbles> performance test for zhwiki was good. trying dewiki [production]
14:55 <manybubbles> running performance test for Cirrus taking zhwiki [production]