|
2016-03-29
§
|
| 14:54 |
<moritzm> |
depooled maps-test2003 (to apply ferm) |
[production] |
| 13:53 |
<moritzm> |
repooled maps-test2002 (nodejs being put on hold prevented generation of ferm rules provided by service::node, this has been fixed) |
[production] |
| 13:14 |
<jynus> |
restoring (but not provisioning) older backup from labswiki sql |
[production] |
| 13:08 |
<moritzm> |
unheld nodejs on maps* |
[production] |
| 11:51 |
<hashar> |
Jenkins / Zuul lagging out trying to catch up with a huge number of changes I have sent |
[production] |
| 10:11 |
<reedy@tin> |
Synchronized wmf-config/throttle.php: Remove old throttle rules (duration: 00m 44s) |
[production] |
| 09:50 |
<godog> |
powercycle ms-be2008, got stuck again while diagnosing failed /dev/sdl |
[production] |
| 09:08 |
<paravoid> |
hard-resetting alsafi, I/O-stuck (qemu bug?) |
[production] |
| 09:05 |
<paravoid> |
hard-resetting mx1001, I/O-stuck (qemu bug?) |
[production] |
| 08:45 |
<moritzm> |
upgraded chromium on osmium |
[production] |
| 03:29 |
<bblack> |
cr1/2-eqiad: re-activating static routes for ns0/ns1 (ipv4/ipv6) pointing to radon - |
[production] |
| 02:45 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki zhwikisource |
[production] |
| 02:45 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki zhwikinews |
[production] |
| 02:36 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue Mar 29 02:36:39 UTC 2016 (duration 8m 46s) |
[production] |
| 02:27 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.18) (duration: 11m 34s) |
[production] |
| 01:07 |
<Pchelolo> |
restbase finish update to 3ea08751a8 |
[production] |
| 01:00 |
<Pchelolo> |
restbase start update to 3ea08751a8 |
[production] |
| 00:15 |
<krinkle@tin> |
Synchronized docroot/wikipedia.org/speed-tests/: (no message) (duration: 00m 32s) |
[production] |
| 00:12 |
<mutante> |
mw2090 - reinstalled, re-signed puppet/salt |
[production] |
|
2016-03-28
§
|
| 23:38 |
<maxsem@tin> |
Synchronized php-1.27.0-wmf.18/includes/api/ApiUpload.php: https://gerrit.wikimedia.org/r/#/c/280092/ (duration: 02m 26s) |
[production] |
| 23:34 |
<mutante> |
mw2090 - reboot, reinstall as regular appserver |
[production] |
| 23:27 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/280115/ (duration: 00m 35s) |
[production] |
| 23:18 |
<maxsem@tin> |
Synchronized php-1.27.0-wmf.18/extensions/Echo/: https://gerrit.wikimedia.org/r/#/c/280112/ (duration: 00m 43s) |
[production] |
| 23:16 |
<maxsem@tin> |
Synchronized php-1.27.0-wmf.18/extensions/MobileApp/: https://gerrit.wikimedia.org/r/#q,279980,n,z (duration: 01m 07s) |
[production] |
| 22:03 |
<mutante> |
mw2090 - disable puppet/cron, crons now on wasat |
[production] |
| 21:46 |
<mutante> |
mw1119 restarted hhvm |
[production] |
| 21:26 |
<bblack> |
http/2 enabled on pinkunicorn.wikimedia.org for testing - T96848 |
[production] |
| 21:26 |
<bblack> |
http/2 enabled on pinkunicorn.wikimedia.org for testing |
[production] |
| 21:12 |
<Pchelolo> |
restbase finised deploy of 78e6eab37c |
[production] |
| 21:04 |
<Pchelolo> |
restbase start deploy of 78e6eab37c |
[production] |
| 21:04 |
<mdholloway> |
mobileapps deployed 90cfdcd |
[production] |
| 20:37 |
<Pchelolo> |
restbase canary deploy of 78e6eab37c to restbase1005 |
[production] |
| 20:35 |
<mdholloway> |
starting mobileapps deployment |
[production] |
| 18:30 |
<andrewbogott> |
labs public DNS will remain in a transitional state until MarkMonitor updates their records and the TTL expires. DNS resolution should proceed as usual in the meantime. |
[production] |
| 18:03 |
<mutante> |
mw1140 restart hhvm |
[production] |
| 17:52 |
<cmjohnson1> |
shutting down radon to re-apply thermal paste: |
[production] |
| 17:45 |
<mutante> |
mw1152,mw2090,terbium apt-get remove python-mysqldb (T84075) |
[production] |
| 17:44 |
<legoktm> |
manually fixed T129881 rename |
[production] |
| 17:20 |
<paravoid> |
cr1/2-eqiad: deactivating static routes for ns0/ns1 (ipv4/ipv6) pointing to radon |
[production] |
| 17:18 |
<bblack> |
rebooting radon again |
[production] |
| 15:31 |
<thcipriani@tin> |
Synchronized php-1.27.0-wmf.18/extensions/OpenStackManager/OpenStackManager.php: SWAT: Wikitech: Remove address, domain, proxy special pages [[gerrit:279569]] (duration: 00m 31s) |
[production] |
| 13:32 |
<bblack> |
rebooting radon (unresponsive console) |
[production] |
| 08:44 |
<paravoid> |
force-rebooting ms-be2008 to fix hundreds of unkillable mkfs.xfs stuck due to a hosed disk |
[production] |
| 08:21 |
<paravoid> |
powercycling ms-be2016, down for ~1d, unresponsive on console |
[production] |
| 02:35 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Mar 28 02:35:00 UTC 2016 (duration 8m 26s) |
[production] |
| 02:26 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.18) (duration: 11m 32s) |
[production] |