|
2020-12-22
§
|
| 19:32 |
<mutante> |
restarting gerrit to pick up config change in gitiles for T269300 |
[production] |
| 18:29 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on labstore1004.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:27 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on labstore1004.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:22 |
<bstorm> |
rebooting the grid master because it is misbehaving following the NFS outage |
[tools] |
| 18:13 |
<elukey> |
failover analytics-hive.eqiad.wmnet to an-coord1002 (to allow maintenance on an-coord1001) |
[analytics] |
| 18:07 |
<elukey> |
restart hive server on an-coord1002 (current standby - no traffic) to pick up the new config (use the local metastore as opposed to what it is pointed by analytics-hive) |
[analytics] |
| 17:27 |
<andrewbogott> |
shutting down labstore1004 in preparation for move and reimage |
[production] |
| 17:00 |
<mforns> |
Deployed refinery as part of weekly train (v0.0.142) |
[analytics] |
| 16:51 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@21c0c89] (thin): Regular analytics weekly train THIN [analytics/refinery@Ie7bce02179547ee4c6756d52f9956f492c5b4df6] (duration: 00m 08s) |
[production] |
| 16:51 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@21c0c89] (thin): Regular analytics weekly train THIN [analytics/refinery@Ie7bce02179547ee4c6756d52f9956f492c5b4df6] |
[production] |
| 16:48 |
<volans> |
restarted ferm on ms-be1026 (failed with DNS query for 'ms-be1055.eqiad.wmnet' failed: query timed out ) |
[production] |
| 16:42 |
<mforns> |
Deployed refinery-source v0.0.142 |
[analytics] |
| 16:30 |
<mforns> |
Deployed refinery-source v0.0.142 |
[analytics] |
| 16:15 |
<bstorm> |
downtimed and stopped puppet on labstore1004 and labstore1005 for failover T266202 |
[production] |
| 15:30 |
<dcaro> |
cleaning up 6778 dangling snapshots for glance images in eqiad (T270478) |
[admin] |
| 15:23 |
<jgiannelos@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 15:12 |
<jgiannelos@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 15:08 |
<jgiannelos@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 15:00 |
<razzi> |
stopping superset server on analytics-tool1004 |
[analytics] |
| 13:51 |
<dcaro> |
merged patch to move wikidumpparse backups to cloudvirt1025 to free space on cloudvirt1026 |
[admin] |
| 11:52 |
<marostegui> |
Set db1151 to writable T269324 |
[production] |
| 11:16 |
<wm-bot> |
<lucaswerkmeister> deployed 6e1185532d (Basque adjective) |
[tools.lexeme-forms] |
| 11:10 |
<jbond42> |
upload puppet 5.5.22 to jessie-wikimedia |
[production] |
| 11:02 |
<jbond42> |
upload puppet 5.5.22 to stretch-wikimedia |
[production] |
| 10:53 |
<arturo> |
rebase & resolve ugly git merge conflict in labs/private.git |
[tools] |
| 10:51 |
<volans@cumin2001> |
test SAL message from wmflib, please ignore |
[production] |
| 10:48 |
<arturo> |
rebase & resolve ugly git merge conflict in labs/private.git |
[toolsbeta] |
| 10:36 |
<elukey> |
restart presto coordinator to pick up analytics-hive settings |
[analytics] |
| 10:25 |
<elukey> |
failover analytics-hive.eqiad.wmnet to an-coord1001 |
[analytics] |
| 10:06 |
<volans> |
upgraded python3-wmflib to 0.0.5 on cumin2001 |
[production] |
| 09:56 |
<elukey> |
restart hive daemons on an-coord1001 to pick up analytics-hive settings |
[analytics] |
| 08:52 |
<hashar> |
gerrit: running jhat heap analyzer on gerrit2001 # T263008 |
[production] |
| 07:27 |
<elukey> |
reboot stat100[4-8] (analytics hadoop clients) for kernel upgrades |
[analytics] |
| 07:27 |
<elukey> |
reboot stat100[4-8] (analytics hadoop clients) for kernel upgrades |
[production] |
| 07:23 |
<elukey> |
move all analytics clients (spark refine, stat100x, hive-site.xml on hdfs, etc..) to analytics-hive.eqiad.wmnet |
[analytics] |
| 00:28 |
<wikibugs> |
Updated channels.yaml to: 6687ebb2a6dc348504af31c8766820eb529cfbf7 Notify analytics for schema changes |
[tools.wikibugs] |
| 00:20 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@b17db99]: Redeploy of 2.9.10 to netbox-dev for dep test (duration: 00m 54s) |
[production] |
| 00:19 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@b17db99]: Redeploy of 2.9.10 to netbox-dev for dep test |
[production] |
|
2020-12-21
§
|
| 23:20 |
<legoktm@deploy1001> |
Synchronized docroot/noc/conf/index.php: noc: Fix "Currently active MediaWiki versions" (T235338) (duration: 00m 54s) |
[production] |
| 23:14 |
<James_F> |
Docker: Building and publishing tox-pywikibot 0.5.0 T270389 |
[releng] |
| 22:54 |
<James_F> |
Docker: Building and publishing tox-buster |
[releng] |
| 22:38 |
<bd808> |
Stopped webservice and cleaned up $HOME as tool is abandoned. |
[tools.google-drive-photos-to-commons] |
| 22:26 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@0362a12]: Deploy of 2.9.10 to netbox-dev for script testing p2 (duration: 00m 05s) |
[production] |
| 22:26 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@0362a12]: Deploy of 2.9.10 to netbox-dev for script testing p2 |
[production] |
| 22:26 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@0362a12]: Deploy of 2.9.10 to netbox-dev for script testing (duration: 01m 01s) |
[production] |
| 22:25 |
<sbassett> |
Deployed security patch T270453 |
[production] |
| 22:25 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@0362a12]: Deploy of 2.9.10 to netbox-dev for script testing |
[production] |
| 22:18 |
<chaomodus> |
Re-enabling puppet on Netbox production instances after havintg tested netbox2001 with new puppet code T266487 |
[production] |
| 22:11 |
<bstorm> |
rebuilt virtualenv for python3.5 tool to try and get it working. Tool is abandoned. |
[tools.google-drive-photos-to-commons] |
| 21:42 |
<legoktm> |
manually imported debs to buster-wikimedia thirdparty/pyall component (T241195) |
[production] |