Thank you both, this is helpful!

I'm sorry for being still very clueless about databases in mediawiki (and in wmf prod in particular). But from the parent task I gather that the LBFactory class (or service?) is the problem?

Looking at logstash with the link provided by Gergő (https://logstash.wikimedia.org/goto/facfbc8c0ea8f6503302c13a60a6e6e5), I can confirm that this is still happening at the order of ~1250 events in the last 30 days.

Let's wait for at least one, but maybe even 2 branch-cuts before starting the work to remove this feature flag. So I'm setting this to stalled until Wednesday, November 20th, 2024.

In T367724#10310085, @Urbanecm_WMF wrote:

In T367724#10309993, @Michael wrote:

Looking through this, I notice that GEStructuredTaskRejectionReasonTextInputEnabled seems to be false everywhere!?! Is that actually a feature that is intended to be removed?

At this point, quite probably! This is a perfect example of the first Tim Starling's law :/. We originally added this in 2022 as part of T304099: Structured tasks: temporary free text for "other" rejection reason, to research what reasons we might be missing. Users enter surprising things, so we turned it off as fast as we could. Removing the feature seems like a good idea, unless @KStoller-WMF thinks otherwise. Let's have a separate task to make+implement that decision?

Looking through this, I notice that GEStructuredTaskRejectionReasonTextInputEnabled seems to be false everywhere!?! Is that actually a feature that is intended to be removed?

In T379345#10304487, @CCiufo-WMF wrote:

@Michael just want to confirm you're hoping to use the Codex Card for the "showcase" of each suggested edit, and not the entire "container" that the feature is within, right? (the part in red in the below screenshot).

Waiting on T310632: Card: Add the "portrait" orientation.

Not yet fully done, but ready for a first round of overall feedback.

I just checked our logstash again and can't find the expected error message. Strange. But I think we can close this for now, no point in keeping it open.

I noticed a couple more of these, various usernames, all without colons:

In T378983#10291658, @dcausse wrote:

In T378983#10289740, @Michael wrote:

Playing around with the dashboard, I'm noticing that there seem to be no set-events for recommendation_image and recommendation_image_section either.
So maybe those are affected as well? But maybe I'm misunderstanding how they work.

I believe that these ones are handled by a spark job of the SD team, these are not yet using the Search Update Pipeline and thus not covered by this metric, @pfischer is working on unifying all these sources so hopefully we should see those in this graph at some point.

Not sure which Wikidata team owns this. Feel free to adjust the tags as appropriate and prioritize as you see fit.

Might have caused T379095: CommunityConfiguration merges are blocked by a composer dependency mismatch

This is blocking us from fixing a production error (T379094) that would in turn be likely a significant degradation of our features (train blocker?)

In T373459#10289027, @dcausse wrote:

@pfischer/@Michael a dashboard is up at: https://grafana-rw.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1
I'm not sure how to craft an alert based on these numbers, please let me know if you have ideas. Moving to Blocked/Waiting while we decide on if we want to set up an alert and how.

Playing around with the dashboard, I'm noticing that there seem to be no set-events for recommendation_image and recommendation_image_section either.
So maybe those are affected as well? But maybe I'm misunderstanding how they work.

In T378983#10289670, @Michael wrote:

I'm seeing over 10.000 errors stemming from this in MediaWiki Logstash too, in the EventBus channel. The first event is from Oct 30, 2024 @ 14:19:03.375 (UTC, I think)

[...]

This is being worked on in T378983: Add Link recommendation are not being processed by CirrusSearch (November 2024).

I'm seeing over 10.000 errors stemming from this in MediaWiki Logstash too, in the EventBus channel. The first event is from Oct 30, 2024 @ 14:19:03.375 (UTC, I think)

What is the state of this? I'm asking, because looking at things on our end, it seems that the "database-and-search-index"-pairs are missing the search-index part. And the staircase-pattern makes me suspect that something is going wrong when the maintenance script every couple of hours is trying to add new records both to the database and to the search-index. Adding to the database succeeds, adding to the search-index seems to fail, thus the dangling record.

In T378177#10279227, @AnneT wrote:

[...]
Even weirder: we have unit tests that check the input event that's emitted when an item is selected, including the value attached to the event, and they work. cc @egardner

In T378770#10282953, @KStoller-WMF wrote:

@Cyndymediawiksim, @Sgs - I can't remember, was there a technical reason why we aren't respecting image transparency, or is this a bug?

In T371678#10275309, @Etonkovidova wrote:

@Michael - it seems that I cannot correctly supply the provider's name for the script. I thought that the field's name is a provider's name, e.g.EHomepageSuggestedEditsIntroLinks - but it's not correct.
Where the providers' names are stored? I couldn't find them in extension.json, but probably I just didn't know how to look.

QA Note: This change should not cause any user-visible changes at all. But if anything seems out of the ordinary with Notifications (or emails or push messages), then please highlight that here.

I've marked this as a risky change for the train next week: T375661#10277259
I don't expect any problems beyond the existing T378349, but this is changing many aspects for Echo, not sure that I've covered them all.

1. Risky Patch! 🚂🔥

nevermind, wrong week

QA Note: This change only affects the PageTitleControl and CommonsFileControl components. They can be both tested on the CommunityConfiguration form for CommunityUpdates, for example on https://cs.wikipedia.beta.wmflabs.org/wiki/Speci%C3%A1ln%C3%AD:Komunitn%C3%AD_konfigurace/CommunityUpdates

Running the fixLinkRecommendationData.php maintenance script again on eswiki has reduced the number of dangling recommendations by another order of magnitude down to 1000:

migr@mwmaint2002:/srv/mediawiki/php-1.43.0-wmf.28$ mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=eswiki --search-index --dry-run
DEPRECATION WARNING: Maintenance scripts are moving to Kubernetes. See
https://wikitech.wikimedia.org/wiki/Maintenance_scripts for the new process.
Maintenance hosts will be going away; please submit feedback promptly if
maintenance scripts on Kubernetes don't work for you. (T341553)
  topic biography had more than 10K tasks
  topic media had more than 10K tasks
  topic europe had more than 10K tasks
  topic stem had more than 10K tasks
Total number of OK search index entries: 134126
 (results in multiple topics counted multiple times)Total number of dangling search-index entries: 10673
migr@mwmaint2002:/srv/mediawiki/php-1.43.0-wmf.28$ mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=eswiki --search-index
DEPRECATION WARNING: Maintenance scripts are moving to Kubernetes. See
https://wikitech.wikimedia.org/wiki/Maintenance_scripts for the new process.
Maintenance hosts will be going away; please submit feedback promptly if
maintenance scripts on Kubernetes don't work for you. (T341553)
  topic biography had more than 10K tasks
  topic media had more than 10K tasks
  topic europe had more than 10K tasks
  topic stem had more than 10K tasks
Total number of OK search index entries: 126453
 (results in multiple topics counted multiple times)Total number of dangling search-index entries: 9920
migr@mwmaint2002:/srv/mediawiki/php-1.43.0-wmf.28$ mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=eswiki --search-index --dry-run
DEPRECATION WARNING: Maintenance scripts are moving to Kubernetes. See
https://wikitech.wikimedia.org/wiki/Maintenance_scripts for the new process.
Maintenance hosts will be going away; please submit feedback promptly if
maintenance scripts on Kubernetes don't work for you. (T341553)
  topic biography had more than 10K tasks
  topic media had more than 10K tasks
  topic europe had more than 10K tasks
  topic stem had more than 10K tasks
Total number of OK search index entries: 133896
 (results in multiple topics counted multiple times)Total number of dangling search-index entries: 1016
migr@mwmaint2002:/srv/mediawiki/php-1.43.0-wmf.28$

Recording here that I'm noticing myself still running one-off scripts on the maint-hosts because, as I understand it, for the new way of running them, I would need deployer-rights, and I do not have (and do not really want) those.

QA Note: The way to try this out, is by manually deleting some of the data from .json page (not via the form, but via editing the MediaWiki:Something.json page directly), or finding a page that is already incomplete, like https://cs.wikipedia.beta.wmflabs.org/wiki/MediaWiki:GrowthExperimentsHomepage.json .

The description lists a bunch of things as "examples", but I think what the changes up for review are currently covering is a good start.

Thanks!

In T276438#10258492, @MGerlach wrote:

[...]
@Michael I am curious to understand your use-case better: Are you looking for a specific dataset, or would you want to update the datasets for all models (or something different)? I am happy to support.

The editnotice seems to be a red herring. It exists on mobile already for at least half a year (see T312587: Show edit notices within mobile editing interfaces), and the respective page on testwiki was last edited 2019: https://test.wikipedia.org/w/index.php?title=MediaWiki:Editnotice-0&action=history

I tried it on dewiki and there the mobile interface works. I looked for the API GET-request above and compared them to test-wiki, and so only difference that looks maybe meaningful is that testwiki has a notice with key editnotice-0, and dewiki as a notice with key flaggedrevs_editnotice. Are we maybe handling flaggedrevs_editnotice but not editnotice-0 or something?

I'm noticing that the editor keeps making the same two VisualEditor requests:

could it be similar to T377783?

Mh, no. Even when I clicked this popup away beforehand, I still get this error.

I can reproduce the problem on testwiki, unsure what the cause is. When I try to edit another article, I get an "Editnotice links" pop-up:

Yes, what @KStoller-WMF writes is already a very good start!
From my perspective we want the following:

I very much appreciate the work on this 💚 ! But I think closing it might be a tiny bit premature.
Let's wait till these changes reach at least Group 2 on Thursday. Then we can keep our eyes open for the Unable to fetch suggested edits count for user {userId}; no user impact found.-message and start figuring out what is going on with that. In that context, we can then open a new task for that and close this one.

This maintenance tasks should probably provide a histogram / overview of how the minimum link-score of a page is distributed across the suggestions in the database (see T316079#9091797 for reference).
Also, I think it would maybe be nice to have a breakdown of the number of suggestions per topic.

Growth is working on surfacing link-recommendations in new ways (T362584), and so I'm trying to get a grasp on how this service is evolving. Where can I get insights into the latest datasets?

Thank you! This backport enabled our maintenance script to add back task recommendations! 🙏

@dcausse Is this something that can be back-ported on Monday, or should it better ride the train normally?

In T376715#10213965, @Lucas_Werkmeister_WMDE wrote:

Impact on the maintenance script is unknown.

Looks like the error gets caught and logged in link RecommendationUpdater::processCandidate(), which is why the maintenance script continues running. Beyond that, I still don’t know what the impact of this is (presumably the pages that experience this error will be missing some kind of update).

Adding User-notice here, because there have been a couple of changes to GuidedTours that will roll out with the next train:

In T374402#10240009, @AnneT wrote:

Is it possible for the Component and the HTML Element to *disagree* about the validation state?

FWIW, this is more specific to the status prop that Codex components have and the validity state of their internal HTML elements.

Not sure the root cause lies with GrowthExperiments, but maybe we triggered a code-path that wasn't executed like that before. This seems similar to (though distinct from) T376715: TypeError: Argument 3 passed to CirrusSearch\DataSender::sendWeightedTagsUpdate() must be of the type array, null given, called in /srv/mediawiki/php-1.43.0-wmf.25/extensions/CirrusSearch/includes/Job/ElasticaWrite.php on line that I'll boldly add Discovery-Search and CirrusSearch back. Even if our code is wrong, it would probably be good to have those extra eyes on it.

I would propose going for requiring a value for these fields. Conceptually it does not make sense to have no value for "Try Suggested edits notification" or "Keep going notification" or "minimumLinkScore" or "underlinkedWeight", etc.
We need/want a value for all of them. I think it is ok to require that value, but maybe we can mention the default in help-text.

This time only Wikibase involved, as far as I can tell: