Getting deadlock errors in logs

Job exception: PG::TRDeadlockDetected: ERROR: deadlock detectedDETAIL: Process 29659 waits for ShareLock on transaction 892416679; blocked by process 29650.Process 29650 waits for ShareLock on transaction 892416541; blocked by process 29659.HINT: See server log for query details.CONTEXT: while rechecking updated tuple (36923,3) in relation "discourse_activity_pub_actors"

These are showing up in /logs @angus, I'm not sure if there are any user-facing issues. Stack trace points to a job handling deliveries:

rack-mini-profiler-3.3.1/lib/patches/db/pg.rb:69:in `exec_params' rack-mini-profiler-3.3.1/lib/patches/db/pg.rb:69:in `exec_params' (eval at /var/www/discourse/lib/method_profiler.rb:38):105:in `exec_params'activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:894:in `block (2 levels) in exec_no_cache' activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:1004:in `block in with_raw_connection' activesupport-7.2.2.1/lib/active_support/concurrency/null_lock.rb:9:in `synchronize' activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:976:in `with_raw_connection' activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:893:in `block in exec_no_cache' activesupport-7.2.2.1/lib/active_support/notifications/instrumenter.rb:58:in `instrument' activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:1119:in `log' activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:892:in `exec_no_cache' activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:872:in `execute_and_clear' activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:79:in `exec_delete' activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:208:in `update' activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:27:in `update' activerecord-7.2.2.1/lib/active_record/relation.rb:610:in `block in update_all' activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:421:in `with_connection' activerecord-7.2.2.1/lib/active_record/connection_handling.rb:296:in `with_connection' activerecord-7.2.2.1/lib/active_record/relation.rb:598:in `update_all' /var/www/discourse/plugins/discourse-activity-pub/lib/discourse_activity_pub/delivery_failure_tracker.rb:36:in `set_domain_actors_as_available' /var/www/discourse/plugins/discourse-activity-pub/lib/discourse_activity_pub/delivery_failure_tracker.rb:22:in `track_success' /var/www/discourse/plugins/discourse-activity-pub/app/jobs/discourse_activity_pub_deliver.rb:50:in `perform_request' /var/www/discourse/plugins/discourse-activity-pub/app/jobs/discourse_activity_pub_deliver.rb:15:in `execute' /var/www/discourse/app/jobs/base.rb:316:in `block (2 levels) in perform' 


Discuss this on our forum.

This should address it

https://github.com/discourse/discourse-activity-pub/pull/203



Discuss this on our forum.
1 Like

Hmm, that doesn't seem to fix the issue unfortunately @angus. There's a lot of actors, possibly, in the DB, and the batching is still insufficient.

On meta, for example, there are 90k+ rows in that model:

discourse(prod)> DiscourseActivityPubActor.where(domain: 'mastodon.social').size=> 90264

Do we absolutely need to update their available column on every delivery?



Discuss this on our forum.

I see. Hopefully this, together with the previous changes, will address it:

https://github.com/discourse/discourse-activity-pub/pull/204



Discuss this on our forum.
1 Like